This one is a bit unusual. You can't set the record selection formula to use data from the subreports because at the time the report is run, during Pass #1, the subreport data isn't available yet. So you can't filter on it.

A way around this is to use Shared variables in the subreport and then in the main report I would use conditional formatting on the Details section. Create a formula which uses the Shared variable to determine if the record should be excluded or not. If so, return True so that the record is suppressed. This effectively lets you apply a manual record selection filter on the data and hide the records you don't want shown.

HTH.