High-energy cosmic rays are informative probes of astrophysical sources in our galaxy. A main challenge is to separate gamma showers (extremely rare events of interest) from the vast majority of hadron showers, when we have access to realistic simulations of the shower production (forward process) but the prior distribution on the shower parameters is unknown. Direct classification of the showers using output data leads to biased predictions and invalid uncertainty estimates, since the prior is chosen by design and is different from the true distribution. We overcome these biases by proposing a new method that casts classification as a hypothesis testing problem under nuisance parameters. The main idea is to estimate ROC curves as a function of all nuisances, devising selection criteria that are valid under a generalized prior probability shift over both shower label and nuisance parameters. Our method yields a set-valued classifier that returns valid confidence sets for all levels alpha simultaneously without having to retrain the classifier for each level.