In order to capitalize on the numerous applications of machine learning for medical imaging analysis, clinicians need to understand the clinical decisions made by machine learning (ML) models. This allows clinicians to trust ML models, understand their failure modes, and ideally learn from their superhuman capabilities and expand clinical knowledge. Providing explanations for each high resolution image in a large medical database can be computationally expensive. Recent methods amortize this cost by learning a selector model that takes a sample of data and selects the subset of its features that is important. We show that while the selector model learned by these methods make it simple for practitioners to explain new images, the model learns to counterintuitively encode predictions within its selections, omitting the important features. We demonstrate that this phenomenon can occur even with simple medical imaging tasks, such as detecting cardiomegaly in chest X-Rays. We propose REAL-X to address these issues and show that our method provides trustworthy explanations through quantitative and expert radiologist evaluation.