Convolutional Neural Networks are particularly suited for image analysis tasks, such as Image Classification, Object Recognition or Image Segmentation. Like all Artificial Neural Networks, however, they are "black box" models, and suffer from poor explainability. This work is concerned with the specific downstream task of Emotion Recognition from images, and proposes a framework that combines CAM-based techniques with Object Detection on a corpus level to better understand on which image cues a particular model relies to assign a specific emotion to an image. We demonstrate our framework using the EmoNet model and show that it mostly focuses on human characteristics, but also explore the pronounced effect of specific image modifications.