In this paper, we develop attribution-based confidence (ABC) metric to detect black-box adversarial attacks in breast histopathology images used to detect cancer. Due to the lack of data for this problem, we subjected histopathological images to adversarial attacks using the state-of-the-art technique Meta-Learning the Search Distribution (Meta-RS) and generated a new dataset. We adopt the Sobol Attribution Method to the problem of cancer detection. The output helps the user to understand those parts of the images that determine the output of a classification model. The ABC metric characterizes whether the output of a deep learning network can be trusted. We can accurately identify whether an image is adversarial or original with the proposed approach. The proposed approach is validated with eight different deep learning-based classifiers. The ABC metric for all original images is greater or equal to 0.8 and less for adversarial images. To the best of our knowledge, this is the first work to detect attacks on medical systems for breast cancer detection using the ABC metric.