The interpretability of deep learning (DL) for Alzheimer's disease (AD) prediction has provided supporting evidence for the timely intervention of disease progression. In particular, counterfactual reasoning is gradually being employed in the medical field, providing refined visual explanatory maps. However, most visual explanatory maps still rely on visual inspection without quantifying their validity, being a barrier for non-expert individuals. To this end, we propose a novel framework to analyze the counterfactual reasoning-based visual explanation by transforming them into quantitative features. Furthermore, we develop a simple shallow linear classifier to boost the effectiveness of quantitative features while promoting the model's interpretability and achieving superior predictive performance compared to the DL model. By doing so, our method further provides an ADness index that can be used to intuitively comprehend a patient's brain status with respect to AD.