Convolutional neural networks (CNNs) have become essential models for predicting neural activity and behavior in visual tasks. However, their ability to capture complex cognitive functions, such as numerosity discrimination, remains debated. Numerosity, the ability to perceive and estimate the number of items in a visual scene, is thought to be represented by specialized "number-detector" units in CNNs. In this study, we address the limitations of classical Representational Similarity Analysis (RSA), which assumes equal importance for all features, by applying pruning - a feature selection technique that identifies and retains the most behaviorally relevant units. We applied pruning to retain only the most behaviorally relevant units in the CNNs. Our results show that number-detector units are not critical for population-level representations of numerosity, challenging their proposed significance in previous studies.