Effective machine learning models learn both robust features that directly determine the outcome of interest (e.g., an object with wheels is more likely to be a car), and shortcut features (e.g., an object on a road is more likely to be a car). The latter can be a source of error under distributional shift, when the correlations change at test-time. The prevailing sentiment in the robustness literature is to avoid such correlative shortcut features and learn robust predictors. However, while robust predictors perform better on worst-case distributional shifts, they often sacrifice accuracy on majority subpopulations. In this paper, we argue that shortcut features should not be entirely discarded. Instead, if we can identify the subpopulation to which an input belongs, we can adaptively choose among models with different strengths to achieve high performance on both majority and minority subpopulations. We propose COnfidence-baSed MOdel Selection (COSMOS), where we observe that model confidence can effectively guide model selection. Notably, COSMOS does not require any target labels or group annotations, either of which may be difficult to obtain or unavailable. We evaluate COSMOS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift. We find that COSMOS achieves 2-5% lower average regret across all subpopulations, compared to using only robust predictors or other model aggregation methods.