Neuroimaging-based biomarkers of brain health are necessary for early diagnosis of cognitive decline in the aging population. While many recent studies have investigated whether an individual's "brain-age'' can be accurately predicted based on anatomical or functional brain biomarkers, comparatively few studies have sought to predict brain-age with structural connectivity features alone. Here, we investigated this question with data from a large cross-sectional study of elderly volunteers in India (n=158 participants, age-range=51-86 yrs, 66 females). We analyzed 23 standardized cognitive test scores obtained from these participants with factor analysis. All test score variations could be explained with just three latent cognitive factors, each of which declined markedly with age. Next, using diffusion magnetic resonance imaging (dMRI) and tractography we estimated the structural brain connectome in a subset of n=101 individuals. Structural connectivity features robustly predicted inter-individual variations in cognitive factor scores (r=0.293-0.407, p<0.001) and chronological age (r=0.517-0.535, p<0.001), and identified critical connections in the prefrontal and parietal cortex whose strength most strongly predicted each of these variables. dMRI structural connectivity may serve as a reliable tool for predicting age-related cognitive decline in healthy individuals, as well as accelerated decline in patient populations.