Cataract is the leading cause of blindness worldwide. Access to cataract screening is important to enable treatment and vision restoration to eliminate avoidable blindness. This paper introduces an artificial intelligence (AI)-driven approach designed to improve access to cataract screening, using external ocular images captured by community health workers utilizing a smartphone-based anterior segment eye imaging modality. The platform integrates segmentation and classification networks by leveraging pretrained foundation models to accurately differentiate between healthy eyes, immature cataracts, and mature cataracts. We evaluated several fine-tuning strategies and proposed projection tuning as an efficient and lightweight approach to tackle distribution shift challenges among datasets. In combination with a Vision Transformer model, we demonstrate exceptional lens classification performance using a small cataract image database. Our investigation confirms that our smartphone-based imaging system combined with the proposed framework offers a effective and accurate solution for cataract detection, addressing distribution shift challenges.