Accurate, reliable and scalable predictions of protein ligand binding affinity through artificial intelligence have a great potential to accelerate drug discovery process. While many works have been introduced for this purpose, their performance remains poor when applied to new out-of-distribution (OOD) cases where new unseen chemicals belong to a new chemical scaffold. Moreover, they neither account for uncertainty nor quantify the uncertainty associated with individual predictions. To address these issues, we propose a sequence-based novel deep learning framework, TrustAffinity, to predict the binding affinity and the uncertainty of the prediction. TrustAffinity employs a novel uncertainty-based loss function to leverage the uncertainty for improving OOD generalizations. We perform extensive validations of TrustAffinity in multiple OOD settings. TrustAffinity significantly outperforms state-of-the-art deep learning methods and protein-ligand docking in the prediction of binding affinity. Moreover, TrustAffinity is able to perform predictions at least three orders of magnitude of faster than protein-ligand docking, highlighting its suitability for integrating TrustAffinity into a real-time drug discovery pipeline. Notably, we successfully illustrate the practical utility of TrustAffinity through a case study focused on lead discovery in the context of opioid use disorder.