Protein Language Models (PLMs) trained on large databases of protein sequences have proven effective for modeling protein biology across a wide range of applications. However, existing PLMs primarily focus on individual sequences, despite many critical functional aspects of proteins arising from their interactions with other proteins. In this work we introduce ESM-Multimer, the first PLM that is trained on protein-protein interactions from STRING-DB that uses a masked language modeling objective. We show that ESM-Multimer outperforms existing PLMs and domain-specific methods on a variety of tasks including predicting general protein-protein interactions, antibody binding, mutational effects, TCR-epitope binding and antibody structure prediction. These findings highlight the significance of incorporating protein-protein interactions into PLMs and pave the way for improved understanding of protein function.