Discovery of high-affinity peptide binders is broadly useful for designing novel therapeutics. Machine learning-based in silico screening is a promising approach for increasing the success rate of therapeutic peptide design. Structure-based prediction models, such as AlphaFold-multimer, have shown promising though insufficient zero-shot performance for in silico screens of diverse peptides. Incorporating interaction data produced during peptide discovery campaigns, we develop a low-N OpenFold fine-tuning procedure on the peptide recognition modules database (PRM-db). With a relatively small dataset, we find 13-60x fold increase in design hit rate with the fine-tuned model making a powerful model for improving peptide design success rates. Unexpectedly, we also find that interaction data also improves structure complex predictions critical for targeting binding sites during design campaigns. The framework introduced here demonstrates a data-efficient, structure-free recipe for dramatically improving peptide-protein prediction and ultimately the success rate of peptide binder design.