Protein-small molecule interactions, or receptor-ligand interactions, are essential for understanding biological processes and advancing drug design. In this paper, we introduce DockFormer, a method that leverages multi-modal learning to predict both the binding affinity and structure of these interactions. DockFormer employs fully flexible docking, where no part of the receptor remains rigid, by adapting the AlphaFold2 architecture. Instead of relying on protein sequences and Multiple Sequence Alignments, DockFormer uses predicted receptor structures as input. This modification enables the model to concentrate on ligand docking prediction rather than protein folding, while preserving full receptor flexibility. The streamlined design also reduces the model size to just 8 layers, compared to AlphaFold2's 48 layers, greatly accelerating the inference process and making it more efficient for large-scale screening.Benchmark tests on the PoseBusters dataset demonstrated a 27% success rate, while on an affinity benchmark, DockFormer achieved a Pearson correlation of 0.91. This optimized architecture offers a valuable tool for rapid and accurate virtual screening in drug discovery.