The effective and efficient transfer of knowledge from a foundation model using Convolutional Neural Networks (CNNs) to a new task is a significant challenge in computer vision. Initial approaches include side-tuning, a straightforward method that attaches a lightweight, modular side-network to the original model and interpolates the two outputs for transference to the new task. They fell to the wayside when visual prompt tuning (VPT) was introduced due to VPT’s superior performance, significantly fewer additional parameters, and improved ability to generalize to multiple datasets. This paper presents the Parasite Network, an alternative side-network approach using CNNs that leverages a small ‘parasite’ model that extracts knowledge at points along the original larger ‘host’ network without adapting the original model. We show that parasite networks have a significant reduction in the number of training parameters and are able to generalize across multiple datasets as compared to side-tuning. The parasite approach was experimentally validated against both substitutive and additive transfer learning methods using various VTAB-1K datasets. We show that the parasite approach outperforms VPT for CNNs and has superior GPU utilization and competitive latency.