Traditional spectral analysis methods are being pushed to their limits by exploding survey volumes. For efficient stellar characterization, accurate synthetic libraries and automated, interpretable techniques are needed. We develop a novel framework - deep-\underline{R}egularized \underline{E}nsemble-based \underline{M}ulti-task Learning with \underline{A}symmetric Loss for \underline{P}robabilistic Inference (deep-REMAP) - and show its effectiveness in predicting atmospheric parameters from observed spectra. We train our deep convolution neural network on PHOENIX, fine-tune on MARVELS FGK dwarfs, then predict effective temperature, surface gravity, and metallicity for FGK giants. To incorporate MARVELS peculiarities, we augment PHOENIX with realistic signatures. When validated on MARVELS calibration stars, the fine-tuned model recovers parameters and uncertainties, demonstrating effective transfer learning. While trained on PHOENIX for MARVELS, deep-REMAP is easily extended to other libraries, wavelengths, resolutions, and wider stellar properties.