Abstract: From common-sense reasoning to domain-specific tasks, parameter-efficient fine tuning (PEFT) methods for large language models (LLMs) have showcased significant performance improvements on downstream tasks. However, fine-tuned LLMs are not always correct due to the sparsity of training data and they often become overconfident on uncertain answers. These deficiencies resembles epistemic uncertainty, which arises from limitations in the model's ability to learn from data and generalize to new situations. Existing methods focus on the post-hoc stage and thus have limited capability in calibrating epistemic uncertainty. To address these limitations, we propose Functional-Level Uncertainty Quantification for Calibrated Fine-Tuning (UQ4CT), which refines functional-level epistemic uncertainty during the fine-tuning stage via a mixture-of-expert framework. We show that UQ4CT reduces Expected Calibration Error (ECE) by more than $25$% while maintaining high accuracy across $5$ benchmarks.