Multitask models have been the key to the most AI-driven applications on smart devices like phones. Such applications often infer on-devices using a pre-trained model. However, pre-trained multitask models fail when in-the-wild data distribution differs from the training data. While adapting to the target test data is a natural solution, conventional algorithms from transfer learning and unsupervised domain adaptation are impractical in the above on-device adaptation requirement due to the unavailability of labeled runtime data and limited resources at the devices. Recent methods in \textit{Test-time Adaptation} (TTA) are deemed suitable as they neither require access to labels for test data nor the training data. However, the current state-of-the-art (SOTA) TTA approaches only consider models with single-task objectives and thus may fail to capture the nuances of multitask modeling. For the first time in literature, we systematically explore this novel but practical problem regime. Firstly, we investigate the impact of different tasks on the entropy of class probability distribution, a key optimization criterion for many TTA approaches. Next, we extend a popular SOTA TTA approach and systematically investigate its performance on a benchmark multitask image dataset under various domain shifts. With different experiments, we observe that the current TTA approaches fail to capture the intricacies of the different tasks. We envision this study will pave the way for further investigation and development of TTA approaches designed explicitly for multitask architectures.