In-context learning (ICL) has emerged as a powerful paradigm enabling Large Language Models (LLMs) to perform new tasks by prompting them with few training examples, and no need to fine-tune them. Given its potential to adapt and personalize the model's behaviour using private user data, recent studies have introduced techniques for ICL that satisfy Differential Privacy guarantees (DP ICL). Existing DP ICL approaches claim to attain such guarantees while maintaining negligible utility degradations when adapting the models to perform new tasks. In this paper, we present preliminary empirical evidence suggesting that these claims may hold only for tasks aligned with the model's pre-training knowledge and biases. We do so by showing the performance of DP ICL significantly degrades with respect to the non-private counterpart in scenarios that introduce tasks and distribution shifts that challenge the model's prior knowledge.To mitigate the risk of overly optimistic evaluations of DP ICL, we invite the community to consider our sanity checks to attain a more accurate understanding of its capabilities and limitations.