In this paper, we introduce a dynamic semantic clustering approach, based on the Chinese Restaurant Process, to address uncertainty in the inference of Large Language Models (LLMs). Uncertainty quantification of an LLM is performed by calculating entropy over the semantic clusters. Further, we propose using (negative) likelihood of the generated clusters as the (non)conformity score in the Conformal Prediction framework for predicting a set (instead of a point prediction) by an LLM to account for uncertainty in its prediction. We demonstrate the efficacy of our approach on two question-answering benchmarks, COQA and TriviaQA, using two LLMs, Llama-2-13b and Mistral-7b, achieving state-of-the-art (SOTA) results on uncertainty quantification using AUROC, AUARC and AUARC as metrics. We also show that our conformal predictor generates smaller prediction sets for the same probabilistic guarantee of including correct response compared to the SOTA conformal prediction baseline. Our code is publicly available at https://shorturl.at/7yHSq.