The emergence of Foundation Models has radically transformed the Deep Learning scene and also accelerated its adoption in other domains. In particular, Large Language Models (LLMs) are being used in many time series forecasting tasks including Epidemic Forecasting. While the adoption of a new technology is generally a good sign, we must be scientific in analysing the benefits of doing so. We try two LLMs used in time series forecasting and show that on average, they perform almost similar or marginally better to the very popular classical statistical method ARIMA when applied to epidemic forecasting. We have performed extensive experiments on many Epidemic Forecasting datasets and thoroughly validated our conclusion that we need Foundation Models like LLMs for Epidemic Forecasting for the growth of the field even if the benefits are not proportionate to the costs immediately.