The quantity and quality of literature around climate change (CC) and its impacts are increasing yearly. Yet, this field has received limited attention in the Natural Language Processing (NLP) community. With the help of large Language Models (LMs) and transfer learning, NLP can support policymakers, researchers, and climate activists in making sense of large-scale and complex CC-related texts. CC-related texts include specific language that general language models cannot represent accurately. Therefore we collected a climate change corpus consisting of over 360 thousand abstracts of top climate scientists' articles from trustable sources covering large temporal and spatial scales. Comparison of the performance of GPT2 LM and our 'climateGPT2 models', fine-tuned on the CC-related corpus, on claim generation (text generation) and fact-checking, downstream tasks show the better performance of the climateGPT2 models compared to the GPT2. The climateGPT2 models decrease the validation loss to 1.08 for claim generation from 43.4 obtained by GPT2. We found that climateGPT2 models improved the masked language model objective for the fact-checking task by increasing the F1 score from 0.67 to 0.72.