The rapid growth of machine learning (ML) has led to an increased demand for computational power, resulting in larger data centers (DCs) and higher energy consumption. To address this issue and reduce carbon emissions, intelligent control of DC components such as cooling, load shifting, and energy storage is essential. However, the complexity of managing these controls in tandem with external factors like weather and green energy availability presents a significant challenge. While some individual components like HVAC control have seen research in Reinforcement Learning (RL), there's a gap in holistic optimization covering all elements simultaneously. To tackle this, we've developed DCRL, a multi-agent RL environment that empowers the ML community to research, develop, and refine RL controllers for carbon footprint reduction in DCs. DCRL is a flexible, modular, scalable, and configurable platform that can handle large High Performance Computing (HPC) clusters. In its default setup, DCRL also provides a benchmark for evaluating multi-agent RL algorithms, facilitating collaboration and progress in green computing research.