As machine learning workloads significantly increase energy consumption, sustainable data centers with low carbon emissions are becoming a top priority for governments and corporations worldwide. There is a pressing need to optimize energy usage in these centers, especially considering factors like cooling, load flexibility based on renewable energy availability, and battery storage utilization. The challenge arises due to the interdependencies of these strategies with fluctuating external factors such as weather and grid carbon intensity. Although there's currently no real-time solution that addresses all these aspects, our proposed Data Center Carbon Footprint Reduction (DCCFR) framework, based on multi-agent Reinforcement Learning (MARL), targets carbon footprint reduction, energy conservation, and cost. Our findings reveal that DCCFR's MARL agents efficiently navigate these complexities, optimizing energy in real-time. Compared to the industry standard ASHRAE controller controlling HVAC for a year in various regions, DCCFR reduced carbon emissions, energy consumption, and energy costs by over 10% with EnergyPlus simulation.