High fidelity simulation of detector components in collider physics is computationally expensive and often not scalable to the requirements of future experimental facilities. In this work, we present a fast and accurate alternative for detector simulation based on continuous normalizing flows for calorimeter simulation named CaloFFJORD, able to reproduce high-fidelity calorimeter responses in a fraction of the time compared to full simulation routines. We evaluate our model using different detector simulations and show that CaloFFJORD can improve the fidelity of detector simulations by incorporating data symmetries that are harder to encode within standard normalizing flow architectures.