Ordinary Differential Equation Variational Auto-Encoder (ODE2VAE) is a deep latent variable model that aims to learn complex distributions over high-dimensional sequential data and their low-dimensional representations in a hierarchical latent space. The hierarchical organization of the latent space embeds a physics-guided inductive bias in the model. In this paper, we analyze the latent representations inferred by the ODE2VAE model over three different physical motion datasets: bouncing balls, projectile motion, and simple pendulum. We show that the model is able to learn meaningful latent representations to an extent without any supervision.