We present a new class of learnable Riemannian manifolds with a metric parameterized by a deep neural network. The core manifold operations--specifically the Riemannian exponential and logarithmic maps--are solved using approximate numerical techniques. Input and parameter gradients are computed with an adjoint sensitivity analysis. This enables us to fit geodesics and distances with gradient-based optimization of both on-manifold values and the manifold itself. We demonstrate our method's capability to model smooth, flexible metric structures in graph and dynamical system embedding tasks.