Gaussian processes (GPs) are a widely-used model class for approximating unknown functions, especially useful in tasks such as Bayesian optimisation, where accurate uncertainty estimates are key. Deep Gaussian processes (DGPs) are a multi-layered generalisation of GPs, which promises improved performance at modelling complex functions. Some of the problems where GPs and DGPs may be utilised involve data on manifolds like hyperspheres. Recent work has recognised this, generalising scalar-valued and vector-valued Matérn GPs to a broad class of Riemannian manifolds. Despite that, an appropriate analogue of DGP for Riemannian manifolds is missing. We introduce a new model, residual manifold DGP, and a suitable doubly stochastic variational inference technique that helps train and deploy it on hyperspheres. Through examination on stylised examples, we highlight the usefulness of residual deep manifold GPs on regression tasks and in Bayesian optimisation.