Synthesizing optimal controllers for dynamical systems in practice involves solving real-time optimization problems with hard time constraints. These constraints restrict the class of numerical methods that can be applied; indeed, computationally expensive but accurate numerical routines often have to be replaced with fast and inaccurate methods, trading inference time for worse theoretical guarantees on solution accuracy. This paper proposes a novel methodology to accelerate numerical optimization of optimal control policies via hypersolvers, hybrids of a base solver and a neural network. In particular, we apply low–order explicit numerical methods for the ordinary differential equation (ODE) associated to the numerical optimal control problem, augmented with an additional parametric approximator trained to reduce local truncation errors introduced by the base solver. Given a target system to control, we first pre-train hypersolvers to approximate base solver residuals by sampling plausible control inputs. Then, we use the trained hypersolver to obtain fast and accurate solutions of the target system during optimization of the controller. The performance of our approach is evaluated in direct and model predictive optimal control settings, where we show consistent Pareto improvements in terms of solution accuracy and control performance.