Stochastic variational Gaussian processes (SVGP) scale Gaussian process inference up to large datasets through inducing points and stochastic training. However, the training process involves hard multimodal optimization, and often suffers from slow and suboptimal convergence by initializing inducing points directly from training data. We provide a better initialization of inducing points from kernel-based least squares fitting. We show empirically that our approach consistently reaches better prediction performance with much fewer training epochs. Our initialization saves up to 38% of the total time cost as compared to standard SVGP training.