We discuss the use of the sparse Gaussian process regression (SGPR) method introduced by Titsias (2009) as a baseline for approximate Gaussian processes. We make concrete recommendations to ensure that it is a strong baseline, ensuring that meaningful comparisons can be made. In doing so, we provide recommendations for comparing Gaussian process approximations, designed to explore both the limitations of methods as well as understand their computation-accuracy tradeoffs. This is particularly important now that highly accurate GP approximations are available, so that the literature provides a clear picture of currently achievable results.