Bayesian Optimization (BO) is a powerful tool for tackling optimization problems involving limited black-box function evaluations. However, it suffers from high computational complexity and struggles to scale efficiently on high-dimensional problems when fitting a Gaussian process surrogate model. We address these issues by proposing a fast acquisition function maximization procedure. We leverage the fact that Probability Improvement (PI) acquisition function is a likelihood function whose score can be estimated through a simple linear regression problem called local score matching. This enables fast gradient-based optimization of the acquisition function, and a competitive BO procedure which performs similarly to that of computationally expensive neural networks.