Self-improvement is an promising mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we coin as the generation-verification gap. Through experiments with various model families and tasks, we examine scaling properties, an iterative self-improvement procedure, and ways to improve its performance. We believe our results have several empirical implications, and our study leaves several exciting future directions for understanding the potential and limits of LLM self-improvement.