The recent explosion in the capabilities of large language models has led to a wave of interest in how best to prompt the model to perform a given task. While it may be tempting to choose a prompt based on average empirical results on a validation set, this can lead to a deployment where unexpectedly poor responses are generated. To mitigate this prospect, we propose a lightweight framework, Prompt Risk Control, for selecting a prompt based on rigorous upper bounds on families of informative risk measures. We provide and compare different methods for producing bounds on a diverse set of metrics measuring quantities such as worst-case response and disparities in generation quality across the population of users. In addition, we extend the underlying statistical bounding techniques to accommodate the possibility of distribution shifts in deployment. Experiments on applications such as chatbots, medical question summarization, and code generation highlight how such a framework can reduce the risk of the worst outcomes.