Contextual Bayesian Optimization (CBO) efficiently optimizes black-box, expensive-to-evaluate functions with respect to design variables, while simultaneously integrating relevant contextual information regarding the environment, such as experimental conditions.However, the relevance of contextual variables is not necessarily known beforehand. Moreover, contextual variables can sometimes be optimized themselves, an overlooked setting bycurrent CBO algorithms. Optimizing contextual variables may be costly, which raises thequestion of determining a minimal relevant subset. We address this problem using a novelmethod, Sensitivity-Analysis-Driven Contextual BO (SADCBO). We learn the relevance ofcontext variables by sensitivity analysis of the posterior surrogate model, whilst minimizing the cost of optimization by leveraging recent developments on early stopping for BO.We empirically evaluate our proposed SADCBO against alternatives on both synthetic andreal-world experiments, and demonstrate a consistent improvement across examples.