Abstract: The power of digital platforms is at the center of major ongoing policy and regulatory efforts. To advance existing debates, we designed and executed an experiment to measure the power of online search providers, building on the recent definition of performative power. Instantiated in our setting, performative power quantifies the ability of a search engine to steer web traffic by rearranging results. To operationalize this definition we developed a browser extension that performs unassuming randomized experiments in the background. These randomized experiments emulate updates to the search algorithm and identify the causal effect of different content arrangements on clicks. Analyzing tens of thousands of clicks, we consistently find significant effects of minimal arrangement changes. For example, down-ranking the first item by one position and adding visually appealing elements on top of the page results in a click reduction of more than $50%$ on average across queries. Relating our quantitative findings to performative power, we discuss what our robust quantitative insights say about the power of online search engines.