Existing benchmarks for grounding language in interactive environments either lack real-world linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. The web provides the perfect balance between the two - it is very practical for the deployment of autonomous agents for helping reduce human effort, while being more scalable than physical setups like robotics. In this talk, I will first describe WebShop, a new RL environment based on a simulated e-commerce website containing >1 million real-world products and >12000 crowd-sourced text instructions. WebShop provides several challenges for language grounding including understanding compositional instructions, query (re-)formulation, comprehending and acting on noisy text in webpages, and performing strategic exploration. Then, I will introduce ReAct, a new approach for large language models (LLMs) to perform both reasoning and acting over the web (or any API) and gain external knowledge for a task.