Buildings produce more U.S. greenhouse gas emissions through electricity generation than any other economic sector. To improve the energy efficiency of buildings, engineers often rely on physics-based building simulations to predict the impacts of retrofits in individual buildings. In dense urban areas, these models suffer from inaccuracy due to imprecise parameterization or external, unmodeled urban context factors such as inter-building effects and urban microclimates. In a case study of approximately 30 buildings in Sacramento, California, we demonstrate how our hybrid physics-driven deep learning framework can use these external factors advantageously to identify a more optimal energy efficiency retrofit installation strategy and achieve significant savings in both energy and cost.