Conventional electromagnetic wave simulators often have long simulation times, so are not suitable for computational imaging and photonic inverse problems (e.g. end-to-end design, iterative reconstruction) that require evaluating the forward model many times. Electromagnetic wave simulators based on neural networks promise speed improvements of several orders-of-magnitude, but standard supervised training approaches have difficulty fitting the true physics. Physics-informed approaches help, but existing residual-based methods use only local information and must be used in conjunction with standard supervised loss. In this work, we introduce Time Reversal Consistency (TReC), a new physics-based training method based on the time reversibility of Maxwell's equations. TReC uses a time-reversed, differentiable finite-difference simulator to compare neural network predictions with a known initial condition. TReC provides both global physics guidance and supervision in a single function. When trained only on randomized scatterers, we find that networks trained with TReC generalize well to a range of arbitrary structured media. We validate the method on the inverse design of a set of angle-to-angle couplers, addressing almost two magnitudes more parameters than previous methods, and find that the design quality corresponds closely with designs based on a conventional simulator while requiring 5\% of the design time.