Photonic inverse design has emerged as a powerful technique for creating non-intuitive optical devices, revolutionizing traditional design methodologies. However, the bottleneck in photonic inverse design lies in the computational cost of physically accurate 3D simulations as the high number of optimization iterations or large design foot-print becomes a limiting factor on photonic device design. Here we introduce a novel approach, utilizing a two-staged model combined with a 2D-FDFD simulator, to achieve accurate field predictions in a much smaller amount of time than 3D FDTD. The model is trained to emphasize field properties crucial for photonic device optimization such as mode overlap and transmission, enabling rapid and physically accurate simulations. Results demonstrate a remarkable speedup of up to 264.53 times compared to 3D FDTD simulations, opening new avenues for the design of complex and accurate photonic devices.