Neural Operators can learn operators from data, for example, to solve partial differential equations (PDEs). In some cases, this data-driven approach is not sufficient, e.g., if the data is limited, or only available at a resolution that does not permit resolving the underlying physics. The Physics-Informed Neural Operator (PINO) aims to solve this issue by adding the PDE residual as a loss to the Fourier Neural Operator (FNO). Several methods have been proposed to compute the derivatives appearing in the PDE, such as finite differences and Fourier differentiation. However, these methods are limited to regular grids and suffer from inaccuracies. In this work, we propose the first method capable of exact derivative computations for general functions on arbitrary geometries. We leverage the Geometry Informed Neural Operator (GINO), a recently proposed graph-based extension of FNO. While GINO can be queried at arbitrary points in the output domain, it is not differentiable with respect to those points due to a discrete neighbor search procedure. We introduce a fully differentiable extension of GINO that uses a differentiable weight function and neighbor caching in order to maintain the efficiency of GINO while allowing for exact derivatives. We empirically show that our method matches prior PINO methods while being the first to compute exact derivatives for arbitrary query points.