Physics-informed neural networks have been recognized as a viable alternative to conventional numerical solvers for Partial Differential Equations (PDEs). However, a key challenge is their limited generalization across varied initial conditions. Addressing this, our study presents a novel physics-informed transformer model for learning the solution operator for PDEs. Leveraging the attention mechanism, our model is able to explore the relationships between its inputs. Furthermore, by using a physics-informed loss, our model is able to train without requiring ground-truth solutions as labelled training data, which are often costly to obtain. Additionally, our model is invariant to the discretization of the input domain, thus providing great flexibility. We validated our proposed method on the 1D Burgers' and the 2D Heat equations, demonstrating the model's competitive results compared to other standard physics-informed models for operator learning.