The reconstruction of complex time-evolving fields from a small number of sensor observations is a grand challenge in a wide range of scientific and industrial applications. Frequently, sensors have very sparse spatial coverage, and report noisy observations from highly non-linear phenomena. While numerical simulations can model some of these phenomena in a classical manner, the inverse problem is not well-posed, hence data-driven modeling can provide crucial disambiguation. Here we present the \textit{Senseiver}, an attention-based framework that excels in the task of reconstructing spatially-complex fields from a small number of observations. Building on the \textit{Perceiver IO} model, the Senseiver reconstructs complex \textit{n}-dimensional fields accurately using a small number of sensor observations by encoding arbitrarily-sized sparse sets of inputs into a latent space using cross-attention, which produces a uniform-sized space regardless of the number of observations. This same property allows very efficient training as a consequence of the being able to decode only a sparse set of observations as outputs. This enables efficient training of data with complex boundary conditions (sea temperature) and to extremely large and complex domains (3D porous media). We show that the Senseiver sets a new state of the art for three existing datasets, including real-world sea temperature observations, and pushes the bounds of sparse reconstruction using a large-scale simulation of two fluids flowing through a complex 3D domain.