Neuronal dynamics are highly nonlinear and nonstationary. Traditional methods for extracting the underlying network structure from activity recordings mainly concentrate on modeling static connectivity, without accounting for key nonstationary aspects of biological neural systems, such as ongoing synaptic plasticity and neuronal modulation. To bridge this gap, we introduce the NetFormer model, an interpretable approach applicable to such systems. In our model, activity of each neuron across a series of historical time steps is defined as a token. These tokens are then linearly mapped through a query and key mechanism to generate a state- (and hence time-) dependent attention matrix that directly encodes nonstationary connectivity structures. We analyzed our formulation from the perspective of nonstationary and nonlinear networked dynamical systems. We then applied NetFormer to a large-scale, multi-modal dataset of neural activity patterns across populations of neurons in mouse visual cortex. By comparing against an allied dataset containing ground-truth baselines for connectivity between cell types, we demonstrated the effectiveness of NetFormer in predicting neural dynamics and recovering the underlying structural information about the molecular identity.