Frame averaging (FA), a recent progress in geometric deep learning, is a general framework that endows a given architecture with the ability to transform data equivariantly. However, serving FA as a model wrapper introduces additional computation that grows linearly with the group's cardinality and may hinder the exploitation of 3D structures, making it challenging to model macro-molecules such as proteins and nucleic acids. In this paper, we present FAFormer, an equivariant Transformer model that incorporates FA as a basic component within each layer. Such incorporation allows FAFormer to model the coordinates in the latent space directly without using other elaborate geometric features. Building on this foundation, we introduce an equivariant cross-attention module to FAFormer to capture the interactions between node and coordinate representations. Besides, an equivariant feed-forward network is proposed for enhancing the communication between them. To evaluate FAFormer's performance, we establish two benchmark datasets for nucleic acid-protein contact prediction and compare FAFormer with 8 different baseline models. With these two innovations, FAFormer outperforms all the baselines and achieves state-of-the-art performance.