Seeking effective numerical approximations for partial differential equations (PDEs) is a major challenge in modern science and technology. Recently, AI-inspired data-driven solvers, such as neural operators, have achieved great success in quickly PDE solving. However, in the design of neural operators, the processing of frequency domain information is crucial, and a single processing method is difficult to comprehensively handle frequency domain information of different components. We present the V-shaped spectral mixture neural operator (VSMNO) architecture which combines spectral learning modes of different neural operators to process frequency domain information in PDE solving at various levels. For general PDE solving, we propose a residual learning structure that can transfer residuals in V-Cycle by combining frequency domain learning patterns of different neural operators to reduce high-frequency and low-frequency error. For differences in PDEs, we propose a neural operator correction strategy based on the correspondence between the PDE spectrum distribution and the neural operator spectral pattern, to correct the results by utilizing the prior knowledge of the PDE system. Experimentally, VSMNO achieves state-of-the-art and yields a relative error reduction of 22% averaged on four classical benchmarks.