Material discovery is a critical research area with profound implications for various industries. In this work, we introduce MatExpert, a novel framework that leverages Large Language Models (LLMs) and contrastive learning to accelerate the discovery and design of new materials. Inspired by the workflow of human material experts, our approach integrates three key stages: retrieval, transition, and generation. In the initial retrieval stage, MatExpert identifies an existing material that closely matches the desired criteria. Subsequently, in the transition stage, MatExpert outlines the necessary modifications to transform this material into one that meets specific requirements. Finally, in the generation state, MatExpert handles the detailed computations and structural generation. Ourexperimental results demonstrate that MatExpert outperforms state-of-the-art methods in material generation tasks, achieving superior performance across various metrics including validity, distribution, and stability. As such, MatExpert represents a meaningful advancement in computational material discovery using modern machine learning.