Large Language Models (LLMs) have shown great promise in building various educational applications, improving learning experiences, and providing resources for both students and educators. According to Pew Research Center, one-in-five U.S. adults said they have used ChatGPT for learning by August 2023, and the trend continues to grow. In this talk, I will discuss our ongoing efforts on leveraging LLMs to build personalized argumentative writing assistants based on feedback giving. I will also present our recent work on building human-NLP collaborative tools to help educator create active learning opportunities at scale. Finally, I will highlight challenges associated with employing LLMs for educational support, as evidenced by our research findings. Specifically, I will discuss the issues and impacts of "hallucination" and outdated knowledge as well as the lack of complex reasoning abilities and creativity.