Hierarchical Reinforcement Learning (HRL) breaks down complex tasks into manageable subtasks, but faces challenges with efficiency and generalization in high-dimensional, open-ended environments. Human In The Loop approaches offer a potential solution to these limitations. In this work, we propose the integration of large language models (LLM) with HRL, leveraging LLM's natural language and reasoning capabilities and study how to bridge the gap between human instructions and HRL's learned abstract representations. By translating human demonstrations into actionable reinforcement learning signals, LLM can improve task abstraction and planning within HRL. Our approach builds upon the Spatial-Temporal Abstraction via Reachability (STAR) algorithm, using LLM to optimize the hierarchical planning process. We conduct experiments in ant robot environments, showing how emergent symbolic representations can be used by LLM to assist with task planning. The results illustrate the potential of LLM to enhance HRL in complex, real-world tasks, particularly in environments requiring spatial reasoning and hierarchical control.