Compositionality is addressed by Hierarchical Reinforcement Learning (HRL) by breaking down complex tasks into manageable subtasks, but faces challenges with efficiency and generalization in continual learning environments. Potential solutions to these limitations include a dimensional reduction of the high-level state space through a symbolic representation and region of interest identification through language input for imitation learning. In this work, we propose the integration of a dynamic symbolic representation and large language models (LLM) in the framework of HRL, leveraging LLM's natural language and reasoning capabilities to bridge the gap between human instructions and an emerging abstract representation. By acting as an interface for translating human demonstrations into actionable reinforcement learning signals, LLM can improve task abstraction and planning within HRL. Our approach builds upon the Spatial-Temporal Abstraction via Reachability (STAR) algorithm, using LLM to optimize the hierarchical planning process. We conduct experiments in ant robot environments, showing how LLM can translate abstract spatial states into symbol representations and assist with task planning. The results demonstrate the potential of LLM to enhance HRL in complex, real-world tasks, particularly in environments requiring spatial reasoning and hierarchical control.