As large language models (LLMs) increasingly influence decision-making processes, critical questions arise about their safety, security, and evaluation in nuanced, collaborative deliberations. We present the LLM-Deliberation Quality Index, a novel framework for evaluating the deliberative capabilities of LLMs as a step towards both assessing deliberation capability and ultimately developing safer, more aligned AI systems through improved deliberative reasoning. Our approach focuses on observing multi-agent interactions and social dynamics, combining aspects of the Deliberation Quality Index from political science literature with LLM-specific measures to assess both the quality of deliberation and the believability of AI agents in simulated policy discussions.We introduce a controlled simulation environment featuring complex public policy scenarios and conduct experiments using various LLMs as deliberative agents. Our findings reveal both promising capabilities and notable limitations in current LLMs' deliberative abilities. While models like GPT-4 demonstrate high performance in providing justified reasoning (9.41/10), they struggle with social aspects of deliberation such as storytelling (2.43/10) and active questioning (3.41/10). This contrasts sharply with typical human performance in deliberations. We also observe a strong correlation between an LLM's ability to respect others' arguments and its propensity for opinion change, indicating a potential limitation in LLMs' capacity to acknowledge valid counterarguments without altering their core stance, suggesting vulnerabilities to adversarial influences.Overall, our work offers a comprehensive framework for evaluating the deliberative abilities of LLM agents across various policy domains. By assessing these capabilities, we not only gain insights into the current state of LLM reasoning but also provide a foundation for developing safer, more socially aware, and better-aligned AI systems capable of nuanced deliberation in complex decision-making scenarios. These findings could then further be used to make more trustworthy AI systems, enhancing their reliability and safety in critical deliberative contexts.