Large Language Models (LLMs) have shown promising capabilities in using external tools.However, existing approaches rely on fine-tuning or in-context learning to use tools, which make syntactic mistakes and are difficult to generalize.In this paper, we propose ToolDec, a finite-state machine-guided decoding algorithm for tool-augmented LLMs.ToolDec eliminates tool-related errors by ensuring valid tool names and type-conforming arguments.Furthermore, ToolDec enables LLM to effectively select tools using only the information contained in their names, with no need for tool-specific fine-tuning.Our experiments on multiple word problem datasets show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup.We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baseline.