Poster
in
Workshop: Statistical Frontiers in LLMs and Foundation Models
CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?
Vaishnavi Bhargava · Rajat Ghosh · Debojyoti Dutta
Keywords: [ Benchmark dataset ] [ C++ unit test ] [ Full-parameter fine-tuning ] [ PEFT/LoRA ] [ Code generation ] [ Code LLM ] [ Evaluation ] [ Few-shot learning ] [ Long context ]
We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70\%.