Using Large Language Models for Commit Message Generation: A Preliminary Study
CoRR(2024)
摘要
A commit message is a textual description of the code changes in a commit,
which is a key part of the Git version control system (VCS). It captures the
essence of software updating. Therefore, it can help developers understand code
evolution and facilitate efficient collaboration between developers. However,
it is time-consuming and labor-intensive to write good and valuable commit
messages. Some researchers have conducted extensive studies on the automatic
generation of commit messages and proposed several methods for this purpose,
such as generation-based and retrieval-based models. However, seldom studies
explored whether large language models (LLMs) can be effectively used for the
automatic generation of commit messages. To this end, this paper designed and
conducted a series of experiments to comprehensively evaluate the performance
of popular open-source and closed-source LLMs, i.e., Llama 2 and ChatGPT, in
commit message generation. The results indicate that considering the BLEU and
Rouge-L metrics, LLMs surpass existing methods in certain indicators but lag
behind in others. After human evaluations, however, LLMs show a distinct
advantage over all these existing methods. Especially, in 78
samples, the commit messages generated by LLMs were evaluated by humans as the
best. This work not only reveals the promising potential of using LLMs to
generate commit messages, but also explores the limitations of commonly used
metrics in evaluating the quality of automatically generated commit messages.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要