Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models
CoRR(2023)
摘要
Large Language Models (LLMs) have demonstrated remarkable potential in code
generation. The integration of Chain of Thought (CoT) reasoning can further
boost their performance. However, current CoT methods often require manual
writing or LLMs with over 100 billion parameters to generate, impeding their
applicability in resource-constrained scenarios. In this study, we investigate
lightweight Language Models (lLMs), which are defined to have fewer than 10
billion parameters. Empirically, we find that most lLMs cannot generate
high-quality CoTs when prompted by the few-shot method, but can take advantage
of high-quality CoTs generated elsewhere to improve their performance in code
generation. Based on these findings, we design a novel approach COTTON which
can leverage lLMs to automatically generate CoTs for code generation. We
synthesize new datasets and conduct extensive experiments on various
benchmarks. The results show that the CoTs generated by COTTON outperform the
baselines in terms of automated and human evaluation metrics. In particular,
the CoTs generated by COTTON boost various lLMs to achieve higher performance
gains than those generated by LLMs such as ChatGLM (130B), and are competitive
with those generated by gpt-3.5-turbo (175B). Our study also showcases the
potential of lLMs in software engineering applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要