CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
arxiv(2024)
摘要
Large Language Models (LLMs) have proved effective and efficient in
generating code, leading to their utilization within the hardware design
process. Prior works evaluating LLMs' abilities for register transfer level
code generation solely focus on functional correctness. However, the creativity
associated with these LLMs, or the ability to generate novel and unique
solutions, is a metric not as well understood, in part due to the challenge of
quantifying this quality.
To address this research gap, we present CreativeEval, a framework for
evaluating the creativity of LLMs within the context of generating hardware
designs. We quantify four creative sub-components, fluency, flexibility,
originality, and elaboration, through various prompting and post-processing
techniques. We then evaluate multiple popular LLMs (including GPT models,
CodeLlama, and VeriGen) upon this creativity metric, with results indicating
GPT-3.5 as the most creative model in generating hardware designs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要