CHEMFUZZ: Large Language Models-Assisted Fuzzing for Quantum Chemistry Software Bug Detection.

Quantum chemistry software implements the first principle quantum computation and is indispensable in both scientific research and chemical industries. Any bugs in such software will lead to serious consequences, thus defeating its trustworthiness and reliability. However, bug detection techniques for such software have not been fully investigated. In this paper, to fill this gap, we propose a novel approach to fuzz quantum chemistry software with the aid of Large Language Models (LLMs). Our basic idea is utilize LLMs to mutate and generate syntactic and semantic valid input files from seed inputs, by proving valuable domain-specific knowledge of chemistry. With this basic idea, we have designed and implemented CHEMFuzz, a fully automatic fuzzing framework to fuzz quantum chemistry software for bugs. Our evaluation of CHEMFUZZ leverages popular LLMs including GPT3.5, Claude-2, and Bart as test oracles to generate parameters to mutate inputs and analyze computation results. CHEMFUZZ detected 40 unique bugs, which have been classified and reported to developers, with a code coverage of 17.4%.