Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report.

This paper reports on a pilot study of using ChatGPT, a language model based on GPT-3.5 architecture, for automatic generation of metamorphic relations (MRs), in the context of testing of autonomous driving systems (ADSs). The oracle problem is a major challenge in testing such systems, where it is difficult to determine whether or not the output of a system is correct. Metamorphic testing (MT) can alleviate this problem by checking the consistency of the system's outputs under various transformations. However, manual generation of MRs is often a time-consuming and error-prone process. Automated MR generation can yield several benefits, including enhanced efficiency, quality, coverage, scalability, and reusability in software testing, thereby facilitating a more comprehensive and effective testing process. In this paper, we investigate the effectiveness of using ChatGPT for automatic generation of MRs for ADSs. We provide a detailed methodology for generating MRs using ChatGPT and evaluate the generated MRs using our domain knowledge and existing MRs. The results of our study indicate that our proposed approach is effective at generating high-quality MRs, and can significantly reduce the manual effort required for MR generation. Furthermore, we discuss the practical implications and limitations of using ChatGPT for MR generation and provide recommendations for future research. Our study contributes to the advancement of automated testing of ADSs, which is crucial for ensuring their safety and reliability in real-world scenarios.