Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report
COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference, August 2023
Yifan Zhang, Dave Towey, Matthew Pike. 2023. Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report. In COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference. DOI:https://doi.org/10.1109/COMPSAC57700.2023.00275
Yifan Zhang and Dave Towey and Matthew Pike. (2023). Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report. COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference. https://doi.org/10.1109/COMPSAC57700.2023.00275
Yifan Zhang and Dave Towey and Matthew Pike. "Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report." COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference, 2023. https://doi.org/10.1109/COMPSAC57700.2023.00275
Yifan Zhang, Dave Towey, Matthew Pike. 2023. Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report. COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference. doi:10.1109/COMPSAC57700.2023.00275
Yifan Zhang and Dave Towey and Matthew Pike, "Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report," COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference, 2023. doi: 10.1109/COMPSAC57700.2023.00275
@inproceedings{compsac-2023-1,
title={Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report},
author={Yifan Zhang and Dave Towey and Matthew Pike},
booktitle={COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference},
year={2023},
doi={10.1109/COMPSAC57700.2023.00275}
}
Autonomous driving systems, Metamorphic testing, Metamorphic relation generation, Oracle problem, ChatGPT, Large language models, Software testing automation
Abstract
This paper presents a pilot study on using ChatGPT, a GPT-3.5-based language model, for automated generation of metamorphic relations (MRs) in the testing of autonomous driving systems (ADSs). The oracle problem remains a major challenge in ADS testing, making metamorphic testing (MT) a useful approach for verifying system correctness through input-output transformations. However, manually defining MRs is labor-intensive and prone to errors. This study evaluates the feasibility of leveraging ChatGPT for automatic MR generation, examining the effectiveness, efficiency, and limitations of the approach. The results suggest that ChatGPT can generate high-quality MRs, reducing the manual effort required for MR creation. However, while automation enhances efficiency, expert validation remains necessary to ensure correctness and applicability. This work contributes to advancing automated software testing methodologies, particularly in complex, safety-critical domains such as ADSs.