Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report

<- Publications

Automated Metamorphic-Relation Generation with ChatGPT: An Experience Report

Yifan Zhang, Dave Towey, Matthew Pike

COMPSAC 2023 - 47th Annual Computers, Software, and Applications Conference | 2023 | View on Publisher's Website

Abstract

This paper presents a pilot study on using ChatGPT, a GPT-3.5-based language model, for automated generation of metamorphic relations (MRs) in the testing of autonomous driving systems (ADSs). The oracle problem remains a major challenge in ADS testing, making metamorphic testing (MT) a useful approach for verifying system correctness through input-output transformations. However, manually defining MRs is labor-intensive and prone to errors. This study evaluates the feasibility of leveraging ChatGPT for automatic MR generation, examining the effectiveness, efficiency, and limitations of the approach. The results suggest that ChatGPT can generate high-quality MRs, reducing the manual effort required for MR creation. However, while automation enhances efficiency, expert validation remains necessary to ensure correctness and applicability. This work contributes to advancing automated software testing methodologies, particularly in complex, safety-critical domains such as ADSs.