Can ChatGPT Fix My Code?

Viktor Csuvik, Tibor Gyimóthy, László Vidács, László Vidács

2023

Abstract

ChatGPT, a large language model (LLM) developed by OpenAI, fine-tuned on a massive dataset of text and source code, has recently gained significant attention on the internet. The model, built using the Transformer architecture, is capable of generating human-like text in a variety of tasks. In this paper, we explore the use of ChatGPT for Automated Program Repair (APR); that is, we ask the model to generate repair suggestions for instances of buggy code. We evaluate the effectiveness of our approach by comparing the repair suggestions to those made by human developers. Our results show that ChatGPT is able to generate fixes that are on par with those made by humans. Choosing the right prompt is a key aspect: on average, it was able to propose corrections in 19% of cases, but choosing the wrong input format can drop the performance to as low as 6%. By sampling real-world bugs from seminal APR datasets, generating 1000 input examples for the model, and evaluating the output manually, our study demonstrates the potential of language models for Automated Program Repair and highlights the need for further research in this area.

Paper Citation

in Harvard Style

Csuvik V., Gyimóthy T. and Vidács L. (2023). Can ChatGPT Fix My Code?. In Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT; ISBN 978-989-758-665-1, SciTePress, pages 478-485. DOI: 10.5220/0012120800003538

in Bibtex Style

@conference{icsoft23,
author={Viktor Csuvik and Tibor Gyimóthy and László Vidács},
title={Can ChatGPT Fix My Code?},
booktitle={Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT},
year={2023},
pages={478-485},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012120800003538},
isbn={978-989-758-665-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT
TI - Can ChatGPT Fix My Code?
SN - 978-989-758-665-1
AU - Csuvik V.
AU - Gyimóthy T.
AU - Vidács L.
PY - 2023
SP - 478
EP - 485
DO - 10.5220/0012120800003538
PB - SciTePress