Commit Classification into Maintenance Activities Using In-Context Learning Capabilities of Large Language Models

Yasin Sazid, Sharmista Kuri, Kazi Ahmed, Abdus Satter

2024

Abstract

Classifying software changes, i.e., commits into maintenance activities enables improved decision-making in software maintenance, thereby decreasing maintenance costs. Commonly, researchers have tried commit classification using keyword-based analysis of commit messages. Source code changes and density data have also been used for this purpose. Recent works have leveraged contextual semantic analysis of commit messages using pre-trained language models. But these approaches mostly depend on training data, making their ability to generalize a matter of concern. In this study, we explore the possibility of using in-context learning capabilities of large language models in commit classification. In-context learning does not require training data, making our approach less prone to data overfitting and more generalized. Experimental results using GPT-3 achieves a highest accuracy of 75.7% and kappa of 61.7%. It is similar to performances of other baseline models except one, highlighting the applicability of in-context learning in commit classification.

Paper Citation

in Harvard Style

Sazid Y., Kuri S., Ahmed K. and Satter A. (2024). Commit Classification into Maintenance Activities Using In-Context Learning Capabilities of Large Language Models. In Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE; ISBN 978-989-758-696-5, SciTePress, pages 506-512. DOI: 10.5220/0012686700003687

in Bibtex Style

@conference{enase24,
author={Yasin Sazid and Sharmista Kuri and Kazi Ahmed and Abdus Satter},
title={Commit Classification into Maintenance Activities Using In-Context Learning Capabilities of Large Language Models},
booktitle={Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE},
year={2024},
pages={506-512},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012686700003687},
isbn={978-989-758-696-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE
TI - Commit Classification into Maintenance Activities Using In-Context Learning Capabilities of Large Language Models
SN - 978-989-758-696-5
AU - Sazid Y.
AU - Kuri S.
AU - Ahmed K.
AU - Satter A.
PY - 2024
SP - 506
EP - 512
DO - 10.5220/0012686700003687
PB - SciTePress