Unmasking the Giant: A Comprehensive Evaluation of ChatGPT’s Proficiency in Coding Algorithms and Data Structures

Sayed Arefin, Tasnia Heya, Hasan Al-Qudah, Ynes Ineza, Abdul Serwadda

2024

Abstract

We conduct an extensive analysis of ChatGPT, a standout Large Language Model (LLM), particularly in coding within the Python language, focusing on data structures and algorithms. We assess ChatGPT’s ability to accurately solve coding problems, its code quality, and the nature of run-time errors. Additionally, we examine how ChatGPT’s code performs when it executes but doesn’t solve the problem, identifying error patterns. We also explore whether ChatGPT has memorized training data through a structured experiment. Comparing with human performance where possible, our study encompasses both GPT-3.5 and GPT-4 models, various subtopics within the main areas, and problems of different complexities.

Paper Citation

in Harvard Style

Arefin S., Heya T., Al-Qudah H., Ineza Y. and Serwadda A. (2024). Unmasking the Giant: A Comprehensive Evaluation of ChatGPT’s Proficiency in Coding Algorithms and Data Structures. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 412-419. DOI: 10.5220/0012467100003636

in Bibtex Style

@conference{icaart24,
author={Sayed Arefin and Tasnia Heya and Hasan Al-Qudah and Ynes Ineza and Abdul Serwadda},
title={Unmasking the Giant: A Comprehensive Evaluation of ChatGPT’s Proficiency in Coding Algorithms and Data Structures},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART},
year={2024},
pages={412-419},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012467100003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART
TI - Unmasking the Giant: A Comprehensive Evaluation of ChatGPT’s Proficiency in Coding Algorithms and Data Structures
SN - 978-989-758-680-4
AU - Arefin S.
AU - Heya T.
AU - Al-Qudah H.
AU - Ineza Y.
AU - Serwadda A.
PY - 2024
SP - 412
EP - 419
DO - 10.5220/0012467100003636
PB - SciTePress