How Proficient Is Generative AI in an Introductory Object-Oriented Programming Course?

Marina Lepp, Joosep Kaimre

2025

Abstract

In 2022, the release of ChatGPT marked a significant breakthrough in Artificial Intelligence (AI) chatbot usage, particularly impacting computer science education. AI chatbots can now generate code snippets, but their proficiency in solving various tasks remains debated. This study examines how well AI-based chatbots, including ChatGPT and Microsoft Copilot, perform in solving tasks in the "Object-Oriented Programming" course. Both tools were tested on multiple programming tasks and exam questions, with their results compared to those of students. Currently, ChatGPT-3.5 performs below the average student, while Copilot is on par. The chatbots performed better on introductory topics, though their performance varied as task difficulty increased. They also fared better on longer programming test tasks than on shorter exam tasks. A common error was failing to provide all possible solutions and misinterpreting implied requirements. Despite these challenges, both AI tools are capable of passing the course. These findings offer valuable insights for programming instructors by highlighting the strengths and weaknesses of AI chatbots, helping guide potential improvements in computer science education.

Paper Citation

in Harvard Style

Lepp M. and Kaimre J. (2025). How Proficient Is Generative AI in an Introductory Object-Oriented Programming Course?. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 216-227. DOI: 10.5220/0013199200003932

in Bibtex Style

@conference{csedu25,
author={Marina Lepp and Joosep Kaimre},
title={How Proficient Is Generative AI in an Introductory Object-Oriented Programming Course?},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={216-227},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013199200003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - How Proficient Is Generative AI in an Introductory Object-Oriented Programming Course?
SN - 978-989-758-746-7
AU - Lepp M.
AU - Kaimre J.
PY - 2025
SP - 216
EP - 227
DO - 10.5220/0013199200003932
PB - SciTePress