Can ChatGPT Generate Code Tasks? An Empirical Study on Using ChatGPT for Generating Tasks for SQL Queries

Ole Werger, Stefan Hanenberg, Ole Meyer, Nils Schwenzfeier, Volker Gruhn

2023

Abstract

It is now widely accepted that ML models can solve tasks that deal with the generation of source code. Now it is interesting to know whether the related tasks can be generated as well. In this paper, we evaluate how well ChatGPT can generate tasks that deal with generating simple SQL statements. To do this, ChatGPT generated for 10 different database schemas tasks with three different difficulty levels (easy, medium, hard). The generated tasks are then evaluated for suitability and difficulty by exam-correction-experienced raters. With a substantial raters agreement (α=.731), 90.67% of the tasks were considered appropriate (p<.001). However, while raters agreed that tasks, that ChatGPT considers as more difficult, are actually more difficult (p<.001), there is in general no agreement between ChatGPT’s task difficulty and rated difficulty (α=.310). Additionally, we checked in an N-of-1 experiment, whether the use of ChatGPT helped in the design of exams. It turned out that ChatGPT increased the time required to design an experiment by 40% (p=.036; d=-1.014). Altogether the present study rather raises doubts whether ChatGPT is in its current version a practical tool for the design of source code tasks.

Paper Citation

in Harvard Style

Werger O., Hanenberg S., Meyer O., Schwenzfeier N. and Gruhn V. (2023). Can ChatGPT Generate Code Tasks? An Empirical Study on Using ChatGPT for Generating Tasks for SQL Queries. In Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT; ISBN 978-989-758-665-1, SciTePress, pages 303-312. DOI: 10.5220/0012076900003538

in Bibtex Style

@conference{icsoft23,
author={Ole Werger and Stefan Hanenberg and Ole Meyer and Nils Schwenzfeier and Volker Gruhn},
title={Can ChatGPT Generate Code Tasks? An Empirical Study on Using ChatGPT for Generating Tasks for SQL Queries},
booktitle={Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT},
year={2023},
pages={303-312},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012076900003538},
isbn={978-989-758-665-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT
TI - Can ChatGPT Generate Code Tasks? An Empirical Study on Using ChatGPT for Generating Tasks for SQL Queries
SN - 978-989-758-665-1
AU - Werger O.
AU - Hanenberg S.
AU - Meyer O.
AU - Schwenzfeier N.
AU - Gruhn V.
PY - 2023
SP - 303
EP - 312
DO - 10.5220/0012076900003538
PB - SciTePress