Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3

Carlo Bellettini, Michael Lodi, Michael Lodi, Michael Lodi, Violetta Lonati, Violetta Lonati, Mattia Monga, Mattia Monga, Anna Morpurgo, Anna Morpurgo

2023

Abstract

In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the “Bebras International Challenge on Informatics and Computational Thinking”. In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.

Paper Citation

in Harvard Style

Bellettini C., Lodi M., Lonati V., Monga M. and Morpurgo A. (2023). Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. In Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-641-5, SciTePress, pages 59-69. DOI: 10.5220/0012007500003470

in Bibtex Style

@conference{csedu23,
author={Carlo Bellettini and Michael Lodi and Violetta Lonati and Mattia Monga and Anna Morpurgo},
title={Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3},
booktitle={Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2023},
pages={59-69},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012007500003470},
isbn={978-989-758-641-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3
SN - 978-989-758-641-5
AU - Bellettini C.
AU - Lodi M.
AU - Lonati V.
AU - Monga M.
AU - Morpurgo A.
PY - 2023
SP - 59
EP - 69
DO - 10.5220/0012007500003470
PB - SciTePress