LLMs Take on the Bebras Challenge: How Do Machines Compare to Students?

Germán Capdehourat, María Eugenia Curi, Víctor Koleszar

2025

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains. However, their performance in tasks involving logical reasoning and computational thinking continues to be an active area of research. This study analyzes the behaviour of state-of-the-art LLMs on tasks from Bebras Challenge, a test designed to promote computational thinking skills. We compare the outcomes of LLMs and primary and secondary school students from grades 3rd through 9th in Uruguay, who participated in the Bebras Challenge as part of the country’s Computational Thinking and Artificial Intelligence program. The results reveal that LLMs achieve an increasing performance as the model complexity increases, with the most advanced ones outperforming the average younger students' results. Our findings highlight both the promise and the current limitations of LLMs in tackling computational thinking challenges, providing valuable insights for their integration into educational contexts. In particular, the results suggest that LLMs could be used as a complementary tool to analyse the task's difficulty level, which could be very helpful to accelerate the time-consuming exchange and discussion process actually required to categorize the tasks.

Paper Citation

in Harvard Style

Capdehourat G., Curi M. and Koleszar V. (2025). LLMs Take on the Bebras Challenge: How Do Machines Compare to Students?. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 338-346. DOI: 10.5220/0013364100003932

in Bibtex Style

@conference{csedu25,
author={Germán Capdehourat and María Curi and Víctor Koleszar},
title={LLMs Take on the Bebras Challenge: How Do Machines Compare to Students?},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={338-346},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013364100003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - LLMs Take on the Bebras Challenge: How Do Machines Compare to Students?
SN - 978-989-758-746-7
AU - Capdehourat G.
AU - Curi M.
AU - Koleszar V.
PY - 2025
SP - 338
EP - 346
DO - 10.5220/0013364100003932
PB - SciTePress