Beyond Functionality: Automating Algorithm Design Evaluation in Introductory Programming Courses

Caio Oliveira, Leandra Silva, João Brunet

2025

Abstract

Introductory programming courses often enroll large numbers of students, making individualized feedback challenging. Automated grading tools are widely used, yet they typically focus on functional correctness rather than the structural aspects of algorithms—such as the use of appropriate functions or data structures. This limitation can lead students to create solutions that, while correct in functionality, contain structural flaws that might hinder their learning. To address this gap, we applied the concept of design tests to automatically detect structural issues in students’ algorithms. Our tool, PyDesignWizard, provides an API leveraging Python’s Abstract Syntax Tree (AST) to streamline the creation of design tests. We quantitatively evaluated this approach on 1,714 student programs from an introductory course at the Federal University of Campina Grande, achieving 98.6% accuracy, a 0.99 true negative rate, 77% precision, and 84% recall. A qualitative evaluation was also conducted through interviews with 16 programming educators using the Think Aloud Protocol. The results indicated a high level of understanding and positive feedback: 15 educators grasped the concept quickly, and 8 highlighted the educational benefits of identifying design issues. In summary, our approach empowers educators to write structural tests for automated feedback, enhancing the quality of grading in programming education.

Paper Citation

in Harvard Style

Oliveira C., Silva L. and Brunet J. (2025). Beyond Functionality: Automating Algorithm Design Evaluation in Introductory Programming Courses. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 516-525. DOI: 10.5220/0013242000003932

in Bibtex Style

@conference{csedu25,
author={Caio Oliveira and Leandra Silva and João Brunet},
title={Beyond Functionality: Automating Algorithm Design Evaluation in Introductory Programming Courses},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={516-525},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013242000003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - Beyond Functionality: Automating Algorithm Design Evaluation in Introductory Programming Courses
SN - 978-989-758-746-7
AU - Oliveira C.
AU - Silva L.
AU - Brunet J.
PY - 2025
SP - 516
EP - 525
DO - 10.5220/0013242000003932
PB - SciTePress