Do Rules Still Rule? Comprehensive Evaluation of a Rule-Based Question Generation System

Bernardo Leite, Bernardo Leite, Henrique Cardoso, Henrique Cardoso

2023

Abstract

The task of Question Generation (QG) has attracted the interest of the natural language processing community in recent years. QG aims to automatically generate well-formed questions from an input (e.g., text), which can be especially relevant for computer-supported educational platforms. Recent work relies on large-scale question-answering (QA) datasets (in English) to train and build the QG systems. However, large-scale quality QA datasets are not widely available for lower-resourced languages. In this respect, this research addresses the task of QG in a lower-resourced language — Portuguese — using a traditional rule-based approach for generating wh-questions. We perform a feasibility analysis of the approach through a comprehensive evaluation supported by two studies: (1) comparing the similarity between machine-generated and human-authored questions using automatic metrics, and (2) comparing the perceived quality of machine-generated questions to those elaborated by humans. Although the results show that rule-based generated questions fall short in quality compared to those authored by humans, they also suggest that a rule-based approach remains a feasible alternative to neural-based techniques when these are not viable. The code is publicly available at https://github.com/bernardoleite/question-generation-portuguese.

Paper Citation

in Harvard Style

Leite B. and Cardoso H. (2023). Do Rules Still Rule? Comprehensive Evaluation of a Rule-Based Question Generation System. In Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-641-5, SciTePress, pages 27-38. DOI: 10.5220/0011852100003470

in Bibtex Style

@conference{csedu23,
author={Bernardo Leite and Henrique Cardoso},
title={Do Rules Still Rule? Comprehensive Evaluation of a Rule-Based Question Generation System},
booktitle={Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2023},
pages={27-38},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011852100003470},
isbn={978-989-758-641-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Do Rules Still Rule? Comprehensive Evaluation of a Rule-Based Question Generation System
SN - 978-989-758-641-5
AU - Leite B.
AU - Cardoso H.
PY - 2023
SP - 27
EP - 38
DO - 10.5220/0011852100003470
PB - SciTePress