Investigating Answer Validation Using Noise Identification and Classification in Goal-Oriented Dialogues

Sara Mirabi, Bahadorreza Ofoghi, John Yearwood, Diego Molla-Aliod, Vicky Mak-Hau

2025

Abstract

Goal-oriented conversational systems based on large language models (LLMs) provide the potential capability to gather the necessary requirements for solving tasks or developing solutions. However, in real-world scenarios, non-expert users may respond incorrectly to dialogue questions, which can impede the system’s effectiveness in eliciting accurate information. This paper presents a novel approach to detecting and categorizing noisy answers in goal-oriented conversations, with a focus on modeling linear programming problems. Using a current LLM, Gemini, we develop multi-agent synthetic conversations based on problem statements from the benchmark optimization modeling dataset NL4Opt to generate dialogues in the presence of noisy answers too. Our experiments show the LLM is not sufficiently equipped with the capabilities to detect noisy answers and hence, in almost 59% of the cases where there is a noisy answer, the LLM continues with the conversation without any attempts at resolving the noise. Thus, we also propose a two-step answer validation method for the identification and classification of noisy answers. Our findings demonstrate that while some LLM and non-LLM-based models perform well in detecting answer inaccuracies, there is a need for further improvements in classifying noisy answers into fine-grained stress types.

Paper Citation

in Harvard Style

Mirabi S., Ofoghi B., Yearwood J., Molla-Aliod D. and Mak-Hau V. (2025). Investigating Answer Validation Using Noise Identification and Classification in Goal-Oriented Dialogues. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 658-669. DOI: 10.5220/0013304000003890

in Bibtex Style

@conference{icaart25,
author={Sara Mirabi and Bahadorreza Ofoghi and John Yearwood and Diego Molla-Aliod and Vicky Mak-Hau},
title={Investigating Answer Validation Using Noise Identification and Classification in Goal-Oriented Dialogues},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2025},
pages={658-669},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013304000003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Investigating Answer Validation Using Noise Identification and Classification in Goal-Oriented Dialogues
SN - 978-989-758-737-5
AU - Mirabi S.
AU - Ofoghi B.
AU - Yearwood J.
AU - Molla-Aliod D.
AU - Mak-Hau V.
PY - 2025
SP - 658
EP - 669
DO - 10.5220/0013304000003890
PB - SciTePress