Intrinsic Evaluation of RAG Systems for Deep-Logic Questions

Junyi Hu, You Zhou, Jie Wang

2024

Abstract

We introduce the Overall Performance Index (OPI), an intrinsic metric to evaluate retrieval-augmented generation (RAG) mechanisms for applications involving deep-logic queries. OPI is computed as the harmonic mean of two key metrics: the Logical-Relation Correctness Ratio and the average of BERT embedding similarity scores between ground-truth and generated answers. We apply OPI to assess the performance of LangChain, a popular RAG tool, using a logical relations classifier fine-tuned from GPT-4o on the RAG-Dataset-12000 from Hugging Face. Our findings show a strong correlation between BERT embedding similarity scores and extrinsic evaluation scores. Among the commonly used retrievers, the cosine similarity retriever using BERT-based embeddings outperforms others, while the Euclidean distance-based retriever exhibits the weakest performance. Furthermore, we demonstrate that combining multiple retrievers, either algorithmically or by merging retrieved sentences, yields superior performance compared to using any single retriever alone.

Download


Paper Citation


in Harvard Style

Hu J., Zhou Y. and Wang J. (2024). Intrinsic Evaluation of RAG Systems for Deep-Logic Questions. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-716-0, SciTePress, pages 489-496. DOI: 10.5220/0013070300003838


in Bibtex Style

@conference{kdir24,
author={Junyi Hu and You Zhou and Jie Wang},
title={Intrinsic Evaluation of RAG Systems for Deep-Logic Questions},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2024},
pages={489-496},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013070300003838},
isbn={978-989-758-716-0},
}


in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Intrinsic Evaluation of RAG Systems for Deep-Logic Questions
SN - 978-989-758-716-0
AU - Hu J.
AU - Zhou Y.
AU - Wang J.
PY - 2024
SP - 489
EP - 496
DO - 10.5220/0013070300003838
PB - SciTePress