Authors:
Junyi Hu
;
You Zhou
and
Jie Wang
Affiliation:
Miner School of Computer & Information Sciences, University of Massachusetts, Lowell, MA, U.S.A.
Keyword(s):
Retrieval Augmented Generation, Logical-Relation Correctness Ratio, Overall Performance Index.
Abstract:
We introduce the Overall Performance Index (OPI), an intrinsic metric to evaluate retrieval-augmented generation (RAG) mechanisms for applications involving deep-logic queries. OPI is computed as the harmonic mean of two key metrics: the Logical-Relation Correctness Ratio and the average of BERT embedding similarity scores between ground-truth and generated answers. We apply OPI to assess the performance of LangChain, a popular RAG tool, using a logical relations classifier fine-tuned from GPT-4o on the RAG-Dataset-12000 from Hugging Face. Our findings show a strong correlation between BERT embedding similarity scores and extrinsic evaluation scores. Among the commonly used retrievers, the cosine similarity retriever using BERT-based embeddings outperforms others, while the Euclidean distance-based retriever exhibits the weakest performance. Furthermore, we demonstrate that combining multiple retrievers, either algorithmically or by merging retrieved sentences, yields superior perfor
mance compared to using any single retriever alone.
(More)