Leveraging Out-of-the-Box Retrieval Models to Improve Mental Health Support

Theo Rummer-Downing, Theo Rummer-Downing, Julie Weeds

2023

Abstract

This work compares the performance of several information retrieval (IR) models in the search for relevant mental health documents based on relevance to forum post queries from a fully-moderated online mental health service. Three different architectures are assessed: a sparse lexical model, BM25, is used as a baseline, alongside two neural SBERT-based architectures - the bi-encoder and the cross-encoder. We highlight the credibility of using pretrained language models (PLMs) out-of-the-box, without an additional fine-tuning stage, to achieve high retrieval quality across a limited set of resources. Error analysis of the ranking results suggested PLMs make errors on documents which contain so called red-herrings - words which are semantically related but irrelevant to the query - whereas human judgements were found to suffer when queries are vague and present no clear information need. Further, we show that bias towards an author’s writing style within a PLM affects retrieval quality and, therefore, can impact on the success of mental health support if left unaddressed.

Paper Citation

in Harvard Style

Rummer-Downing T. and Weeds J. (2023). Leveraging Out-of-the-Box Retrieval Models to Improve Mental Health Support. In Proceedings of the 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2023) - Volume 5: HEALTHINF; ISBN 978-989-758-631-6, SciTePress, pages 64-73. DOI: 10.5220/0011634300003414

in Bibtex Style

@conference{healthinf23,
author={Theo Rummer-Downing and Julie Weeds},
title={Leveraging Out-of-the-Box Retrieval Models to Improve Mental Health Support},
booktitle={Proceedings of the 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2023) - Volume 5: HEALTHINF},
year={2023},
pages={64-73},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011634300003414},
isbn={978-989-758-631-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2023) - Volume 5: HEALTHINF
TI - Leveraging Out-of-the-Box Retrieval Models to Improve Mental Health Support
SN - 978-989-758-631-6
AU - Rummer-Downing T.
AU - Weeds J.
PY - 2023
SP - 64
EP - 73
DO - 10.5220/0011634300003414
PB - SciTePress