Learning Embeddings from Free-text Triage Notes using Pretrained Transformer Models

Émilien Arnaud, Mahmoud Elbattah, Mahmoud Elbattah, Maxime Gignon, Gilles Dequen

2022

Abstract

The advent of transformer models has allowed for tremendous progress in the Natural Language Processing (NLP) domain. Pretrained transformers could successfully deliver the state-of-the-art performance in a myriad of NLP tasks. This study presents an application of transformers to learn contextual embeddings from free-text triage notes, widely recorded at the emergency department. A large-scale retrospective cohort of triage notes of more than 260K records was provided by the University Hospital of Amiens-Picardy in France. We utilize a set of Bidirectional Encoder Representations from Transformers (BERT) for the French language. The quality of embeddings is empirically examined based on a set of clustering models. In this regard, we provide a comparative analysis of popular models including CamemBERT, FlauBERT, and mBART. The study could be generally regarded as an addition to the ongoing contributions of applying the BERT approach in the healthcare context.

Paper Citation

in Harvard Style

Arnaud É., Elbattah M., Gignon M. and Dequen G. (2022). Learning Embeddings from Free-text Triage Notes using Pretrained Transformer Models. In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: Scale-IT-up, ISBN 978-989-758-552-4, pages 835-841. DOI: 10.5220/0011012800003123

in Bibtex Style

@conference{scale-it-up22,
author={Émilien Arnaud and Mahmoud Elbattah and Maxime Gignon and Gilles Dequen},
title={Learning Embeddings from Free-text Triage Notes using Pretrained Transformer Models},
booktitle={Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: Scale-IT-up,},
year={2022},
pages={835-841},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011012800003123},
isbn={978-989-758-552-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: Scale-IT-up,
TI - Learning Embeddings from Free-text Triage Notes using Pretrained Transformer Models
SN - 978-989-758-552-4
AU - Arnaud É.
AU - Elbattah M.
AU - Gignon M.
AU - Dequen G.
PY - 2022
SP - 835
EP - 841
DO - 10.5220/0011012800003123