Integrated Evaluation of Semantic Representation Learning, BERT, and Generative AI for Disease Name Estimation Based on Chief Complaints

Ikuo Keshi, Ikuo Keshi, Ryota Daimon, Yutaka Takaoka, Yutaka Takaoka, Atsushi Hayashi, Atsushi Hayashi

2024

Abstract

This study compared semantic representation learning + machine learning, BERT, and GPT-4 to estimate disease names from chief complaints and evaluate their accuracy. Semantic representation learning + machine learning showed high accuracy for chief complaints of at least 10 characters in the International Classification of Diseases 10th Revision (ICD-10) codes middle categories, slightly surpassing BERT. For GPT-4, the Retrieval Augmented Generation (RAG) method achieved the best performance, with a Top-5 accuracy of 84.5% when all chief complaints, including the evaluation data, were used. Additionally, the latest GPT-4o model further improved the Top-5 accuracy to 90.0%. These results suggest the potential of these methods as diagnostic support tools. Future work aims to enhance disease name estimation through more extensive evaluations by experienced physicians.

Paper Citation

in Harvard Style

Keshi I., Daimon R., Takaoka Y. and Hayashi A. (2024). Integrated Evaluation of Semantic Representation Learning, BERT, and Generative AI for Disease Name Estimation Based on Chief Complaints. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-716-0, SciTePress, pages 294-301. DOI: 10.5220/0012927100003838

in Bibtex Style

@conference{kdir24,
author={Ikuo Keshi and Ryota Daimon and Yutaka Takaoka and Atsushi Hayashi},
title={Integrated Evaluation of Semantic Representation Learning, BERT, and Generative AI for Disease Name Estimation Based on Chief Complaints},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2024},
pages={294-301},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012927100003838},
isbn={978-989-758-716-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Integrated Evaluation of Semantic Representation Learning, BERT, and Generative AI for Disease Name Estimation Based on Chief Complaints
SN - 978-989-758-716-0
AU - Keshi I.
AU - Daimon R.
AU - Takaoka Y.
AU - Hayashi A.
PY - 2024
SP - 294
EP - 301
DO - 10.5220/0012927100003838
PB - SciTePress