Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System

Derek Sleeman, Kiril Kostadinov, Laura Moss, Laura Moss, Malcolm Sim

2020

Abstract

Knowledge intensive clinical systems, as well as machine learning algorithms, have become more widely used over the last decade or so. These systems often need access to sizable labelled datasets which could be more useful if their instances are accurately labelled / annotated. A variety of approaches, including statistical ones, have been used to label instances. In this paper, we discuss the use of domain experts, in this case clinicians, to perform this task. Here we recognize that even highly rated domain experts can have differences of opinion on certain instances; we discuss a system inspired by the Delphi approaches which helps experts resolve their differences of opinion on classification tasks. The focus of this paper is the IS-DELPHI tool which we have implemented to address the labelling issue; we report its use in a medical domain in a study involving 12 Intensive Care Unit clinicians. The several pairs of experts initially disagreed on the classification of 11 instances but as a result of using IS-DELPHI all those disagreements were resolved. From participant feedback (questionnaires), we have concluded that the medical experts understood the task and were comfortable with the functionality provided by IS-DELPHI. We plan to further enhance the system’s capabilities and usability, and then use IS-DELPHI, which is a domain independent tool, in a number of further medical domains.

Paper Citation

in Harvard Style

Sleeman D., Kostadinov K., Moss L. and Sim M. (2020). Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System. In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF; ISBN 978-989-758-398-8, SciTePress, pages 66-76. DOI: 10.5220/0008922000660076

in Bibtex Style

@conference{healthinf20,
author={Derek Sleeman and Kiril Kostadinov and Laura Moss and Malcolm Sim},
title={Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System},
booktitle={Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF},
year={2020},
pages={66-76},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008922000660076},
isbn={978-989-758-398-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF
TI - Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System
SN - 978-989-758-398-8
AU - Sleeman D.
AU - Kostadinov K.
AU - Moss L.
AU - Sim M.
PY - 2020
SP - 66
EP - 76
DO - 10.5220/0008922000660076
PB - SciTePress