ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning

Jasper van der Waa, Jurriaan van Diggelen, Mark Neerincx, Stephan Raaijmakers

2018

Abstract

End-users of machine learning-based systems benefit from measures that quantify the trustworthiness of the underlying models. Measures like accuracy provide for a general sense of model performance, but offer no detailed information on specific model outputs. Probabilistic outputs, on the other hand, express such details, but they are not available for all types of machine learning, and can be heavily influenced by bias and lack of representative training data. Further, they are often difficult to understand for non-experts. This study proposes an intuitive certainty measure (ICM) that produces an accurate estimate of how certain a machine learning model is for a specific output, based on errors it made in the past. It is designed to be easily explainable to non-experts and to act in a predictable, reproducible way. ICM was tested on four synthetic tasks solved by support vector machines, and a real-world task solved by a deep neural network. Our results show that ICM is both more accurate and intuitive than related approaches. Moreover, ICM is neutral with respect to the chosen machine learning model, making it widely applicable.

Paper Citation

in Harvard Style

van der Waa J., van Diggelen J., Neerincx M. and Raaijmakers S. (2018). ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning.In Proceedings of the 10th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-275-2, pages 314-321. DOI: 10.5220/0006542603140321

in Bibtex Style

@conference{icaart18,
author={Jasper van der Waa and Jurriaan van Diggelen and Mark Neerincx and Stephan Raaijmakers},
title={ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning},
booktitle={Proceedings of the 10th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2018},
pages={314-321},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006542603140321},
isbn={978-989-758-275-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning
SN - 978-989-758-275-2
AU - van der Waa J.
AU - van Diggelen J.
AU - Neerincx M.
AU - Raaijmakers S.
PY - 2018
SP - 314
EP - 321
DO - 10.5220/0006542603140321