Invasive Measurements Can Provide an Objective Ceiling for Non-invasive Machine Learning Predictions

Christopher Bartlett, Jamie Bossenbroek, Yukie Ueyama, Patricia Mccallinhart, Aaron Trask, William Ray

2021

Abstract

Early stopping is an extremely common tool to minimize overfitting, which would otherwise be a cause of poor generalization of the model to novel data. However, early stopping is a heuristic that, while effective, primarily relies on ad hoc parameters and metrics. Optimizing when to stop remains a challenge. In this paper, we suggest that for some biomedical applications, a natural dichotomy of invasive/non-invasive measurements of a biological system can be exploited to provide objective advice on early stopping. We discuss the conditions where invasive measurements of a biological process should provide better predictions than non-invasive measurements, or at best offer parity. Hence, if data from an invasive measurement is available locally, or from the literature, that information can be leveraged to know with high certainty whether a model of non-invasive data is overfitted. We present paired invasive/non-invasive cardiac and coronary artery measurements from two mouse strains, one of which spontaneously develops type 2 diabetes, posed as a classification problem. Examination of the various stopping rules shows that generalization is reduced with more training epochs and commonly applied stopping rules give widely different generalization error estimates. The use of an empirically derived training ceiling is demonstrated to be helpful as added information to leverage early stopping in order to reduce overfitting.

Paper Citation

in Harvard Style

Bartlett C., Bossenbroek J., Ueyama Y., Mccallinhart P., Trask A. and Ray W. (2021). Invasive Measurements Can Provide an Objective Ceiling for Non-invasive Machine Learning Predictions. In Proceedings of the 18th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP, ISBN 978-989-758-525-8, pages 73-80. DOI: 10.5220/0010582000730080

in Bibtex Style

@conference{sigmap21,
author={Christopher Bartlett and Jamie Bossenbroek and Yukie Ueyama and Patricia Mccallinhart and Aaron Trask and William Ray},
title={Invasive Measurements Can Provide an Objective Ceiling for Non-invasive Machine Learning Predictions},
booktitle={Proceedings of the 18th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP,},
year={2021},
pages={73-80},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010582000730080},
isbn={978-989-758-525-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP,
TI - Invasive Measurements Can Provide an Objective Ceiling for Non-invasive Machine Learning Predictions
SN - 978-989-758-525-8
AU - Bartlett C.
AU - Bossenbroek J.
AU - Ueyama Y.
AU - Mccallinhart P.
AU - Trask A.
AU - Ray W.
PY - 2021
SP - 73
EP - 80
DO - 10.5220/0010582000730080