Reduction of Variance-related Error through Ensembling: Deep Double Descent and Out-of-Distribution Generalization

Pavlos Rath-Manakidis, Hlynur Hlynsson, Laurenz Wiskott

2022

Abstract

Prediction variance on unseen data harms the generalization performance of deep neural network classifiers. We assess the utility of forming ensembles of deep neural networks in the context of double descent (DD) on image classification tasks to mitigate the effects of model variance. To that end, we propose a method for using geometric-mean based ensembling as an approximate bias-variance decomposition of a training procedure’s test error. In ensembling equivalent models we observe that ensemble formation is more beneficial the more the models are correlated with each other. Our results show that small models afford ensembles that outperform single large models while requiring considerably fewer parameters and computational steps. We offer an explanation for this phenomenon in terms of model-internal correlations. We also find that deep DD that depends on the existence of label noise can be mitigated by using ensembles of models subject to identical label noise almost as thoroughly as by ensembles of networks each trained subject to i.i.d. noise. In the context of data drift, we find that out-of-distribution performance of ensembles can be assessed by their in-distribution performance. This aids in ascertaining the utility of ensembling for generalization.

Paper Citation

in Harvard Style

Rath-Manakidis P., Hlynsson H. and Wiskott L. (2022). Reduction of Variance-related Error through Ensembling: Deep Double Descent and Out-of-Distribution Generalization. In Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-549-4, pages 31-40. DOI: 10.5220/0010821300003122

in Bibtex Style

@conference{icpram22,
author={Pavlos Rath-Manakidis and Hlynur Hlynsson and Laurenz Wiskott},
title={Reduction of Variance-related Error through Ensembling: Deep Double Descent and Out-of-Distribution Generalization},
booktitle={Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2022},
pages={31-40},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010821300003122},
isbn={978-989-758-549-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Reduction of Variance-related Error through Ensembling: Deep Double Descent and Out-of-Distribution Generalization
SN - 978-989-758-549-4
AU - Rath-Manakidis P.
AU - Hlynsson H.
AU - Wiskott L.
PY - 2022
SP - 31
EP - 40
DO - 10.5220/0010821300003122