Online Importance Sampling for Stochastic Gradient Optimization

Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy Mitra, Niloy Mitra, Gurprit Singh

2025

Abstract

Machine learning optimization often depends on stochastic gradient descent, where the precision of gradient estimation is vital for model performance. Gradients are calculated from mini-batches formed by uniformly selecting data samples from the training dataset. However, not all data samples contribute equally to gradient estimation. To address this, various importance sampling strategies have been developed to prioritize more significant samples. Despite these advancements, all current importance sampling methods encounter challenges related to computational efficiency and seamless integration into practical machine learning pipelines. In this work, we propose a practical algorithm that efficiently computes data importance on-the-fly during training, eliminating the need for dataset preprocessing. We also introduce a novel metric based on the derivative of the loss w.r.t. the network output, designed for mini-batch importance sampling. Our metric prioritizes influential data points, thereby enhancing gradient estimation accuracy. We demonstrate the effectiveness of our approach across various applications. We first perform classification and regression tasks to demonstrate improvements in accuracy. Then, we show how our approach can also be used for online data pruning by identifying and discarding data samples that contribute minimally towards the training loss. This significantly reduce training time with negligible loss in the accuracy of the model.

Paper Citation

in Harvard Style

Salaün C., Huang X., Georgiev I., Mitra N. and Singh G. (2025). Online Importance Sampling for Stochastic Gradient Optimization. In Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-730-6, SciTePress, pages 130-140. DOI: 10.5220/0013311100003905

in Bibtex Style

@conference{icpram25,
author={Corentin Salaün and Xingchang Huang and Iliyan Georgiev and Niloy Mitra and Gurprit Singh},
title={Online Importance Sampling for Stochastic Gradient Optimization},
booktitle={Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2025},
pages={130-140},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013311100003905},
isbn={978-989-758-730-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Online Importance Sampling for Stochastic Gradient Optimization
SN - 978-989-758-730-6
AU - Salaün C.
AU - Huang X.
AU - Georgiev I.
AU - Mitra N.
AU - Singh G.
PY - 2025
SP - 130
EP - 140
DO - 10.5220/0013311100003905
PB - SciTePress