Multiple Importance Sampling for Stochastic Gradient Estimation

Corentin Salaün, Xingchang Huang, Iliyan Georgiev, Niloy Mitra, Niloy Mitra, Gurprit Singh

2025

Abstract

We introduce a theoretical and practical framework for efficient importance sampling of mini-batch samples for gradient estimation from single and multiple probability distributions. To handle noisy gradients, our framework dynamically evolves the importance distribution during training by utilizing a self-adaptive metric. Our framework combines multiple, diverse sampling distributions, each tailored to specific parameter gradients. This approach facilitates the importance sampling of vector-valued gradient estimation. Rather than naively combining multiple distributions, our framework involves optimally weighting data contribution across multiple distributions. This adapted combination of multiple importance yields superior gradient estimates, leading to faster training convergence. We demonstrate the effectiveness of our approach through empirical evaluations across a range of optimization tasks like classification and regression on both image and point cloud datasets.

Paper Citation

in Harvard Style

Salaün C., Huang X., Georgiev I., Mitra N. and Singh G. (2025). Multiple Importance Sampling for Stochastic Gradient Estimation. In Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-730-6, SciTePress, pages 401-409. DOI: 10.5220/0013311200003905

in Bibtex Style

@conference{icpram25,
author={Corentin Salaün and Xingchang Huang and Iliyan Georgiev and Niloy Mitra and Gurprit Singh},
title={Multiple Importance Sampling for Stochastic Gradient Estimation},
booktitle={Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2025},
pages={401-409},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013311200003905},
isbn={978-989-758-730-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Multiple Importance Sampling for Stochastic Gradient Estimation
SN - 978-989-758-730-6
AU - Salaün C.
AU - Huang X.
AU - Georgiev I.
AU - Mitra N.
AU - Singh G.
PY - 2025
SP - 401
EP - 409
DO - 10.5220/0013311200003905
PB - SciTePress