Investigation into the Training Dynamics of Learned Optimizers

Jan Sobotka, Petr Šimánek, Daniel Vašata

2024

Abstract

Optimization is an integral part of modern deep learning. Recently, the concept of learned optimizers has emerged as a way to accelerate this optimization process by replacing traditional, hand-crafted algorithms with meta-learned functions. Despite the initial promising results of these methods, issues with stability and generalization still remain, limiting their practical use. Moreover, their inner workings and behavior under different conditions are not yet fully understood, making it difficult to come up with improvements. For this reason, our work examines their optimization trajectories from the perspective of network architecture symmetries and parameter update distributions. Furthermore, by contrasting the learned optimizers with their manually designed counterparts, we identify several key insights that demonstrate how each approach can benefit from the strengths of the other.

Paper Citation

in Harvard Style

Sobotka J., Šimánek P. and Vašata D. (2024). Investigation into the Training Dynamics of Learned Optimizers. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 135-146. DOI: 10.5220/0012317000003636

in Bibtex Style

@conference{icaart24,
author={Jan Sobotka and Petr Šimánek and Daniel Vašata},
title={Investigation into the Training Dynamics of Learned Optimizers},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={135-146},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012317000003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Investigation into the Training Dynamics of Learned Optimizers
SN - 978-989-758-680-4
AU - Sobotka J.
AU - Šimánek P.
AU - Vašata D.
PY - 2024
SP - 135
EP - 146
DO - 10.5220/0012317000003636
PB - SciTePress