Step Size Control in Evolutionary Algorithms for Neural Architecture Search

Christian Nieber, Douglas Dias, Enrique Naredo Garcia, Conor Ryan

2024

Abstract

This work examines how evolutionary Neural Architecture Search (NAS) algorithms can be improved by controlling the step size of the mutation of numerical parameters. The proposed NAS algorithms are based on F-DENSER, a variation of Dynamic Structured Grammatical Evolution (DSGE). Overall, a (1+5) Evolutionary Strategy is used. Two methods of controlling the step size of mutations of numeric values are compared to Random Search and F-DENSER: Decay of the step size over time and adaptive step size for mutations. The search for lightweight, LeNet-like CNN architectures for MNIST classification is used as a benchmark, optimizing for both accuracy and small architectures. An architecture is described by about 30 evolvable parameters. Experiments show that with step size control, convergence is faster, better performing neural architectures are found on average, and with lower variance. The smallest architecture found during the experiments reached an accuracy of 98.8% on MNIST with only 5,450 free parameters, compared to the 62,158 parameters of LeNet-5.

Paper Citation

in Harvard Style

Nieber C., Dias D., Naredo Garcia E. and Ryan C. (2024). Step Size Control in Evolutionary Algorithms for Neural Architecture Search. In Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA; ISBN 978-989-758-721-4, SciTePress, pages 288-295. DOI: 10.5220/0013013800003837

in Bibtex Style

@conference{ecta24,
author={Christian Nieber and Douglas Dias and Enrique Naredo Garcia and Conor Ryan},
title={Step Size Control in Evolutionary Algorithms for Neural Architecture Search},
booktitle={Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA},
year={2024},
pages={288-295},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013013800003837},
isbn={978-989-758-721-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA
TI - Step Size Control in Evolutionary Algorithms for Neural Architecture Search
SN - 978-989-758-721-4
AU - Nieber C.
AU - Dias D.
AU - Naredo Garcia E.
AU - Ryan C.
PY - 2024
SP - 288
EP - 295
DO - 10.5220/0013013800003837
PB - SciTePress