S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization

Mina Basirat, Peter M. Roth, Peter M. Roth

2021

Abstract

Recently, it has been shown that properly parametrized Leaky ReLU (LReLU) as an activation function yields significantly better results for a variety of image classification tasks. However, such methods are not feasible in practice. Either the only parameter (i.e., the slope of the negative part) needs to be set manually (L*ReLU), or the approach is vulnerable due to the gradient-based optimization and, thus, highly dependent on a proper initialization (PReLU). In this paper, we would like to exploit the benefits of piecewise linear functions, avoiding these problems. To this end, we propose a fully automatic approach to estimate the slope parameter for LReLU from the data. We realize this via Stochastic Optimization, namely Particle Swarm Optimization (PSO): S*ReLU. In this way, we can show that, compared to widely-used activation functions (including PReLU), better results can be obtained on seven different benchmark datasets. Moreover, the results even match those of L*ReLU, where the optimal parameter is estimated in a brute-force manner. In this way, our fully-automatic approach allows for drastically reducing the computational effort.

Paper Citation

in Harvard Style

Basirat M. and Roth P. (2021). S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 645-652. DOI: 10.5220/0010338506450652

in Bibtex Style

@conference{visapp21,
author={Mina Basirat and Peter M. Roth},
title={S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP},
year={2021},
pages={645-652},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010338506450652},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP
TI - S*ReLU: Learning Piecewise Linear Activation Functions via Particle Swarm Optimization
SN - 978-989-758-488-6
AU - Basirat M.
AU - Roth P.
PY - 2021
SP - 645
EP - 652
DO - 10.5220/0010338506450652
PB - SciTePress