Activation Adaptation in Neural Networks

Farnoush Farhadi, Vahid Nia, Vahid Nia, Andrea Lodi

2020

Abstract

Many neural network architectures rely on the choice of the activation function for each hidden layer. Given the activation function, the neural network is trained over the bias and the weight parameters. The bias catches the center of the activation, and the weights capture the scale. Here we propose to train the network over a shape parameter as well. This view allows each neuron to tune its own activation function and adapt the neuron curvature towards a better prediction. This modification only adds one further equation to the back-propagation for each neuron. Re-formalizing activation functions as a comulative distribution function (cdf) generalizes the class of activation function extensively. We propose to generalizing towards extensive class of activation functions and study: i) skewness and ii) smoothness of activation functions. Here we introduce adaptive Gumbel activation function as a bridge between assymmetric Gumbel and symmetric sigmoid. A similar approach is used to invent a smooth version of ReLU. Our comparison with common activation functions suggests different data representation especially in early neural network layers. This adaptation also provides prediction improvement.

Paper Citation

in Harvard Style

Farhadi F., Nia V. and Lodi A. (2020). Activation Adaptation in Neural Networks. In Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-397-1, pages 249-257. DOI: 10.5220/0009175102490257

in Bibtex Style

@conference{icpram20,
author={Farnoush Farhadi and Vahid Nia and Andrea Lodi},
title={Activation Adaptation in Neural Networks},
booktitle={Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2020},
pages={249-257},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009175102490257},
isbn={978-989-758-397-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Activation Adaptation in Neural Networks
SN - 978-989-758-397-1
AU - Farhadi F.
AU - Nia V.
AU - Lodi A.
PY - 2020
SP - 249
EP - 257
DO - 10.5220/0009175102490257