A Continuum among Logarithmic, Linear, and Exponential Functions, and Its Potential to Improve Generalization in Neural Networks

Luke B. Godfrey, Michael S. Gashler

2015

Abstract

We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, and sinusoids.

References

  1. Berry, M. W., Brown, M., Langville, A. N., Pauca, V. P., and Plemmons, R. J. (2007). Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 52(1):155-173.
  2. Brunet, J. P., Tamayo, P., Golub, T. R., and Mesirov, J. P. (2004). Metagenes and molecular pattern discovery using matrix factorization. Proceedings of the National Academy of Sciences, 101(12):4164-4169.
  3. Cai, J., Candès, E. J., and Shen, Z. (2010). A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956-1982.
  4. Chen, S., Cowan, C. F., and Grant, P. M. (1991). Orthogonal least squares learning algorithm for radial basis function networks. IEEE Transactions on Neural Networks, 2(2):302-309.
  5. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314.
  6. Gashler, M. S. and Ashmore, S. C. (2014). Training deep fourier neural networks to fit time-series data. Lecture Notes in Bioinformatics, 8590:48-55.
  7. Godfrey, L. B. and Gashler, M. S. (2015). Neural decomposition of time-series data for effective generalization. Publication Pending.
  8. Kalman, B. L. and Kwasny, S. C. (1992). Why tanh: choosing a sigmoidal function. In Neural Networks, 1992. IJCNN., International Joint Conference on, volume 4, pages 578-581. IEEE.
  9. Koren, Y., Bell, R., and Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8):30-37.
  10. Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807-814.
  11. Qasem, S. N. and Shamsuddin, S. M. (2011). Radial basis function network based on time variant multiobjective particle swarm optimization for medical diseases diagnosis. Applied Soft Computing, 11(1):1427- 1438.
  12. Schölkopf, B., Sung, K.-K., Burges, C. J., Girosi, F., Niyogi, P., Poggio, T., and Vapnik, V. (1997). Comparing support vector machines with gaussian kernels to radial basis function classifiers. Signal Processing, IEEE Transactions on, 45(11):2758-2765.
  13. Silvescu, A. (1999). Fourier neural networks. In Neural Networks, 1999. IJCNN'99. International Joint Conference on, volume 1, pages 488-491. IEEE.
  14. Tan, H. (2006). Fourier neural networks and generalized single hidden layer networks in aircraft engine fault diagnostics. Journal of engineering for gas turbines and power, 128(4):773-782.
  15. Xu, W., Liu, X., and Gong, Y. (2003). Document clustering based on non-negative matrix factorization. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 267-273. ACM.
  16. Zeiler, M. D., Ranzato, M., Monga, R., Mao, M., Yang, K., Le, Q. V., Nguyen, P., Senior, A., Vanhoucke, V., Dean, J., et al. (2013). On rectified linear units for speech processing. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 3517-3521. IEEE.
  17. Zuo, W., Zhu, Y., and Cai, L. (2009). Fourier-neuralnetwork-based learning control for a class of nonlinear systems with flexible components. Neural Networks, IEEE Transactions on, 20(1):139-151.
Download


Paper Citation


in Harvard Style

Godfrey L. and Gashler M. (2015). A Continuum among Logarithmic, Linear, and Exponential Functions, and Its Potential to Improve Generalization in Neural Networks . In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2015) ISBN 978-989-758-158-8, pages 481-486. DOI: 10.5220/0005635804810486


in Bibtex Style

@conference{kdir15,
author={Luke B. Godfrey and Michael S. Gashler},
title={A Continuum among Logarithmic, Linear, and Exponential Functions, and Its Potential to Improve Generalization in Neural Networks},
booktitle={Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2015)},
year={2015},
pages={481-486},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005635804810486},
isbn={978-989-758-158-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR, (IC3K 2015)
TI - A Continuum among Logarithmic, Linear, and Exponential Functions, and Its Potential to Improve Generalization in Neural Networks
SN - 978-989-758-158-8
AU - Godfrey L.
AU - Gashler M.
PY - 2015
SP - 481
EP - 486
DO - 10.5220/0005635804810486