Singularity Stairs Following with Limited Numbers of Hidden Units

Seiya Satoh, Ryohei Nakano

2014

Abstract

In a search space of a multilayer perceptron having J hidden units, MLP(J), there exist flat areas called singular regions that cause serious stagnation of learning. Recently a method called SSF1.3 utilizing singular regions has been proposed to systematically and stably find excellent solutions. SSF1.3 starts search from a search space of MLP(1), increasing J one by one. This paper proposes SSF2 that performs MLP search by utilizing singular regions with J changed bidirectionally within a certain range. The proposed method was evaluated using artificial and real data sets.

References

  1. Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10 (2):251-276.
  2. Duda, R.O., Hart, P.E. and Stork, D.G. (2001). Pattern classification. John Wiley & Sons, Inc., New York, 2nd edition.
  3. Fukumizu, K. and Amari, S. (2000). Local minima and plateaus in hierarchical structure of multilayer perceptrons. Neural Networks, 13 (3):317-327.
  4. Hecht-Nielsen, R. (1990). Neurocomputing. AddisonWesley Publishing Company, Reading, Massachusetts.
  5. Little, M.A., McSharry, P.E., Roberts, S.J., Costello D.A.E., Moroz, I.M. (2007). Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. BioMedical Engineering OnLine 2007, 6:23.
  6. Nakano, R., Satoh, E. and Ohwaki, T. (2011). Learning method utilizing singular region of multilayer perceptron. Proc. 3rd Int. Conf. on Neural Comput. Theory and Appl., pp.106-111, 2011.
  7. Saito, K. and Nakano, R. (1997). Partial BFGS update and efficient step-length calculation for three-layer neural networks. Neural Compututation, 9 (1): 239-257.
  8. Satoh, S. and Nakano, R. (2012). Eigen vector descent and line search for multilayer perceptron. Proc. Int. MultiConf. of Engineers and Comput. Scientists, vol.1, pp.1-6.
  9. Satoh, S. and Nakano, R. (2013). Fast and stable learning utilizing singular regions of multilayer perceptron. Neural Processing Letters, 38 (2): 99-115.
  10. Satoh, S. and Nakano, R. (2014). Search pruning for a search method utilizing singular regions of multilayer perceptrons (in Japanese). IEICE Trans. on Information and Systems, J97-D (2):330-340.
  11. Sussmann, H.J. (1992). Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks, 5 (4):589-593.
  12. Watanabe, S. (2008). A formula of equations of states in singular learning machines. Proc. Int. Joint Conf. on Neural Networks, pp.2099-2106, 2008.
  13. Watanabe, S. (2009). Algebraic geometry and statistical learning theory. Cambridge Univ. Press, Cambridge.
Download


Paper Citation


in Harvard Style

Satoh S. and Nakano R. (2014). Singularity Stairs Following with Limited Numbers of Hidden Units . In Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014) ISBN 978-989-758-054-3, pages 180-186. DOI: 10.5220/0005075601800186


in Bibtex Style

@conference{ncta14,
author={Seiya Satoh and Ryohei Nakano},
title={Singularity Stairs Following with Limited Numbers of Hidden Units},
booktitle={Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014)},
year={2014},
pages={180-186},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005075601800186},
isbn={978-989-758-054-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014)
TI - Singularity Stairs Following with Limited Numbers of Hidden Units
SN - 978-989-758-054-3
AU - Satoh S.
AU - Nakano R.
PY - 2014
SP - 180
EP - 186
DO - 10.5220/0005075601800186