# Singularity Stairs Following with Limited Numbers of Hidden Units

### Seiya Satoh, Ryohei Nakano

#### Abstract

In a search space of a multilayer perceptron having J hidden units, MLP(J), there exist flat areas called singular regions that cause serious stagnation of learning. Recently a method called SSF1.3 utilizing singular regions has been proposed to systematically and stably find excellent solutions. SSF1.3 starts search from a search space of MLP(1), increasing J one by one. This paper proposes SSF2 that performs MLP search by utilizing singular regions with J changed bidirectionally within a certain range. The proposed method was evaluated using artificial and real data sets.

#### References

- Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10 (2):251-276.
- Duda, R.O., Hart, P.E. and Stork, D.G. (2001). Pattern classification. John Wiley & Sons, Inc., New York, 2nd edition.
- Fukumizu, K. and Amari, S. (2000). Local minima and plateaus in hierarchical structure of multilayer perceptrons. Neural Networks, 13 (3):317-327.
- Hecht-Nielsen, R. (1990). Neurocomputing. AddisonWesley Publishing Company, Reading, Massachusetts.
- Little, M.A., McSharry, P.E., Roberts, S.J., Costello D.A.E., Moroz, I.M. (2007). Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. BioMedical Engineering OnLine 2007, 6:23.
- Nakano, R., Satoh, E. and Ohwaki, T. (2011). Learning method utilizing singular region of multilayer perceptron. Proc. 3rd Int. Conf. on Neural Comput. Theory and Appl., pp.106-111, 2011.
- Saito, K. and Nakano, R. (1997). Partial BFGS update and efficient step-length calculation for three-layer neural networks. Neural Compututation, 9 (1): 239-257.
- Satoh, S. and Nakano, R. (2012). Eigen vector descent and line search for multilayer perceptron. Proc. Int. MultiConf. of Engineers and Comput. Scientists, vol.1, pp.1-6.
- Satoh, S. and Nakano, R. (2013). Fast and stable learning utilizing singular regions of multilayer perceptron. Neural Processing Letters, 38 (2): 99-115.
- Satoh, S. and Nakano, R. (2014). Search pruning for a search method utilizing singular regions of multilayer perceptrons (in Japanese). IEICE Trans. on Information and Systems, J97-D (2):330-340.
- Sussmann, H.J. (1992). Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks, 5 (4):589-593.
- Watanabe, S. (2008). A formula of equations of states in singular learning machines. Proc. Int. Joint Conf. on Neural Networks, pp.2099-2106, 2008.
- Watanabe, S. (2009). Algebraic geometry and statistical learning theory. Cambridge Univ. Press, Cambridge.

#### Paper Citation

#### in Harvard Style

Satoh S. and Nakano R. (2014). **Singularity Stairs Following with Limited Numbers of Hidden Units** . In *Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014)* ISBN 978-989-758-054-3, pages 180-186. DOI: 10.5220/0005075601800186

#### in Bibtex Style

@conference{ncta14,

author={Seiya Satoh and Ryohei Nakano},

title={Singularity Stairs Following with Limited Numbers of Hidden Units},

booktitle={Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014)},

year={2014},

pages={180-186},

publisher={SciTePress},

organization={INSTICC},

doi={10.5220/0005075601800186},

isbn={978-989-758-054-3},

}

#### in EndNote Style

TY - CONF

JO - Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2014)

TI - Singularity Stairs Following with Limited Numbers of Hidden Units

SN - 978-989-758-054-3

AU - Satoh S.

AU - Nakano R.

PY - 2014

SP - 180

EP - 186

DO - 10.5220/0005075601800186