LEARNING METHOD UTILIZING SINGULAR REGION OF MULTILAYER PERCEPTRON

Ryohei Nakano, Seiya Satoh, Takayuki Ohwaki

2011

Abstract

In a search space of multilayer perceptron having J hidden units, MLP(J), there exists a singular flat region created by the projection of the optimal solution of MLP(J-1). Since such a singular region causes serious slowdown for learning methods, a method for avoiding the region has been aspired. However, such avoiding does not guarantee the quality of the final solution. This paper proposes a new learning method which does not avoid but makes good use of singular regions to find a solution good enough for MLP(J). The potential of the method is shown by our experiments using artificial data sets, XOR problem, and a real data set.

References

  1. Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276.
  2. Amari, S., Park, H. and Fukumizu, K. (2000). Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation, 12(6):1399- 1409.
  3. Duda, R. O., Hart, P. E. and Stork, D. G. (2001). Pattern classification, 2nd edition. John Wiley & Sons, Inc.
  4. Fukumizu, K. and Amari,S.(2000). Local minima and plateaus in heirarchical structure of multilayer perceptrons. Neural Networks, 13(3):317-327.
  5. Hamey, L. G. C. (1998). XOR has no local minima: a case study in neural network error surface. Neural Networks, 11(4):669-681.
  6. Minnett, R. C. J., Smith, A. T., Lennon Jr. W. C. and HechtNielsen, R. (2011). Neural network tomography: network replication from output surface geometry. Neural Networks, 24(5):484-492.
  7. Nakano, R. and Saito, K. (2002). Discovering polynomials to fit multivariate data having numeric and nominal variables. LNAI 2281:482-493.
  8. Watanabe, S. (2009). Algebraic geometry and statistical learning theory. Cambridge Univ. Press.
  9. Saito, K. and Nakano, R. (1997). Partial BFGS update and efficient step-length calculation for three-layer neural networks. Neural Computation, 9(1):239-257.
  10. Sussman, H. J. (1992). Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks, 5(4):589-593.
Download


Paper Citation


in Harvard Style

Nakano R., Satoh S. and Ohwaki T. (2011). LEARNING METHOD UTILIZING SINGULAR REGION OF MULTILAYER PERCEPTRON . In Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2011) ISBN 978-989-8425-84-3, pages 106-111. DOI: 10.5220/0003652501060111


in Bibtex Style

@conference{ncta11,
author={Ryohei Nakano and Seiya Satoh and Takayuki Ohwaki},
title={LEARNING METHOD UTILIZING SINGULAR REGION OF MULTILAYER PERCEPTRON},
booktitle={Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2011)},
year={2011},
pages={106-111},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003652501060111},
isbn={978-989-8425-84-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Neural Computation Theory and Applications - Volume 1: NCTA, (IJCCI 2011)
TI - LEARNING METHOD UTILIZING SINGULAR REGION OF MULTILAYER PERCEPTRON
SN - 978-989-8425-84-3
AU - Nakano R.
AU - Satoh S.
AU - Ohwaki T.
PY - 2011
SP - 106
EP - 111
DO - 10.5220/0003652501060111