# AN APPLICATION OF NON-LINEAR PROGRAMMING TO TRAIN RECURRENT NEURAL NETWORKS IN TIME SERIES PREDICTION PROBLEMS

### M. P. Cuéllar, M. Delgado, M. C. Pegalajar

#### Abstract

Artificial Neural Networks are bioinspired mathematical models that have been widely used to solve many complex problems. However, the training of a Neural Network is a difficult task since the traditional training algorithms may get trapped into local solutions easily. This problem is greater in Recurrent Neural Networks, where the traditional training algorithms sometimes provide unsuitable solutions. Some evolutionary techniques have also been used to improve the training stage, and to overcome such local solutions, but they have the disadvantage that the time taken to train the network is high. The objective of this work is to show that the use of some non-linear programming techniques is a good choice to train a Neural Network, since they may provide suitable solutions quickly. In the experimental section, we apply the models proposed to train an Elman Recurrent Neural Network in real-life Time Series Prediction problems.

#### References

- Blanco, Delgado, Pegalajar. 2001. A Real-Coded genetic algorithm for training recurrent neural networks. Neural Networks, vol. 14, pp. 93-105.
- C. Zhu, R. H. Byrd and J. Nocedal. 1997. L-BFGS-B: Algorithm 778: L-BFGS-B, FORTRAN routines for large scale bound constrained optimization, ACM Transactions on Mathematical Software, Vol 23, Num. 4, pp. 550 - 560.
- Cuéllar M.P., Delgado M., Pegalajar M.C.. 2004. A Comparative study of Evolutionary Algorithms for Training Elman Recurrent Neural Networks to predict the Autonomous Indebtedness. in Proc. ICEIS, Porto, Portugal, pp. 457-461.
- Danilo P. Mandic, Jonathon A. Chambers. 2001. Recurrent Neural Networks for Prediction. Wiley, John & Sons, Incorporated.
- D. W. Marquardt. 1963. An algorithm for least-squares estimation of nonlinear parameters, Journal of the Society for Industrialand Applied Mathematics, pp. 11431-441.
- Martin T. Hagan, Mohammed B. Menhaj. 1994. Training FeedForward networks with the Marquardt algorithm, IEEE transactions on Neural networks, vol 5, no. 6, pp. 989-993.
- Michael Hüsken, Peter Stagge. 2003. Recurrent Neural Networks for Time Series classification, Neurocomputing, vol. 50, pp. 223-235.
- More, J. J. 1977. The Levenberg-Marquardt algorithm: Implementation and theory. Lecture notes in mathematics, Edited by G. A. Watson, SpringerVerlag.
- R. H. Byrd, P. Lu and J. Nocedal. 1995. A Limited Memory Algorithm for Bound Constrained Optimization, SIAM Journal on Scientific and Statistical Computing , 16, 5, pp. 1190-1208.
- R. Martí, A. El-Fallahi. 2002. Multilayer Neural Networks: An experimental evaluation of on-line training methods. Computers and Operations Research 31, pp. 1491-1513.
- Ryad Zemomi, Daniel Racaceanu, Nouredalime Zerhonn. 2003. Recurrent Radial Basis fuction network for Time Seties prediction, Engineering appl. Of Artificial Intelligence, vol. 16, no. 5-6, pp. 453-463.
- Simon Haykin. 1999. Neural Networks (a Comprehensive foundation). Second Edition. Prentice Hall.
- Williams R.J., Peng J. 1990. An efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network trajectories,” Neural Computation, vol. 2, pp. 491-501.
- Williams R.J., Zipser D. 1989. A learning algorithm for continually running fully recurrent neural networks, Neural Computation, vol. 1, pp. 270-280.

#### Paper Citation

#### in Harvard Style

P. Cuéllar M., Delgado M. and C. Pegalajar M. (2005). **AN APPLICATION OF NON-LINEAR PROGRAMMING TO TRAIN RECURRENT NEURAL NETWORKS IN TIME SERIES PREDICTION PROBLEMS** . In *Proceedings of the Seventh International Conference on Enterprise Information Systems - Volume 2: ICEIS,* ISBN 972-8865-19-8, pages 35-42. DOI: 10.5220/0002515800350042

#### in Bibtex Style

@conference{iceis05,

author={M. P. Cuéllar and M. Delgado and M. C. Pegalajar},

title={AN APPLICATION OF NON-LINEAR PROGRAMMING TO TRAIN RECURRENT NEURAL NETWORKS IN TIME SERIES PREDICTION PROBLEMS},

booktitle={Proceedings of the Seventh International Conference on Enterprise Information Systems - Volume 2: ICEIS,},

year={2005},

pages={35-42},

publisher={SciTePress},

organization={INSTICC},

doi={10.5220/0002515800350042},

isbn={972-8865-19-8},

}

#### in EndNote Style

TY - CONF

JO - Proceedings of the Seventh International Conference on Enterprise Information Systems - Volume 2: ICEIS,

TI - AN APPLICATION OF NON-LINEAR PROGRAMMING TO TRAIN RECURRENT NEURAL NETWORKS IN TIME SERIES PREDICTION PROBLEMS

SN - 972-8865-19-8

AU - P. Cuéllar M.

AU - Delgado M.

AU - C. Pegalajar M.

PY - 2005

SP - 35

EP - 42

DO - 10.5220/0002515800350042