A Bayesian Approach for Constructing Ensemble Neural Network

Sai Hung Cheung, Yun Zhang, Zhiye Zhao


Ensemble neural networks (ENNs) are commonly used in many engineering applications due to its better generalization properties compared with a single neural network (NN). As the NN architecture has a significant influence on the generalization ability of an NN, it is crucial to develop a proper algorithm to design the NN architecture. In this paper, an ENN which combines the component networks by using the Bayesian approach and stochastic modelling is proposed. The cross validation data set is used not only to stop the network training, but also to determine the weights of the component networks. The proposed ENN searches the best structure of each component network first and employs the Bayesian approach as an automating design tool to determine the best combining weights of the ENN. Peak function is used to assess the accuracy of the proposed ensemble approach. The results show that the proposed ENN outperforms ENN obtained by simple averaging and the single NN.


  1. Beck, J. L., Katafygiotis, L. S., 1998. Updating models and their uncertainties. I: Bayesian statistical framework. Journal of Engineering Mechanics 124(4), 455-461.
  2. Buntine, W. L., Weigend, A.S.1991. Bayesian Backpropagation. Complex Systems 5, 603-643.
  3. Cheung, S. H., Beck, J. L., 2010. Calculation of posterior probabilities for Bayesian model class assessment and averaging from posterior samples based on dynamic system data. Computer-Aided Civil and Infrastructure Engineering 25, 304-321.
  4. Cheung, S. H. and Beck, J. L., 2012. New Bayesian updating methodology for model validation and robust predictions of a target system based on hierarchical subsystem tests. Computer Methods in Applied Mechanics and Engineering, accepted for publication.
  5. Friedman, J. H., 1991. Multivariate adaptive regression splines. Ann Statist 19(1): 1-82.
  6. Hansen, L. K., Salamon, P., 1990. Neural network ensembles. IEEE Transactions on Pattern Analysis Machine Intelligence 12 (10), 993-1001.
  7. Hashem, S., 1993. Optimal Linear Combinations of Neural Networks. PhD thesis, School of Industrial Engineering, Purdue University.
  8. Hippert, H. S., Taylor, J. W., 2010. An evalution of Bayesian techniques for controlling model complexity and selecting inputs in a neural network for short-term load forecasting, Neural Networks 23, 386-395.
  9. Lampinen, J., Vehtari, A., 2001. Bayesian approach for neural networks-review and case studies. Neural Networks 14, 257-274.
  10. Marwala, T., 2007. Bayesian training of neural networks using genetic programming. Pattern Recognition Letters 28, 1452-1458.
  11. McCulloch, W. S., Pitts, W., 1943. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 1943; 5, 115-133. Reprinted in Anderson & Rosenfeld 1988, 18-28.
  12. Posada, D., Buckley, T. P., 2004. Model selection and model averaging in phylogenetics: advantages of Akaike information criterion and Bayesian approaches over likelihood ratio tests. Systematic Biology 53, 793- 808.
  13. Sollich, P., Krogh, A., 1996. Learning with ensembles: How over-fitting can be useful, in: Touretzky, D. S., Mozer, M. C., Hasselmo, M. E. (Eds.), Advances in Neural Information Processing Systems 8, Denver, CO, MIT press, Cambridge, MA, pp. 190-196.
  14. Wang, P., Xu, L., Zhou, S., Fan, Z., Li, Y., Feng, S., 2010. A novel Bayesian learning method for information aggregation in modular neural networks. Expert Systems with Applications 37, 1071-1074.
  15. Zhao, Z. Y., Zhang, Y., Liao, H. J., 2008. Design of ensemble neural network using the Akaike information criterion. Engineering Applications of Artificial Intelligence 21, 1182-1188.

Paper Citation

in Harvard Style

Hung Cheung S., Zhang Y. and Zhao Z. (2012). A Bayesian Approach for Constructing Ensemble Neural Network . In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2012) ISBN 978-989-8565-29-7, pages 374-377. DOI: 10.5220/0004147103740377

in Bibtex Style

author={Sai Hung Cheung and Yun Zhang and Zhiye Zhao},
title={A Bayesian Approach for Constructing Ensemble Neural Network},
booktitle={Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2012)},

in EndNote Style

JO - Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2012)
TI - A Bayesian Approach for Constructing Ensemble Neural Network
SN - 978-989-8565-29-7
AU - Hung Cheung S.
AU - Zhang Y.
AU - Zhao Z.
PY - 2012
SP - 374
EP - 377
DO - 10.5220/0004147103740377