Determining the Near Optimal Architecture of Autoencoder using Correlation Analysis of the Network Weights

Heng Ma, Yonggang Lu, Haitao Zhang

2016

Abstract

Currently, deep learning has already been successfully applied in many fields such as image recognition, recommendation systems and so on. Autoencoder, as an important deep learning model, has attracted a lot of research interests. The performance of the autoencoder can greatly be affected by its architecture. How-ever, how to automatically determine the optimal architecture of the autoencoder is still an open question. Here we propose a novel method for determining the optimal network architecture based on the analysis of the correlation of the network weights. Experiments show that for different datasets the optimal architecture of the autoencoder may be different, and the proposed method can be used to obtain near optimal network architecture separately for different datasets.

References

  1. Chicco, D., Sadowski, P., & Baldi, P. 2014. Deep autoencoder neural networks for gene ontology annotation predictions. In Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. ACM. pp. 533-540.
  2. Ciresan, D., Meier, U., & Schmidhuber, J. 2012. Multicolumn deep neural networks for image classification. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference. IEEE. pp. 3642-3649.
  3. Collobert, R., & Weston, J. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning. ACM. pp. 160-167.
  4. Fahlman, S. E. and Lebiere, C., 1990, The cascadecorrelation learning architecture, in Advances in NIPS, edited by D. S. Touretzky, vol. 2, pp. 524-532.
  5. Fiszelew, A., Britos, P., Ochoa, A., Merlino, H., Fernández, E., & García-Martínez, R. 2007. Finding optimal neural network architecture using genetic algorithms. Advances in computer science and engineering research in computing science, 27, pp. 15-24.
  6. Hassibi, B., Stork, D. G., & Wolff, G. J. 1993. Optimal brain surgeon and general network pruning. In Neural Networks, 1993., IEEE International Conference. IEEE. pp. 293-299.
  7. Hinton, G. E., & Salakhutdinov, R. R. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786), pp. 504-507.
  8. Laurens van der Maaten, Eric Postma & Jaap van den Herik. 2009. Dimensionality reduction: A comparative review. Tilburg, Netherlands: Tilburg Centre for Creative Computing, Tilburg University, Technical Report: 2009-005.
  9. LeCun, Y., Denker, J. S., Solla, S. A., Howard, R. E., & Jackel, L. D. 1989. Optimal brain damage. In Advances in NIPS, vol. 2, pp. 598-605.
  10. Lee, T. C. 2012. Structure level adaptation for artificial neural networks (Vol. 133). Springer Science & Business Media.
  11. Levine, S., Pastor, P., Krizhevsky, A., & Quillen, D. 2016. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. arXiv preprint arXiv:1603.02199.
  12. Mozer, M. C., & Smolensky, P.. Skeletonization. 1989. A technique for trimming the fat from a network via relevance assessment. In Advances in Neural Information Processing Systems. pp. 107-115.
  13. Reed, R. 1993. Pruning algorithms-a survey. IEEE Transactions on Neural Networks, 4(5), pp. 740-747.
  14. Reitermanova, Z. 2008. Feedforward neural networksarchitecture optimization and knowledge extraction. WDS'08 proceedings of contributed papers, Part I, pp. 159-164.
  15. Stigler, S. M. 1989. Francis Galton's account of the invention of correlation.Statistical Science, pp. 73-79.
  16. Van den Oord, A., Dieleman, S., & Schrauwen, B. 2013. Deep content-based music recommendation. In Advances in Neural Information Processing Systems. pp.
Download


Paper Citation


in Harvard Style

Ma H., Lu Y. and Zhang H. (2016). Determining the Near Optimal Architecture of Autoencoder using Correlation Analysis of the Network Weights . In Proceedings of the 8th International Joint Conference on Computational Intelligence - Volume 3: NCTA, (IJCCI 2016) ISBN 978-989-758-201-1, pages 53-61. DOI: 10.5220/0006039000530061


in Bibtex Style

@conference{ncta16,
author={Heng Ma and Yonggang Lu and Haitao Zhang},
title={Determining the Near Optimal Architecture of Autoencoder using Correlation Analysis of the Network Weights},
booktitle={Proceedings of the 8th International Joint Conference on Computational Intelligence - Volume 3: NCTA, (IJCCI 2016)},
year={2016},
pages={53-61},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006039000530061},
isbn={978-989-758-201-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 8th International Joint Conference on Computational Intelligence - Volume 3: NCTA, (IJCCI 2016)
TI - Determining the Near Optimal Architecture of Autoencoder using Correlation Analysis of the Network Weights
SN - 978-989-758-201-1
AU - Ma H.
AU - Lu Y.
AU - Zhang H.
PY - 2016
SP - 53
EP - 61
DO - 10.5220/0006039000530061