Comparison of Data Selection Strategies for Online Support Vector Machine Classification

Mario Michael Krell, Nils Wilshusen, Andrei Cristian Ignat, Su Kyoung Kim

2015

Abstract

It is often the case that practical applications of support vector machines (SVMs) require the capability to perform online learning under limited availability of computational resources. Enabling SVMs for online learning can be done through several strategies. One group thereof manipulates the training data and limits its size. We aim to summarize these existing approaches and compare them, firstly, on several synthetic datasets with different shifts and, secondly, on electroencephalographic (EEG) data. During the manipulation, class imbalance can occur across the training data and it might even happen that all samples of one class are removed. In order to deal with this potential issue, we suggest and compare three balancing criteria.

References

  1. Alippi, C., Liu, D., Zhao, D., Member, S., and Bu, L. (2014). Detecting and Reacting to Changes in Sensing Units: The Active Classifier Case. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 44(3):1- 10.
  2. Blankertz, B., Lemm, S., Treder, M., Haufe, S., and Müller, K.-R. (2011). Single-Trial Analysis and Classification of ERP Components-a Tutorial. NeuroImage, 56(2):814-825.
  3. Bordes, A., Ertekin, S., Weston, J., and Bottou, L. (2005). Fast Kernel Classifiers with Online and Active Learning. The Journal of Machine Learning Research, 6:1579-1619.
  4. Courchesne, E., Hillyard, S. A., and Courchesne, R. Y. (1977). P3 waves to the discrimination of targets in homogeneous and heterogeneous stimulus sequences. Psychophysiology, 14(6):590-597.
  5. Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y. (2006). Online Passive-Aggressive Algorithms. Journal of Machine Learning Research, 7:551 - 585.
  6. Cristianini, N. and Shawe-Taylor, J. (2000). An introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press.
  7. Dekel, O., Shalev-Shwartz, S., and Singer, Y. (2008). The Forgetron: A Kernel-Based Perceptron on a Budget. SIAM Journal on Computing, 37(5):1342-1372.
  8. Funaya, H., Nomura, Y., and Ikeda, K. (2009). A Support Vector Machine with Forgetting Factor and Its Statistical Properties. In K öppen, M., Kasabov, N., and Coghill, G., editors, Advances in Neuro-Information Processing, volume 5506 of Lecture Notes in Computer Science, pages 929-936. Springer Berlin Heidelberg.
  9. Gretton, A. and Desobry, F. (2003). On-line one-class support vector machines. An application to signal segmentation. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 7803), volume 2, pages 709-712. IEEE.
  10. Hoens, T. R., Polikar, R., and Chawla, N. V. (2012). Learning from streaming data with concept drift and imbalance: an overview. Progress in Artificial Intelligence, 1(1):89-101.
  11. Hsieh, C.-J., Chang, K.-W., Lin, C.-J., Keerthi, S. S., and Sundararajan, S. (2008). A dual coordinate descent method for large-scale linear SVM. In Proceedings of the 25th International Conference on Machine learning (ICML 2008), pages 408-415. ACM Press.
  12. Kirchner, E. A., Kim, S. K., Straube, S., Seeland, A., W öhrle, H., Krell, M. M., Tabie, M., and Fahle, M. (2013). On the applicability of brain reading for predictive human-machine interfaces in robotics. PloS ONE, 8(12):e81732.
  13. Kirchner, E. A., Tabie, M., and Seeland, A. (2014). Multimodal movement prediction - towards an individual assistance of patients. PloS one, 9(1):e85060.
  14. Krell, M. M. (2015). Generalizing, Decoding, and Optimizing Support Vector Machine Classification. Phd thesis, University of Bremen, Bremen.
  15. Krell, M. M., Straube, S., Seeland, A., W öhrle, H., Teiwes, J., Metzen, J. H., Kirchner, E. A., and Kirchner, F. (2013). pySPACE a signal processing and classification environment in Python. Frontiers in Neuroinformatics, 7(40):1-11.
  16. Laskov, P., Gehl, C., Kr üger, S., and M üller, K.-R. (2006). Incremental Support Vector Learning: Analysis, Implementation and Applications. Journal of Machine Learning Research, 7:1909-1936.
  17. Li, Y., Guan, C., Li, H., and Chin, Z. (2008). A self-training semi-supervised SVM algorithm and its application in an EEG-based brain computer interface speller system. Pattern Recognition Letters, 29(9):1285-1294.
  18. Liang, Z. and Li, Y. (2009). Incremental support vector machine learning in the primal and applications. Neurocomputing, 72(10-12):2249-2258.
  19. Lin, H.-T., Lin, C.-J., and Weng, R. C. (2007). A note on Platts probabilistic outputs for support vector machines. Machine Learning, 68(3):267-276.
  20. Mangasarian, O. L. and Musicant, D. R. (1998). Successive Overrelaxation for Support Vector Machines. IEEE Transactions on Neural Networks, 10:1032 - 1037.
  21. Müller, K.-R., Mika, S., Rätsch, G., Tsuda, K., and Schölkopf, B. (2001). An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12(2):181-201.
  22. Nguyen-Tuong, D. and Peters, J. (2011). Incremental online sparsification for model learning in real-time robot control. Neurocomputing, 74(11):1859-1867.
  23. Oskoei, M. A., Gan, J. Q., and Hu, O. (2009). Adaptive schemes applied to online SVM for BCI data classification. In Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society: Engineering the Future of Biomedicine, EMBC 2009, volume 2009, pages 2600-2603.
  24. Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (2009). Dataset Shift in Machine Learning. MIT Press.
  25. Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C. (2001). Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471.
  26. Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, USA.
  27. Spüler, M., Rosenstiel, W., and Bogdan, M. (2012). Adaptive SVM-Based Classification Increases Performance of a MEG-Based Brain-Computer Interface (BCI). In Villa, A., Duch, W., Órdi, P., Masulli, F., and Palm, G., editors, Artificial Neural Networks and Machine Learning ICANN 2012, volume 7552 of Lecture Notes in Computer Science, pages 669-676. Springer Berlin Heidelberg.
  28. Steinwart, I., Hush, D., and Scovel, C. (2009). Training SVMs without offset. Journal of Machine Learning Research, 12:141-202.
  29. Straube, S. and Krell, M. M. (2014). How to evaluate an agent's behaviour to infrequent events? - Reliable performance estimation insensitive to class distribution. Frontiers in Computational Neuroscience, 8(43):1-6.
  30. Street, W. N. and Kim, Y. (2001). A Streaming Ensemble Algorithm (SEA) for Large-scale Classification. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 7801, pages 377-382, New York, NY, USA. ACM.
  31. (2006). Online weighted LS-SVM for hysteretic structural system identification. Engineering Structures, 28(12):1728-1735.
  32. Van Vaerenbergh, S., Santamaria, I., Liu, W., and Principe, J. C. (2010). Fixed-budget kernel recursive leastsquares. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1882-1885. IEEE.
  33. Van Vaerenbergh, S., Via, J., and Santamaria, I. (2006). A Sliding-Window Kernel RLS Algorithm and Its Application to Nonlinear Channel Identification. In2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings, volume 5, pages 789-792. IEEE.
  34. Vapnik, V. (2000). The Nature of Statistical Learning Theory. Springer.
  35. Wöhrle, H., Krell, M. M., Straube, S., Kim, S. K., Kirchner, E. A., and Kirchner, F. (2015). An Adaptive Spatial Filter for User-Independent Single Trial Detection of Event-Related Potentials. IEEE transactions on biomedical engineering, PP(99):1.
  36. Wöhrle, H., Teiwes, J., Krell, M. M., Kirchner, E. A., and Kirchner, F. (2013). A Dataflow-based Mobile Brain Reading System on Chip with Supervised Online Calibration - For Usage without Acquisition of Training Data. In Proceedings of the International Congress on Neurotechnology, Electronics and Informatics, pages 46-53, Vilamoura, Portugal. SciTePress.
  37. Wöhrle, H., Teiwes, J., Krell, M. M., Seeland, A., Kirchner, E. A., and Kirchner, F. (2014). Reconfigurable Dataflow Hardware Accelerators for Machine Learning and Robotics. In ECML/PKDD-2014 PhD Session Proceedings, Nancy.
  38. Yi, Y., Wu, J., and Xu, W. (2011). Incremental SVM based on reserved set for network intrusion detection. Expert Systems with Applications, 38(6):7698-7707.
  39. Zander, T. O. and Kothe, C. (2011). Towards passive braincomputer interfaces: applying brain-computer interface technology to human-machine systems in general. Journal of Neural Engineering, 8(2):025005.
Download


Paper Citation


in Harvard Style

Michael Krell M., Wilshusen N., Cristian Ignat A. and Kyoung Kim S. (2015). Comparison of Data Selection Strategies for Online Support Vector Machine Classification . In Proceedings of the 3rd International Congress on Neurotechnology, Electronics and Informatics - Volume 1: NEUROTECHNIX, ISBN 978-989-758-161-8, pages 59-67. DOI: 10.5220/0005650700590067


in Bibtex Style

@conference{neurotechnix15,
author={Mario Michael Krell and Nils Wilshusen and Andrei Cristian Ignat and Su Kyoung Kim},
title={Comparison of Data Selection Strategies for Online Support Vector Machine Classification},
booktitle={Proceedings of the 3rd International Congress on Neurotechnology, Electronics and Informatics - Volume 1: NEUROTECHNIX,},
year={2015},
pages={59-67},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005650700590067},
isbn={978-989-758-161-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Congress on Neurotechnology, Electronics and Informatics - Volume 1: NEUROTECHNIX,
TI - Comparison of Data Selection Strategies for Online Support Vector Machine Classification
SN - 978-989-758-161-8
AU - Michael Krell M.
AU - Wilshusen N.
AU - Cristian Ignat A.
AU - Kyoung Kim S.
PY - 2015
SP - 59
EP - 67
DO - 10.5220/0005650700590067