ASSET: APPROXIMATE STOCHASTIC SUBGRADIENT ESTIMATION TRAINING FOR SUPPORT VECTOR MACHINES

Sangkyun Lee, Stephen Wright

2012

Abstract

Subgradient methods for training support vector machines have been quite successful for solving large-scale and online learning problems. However, they have been restricted to linear kernels and strongly convex formulations. This paper describes efficient subgradient approaches without such limitations, making use of randomized low-dimensional approximations to nonlinear kernels, and minimization of a reduced primal formulation using an algorithm based on robust stochastic approximation, which do not require strong convexity.

References

  1. Bordes, A., Ertekin, S., Weston, J., and Bottou, L. (2005). Fast kernel classifiers with online and active learning. Journal of Machine Learning Research, 6:1579-1619.
  2. Bottou, L. (2005). SGD: Stochastic gradient descent. http://leon.bottou.org/projects/sgd.
  3. Chapelle, O. (2007). Training a support vector machine in the primal. Neural Computation, 19:1155-1178.
  4. Drineas, P. and Mahoney, M. W. (2005). On the nystrom method for approximating a gram matrix for improved kernel-based learning. Journal of Machine Learning Research, 6:2153-2175.
  5. Franc, V. and Sonnenburg, S. (2008). Optimized cutting plane algorithm for support vector machines. In Proceedings of the 25th International Conference on Machine Learning, pages 320-327.
  6. Joachims, T. (1999). Making large-scale support vector machine learning practical. In Advances in Kernel Methods - Support Vector Learning, pages 169-184. MIT Press.
  7. Joachims, T. (2006). Training linear SVMs in linear time. In International Conference On Knowledge Discovery and Data Mining, pages 217-226.
  8. Joachims, T., Finley, T., and Yu, C.-N. (2009). Cuttingplane training of structural svms. Machine learning, 77(1):27-59.
  9. Joachims, T. and Yu, C.-N. J. (2009). Sparse kernel svms via cutting-plane training. Machine Learning, 76(2- 3):179-193.
  10. Lee, S. and Wright, S. J. (2011). Approximate stochastic subgradient estimation training for support vector machines. http://arxiv.org/abs/1111.0432.
  11. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. (2009). Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609.
  12. Nemirovski, A. and Yudin, D. B. (1983). Problem complexity and method efficiency in optimization. John Wiley.
  13. Rahimi, A. and Recht, B. (2008). Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20, pages 1177- 1184. MIT Press.
  14. Shalev-Shwartz, S., Singer, Y., and Srebro, N. (2007). Pegasos: Primal estimated sub-gradient solver for svm. In Proceedings of the 24th International Conference on Machine Learning, pages 807-814.
  15. Shalev-Shwartz, S., Singer, Y., Srebro, N., and Cotter, A. (2011). Pegasos: Primal estimated sub-gradient solver for svm. Mathematical Programming, Series B, 127(1):3-30.
  16. Zinkevich, M. (2003). Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning, pages 928-936.
Download


Paper Citation


in Harvard Style

Lee S. and Wright S. (2012). ASSET: APPROXIMATE STOCHASTIC SUBGRADIENT ESTIMATION TRAINING FOR SUPPORT VECTOR MACHINES . In Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-8425-98-0, pages 223-228. DOI: 10.5220/0003786202230228


in Bibtex Style

@conference{icpram12,
author={Sangkyun Lee and Stephen Wright},
title={ASSET: APPROXIMATE STOCHASTIC SUBGRADIENT ESTIMATION TRAINING FOR SUPPORT VECTOR MACHINES},
booktitle={Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2012},
pages={223-228},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003786202230228},
isbn={978-989-8425-98-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - ASSET: APPROXIMATE STOCHASTIC SUBGRADIENT ESTIMATION TRAINING FOR SUPPORT VECTOR MACHINES
SN - 978-989-8425-98-0
AU - Lee S.
AU - Wright S.
PY - 2012
SP - 223
EP - 228
DO - 10.5220/0003786202230228