Bridging the Gap between Naive Bayes and Maximum Entropy Text Classification

Alfons Juan, David Vilar, Hermann Ney

Abstract

The naive Bayes and maximum entropy approaches to text classification are typically discussed as completely unrelated techniques. In this paper, however, we show that both approaches are simply two different ways of doing parameter estimation for a common log-linear model of class posteriors. In particular, we show how to map the solution given by maximum entropy into an optimal solution for naive Bayes according to the conditional maximum likelihood criterion.

References

  1. Lewis, D.: Naive Bayes at Forty: The Independence Assumption in Information Retrieval. In: Proc. of ECML-98. (1998) 4-15
  2. Nigam, K., Lafferty, J., McCallum, A.: Using Maximum Entropy for Text Classification. In: Proc. of IJCAI-99 Workshop on Machine Learning for Information Filtering. (1999) 61-67
  3. McCallum, A., Nigam, K.: A Comparison of Event Models for Naive Bayes Text Classification. In: Proc. of AAAI/ICML-98 Workshop on Learning for Text Categorization. (1998) 41-48
  4. Juan, A., Ney, H.: Reversing and Smoothing the Multinomial Naive Bayes Text Classifier. In: Proc. of PRIS-02, Alacant (Spain) (2002) 200-212
  5. Vilar, D., Ney, H., Juan, A., Vidal, E.: Effect of Feature Smoothing Methods in Text Classification Tasks. In: Proc. of PRIS-04, Porto (Portugal) (2004) 108-117
  6. Ghani, R.: World Wide Knowledge Base (Web?KB) project. www-2.cs.cmu.edu/ afs/cs.cmu.edu/project/theo-11/www/wwkb (2001)
  7. Rennie, J.: Original 20 Newsgroups data set. (www.ai.mit.edu/~jrennie) (2001)
  8. McCallum, A.: Rainbow. (www.cs.umass.edu/~mccallum/bow/rainbow) (1998)
  9. Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Proc. of ICML-97. (1997) 412-420
  10. Darroch, J., Ratcliff, D.: Generalized Iterative Scaling for Log-linear Models. Annals of Mathematical Statistics 43 (1972) 1470-1480
Download


Paper Citation


in Harvard Style

Juan A., Vilar D. and Ney H. (2007). Bridging the Gap between Naive Bayes and Maximum Entropy Text Classification . In Proceedings of the 7th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2007) ISBN 978-972-8865-93-1, pages 59-65. DOI: 10.5220/0002425700590065


in Bibtex Style

@conference{pris07,
author={Alfons Juan and David Vilar and Hermann Ney},
title={Bridging the Gap between Naive Bayes and Maximum Entropy Text Classification},
booktitle={Proceedings of the 7th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2007)},
year={2007},
pages={59-65},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0002425700590065},
isbn={978-972-8865-93-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Workshop on Pattern Recognition in Information Systems - Volume 1: PRIS, (ICEIS 2007)
TI - Bridging the Gap between Naive Bayes and Maximum Entropy Text Classification
SN - 978-972-8865-93-1
AU - Juan A.
AU - Vilar D.
AU - Ney H.
PY - 2007
SP - 59
EP - 65
DO - 10.5220/0002425700590065