MULTI-CRITERIA EVALUATION OF INFORMATION RETRIEVAL TOOLS

Nishant Kumar, Jan Vanthienen, Jan De Beer, Marie-Francine Moens

2006

Abstract

We propose a generic methodology for the evaluation of Text Mining/Search and Information Retrieval tools based on their functional conformity to a predefined set of functional requirements prioritized by distinguishable user profiles. The methodology is worked out and applied within the context of a research project concerning the assessment of intelligent exploitation tools for unstructured information sources in the police domain. We present the general setting of our work, give an overview of our evaluation approach, and discuss our methodology for testing in greater detail. These kinds of evaluations are particularly useful for both (potential)purchasers of exploitation tools, given the high investments in time and money required in becoming proficient in their use, and developers who aim at producing better quality software products.

References

  1. Atabakhsh, H., Schroeder, J., Chen, H., Chau, M., Xu, J. J., Zhang, J., and Bi, H. (2001). Coplink knowledge management for law enforcement: Text analysis, visualization and collaboration. In Proceedings of the National Conference for Digital Government Research, volume 1.
  2. Chen, H., Chung, W., Xu, J. J., Wang, G., Qin, Y., and Chau, M. (2004). Crime data mining: a general framework and some examples. 37(4).
  3. Chen, H., Schroeder, J., Hauck, R., Ridgeway, L., Atabakhsh, H., Gupta, H., Boarman, C., Rasmussen, K., and Clements, A. (2002). Coplink connect: information and knowledge management for law enforcement. 34(3):271-285.
  4. Chen, H., Zeng, D., Atabakhsh, H., Wyzga, W., and Schroeder, J. (2003). Coplink managing law enforcement data and knowledge. 46(1):28-34.
  5. Collier, K., Carey, B., Sautter, D., and Marjaniemi, C. (1999). A methodology for evaluating and selecting data mining software. In Proceedings of the International Conference on System Sciences.
  6. Cooper, W. S. (1973). On selecting a measure of retrieval effectiveness. Journal of the American Society for Information Science, 24(2):87-100.
  7. De Beer, J., Kumar, N., Moens, M.-F., and Vanthienen, J. (2006). Assessing the state of the art of commercial tools for unstructured information exploitation.
  8. Elder, J. F. and Abbott, D. W. (1998). A comparison of leading data mining tools. Technical report.
  9. Hauck, R. V., Schroeder, J., and Chen, H. (2001). Coplink: Developing information sharing and criminal intelligence analysis technologies for law enforcement. In Proceedings of the National Conference for Digital Government Research, volume 1, pages 134-140.
  10. Ingwersen, P. (1992). Information Retrieval Interaction. Taylor Graham, London.
  11. Kumar, N., De Beer, J., Vanthienen, J., and Moens, M.-F. (2006). A study on the quality of enterprise search tools.
  12. Lancaster, F. W. (1968). Information Retrieval Systems: Characteristics, Testing and Evaluation. Wiley, New York.
  13. Leary, R. M. The role of the national intelligence model and flints in improving police performance. Online at http://www.homeoffice.gov.uk/docs2/ resconf2002daytwo.html.
  14. Mena, J. (2003). Investigative data mining for security and criminal detection. First edition.
  15. Nakhaeizadeh, G. and Schnabl, A. (1997). Development of multi-criteria metrics for evaluation of data mining algorithms. In Proceedings KDD-97. AAAI Press.
  16. Oatley, G. C., Ewart, B. W., and Zeleznikow, J. (2004). Decision support systems for police: Lessons from the application of data mining techniques to 'soft' forensic evidence.
  17. Van Rijsbergen, C. J. (1979). Information Retrieval. Butterworths London, second edition.
Download


Paper Citation


in Harvard Style

Kumar N., Vanthienen J., De Beer J. and Moens M. (2006). MULTI-CRITERIA EVALUATION OF INFORMATION RETRIEVAL TOOLS . In Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 2: ICEIS, ISBN 978-972-8865-42-9, pages 150-155. DOI: 10.5220/0002463601500155


in Bibtex Style

@conference{iceis06,
author={Nishant Kumar and Jan Vanthienen and Jan De Beer and Marie-Francine Moens},
title={MULTI-CRITERIA EVALUATION OF INFORMATION RETRIEVAL TOOLS},
booktitle={Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 2: ICEIS,},
year={2006},
pages={150-155},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0002463601500155},
isbn={978-972-8865-42-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the Eighth International Conference on Enterprise Information Systems - Volume 2: ICEIS,
TI - MULTI-CRITERIA EVALUATION OF INFORMATION RETRIEVAL TOOLS
SN - 978-972-8865-42-9
AU - Kumar N.
AU - Vanthienen J.
AU - De Beer J.
AU - Moens M.
PY - 2006
SP - 150
EP - 155
DO - 10.5220/0002463601500155