Goal-based Evaluation of Text Mining Results in an Industrial Use Case

Jens Drawehn, Matthias Blohm, Maximilien Kintz, Monika Kochanowski

2020

Abstract

Artificial intelligence boosted the interest in text mining solutions in the last few years. Especially in nonEnglish-speaking countries, where there might not be clear market leaders, a variety of solutions for different text mining scenarios has become available. Most of them support special use cases and have strengths and weaknesses in others. In text or page classification, standard measures like precision, recall, sensitivity or F1-score are prevalent. However, evaluation of feature extraction results requires more tailored approaches. We experienced many issues on the way to benchmarking feature extraction results from text, like whether a result is correct, partly correct, helpful or useless. The main contribution of this work is a method for designing a tailored evaluation procedure in an individual text extraction benchmark for one specific use case. In this context, we propose a general way of mapping the common CRISP-DM process to particularities of text mining projects. Furthermore, we describe possible goals of information extraction, the features to be extracted, suitable evaluation criteria and a corresponding customized scoring system. This is applied in detail in an industrial use case.

Paper Citation

in Harvard Style

Drawehn J., Blohm M., Kintz M. and Kochanowski M. (2020). Goal-based Evaluation of Text Mining Results in an Industrial Use Case. In Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-397-1, pages 183-191. DOI: 10.5220/0008973801830191

in Bibtex Style

@conference{icpram20,
author={Jens Drawehn and Matthias Blohm and Maximilien Kintz and Monika Kochanowski},
title={Goal-based Evaluation of Text Mining Results in an Industrial Use Case},
booktitle={Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2020},
pages={183-191},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008973801830191},
isbn={978-989-758-397-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Goal-based Evaluation of Text Mining Results in an Industrial Use Case
SN - 978-989-758-397-1
AU - Drawehn J.
AU - Blohm M.
AU - Kintz M.
AU - Kochanowski M.
PY - 2020
SP - 183
EP - 191
DO - 10.5220/0008973801830191