Recommendation System for Product Test Failures Using BERT

Xiaolong Sun, Xiaolong Sun, Henrik Holm, Sina Molavipour, Fitsum Gebre, Yash Pawar, Kamiar Radnosrati, Serveh Shalmashi

2023

Abstract

Historical failure records can provide insights to investigate if a similar situation occurred during the troubleshooting process in software. However, in the era of information explosion, massive amounts of data make it unrealistic to rely solely on manual inspection of root causes, not to mention mapping similar records. With the ongoing development and breakthroughs of Natural Language Processing (NLP), we propose an end-to-end recommendation system that can instantly generate a list of similar records given a new raw failure record. The system consists of three stages: 1) general and tailored pre-processing of raw failure records; 2) information retrieval; 3) information re-ranking. In the process of model selection, we undertake a thorough exploration of both frequency-based models and language models. To mitigate issues stemming from imbalances in the available labeled data, we propose an updated Recall@K metric that utilizes an adaptive K. We also develop a multi-stage training pipeline to deal with limited labeled data and investigate how different strategies affect performance. Our comprehensive experiments demonstrate that our two-stage BERT model, fine-tuned on extra domain data, achieves the best score over the baseline models.

Paper Citation

in Harvard Style

Sun X., Holm H., Molavipour S., Gebre F., Pawar Y., Radnosrati K. and Shalmashi S. (2023). Recommendation System for Product Test Failures Using BERT. In Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-671-2, SciTePress, pages 206-213. DOI: 10.5220/0012160800003598

in Bibtex Style

@conference{kdir23,
author={Xiaolong Sun and Henrik Holm and Sina Molavipour and Fitsum Gebre and Yash Pawar and Kamiar Radnosrati and Serveh Shalmashi},
title={Recommendation System for Product Test Failures Using BERT},
booktitle={Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2023},
pages={206-213},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012160800003598},
isbn={978-989-758-671-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Recommendation System for Product Test Failures Using BERT
SN - 978-989-758-671-2
AU - Sun X.
AU - Holm H.
AU - Molavipour S.
AU - Gebre F.
AU - Pawar Y.
AU - Radnosrati K.
AU - Shalmashi S.
PY - 2023
SP - 206
EP - 213
DO - 10.5220/0012160800003598
PB - SciTePress