An Anomaly Prediction of Spark Log Based on Self-Attention GRU Network

Yanyu Gong, Xinjiang Chen, Xiaoli Zhang, Haotian Xu, Xue Zhang, Haifeng Wang

2023

Abstract

This paper proposes a GRU network training prediction model to solve the problem of the difficulty of separating time series in Spark framework, so that the abnormal prediction model of Spark system can be realized in big data frameworks. Task is used to separate log data, SwissLog is used to transform it into a vector, and multiple attention mechanisms are used to deepen the repeated log series [1, 2]. To begin with, this paper solves the problem that Spark log data workflows are difficult to separate due to multi-thread output, and then log data cannot be converted into vectors. The robustness of structured data of log sequence conversion is further improved by optimizing and modifying SwissLog prefix tree and replacing it with Jaccard similarity algorithm, which improves anomaly prediction accuracy. To train the normal prediction model, the repeated time series is taken as the incremental dimension, and a GRU network with multiple attention mechanisms is used. As a result, the operation efficiency and model accuracy are improved, and equipment memory requirements are greatly reduced when training a large number of data sets. Based on the results presented in this paper, the GRU model for repetitive log sequences with multi-attention mechanism achieves the highest accuracy of 86.77% in the general public Spark data set from LogHub, and the accuracy and performance are 1.16 percent higher than the latest benchmark model LSTM, indicating that the proposed model can enhance anomaly detection accuracy and robustness effectively.

Paper Citation

in Harvard Style

Gong Y., Chen X., Zhang X., Xu H., Zhang X. and Wang H. (2023). An Anomaly Prediction of Spark Log Based on Self-Attention GRU Network. In Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT; ISBN 978-989-758-677-4, SciTePress, pages 415-421. DOI: 10.5220/0012285100003807

in Bibtex Style

@conference{anit23,
author={Yanyu Gong and Xinjiang Chen and Xiaoli Zhang and Haotian Xu and Xue Zhang and Haifeng Wang},
title={An Anomaly Prediction of Spark Log Based on Self-Attention GRU Network},
booktitle={Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT},
year={2023},
pages={415-421},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012285100003807},
isbn={978-989-758-677-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT
TI - An Anomaly Prediction of Spark Log Based on Self-Attention GRU Network
SN - 978-989-758-677-4
AU - Gong Y.
AU - Chen X.
AU - Zhang X.
AU - Xu H.
AU - Zhang X.
AU - Wang H.
PY - 2023
SP - 415
EP - 421
DO - 10.5220/0012285100003807
PB - SciTePress