A Study of Classiﬁcation of Texts into Categories of Cybersecurity

Incident and Attack with Topic Models

Masahiro Ishii, Satoshi Matsuura, Kento Mori, Masahiko Tomoishi, Yong Jin and Yoshiaki Kitaguchi

Tokyo Institute of Technology, Meguro-ku, Tokyo, Japan

Keywords:

Text Classiﬁcation, Seeded LDA, Topic Models, Data Mining, Cybersecurity Incidents, CERT.

Abstract:

To improve and automate cybersecurity incident handling in security operations centers (SOCs) and com-

puter emergency response teams (CERTs), security intelligences extracted from various internal and external

sources, including incident response playbooks, incident reports in each SOCs and CERTs, the National Vul-

nerability Database, and social media, must be utilized. In this paper, we apply various topic models to classify

text related to cybersecurity intelligence and incidents according to topics derived from incidents and cyber

attacks. We analyze cybersecurity incident reports and related text in our CERT and security blog posts using

naive latent Dirichlet allocation (LDA), seeded LDA, and labeled LDA topic models. Labeling text based on

designated categories is difﬁcult and time-consuming. Training the seeded model does not require text to be

labeled; instead, seed words are given to allow the model to infer topic-word and document-topic distributions

for the text. We show that a seeded topic model can be used to extract and classify intelligence in our CERT,

and we infer text more precisely compared with a supervised topic model.

1 INTRODUCTION

Cybersecurity incidents have become more compli-

cated as cyber attack methods evolve and latent vul-

nerabilities put organizations at risk. To mitigate the

effects and damages caused by such incidents, com-

puter emergency response teams (CERTs) and se-

curity operation centers (SOCs) have to analyze an

enormous number of security alerts generated by se-

curity systems and devices and those stored in vul-

nerability databases. Security information and event

management systems alone cannot protect against cy-

ber threats because they are based on signatures or

rigid rules and are thus insufﬁciently ﬂexible to de-

tect unknown attacks (Andrade and Torres, 2018).

In addition, the management and customization of

such systems are time-consuming and require exper-

tise (Zhong et al., 2019).

In this paper, we use various topic models to clas-

sify incident reports in our CERT and blog posts by a

security vendor and evaluate their performance using

experiments. We start with a small trial to extract se-

curity intelligence useful incident handling. We focus

on topic models because many data sources related

to security incidents are in natural language form and

the results of classifying text into categories are help-

ful for all CERT members, including non-experts.

The categories are based on incident and cyber at-

tack types. Labeling all training data for predicting

new text with models trained using a supervised algo-

rithm is extremely time-consuming (Li et al., 2018).

We perform experiments to compare a seeded latent

Dirichlet allocation (LDA) topic model with a su-

pervised topic model. For the seeded LDA (SLDA)

model, text does not need to be labeled; instead, seed

words are provided to classify the text into the desig-

nated topic categories. Our experimental results show

that the SLDA topic model can classify our report and

blog posts better than the supervised algorithm. We

also discussed how we set the seed words to enhance

the trained topic model by observing the clustering

results and correlations between the clusters and the

ground truth data

The rest of this paper is organized as follows. Sec-

tion 2 quickly reviews studies on analyzing and uti-

lizing cybersecurity intelligence extracted from data

from various sources. Section 3 describes the LDA-

based topic models used in the experiments. Section 4

presents the data and measures used for experiments

on clustering security text and predicting new text,

and the experimental results. We remark future tasks

and the conclusions in Section 5.

Ishii, M., Matsuura, S., Mori, K., Tomoishi, M., Jin, Y. and Kitaguchi, Y.

A Study of Classiﬁcation of Texts into Categories of Cybersecurity Incident and Attack with Topic Models.

DOI: 10.5220/0009099606390646

In Proceedings of the 6th International Conference on Information Systems Security and Privacy (ICISSP 2020), pages 639-646

ISBN: 978-989-758-399-5; ISSN: 2184-4356

639

2 RELATED WORK

Data mining and machine learning have recently been

applied to social media platforms, forums, and blogs

for extracting security intelligence. Several studies

have analyzed security intelligence and events using

text classiﬁcation based on machine learning or natu-

ral language processing models.

Deliu et al. (Deliu et al., 2017) performed ex-

periments on text classiﬁcation using convolutional

neural network (CNN) and support vector machine

(SVM) methods with text data from a hacker forum.

They found that the SVM method outperformed the

CNN method.

Nagai et al. (Nagai et al., 2018) classiﬁed secu-

rity blogs from different vendors using a guided-topic

model and a naive LDA model. They also analyzed

cybersecurity events by setting seed words for vari-

ous cyber threat categories.

Chambers et al. (Chambers et al., 2018) proposed

a partially labeled LDA model for detecting early cy-

ber attack discussions on Twitter. They used 2 mil-

lion tweets consisting of data related to distributed

denial-of-service (DDoS) attacks labeled binary value

(attack or non-attack). Neither attack dictionaries nor

seed words were used.

We remark that Nagai et al. (Nagai et al., 2018)

also presented classiﬁcation results using the SLDA

model for security text. We evaluated not only clus-

tering measurements but also classifying accuracy for

test text. In addition, we tuned the seed words and

evaluated the SLDA model in several steps. We fur-

thermore performed the experiments to compare the

SLDA model with the supervised topic model (la-

beled LDA).

3 TOPIC MODELS AND

PREPROCESSING TEXTS

We adopted topic models based on LDA to analyze

and classify Japanese text. Here, we brieﬂy describe

the features of the topic models and methods used to

preprocess Japanese text for the models.

3.1 Topic Models based on LDA

LDA (Blei et al., 2003) is a basic unsupervised topic

model algorithm that models a document with a mix-

ture of topics. The model generates topic-word and

document-topic distributions automatically, where the

model explicitly assumes that each word is generated

from one underlying topic. Figure 1a shows a graphi-

cal model of LDA. The topic-document distribution θ

and the topic-word distribution ϕ are derived from the

Dirichlet distributions with hyper parameters α and β,

respectively. Then, a topic in documents and a word

in topics are selected based on multinomial distribu-

tions θ and ϕ, respectively

Ramage et al. (Ramage et al., 2009) proposed the

labeled LDA (LLDA), which is a supervised LDA-

based topic model. In this model, document user tags

are associated with latent topics. Figure 1b shows a

graphical model of LLDA. The document-topic dis-

tribution is restrictedly drawn from the Dirichlet dis-

tribution with parameter α using the label set Λ. We

note that LLDA is appropriate for documents with

multiple labels.

SLDA (Jagarlamudi et al., 2012) is a guided-topic

model that uses sets of seed words that users as-

sume represent the underlying topics in a document.

A graphical model of SLDA is shown in Figure 1b.

This model uses seed words to improve the document-

topic distribution by drawing the group variable g and

the topic-word distribution by deﬁning a topic as a

mixture of seed topic distribution ϕ

and regular topic

distribution ϕ

We used the library GuidedLDA (Singh and am-

rrs, 2017) for the above LDA-based models. Guid-

edLDA cannot be completely implemented by fol-

lowing the algorithm described in (Jagarlamudi et al.,

2012, Section 2.3). Therefore, we implemented the

SLDA algorithm based on GuidedLDA. Furthermore,

we added the methods for computing LLDA model

with labels of document on GuidedLDA.

3.2 Preprocessing of Japanese Text

MeCab (Kudo et al., 2004) is one of the most com-

monly used morphological analyzers for Japanese

text. We used MeCab to parse Japanese text and

extract morphemes, which are useful for training

topic models. To extract proper nouns and recog-

nized concepts, we used the dictionary mecab-ipadic-

NEologd

, which consists of words extracted from

web resources. Stop words were set to enhance the

topic models. We used the words provided in Sloth-

Lib

for Japanese words and those in the Natural Lan-

guage Toolkit (Loper and Bird, 2002) for English

words. We performed basic stemming of words by

using the original form of words and converting the

characters to lowercase and half-width characters for

each Japanese and English word.

Many technical terms and entities appear in

cybersecurity-related text, such as Internet Protocol

(IP) addresses, Uniform Resource Locators (URLs),

https://github.com/neologd/mecab-ipadic-neologd

http://www.dl.kuis.kyoto-u.ac.jp/slothlib/

ICISSP 2020 - 6th International Conference on Information Systems Security and Privacy

640

Labeled LDA

LDA Seeded LDA

<latexit sha1_base64="FnwjOJKlE9EnO6pemzKI5tt+4d0=">AAACa3icZZHLSsNAFIan8VbjLbU7dSG2gquSRMFtsRtBChXsBdpSJtPTdOwkE2amJSXkTdzqO/kQvoNJDUKaf/XznTlzzvzjBIxKZZrfJW1nd2//oHyoHx2fnJ4ZlfOe5EtBoEs442LgYAmM+tBVVDEYBAKw5zDoO4tWWu+vQEjK/Te1DmDsYdenM0qwStDEMKKRw9nUw2qu11/qejwxambD3Oi6aKzM1FCmzqRSao2mnCw98BVhWMohfngfR+9YKEoYxPpoKSHAZIFdiDYL5xH2ZDo9gRIU4UtfgYjaOGxjJWiYvEDGkW0WeuTacwpQzb08S2/GTliEQs5kkU652qIOdam/gWoOXEAyY80gmsKM+jSNcGsH5nJBC2v8Y0pyhSFmbByF61hPYre2Qy6ant2w7hv2q11rPmUfUEaX6AbdIQs9oiZ6Rh3URQSt0Af6RF+lH62qXWhXf0e1UtZTRTlpt79H5cCt</latexit>

<latexit sha1_base64="FY6oCTTNvJhW6CWVxyGVztXQjSk=">AAACa3icZZHLTsJAFIaHesN6K7JTF0YwcUXaauKWiAs3JJjIJQFCpsOhjEw7zcxASpq+iVt9Jx/Cd7DFxqT0X/35zpw5Z/5xAkalMs3vkrazu7d/UD7Uj45PTs+MynlP8qUg0CWccTFwsARGfegqqhgMAgHYcxj0nUUrrfdXICTl/ptaBzD2sOvTGSVYJWhiGNHI4WzqYTXX6891PZ4YNbNhbnRdNFZmaihTZ1IptUZTTpYe+IowLOUQP7yPo3csFCUMYn20lBBgssAuRJuF8wh7Mp2eQAmK8KWvQERtHLaxEjRMXiDjyDYLPXLtOQWo5l6epTdjJyxCIWeySKdcbVGHutTfQDUHLiCZsWYQTWFGfZpGuLUDc7mghTX+MSW5whAzNo7CdawnsVvbIRdNz25Y9w371a41n7IPKKNLdIPukIUeURO9oA7qIoJW6AN9oq/Sj1bVLrSrv6NaKeupopy02186FsCm</latexit>

<latexit sha1_base64="tid3WGgQ7DPGvtX84A3ToDdDNUA=">AAACbXicZZHLTsJAFIYP9Yb1BhhXGmOERFekoIlbIhs3GE3kkgAh0+mhDEw7ZGZqSpq+ilt9JZ/CV7BFYgL9V3++M2fOmX/sOWdKW9Z3ztja3tndy++bB4dHxyeFYqmjRCAptqngQvZsopAzH9uaaY69uUTi2Ry79qyZ1rvvKBUT/ptezHHoEddnY0aJTtCoUIoGtuCOR/TErDyPnIoZjwplq2otdZU1tZUpw0ovo2KuOXAEDTz0NeVEqT65nw6jKZGaUY6xOQgUzgmdERej5crriHgqnZ9AhZqKwNcooxYJW0RLFiZvUHFUtzI9auHZGagn3jpLbyZ2mIVSjVWWOkJvUJu5zF9CPUEhMZmx4Bg5OGY+S0Pc2IG7QrLMGv+Y0bVCn3A+jMJFbCax1zZDzppOvVq7q9Zf6+XG4+oD8nAO13ALNXiABjzBC7SBQggf8AlfuR/jzLgwLv+OGrlVzymsybj5BSNcwYc=</latexit>



<latexit sha1_base64="Pzv/mrzyY2wHkx8A1Byx7ZCJm/o=">AAACcHicZZHLTsJAFIaHesN6A42JiQsvsDAuSFtN3BLZuCHBRC4JEHI6HGBg2mlmpgbS9GHc6hP5Gj6BBRsT6L86+c6cOf/84wacKW1Z3zlja3tndy+/bx4cHh2fFIqnLSVCSbFJBRey44JCznxsaqY5dgKJ4Lkc2+6stuy331EqJvw3vQiw78HYZyNGQSdoUDiPeq7gQw/0xCz3gAcTKJvxoFCyKtZK19nCTosSSdUYFHO13lDQ0ENfUw5KdeFx2o+mIDWjHGOzFyoMgM5gjNHK9ToCTy0tJFChpiL0NcqoDvM6aMnmyTNUHDlWZkYtPDcD9cRbZ8ubwZ1noVQjlaVDoTeoy8bMX0E9QSEx2bHgGA1xxHy2zHHDAx8LyTI2/jGja40ucN6P5ovYTGK3N0POFi2nYj9UnFenVH1OPyBPLsktuSM2eSJV8kIapEkoicgH+SRfuR/jwrgybv6OGrl05oysybj/BT7uwvY=</latexit>



<latexit sha1_base64="3Eat2YKfp79/Ch+LOSN/8jqLOlA=">AAACcXicZZHLTsJAFIaHesN6A01MjBsCLExMSKkmbols3JBgIpeEIpmZHmBg2mlmBlNS+zJu9YV8Dl/AFokJ9F+dfGfOnH/+IQFnSlvWd87Y2d3bP8gfmkfHJ6dnheJ5V4mFpNChggvZJ1gBZz50NNMc+oEE7BEOPTJvpv3eG0jFhP+ilwEMPTzx2ZhRrBM0KlxGDhHc9bCemlWHgMavsmrGo0LFqlkrlbJFfV1U0FrtUTHXdFxBFx74mnKs1ADfz4bRDEvNKIfYdBYKAkzneALRyvYmwp5KPSRQgaZi4WuQUQuHLawlC5N3qDiyrcyMWnokA/XU22TpzZiEWSjVWGWpK/QWJWzC/BXUUxASkh1LDpELY+azNMgtD3wiJMvY+MeMbjQGmPNhFC5jM4m9vh1ytujatfpdzX62K43H9Qfk0TUqoxtURw+ogZ5QG3UQRe/oA32ir9yPcWWUjPLfUSO3nrlAGzJufwE0ZsNm</latexit>



<latexit sha1_base64="D0ly8UQniO3m1qzXobGx4g21hTw=">AAACcHicZZHLSsNAFIan8VbjrVUEwYWXuhAXJYmC22I3bgoV7AWaWiaT03TaSSbMTCUh9GHc6hP5Gj6BaQ1Cmn91+M6cOf/844SMSmUY3yVtY3Nre6e8q+/tHxweVarHXcnngkCHcMZF38ESGA2go6hi0A8FYN9h0HNmzWW/9w5CUh68qjiEoY+9gI4pwSpFo8ppYjucuT5WE/3GDif0Tdzoi1GlZtSNlS6LhZkVNZSpPaqWmrbLydyHQBGGpRzgh+kwmWKhKGGw0O25hBCTGfYgWbnOI+zLpYUUSlCEzwMFImnhqIWVoFH6DLlILKMwI2PfKUA18fNseTN2oiIUciyL1OVqjTrUo8EKqglwAemOmEHiwpgGdJnjmgfmcUELNv4xJbnGADM2TKJ4oaexm+shF4uuVTfv69aLVWs8ZR9QRufoGt0iEz2iBnpGbdRBBCXoA32ir9KPdqZdaFd/R7VSNnOCctLufgFUucMB</latexit>



<latexit sha1_base64="a8ZzrSU5ilyQ6UKTcb7bWcUyE1A=">AAACcHicZZHLSsNAFIan8VbjrVUEwYWXuhAXJYmC22I3bgoV7AWaWiaT03TaSSbMTCUh9GHc6hP5Gj6BaQ1Cmn91+M6cOf/844SMSmUY3yVtY3Nre6e8q+/tHxweVarHXcnngkCHcMZF38ESGA2go6hi0A8FYN9h0HNmzWW/9w5CUh68qjiEoY+9gI4pwSpFo8ppYjucuT5WE/3GDif0Td7oi1GlZtSNlS6LhZkVNZSpPaqWmrbLydyHQBGGpRzgh+kwmWKhKGGw0O25hBCTGfYgWbnOI+zLpYUUSlCEzwMFImnhqIWVoFH6DLlILKMwI2PfKUA18fNseTN2oiIUciyL1OVqjTrUo8EKqglwAemOmEHiwpgGdJnjmgfmcUELNv4xJbnGADM2TKJ4oaexm+shF4uuVTfv69aLVWs8ZR9QRufoGt0iEz2iBnpGbdRBBCXoA32ir9KPdqZdaFd/R7VSNnOCctLufgFWssMC</latexit>



<latexit sha1_base64="eF3yQIi9D43M9eQ2JDuK+Xpl2BQ=">AAACbnicZZHLTsJAFIan9Yb1BpK4MUYjLFiRtpq4JbJxQ4KJXBJKyHR6KAPTTjMzmDYNz+JWH8m38BEsSExK/9XJd+bM+ecfN2JUKtP81vS9/YPDo9KxcXJ6dn5Rrlz2JV8KAj3CGRdDF0tgNISeoorBMBKAA5fBwF201/3BOwhJefimkgjGAfZDOqUEqwxNytXUcTnzAqxmRt2JJK0bq0m5ZjbNje6KhbUtamir7qSitR2Pk2UAoSIMSznCj/NxOsdCUcJgZThLCREmC+xDuvGcRziQawMZlKAIX4YKRNrBcQcrQePsEXKV2mZhRiaBW4BqFuTZ+mbsxkUo5FQWqcfVDnWpT8MNVDPgArIdCYPUgykN6TrFHQ/M54IWbPxjSnKNEWZsnMbJyshit3ZDLhZ9u2k9NO1Xu9Z63n5ACV2je9RAFnpCLfSCuqiHCErQB/pEX9qPfqXf6Ld/R3VtO1NFOemNX3idwig=</latexit>

<latexit sha1_base64="9OstR+bEV+nDS0Du4O4kvdn618g=">AAACa3icZZHLTsJAFIaHesN6K7JTF0YwcUXaauKWyMYNCSZySYCQ6fQAA9NOMzMlJU3fxK2+kw/hO9giMSn9V3++M2fOmX+cgFGpTPO7pO3tHxwelY/1k9Oz8wujctmTPBQEuoQzLgYOlsCoD11FFYNBIAB7DoO+s2xl9f4KhKTcf1frAMYenvl0SglWKZoYRjxyOHM9rOZ6ParrycSomQ1zo9uisbamhrbqTCql1sjlJPTAV4RhKYf4aTGOF1goShgk+iiUEGCyxDOINwvnEfZkNj2FEhThoa9AxG0ctbESNEpfIJPYNgs9cu05BajmXp5lN2MnKkIhp7JIXa52qENn1N9ANQcuIJ2xZhC7MKU+zSLc2YHNuKCFNf4xJbnCEDM2jqN1oqexW7shF03PbliPDfvNrjVfth9QRtfoDj0gCz2jJnpFHdRFBK3QB/pEX6UfrapdaTd/R7XStqeKctLufwGgqsDa</latexit>

<latexit sha1_base64="wVRBTc5BpSIugg7PEEaKsc4QVFc=">AAACa3icZZHLTsJAFIZP6w3rDWSnLoxg4ooUNHFLZOOGBBO5JEDIdHoog9MOmZkSasObuNV38iF8B1skJtB/9ec7c+ac+ceZcaa0bX8b5s7u3v5B7tA6Oj45PcsXzjtKhJJimwouZM8hCjkLsK2Z5tibSSS+w7HrvDXSeneOUjERvOpohkOfeAEbM0p0gkb5fDxwBHd9oidW+b1sLUf5kl2xV7rOmuralGCt1qhgNAauoKGPgaacKNUnD9NhPCVSM8pxaQ1ChTNC34iH8WrhTUR8lU5PoEJNRRholHGTLJpES7ZIXqCWcc3O9KjIdzJQT/xNlt5MnEUWSjVWWeoKvUUd5rFgBfUEhcRkRsQxdnHMApZGuLUD94RkmTX+MaMbhT7hfBgvoqWVxF7dDjlrOrVK9b5Se6mV6k/rD8jBJdzAHVThEerwDC1oA4U5fMAnfBk/ZtG8MK/+jprGuqcIGzJvfwGknMDc</latexit>

<latexit sha1_base64="hM2iKVi2CpspvbIwLHiK8oUrixo=">AAACa3icZZHLTsJAFIan9Yb1BrJTF0YwcUXaauKWyMYNCSZySYCQ6XAog9NOMzPFNg1v4lbfyYfwHWyxMYH+qz/fmTPnzD9OwKhUpvmt6Tu7e/sHpUPj6Pjk9KxcOe9JHgoCXcIZFwMHS2DUh66iisEgEIA9h0HfeWtl9f4ShKTcf1VxAGMPuz6dUYJViiblcjJyOJt6WM2N+nvdWE3KNbNhrnVdNFZuaihXZ1LRWqMpJ6EHviIMSznED4txssBCUcJgZYxCCQEmb9iFZL3wJsKezKanUIIiPPQViKSNozZWgkbpC+Qqsc1Cj4w9pwDV3Ntk2c3YiYpQyJks0ilXW9ShLvXXUM2BC0hnxAySKcyoT7MIt3ZgLhe0sMY/pmSjMMSMjZMoXhlp7NZ2yEXTsxvWfcN+sWvNp/wDSugS3aA7ZKFH1ETPqIO6iKAl+kCf6Ev70av6hX71d1TX8p4q2pB++wuescDZ</latexit>



<latexit sha1_base64="II4UsidxWlZDzwIDEKIvj8XGVLs=">AAACcHicZZHLTgIxFIbLeMPxBhoTExdeYGFckJnRxC2RjRsSTOSSMIS05QCFznTSFjOk4WHc6hP5Gj6BAxKTYf7VyXd6ev7+JRFnSjvOd87a2t7Z3cvv2weHR8cnheJpS4mZpNCkggvZIVgBZyE0NdMcOpEEHBAObTKtLfvtd5CKifBNzyPoBXgUsiGjWCeoXzg3PhF8EGA9tsu+HoPGZXvRL5ScirPSdbZw10UJrdXoF3M1fyDoLIBQU46V6uLHSc9MsNSMcljY/kxBhOkUj8CsXKcRDtTSQgIVaCpmoQZp6jiuYy1ZnDxDLYznZGbUPCAZqMdBmi1vxiTOQqmGKksHQm9QwkYsXMEkICEh2THnYAYwZCFb5rjhgY+EZBkb/5jRVKOLOe+ZeL6wk9jdzZCzRcuruA8V79UrVZ/XH5BHl+gW3SEXPaEqekEN1EQUGfSBPtFX7se6sK6sm7+jVm49c4ZSsu5/AV60wwY=</latexit>

<latexit sha1_base64="1nFScu3gfGuSAdSKUFT2bIRMpmM=">AAACa3icZZHLTsJAFIYP9Yb1BrJTF0YwcUXaauKWyMYNCSZySYCQ6XAoA9NOMzOQkoY3cavv5EP4DhYkJqX/6s935sw5848bcqa0ZX3njL39g8Oj/LF5cnp2flEoXraVmEuKLSq4kF2XKOQswJZmmmM3lEh8l2PHndXX9c4CpWIieNfLEAc+8QI2ZpToBA0LhbjvCj7yiZ6YFa9iroaFslW1NrrNGntryrBVc1jM1fsjQec+BppyolSPPE0H8ZRIzSjHldmfKwwJnREP483CaUR8tZ6eQIWainmgUcYNEjWIlixKXqBWsWNletTSdzNQT/w0W99M3CgLpRqrLB0JvUNd5rFgA/UEhcRkxpJjPMIxC9g6wp0duCcky6zxjxlNFXqE80EcLVdmEru9G3LWtJ2q/Vh13pxy7WX7AXm4hjt4ABueoQav0IQWUFjAB3zCV+7HKBlXxs3fUSO37SlBSsb9L38hwMk=</latexit>



<latexit sha1_base64="v/mk54zej7fa4CgkUKVzIiJazoo=">AAACcHicZZHLTgIxFIbLeMPxBhoTExdeYGFckBk0cUtk44YEE7kkQMiZchgK7XTSFsNkwsO41SfyNXwCByQmMP/q5Ds9PX//eiFn2jjOd8ba2t7Z3cvu2weHR8cnufxpU8upotigkkvV9kAjZwE2DDMc26FCEB7HljepLvqtd1SayeDNRCH2BPgBGzIKJkH93Hnc9SQfCDAju9j1QQgo2vN+ruCUnKWu04W7KgpkpXo/n6l2B5JOBQaGctC6A4/jXjwGZRjlOLe7U40h0An4GC9dryMQemEhgRoNldPAoIprMKuBUWyWPEPP47KTmtGR8FLQjMQ6W9wM3iwNlR7qNB1Is0E95rNgCc0IpcJkR8QxHuCQBWyR44YH7kvFUjb+MaNrjQ5w3otn0dxOYnc3Q04XzXLJfSiVX8uFyvPqA7LkktySO+KSJ1IhL6ROGoSSmHyQT/KV+bEurCvr5u+olVnNnJE1Wfe/OPnC8w==</latexit>



<latexit sha1_base64="X2wfymGPxtMiRl87YK2wU1I4c/c=">AAACbnicZZHLTsJAFIYP9Yb1BpK4MUYjLFiRgiZuiWzckGAil4QSMh0OZWDaaWamhqbhWdzqI/kWPoIFiUnpvzr5zpw5//zjBJwpbVnfOWNv/+DwKH9snpyenV8Uipc9JUJJsUsFF3LgEIWc+djVTHMcBBKJ53DsO4vWut9/R6mY8N90FODII67PpowSnaBxoRTbjuATj+iZWbE1CSvmalwoWzVro7tsUd8WZdiqMy7mWvZE0NBDX1NOlBqSx/konhOpGeW4Mu1QYUDogrgYbzynEfHU2kACFWoqQl+jjNtk2SZasmXyCLWKG1ZmRkWek4F65qXZ+mbiLLNQqqnK0onQO9RhLvM3UM9QSEx2RBzjCU6Zz9Yp7njgrpAsY+MfM5pqDAnno3gZrcwk9vpuyNmi16jVH2qN10a5+bz9gDxcwz1UoQ5P0IQX6EAXKETwAZ/wlfsxrowb4/bvqJHbzpQgJaP6C3ShwiY=</latexit>

<latexit sha1_base64="m8vJugzfgwdcKSHBvb9br7wm/U8=">AAACa3icZZHLTgIxFIYP4w3H2yA7dWEEE1dkQBO3RDZuSDCRSwKEtOUwFDrTSVsIZMKbuNV38iF8BwecmMD8qz/f6ek5/UtDwbVx3e+Mtbd/cHiUPbZPTs/OL5zcZUvLmWLYZFJI1aFEo+ABNg03AjuhQuJTgW06ra3r7TkqzWXwbpYh9n3iBXzEGTExGjhO1KNSDH1ixnaRFu3VwCm4JXej27QpJ6YAiRqDXKbWG0o28zEwTBCtu+Rp0o8mRBnOBK7s3kxjSNiUeBhtFt5GxNfr6THUaJicBQZVVCeLOjGKL+IX6FVUcVM9eunTFDRjf5utbyZ0kYZKj3SaDqXZoZR7PNhAM0apMJ6xFBgNccQDvo5wZwfhScVTa/xjzrYKXSJEP1osV3Yce3k35LRpVUrlx1LlrVKoviQfkIVruIMHKMMzVOEVGtAEBnP4gE/4yvxYeevKuvk7amWSnjxsybr/BXVEwMQ=</latexit>



<latexit sha1_base64="Y/3JhWVR6voQq+MIoGsFG5yuPOg=">AAACbnicZZHLTsJAFIan9Yb1BpK4MUYjLFiRFk3cEtm4IcFELgklZDo9lIFpp5kZTJuGZ3Grj+Rb+Ai2SExK/9XJd+bM+ecfJ2RUKtP81vS9/YPDo9KxcXJ6dn5RrlwOJF8JAn3CGRcjB0tgNIC+oorBKBSAfYfB0Fl2sv7wHYSkPHhTcQgTH3sBnVGCVYqm5WpiO5y5PlZzo26DwnVjPS3XzKa50V2xsLZFDW3Vm1a0ju1ysvIhUIRhKcf4cTFJFlgoShisDXslIcRkiT1INp7zCPsyM5BCCYrwVaBAJF0cdbESNEofIddJyyzMyNh3ClDN/TzLbsZOVIRCzmSRulztUId6NNhANQcuIN0RM0hcmNGAZinueGAeF7Rg4x9TkmuMMWOTJIrXRhq7tRtysRi0mtZDs/XaqrWftx9QQtfoHjWQhZ5QG72gHuojgmL0gT7Rl/ajX+k3+u3fUV3bzlRRTnrjF1UGwhY=</latexit>



<latexit sha1_base64="IwThYXztQzt8BUjDjwwM4dQM4GM=">AAACb3icZZHLTsJAFIaHesN6AzVxYWKIsNANaauJWyIbNySYyCUBQmamBxiZdpqZwYBN38WtvpGP4RvYYmNS+q9OvjNnzj//kIAzpS3ru2Bsbe/s7hX3zYPDo+OTUvm0q8RCUuhQwYXsE6yAMx86mmkO/UAC9giHHpk3k37vDaRiwn/RqwBGHp76bMIo1jEal87DIRHc9bCembUhAY1rZjQuVa26tVYlX9hpUUWp2uNyoTl0BV144GvKsVIDfP86Cl+x1IxyiMzhQkGA6RxPIVybziLsqcRBDBVoKha+Bhm28LKFtWTL+BUqCh0rN6NWHslBPfOyLLkZk2UeSjVReeoKvUEJmzJ/DfUMhIR4x4pD6MKE+SyJccMDnwrJcjb+MaOZxgBzPgqXq8iMY7c3Q84XXadu39WdZ6faeEw/oIgu0TW6QTZ6QA30hNqogyh6Rx/oE30VfowL48qo/B01CunMGcrIuP0FQdLCgg==</latexit>



<latexit sha1_base64="+DxAfQhFxclyCK+INJqlq2uZVZo=">AAACbnicZZHLTsJAFIYP9Yb1BpK4MUYjLFiRtpq4JbJxQ4KJXBJKyHQ4wMC008wMpk3Ds7jVR/ItfAQLEhPovzr5zpw5//zjhZwpbVnfOWNv/+DwKH9snpyenV8UipcdJRaSYpsKLmTPIwo5C7CtmebYCyUS3+PY9eaNVb/7jlIxEbzpOMSBTyYBGzNKdIqGhVLieoKPfKKnZsUNp6xiLoeFslWz1rrLFvamKMNGrWEx13BHgi58DDTlRKk+eZwNkhmRmlGOS9NdKAwJnZMJJmvP24j4amUghQo1FYtAo0yaJGoSLVmUPkItE8fKzKjY9zJQT/1ttrqZeFEWSjVWWToSeod6bMKCNdRTFBLTHTHHZIRjFrBVijse+ERIlrHxjxndavQJ54MkipdmGru9G3K26Dg1+6HmvDrl+vPmA/JwDfdQBRueoA4v0II2UIjhAz7hK/djXBk3xu3fUSO3mSnBlozqL2Lfwh0=</latexit>



<latexit sha1_base64="a6bZETIPCq8P1o7BwRmtu77xJLg=">AAACbnicZZHLTsJAFIan9Yb1BpK4MUYjLFiRFk3cEtm4IcFELgkQMp0eysC008xMDU3Ds7jVR/ItfARbbExK/9XJd+bM+ecfO2BUKtP81vS9/YPDo9KxcXJ6dn5RrlwOJA8FgT7hjIuRjSUw6kNfUcVgFAjAns1gaK86aX/4DkJS7r+pKICph12fzinBKkGzcjWe2Jw5HlYLoz7pLWjd2MzKNbNpbnVXLKysqKFMvVlF60wcTkIPfEUYlnKMH5fTeImFooTBxpiEEgJMVtiFeOs5j7AnUwMJlKAID30FIu7idRcrQdfJI+QmbpmFGRl5dgGqhZdn6c3YXhehkHNZpA5XO9SmLvW3UC2AC0h2RAxiB+bUp2mKOx6YywUt2PjHlOQaY8zYNF5HGyOJ3doNuVgMWk3rodl6bdXaz9kHlNA1ukcNZKEn1EYvqIf6iKAIfaBP9KX96Ff6jX77d1TXspkqyklv/AIjf8H9</latexit>



<latexit sha1_base64="TjjbuIhnCOYM4ZALZBWT3IyDLo4=">AAACcXicZZHLasJAFIZP0ptNb9pCoXQj6qJQkJgWupW66aKChXoBFZlMjjo6yYSZsShpXqbb9oX6HH2BRisFzb86fGf+Of+ccUPOlLbtb8Pc2d3bP8gcWkfHJ6dn2dx5S4mZpNikggvZcYlCzgJsaqY5dkKJxHc5tt1pbdlvv6FUTASvehFi3yejgA0ZJTpBg+xl1HMF93yix1ap95wYPVKy4kG2aJftlfLporIuirBWY5Azaj1P0JmPgaacKNUl95N+NCFSM8oxtnozhSGhUzLCaBV7ExFfLTMkUKGmYhZolFGdzOtESzZP3qHiyLFTHrXw3RTUY3+TLW8m7jwNpRqqNPWE3qIuG7FgBfUYhcRkxoJj5OGQBWy5yK0MfCQkS8X4x4xuNLqE8340X8RWsvbK9pLTRcspV+7KzotTrD6uPyAD11CAG6jAA1ThCRrQBArv8AGf8GX8mFdm3iz8HTWNtecCNmTe/gLe+cM7</latexit>



(a) LDA (Blei et al., 2003).

Labeled LDA

LDA Seeded LDA



(b) LLDA (Ramage et al., 2009).

Labeled LDA

LDA Seeded LDA



Figure 1: Graphical models of topic models.

hashes, and names of malware. In this work, we ex-

tracted IPv4 addresses and URLs from text using reg-

ular expressions and used them along with the above-

mentioned preprocessed words to train the topic mod-

els. In addition, we also extracted MD5, SHA1,

SHA256, and SHA512 hashes of (malicious) ﬁles as

IOC.

In addition to general stop words, we removed

some words extracted using the LLDA model. When

training models with LLDA for labeled text, the

model assigns the corresponding label of text and the

label common topic to each token. Words that appear

in the category common topic also appear in other cat-

egories uniformly, and thus should not be set as seed

words.

4 EXPERIMENTS

We evaluated LDA, LLDA, and SLDA models trained

using labeled cybersecurity text. We conducted exper-

iments on overlapping (or soft) clustering using multi-

labeled text and predictions using test text (i.e., text

not used to train the models).

4.1 Data

We used the following 405 security text from internal

and external sources in the experiments.

• 45 security incident reports written from the be-

ginning of 2016 to March 2019 managed in our

CERT.

• 120 blog posts in Trend Micro Security Blog (in

Japanese)

https://blog.trendmicro.co.jp

• 120 blog posts in Cisco Japan Blog categorized as

security research

• 120 blog posts ﬁltered by LANGUAGE: Japanese,

SYMANTEC BLOGS: Security Response in

Symantec Connect

For the blogs by each security vendor, we col-

lected the recent 120 blog text posted before the end

of September 2019.

In our CERT, the members summarize the results

of incident investigations and responses for incident

handling interactions with other teams (SOC or the

network operations center) or the person who con-

tacted the CERT. The reports are stored as an Excel

ﬁle. We analyzed the following types of text in the

reports using the topic models: abstract of incident,

status of response, cause of incident and possible at-

tacks, information on the compromised system (e.g.,

IP address, network environment, OS version), attack

(source) information (e.g., IP address/domain, name

of malware), and measures implemented to prevent

recurrence.

Next, we consider suitable categories for classify-

ing security text with topic models. In this paper, we

deﬁne categories of cybersecurity incident and attack

that correspond to some standards that are commonly

recognized.

Speciﬁcally, we set the following 6 categories of

incident based on NIST SP800-61 Revision 1

and

FIRST

in Table 1, and 9 categories of attack based

on MITRE CAPEC

in Table 2.

https://gblogs.cisco.com/jp/category/security/research/

https://www.symantec.com/connect/search

https://www.ﬁsmacenter.com/SP800-61rev1.pdf

https://www.ﬁrst.org/resources/guides/csirt case

classiﬁcation.html

https://capec.mitre.org

A Study of Classiﬁcation of Texts into Categories of Cybersecurity Incident and Attack with Topic Models

641

Table 1: Incident categories.

Id Category

0 Others

1 Unauthorized Access, Compromised In-

formation/Asset, Unlawful Activity (Theft,

Fraud, Human Safety, Child Porn), Espi-

onage

2 DoS

3 Malware

4 Scan, Probe, Attempted Access, Reconnais-

sance

5 Improper Usage, Policy Violations

Table 2: Attack categories.

Id Category

0 Others

1 Social Engineering (Phising, Targeted At-

tack, Information Elicitation, Pretexting,

Identity Fraud)

2 DoS

3 Vulnerability Exploit,

4 Malware, File/Conﬁguration/Environment

Manipulation, Manipulation During Distri-

bution

5 Inject Unexpected Items (XSS, Command

Injection, Targeted Malware)

6 Employ Probabilistic Techniques (Brute

Force, Fuzzing)

7 Collect and Analyze Information (Scan,

Snifﬁng, Fingerprinting, Excavation

8 Subvert Access Control (Session Hijacking,

Cross Site Request Forgery, MITM)

For the incident categories, we merged the cate-

gories by NIST and FIRST. We basically deﬁne each

category of attacks corresponding to mechanisms of

attack in MITRE CAPEC.

These categories are rather general and may be

unsuitable for classifying incidents in detail, unlike

those described in the guide

and the Cyber Threat

Categories by SurfWatch Labs Inc.

. In the incident

report, we must check the categories of incidents and

attacks that can be used to label reports to classify

them according to incident type. Because we focus

on observing our security report data and classifying

them, we use the required set.

Although each blog post is tagged, the tags are

rather general, such as cyber attack, attack methodol-

https://www.ﬁrst.org/resources/guides/csirt case

classiﬁcation.html

https://www.surfwatchlabs.com/threat-categories

ogy, or too much detailed such as malware families

for few special documents. Thus, it is thus appro-

priate for labeling according to distinct attack types

and computing good classiﬁer with topic models. We

manually labeled the security text and set appropri-

ate multi-labels at most 4. These multi-labels are re-

garded as ground truth data in our experiments.

4.2 Measurements

To measure overlapping clustering and predict test

text with the topic models, we evaluated the mod-

els using purity, inverse purity, F1-score, and normal-

ized mutual information (NMI). Speciﬁcally, we use

macro (inverse) purity and macro F1-score which is

the harmonic mean of purity and inverse purity. Let

C be the cluster sets formed by clustering with the

topic models, and C

′

be the ground truth class sets.

Then the macro purity and inverse purity correspond

to F

C,C

′

and F

′

, respectively where F

X,Y

is deﬁned

as Eq. 7 in (Lutov et al., 2019).

We also use normalized mutual information

NMI(C,C

′

) to evaluate overlapping clusterings. De-

tailed formulae to compute NMI are described in (Lu-

tov et al., 2019) (see Section IV.C.1 and Eq.11, 12).

We note that we did not evaluate generalized NMI but

naive NMI in the experiments.

4.3 Initial Seed Words

The quality of the SLDA model greatly depends on

the deﬁned seed words. We ﬁrst observed words that

frequently appeared for the given ground truth labels.

In (Jagarlamudi et al., 2012, Section 2.4), the authors

describe the technique to select initial seed words au-

tomatically by using information gain.

We computed the information gain of words in

text and their ground truth labels, and extract top

ranked 15 words for each category of incident and at-

tack. The sampled seed words translated into English

for the incident and attack topic categories are deﬁned

as shown in Tables 3 and 4. The categories are sim-

pliﬁed and the words in bold and italic style were re-

moved or added for evaluating effects of seed words

in the experiments. The words which are not written

by bold style are sampled extracted words with infor-

mation gain. We describe how we set the seed words

in more detail in Section 4.5.

4.4 Clustering Results

We conducted an experiment with overlapping clus-

tering because the labeled text belongs to one or more

categories. To classify the text into multi-categories,

ICISSP 2020 - 6th International Conference on Information Systems Security and Privacy

642

Table 3: Incident categories and seed words.

Category Seed words

Others ip, “blog author”, supervision , exe, windows, evangelist, system, microsoft, infec-

tion, malware, round up article, comprehension

Unauthorized Access indicator, infection, vulnerable, iocs, compromise, “sha1 hash”,

dnspionage, suspicious, micro, trend, vulnerability, disclosure of information

DoS present, blog, denial, research, prevent, newsroom, vmx, snortsnort

Malware infection, malware, indicator, download, screenshot, ip, ioc, scan, exe, feature,

ﬁrepower

Scan, Attempted Access udp, dictionary attack, port, ssh, ip address, ddos attack, tcp, router, backdoor,

jump server, ﬂood, relay, indicators, bounty

Improper Usage coinhive, rip, virus scan, mac, laboratory, isolation, free, subspecies,

conﬁdential information, personal information, license, version, crack ,

“experimental product”

we extracted the topic category for which the inferred

topic ratio is higher than a given threshold in order of

descending ratios, and we limited the number of in-

ferred topics for each text. Table 5 shows the results

of macro F1-score and NMI for overlapping cluster-

ing.

We denote the number of topic categories as K;

K = 6 for incident categories and K = 9 for attack cat-

egories. We used the standard hyperparameters val-

ues α = 50/K,β = 0.01 and π

= 0.7 (Jagarlamudi

et al., 2012, Section 2.1), and symmetric Dirichlet

distributions. Although we evaluated the clustering

results varying the maximum numbers of inferred top-

ics from 1 to 4, we only showed the cases when then

number is 1 or 4. The results for which the threshold

is 0 and the number of inferred topics is 1 correspond

to those of hard clustering.

We can see that SLDA outperformed LDA in all

cases. When the maximum number of inferred topics

is 4, the F1-scores for threshold 0.01/K and 0.1/K

are higher than the F1-score for threshold 1/K. Ac-

tually, the inverse purity become higher unreasonably

since most of the samples are classiﬁed into 4 cate-

gories, and such a classiﬁer is useless for identifying

topics of security text. In these cases, the NMI values

become lower that means the clusterings are not good

comparing with the ground truth class sets. In our ex-

periments, including the following evaluation for test

text, we set the threshold to 1/K and the maximum

number of inferred topics to 4.

4.5 Update Seed Words

Here, we describe experiments for updating the seed

words by observing the clustering results. For

the trained models with SLDA, we manually corre-

sponded the clusters to the incident or attack cate-

gories by the document-topic distributions and the

seed-topic distributions of the model since SLDA is

an unsupervised algorithm. Then we can evaluate

how the SLDA model classiﬁes security text into

multi-categories. We modiﬁed the seed words and

evaluated clusterings in the following steps.

• Initial Automatic extraction by computing infor-

mation gain using the ground truth labels of text.

• Step 1 We manually remove words that re-

duce overlapping clustering accuracy such as fre-

quently used common words in several categories

or peculiar words.

• Step 2 After training the SLDA model with seed

words deﬁned in Step 1, we select few words ex-

tracted by information gain with classiﬁcation re-

sults.

• Step 3 We manually proper words based on

heuristic knowledge such as an IP address, IOC,

or cybersecurity proper words from security intel-

ligences from internal and external sources.

In Tables 3 and 4, we underline the words that are

actually used in the experiments. For the extracted

words in the initial phase (the non-bold words), the

italic words are removed in Step 1. We then added the

bold words that are selected in Step 2 and 3 together.

We note that the double-quoted words are obfuscated

since these words indicate individual or sensitive in-

formations.

Tables 6 shows the clustering results with F1-

scores and NMI for the seed words deﬁned in each

step.

The evaluated values for the seed words in Step

1 are worse than those for initial seed words. How-

ever, it is necessary to remove some words in Step 1

since the initial seed words totally depend on ground

truth labels of the security text. The F1-scores for the

A Study of Classiﬁcation of Texts into Categories of Cybersecurity Incident and Attack with Topic Models

643

Table 4: Attack categories and seed words.

Category Seed words

Others ﬁle, found, code, vulnerability, exploit, version, execution, remote,

round up article

Social Engineering suspicious, mail, vulnerable, cve, japan, attacker, evangelist, “blog au-

thor”, personal information, disclosure, “IPv4 address (internal)”

DoS ddos attack, ddos, applet, not found, ip address, backdoor, installation,

udp, tcp, dictionary attack, ﬂood, bot, miori, mirai, jump server

Vulnerability Exploit cve, vulnerability, rule, trick, arbitrarily, exploit, spotlight, strategy, advi-

sory, bug, release

Malware infection, malware, indicator, download, trojan, exe, feature, ioc,

trojan, powershell, ﬁrepower

Inject Unexpected Items electronic commerce, ticketmaster, magecart, dpi, security, fashion,

constructor, object, context

Employ Probablistic Techniques dictionary attack, pcastle, yowai, hakai, distributed, put, ssh, gafgyt,

damage, blog, page, ordinary, port, brute, polycom

Collect and Analyze Information layer, lack, exposure, port, udp, shodan, upnp, unit, home, recent, install,

search engine, ngips, disconnection, “public web server (internal)”

Subvert Access Control dynamic, “IPv4 address (external)”, aptdnsdns, intersecdns, authority,

Syria, cctld, rewrite, redirect, domain, electric company,

dnspionage, path, rewrite

Table 5: F1-scores and NMI in parenthesis for various thresholds and maximum numbers of inferred topics.

(threshold, maximum number of inferred topics)

Category Model (0,1) (0.01/K, 4) (0.1/K, 4) (1/K, 4)

Incident

LDA 0.514 (0.097) 0.694 (0.007) 0.66 (0.01) 0.551 (0.072)

Seeded LDA 0.563 (0.103) 0.68 (0.014) 0.659 (0.021) 0.595 (0.074)

Attack

LDA 0.536 (0.137) 0.667 (0.018) 0.657 (0.021) 0.601 (0.072)

Seeded LDA 0.597 (0.145) 0.761 (0.021) 0.761 (0.03) 0.654 (0.11)

words in Step 3 are slightly higher than those for the

initial words.

4.6 Test Texts with Trained Models

SLDA is an unsupervised algorithm and thus an in-

ferred topic cannot be identiﬁed with an actual cat-

egory. In contrast, the inferred topics obtained with

LLDA correspond to actual categories. By corre-

sponding inferred topics to ground truth categories,

we evaluated clustering in terms of precision, recall,

and F1-score for SLDA and LLDA. We computed mi-

cro precision, recall, and F1-score for the evaluation.

The text used for training and testing LLDA contained

the label common topic. We omitted this label when

extracting inferred topics.

Here, we show the results of classiﬁcation of the

incident reports using the model trained using blog

posts. For the incident reports and the blog posts, we

used 270 text for training the model and 135 text as

the test data. The results are shown in Table 7.

SLDA achieved higher precision, recall, and F1

values compared with those for LLDA for all cases.

Nevertheless the classifying accuracy of SLDA is not

at a practical level. We should therefore enhance the

SLDA model by improving seed words and other hy-

perparameters.

4.7 Clustering Analysis

Here, we brieﬂy describe the cluster analysis for the

SLDA topic model and effects by the seed words. In

this paper, we only show the correlations between the

clusters and the ground truth labels for the attack cate-

gories because of limitations of space. Figure 2 shows

the hierarchically-clustered heatmap with the matrix

for ground truth multi-labels of text and clustered text

with the SLDA model. We used Ward’s method and

the standard euclidean distance in hierarchical cluster

analysis.

The text that are tagged ground truth labels “Mal-

ware”, “Social Engineering”, or “Vulnerability Ex-

ploit” by the SLDA with the initial seed are classiﬁed

into similar cluster sets. Thus, it is hard to distinguish

ICISSP 2020 - 6th International Conference on Information Systems Security and Privacy

644

Table 6: Comparing results of F1-score and NMI in parenthesis for each set of seed words.

Seed words

Category Model Initial Step 1 Step 2 Step 3

Incident

LDA 0.551 (0.072)

Seeded LDA 0.595 (0.074) 0.579 (0.073) 0.588 (0.08) 0.644 (0.086)

Attack

LDA 0.601 (0.072)

Seeded LDA 0.654 (0.11) 0.653 (0.106) 0.629 (0.116) 0.673 (0.094)

Table 7: Evaluation of topic models using test text.

Incident categories Attack categories

Seed words Model precision recall F1-score precision recall F1-score

Initial

LLDA 0.175 0.089 0.118 0.115 0.065 0.083

SLDA 0.298 0.354 0.324 0.314 0.43 0.363

Step 3

LLDA 0.241 0.133 0.171 0.137 0.085 0.105

SLDA 0.312 0.411 0.355 0.361 0.46 0.404

clusters each other for such attack categories. For the

results using the SLDA model with the Step 3 seed

set, we can see that the clusters for the text in the cate-

gory “Vulnerability Exploit” are slightly distinguish-

able from those for other categories “Malware” and

“Social Engineering”.

Though the effects by using different seed sets are

not explicitly shown in the stochastic measurement

results for the models and cluster analysis, we remark

how we chose the seed words to be used. We added

the some words for the attack category “Social Engi-

neering” by observing frequently appeared words in

our incident reports with computing information gain

and heuristic knowledges. Furthermore, we could not

classify the text well when we remove the ambigu-

ous seed words which occurred in multiple categories

such as “cve” in the categories “Social Engineering”

and “Vulnerability Exploit”. In our experiment, the

clustering results are better when we set some seed

words for related multi-categories.

5 CONCLUSION AND FUTURE

WORK

We applied several LDA-based topic models to the

classiﬁcation of security text and extraction of intelli-

gence from the text. This work was motivated by an-

alyzing internal data (incident reports) with an SLDA

model trained with external data (e.g., security blog

posts). The results showed that SLDA overcomes

naive LDA in terms of overlapping clustering, and

SLDA classiﬁed the test text with higher accuracy

than LLDA did. Although the results of SLDA are

unsatisfactory to classify security text in a real-world

situation. The followings are future works to enhance

the SLDA model.

We should modify the stop words so that the

SLDA model can classify security text more distin-

guishably. There are many words that have common

meaning or concept all over the categories since the

text used in our experiments are based on cybersecu-

rity topics. We furthermore should consider how we

can deﬁne categories of appropriate size to classify

security text well. Although we used standard cate-

gories in this paper, we will need to consider subdivi-

sion of the categories and its hierarchical structure.

To enhance SLDA models, further experiments

for various seed sets are needed. In addition, we

should tune the hyperparameters of the Dirichlet dis-

tributions α and β and timing of updating these pa-

rameters in the sampling algorithms.

To make the models suitable for systems dedicated

to handling incidents continuously, the seed words

should be updated depending on the situation. Hu et

al. (Hu et al., 2014) proposed a framework that users

can use to iteratively reﬁne topics by observing words

inferred by LDA. It would be helpful to develop a

framework that allows users to interactively reﬁne the

model with seed words. The well reﬁned model and

the extracted intelligences should lead to a system to

handle cybersecurity incidents automatically.

ACKNOWLEDGEMENTS

This work was partially supported by JST CREST

Grant Number JPMJCR1783 and Suematsu prize of

A Study of Classiﬁcation of Texts into Categories of Cybersecurity Incident and Attack with Topic Models

645

(a) Initial seed words (b) Step 3 seed words

Figure 2: Hierarchically-clustered heatmap for the attack categories for the initial and step3 seed words.

Tokyo Tech, Japan.

REFERENCES

Andrade, R. and Torres, J. (2018). Enhancing intelli-

gence soc with big data tools. In 2018 IEEE 9th An-

nual Information Technology, Electronics and Mobile

Communication Conference (IEMCON), pages 1076–

1080.

Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). La-

tent dirichlet allocation. J. Mach. Learn. Res., 3:993–

1022.

Chambers, N., Fry, B., and McMasters, J. (2018). Detecting

denial-of-service attacks from social media text: Ap-

plying NLP to computer security. In Proceedings of

the 2018 Conference of the North American Chapter

of the Association for Computational Linguistics: Hu-

man Language Technologies, Volume 1 (Long Papers),

pages 1626–1635, New Orleans, Louisiana. Associa-

tion for Computational Linguistics.

Deliu, I., Leichter, C., and Franke, K. (2017). Extracting

cyber threat intelligence from hacker forums: Sup-

port vector machines versus convolutional neural net-

works. In 2017 IEEE International Conference on Big

Data (Big Data), pages 3648–3656.

Hu, Y., Boyd-Graber, J., Satinoff, B., and Smith, A.

(2014). Interactive topic modeling. Machine Learn-

ing, 95(3):423–469.

Jagarlamudi, J., Daum

e, III, H., and Udupa, R. (2012).

Incorporating lexical priors into topic models. In

Proceedings of the 13th Conference of the European

Chapter of the Association for Computational Lin-

guistics, EACL ’12, pages 204–213, Stroudsburg, PA,

USA. Association for Computational Linguistics.

Kudo, T., Yamamoto, K., and Matsumoto, Y. (2004). Ap-

plying conditional random ﬁelds to Japanese morpho-

logical analysis. In Proceedings of EMNLP 2004,

pages 230–237, Barcelona, Spain. Association for

Computational Linguistics.

Li, X., Li, C., Chi, J., Ouyang, J., and Li, C. (2018). Data-

less text classiﬁcation: A topic modeling approach

with document manifold. In Proceedings of the 27th

ACM International Conference on Information and

Knowledge Management, CIKM ’18, page 973–982,

New York, NY, USA. Association for Computing Ma-

chinery.

Loper, E. and Bird, S. (2002). Nltk: The natural lan-

guage toolkit. In Proceedings of the ACL-02 Work-

shop on Effective Tools and Methodologies for Teach-

ing Natural Language Processing and Computational

Linguistics - Volume 1, ETMTNLP ’02, pages 63–

70, Stroudsburg, PA, USA. Association for Compu-

tational Linguistics.

Lutov, A., Khayati, M., and Cudr

e-Mauroux, P. (2019). Ac-

curacy evaluation of overlapping and multi-resolution

clustering algorithms on large datasets. In 2019 IEEE

International Conference on Big Data and Smart

Computing (BigComp), pages 1–8.

Nagai, T., Inui, T., Takita, M., Furumoto, K., Shiraishi, Y.,

Takano, Y., Mohri, M., and Morii, M. (2018). Cluster-

ing security blog posts using guided-topic model for

threat analysis. In Proceedings of Computer Security

Symposium 2018, volume 2018, pages 481–488.

Ramage, D., Hall, D., Nallapati, R., and Manning, C. D.

(2009). Labeled lda: A supervised topic model for

credit attribution in multi-labeled corpora. In Proceed-

ings of the 2009 Conference on Empirical Methods in

Natural Language Processing: Volume 1 - Volume 1,

EMNLP ’09, pages 248–256, Stroudsburg, PA, USA.

Association for Computational Linguistics.

Singh, V. and amrrs (2017). Guidedlda: Guided topic mod-

eling with latent dirichlet allocation. https://github.

com/vi3k6i5/guidedlda, Accessed: 2019-11-15.

Zhong, C., Yen, J., Liu, P., and Erbacher, R. F. (2019).

Learning from experts’ experience: Toward auto-

mated cyber security data triage. IEEE Systems Jour-

nal, 13(1):603–614.

ICISSP 2020 - 6th International Conference on Information Systems Security and Privacy

646