Comparative Analysis of Topic Modelling Approaches on Student

Feedback

Faiz Hayat

1 a

, Safwan Shatnawi

2 b

and Ella Haig

1 c

School of Computing, University of Portsmouth, Buckingham Building, Lion Terrace, Portsmouth PO1 3HE, U.K.

Education Practice, International Research and Exchange Board, Amman, Jordan

Keywords:

Topic Modelling, BERT, LDA, LSA, NMF, Education.

Abstract:

Topic modelling, a type of clustering for textual data, is a popular method to extract themes from text. Methods

such as Latent Dirichlet Allocation (LDA), Latent Semantic Analysis (LSA) and Non-negative Matrix Factor-

ization (NMF) have been successfully used across a wide range of applications. Large Language Models, such

as BERT, have led to signiﬁcant improvements in machine learning tasks for textual data in general, as well

as topic modelling, in particular. In this paper, we compare the performance of a BERT-based topic modelling

approach with LDA, LSA and NMF on textual feedback from students about their mental health and remote

learning experience during the COVID-19 pandemic. While all methods lead to coherent and distinct topics,

the BERT-based approach and NMF are able to identify more ﬁne-grained topics. Moreover, while NMF re-

sulted in more detailed topics about the students’ mental health-related experiences, the BERT-based approach

produced more detailed topics about the students’ experiences with remote learning.

1 INTRODUCTION

Machine learning tasks are typically divided into

supervised and unsupervised learning (Berry et al.,

2019). For textual data, one of the most used un-

supervised methods is topic modelling, which is a

type of clustering that extracts topics or themes from

text (Zhao et al., 2021).

Three of the most popular methods for topic mod-

elling are Latent Dirichlet Allocation (LDA) (Blei

et al., 2003), Latent Semantic Analysis (LSA) (Deer-

wester et al., 1990) and Non-negative Matrix Fac-

torization (NMF) (Lee and Seung, 1999). Since

the arrival of Large Language Models (LLMs) in

2017 (Vaswani et al., 2017), pre-trained deep learn-

ing models like BERT (Bidirectional Encoder Rep-

resentations from Transformers) (Devlin et al., 2019)

have shown impressive results for unsupervised learn-

ing across many applications (e.g., (Abuzayed and Al-

Khalifa, 2021; Egger and Yu, 2022; Shariﬁan-Attar

et al., 2022)). Compared with other topic modelling

approaches, BERT-based models have the following

two key advantages: (1) because they were trained

https://orcid.org/0000-0003-0249-4617

https://orcid.org/0000-0002-5063-1295

https://orcid.org/0000-0002-5617-1779

on large amounts of data, they have the capacity to

encode complex semantic relationships, and (2) the

ability to capture both left and right contexts, which

accounts for the term “bidirectional” (Devlin et al.,

2019).

In this paper, we compare a BERT-based topic

modeling approach with LDA, LSA, and NMF to

identify relevant topics from student feedback on their

COVID-19 pandemic experience, focusing on mental

health and remote learning.

The main contribution of the paper is a compar-

ative analysis of topic modeling using LLMs like

BERT against traditional methods. Few studies have

explored this comparison, leaving the superiority of

newer approaches uncertain. We investigate whether

BERT provides an advantage over traditional meth-

ods in analyzing student feedback on pandemic expe-

riences. Our study compares topics identiﬁed using

BERT-based modeling with NMF, LDA, and LSA.

The rest of the paper is structured as follows: Sec-

tion 2 reviews background and related work, Section 3

details the experimental setup, Section 4 presents the

results, Section 5 compares methods and discusses

ﬁndings and Section 6 concludes with future research

directions.

226

Hayat, F., Shatnawi, S. and Haig, E.

Comparative Analysis of Topic Modelling Approaches on Student Feedback.

DOI: 10.5220/0012890400003838

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2024) - Volume 1: KDIR, pages 226-233

ISBN: 978-989-758-716-0; ISSN: 2184-3228

2 RELATED WORK

This section provides an overview of LDA, LSA, and

NMF, reviews research on student feedback, and dis-

cusses evaluation approaches for topic modeling.

LDA is a probabilistic generative model widely

used for topic modeling in natural language process-

ing (NLP) (Blei et al., 2003). It assumes that each

document in a corpus is a mixture of topics, and each

topic is a distribution over words. LDA aims to un-

cover latent topics from a collection of documents by

iteratively assigning words to topics and adjusting as-

signments to maximize the likelihood of the data.

LSA (Deerwester et al., 1990) is a technique used

for dimensionality reduction and semantic analysis of

textual data. It employs Singular Value Decomposi-

tion (SVD) to identify latent semantic structure in a

corpus by capturing relationships between terms and

documents, representing them in a lower-dimensional

space for easier detection of semantic similarities.

NMF (Lee and Seung, 1999) is a dimensionality

reduction technique widely used in natural language

processing (NLP) and other ﬁelds. It decomposes a

non-negative matrix into two lower-dimensional ma-

trices, representing topics and document-topic distri-

butions. NMF is applied to tasks such as topic mod-

eling, document clustering, and feature extraction.

BERT (Grootendorst, 2022) is a pre-trained deep

learning model developed by Google for natural lan-

guage processing tasks. It excels in capturing con-

textual information bidirectionally, enabling it to un-

derstand the meaning of words in context more effec-

tively than previous models. BERT has revolutionized

NLP tasks by leveraging large-scale pre-training on

vast text data and ﬁne-tuning for speciﬁc downstream

tasks, e.g., (Ding et al., 2023; Malladi et al., 2023).

Topic modelling has been used to analyse student

feedback in many studies, e.g., (Buenano-Fernandez

et al., 2020; Hujala et al., 2020; Sun and Yan,

2023). There have also been several studies inves-

tigating student experiences during the COVID-19

pandemic (e.g., (Oliveira et al., 2021; Stevanovi

et al., 2021; Waheeb et al., 2022). Many studies also

employ BERT-based models with educational-related

data (e.g., (Bai and Stede, 2023; Cochran et al., 2023;

Sung et al., 2019)). However, to our knowledge, only

one study has used BERT-based topic modeling on

student feedback (Masala et al., 2021), and none have

focused on students’ COVID-19 experiences through

open-text responses.

We are aware that other studies (e.g., (M

uller

et al., 2023; Wang et al., 2020; Xu et al., 2022))

have used BERT-based topic modeling to examine

COVID-19 experiences in the general population, but

they focus on social media data, not student responses

from open-ended questionnaires. Therefore, these

studies are not directly relevant to our research.

The one study we found using a BERT-based topic

modeling technique (Masala et al., 2021) concen-

trated on examining student textual feedback at the

course level. The researchers developed a tool that an-

alyzed large volumes of student feedback, producing

clusters of similar contexts and recurring keywords

for each course. The processing pipeline involved ex-

tracting general evaluations, restoring diacritics using

RoBERT (a Romanian BERT model), and performing

keyword extraction with KeyBERT (ﬁne-tuned for the

Romanian language). To capture the context around

these keywords, they utilized two methods: extract-

ing sentences containing the keywords and using de-

pendency tree traversal to gather related context. The

extracted contexts were then grouped using K-Means

clustering applied to BERT-generated embeddings.

In contrast, our study applies multiple topic mod-

eling techniques to analyze survey responses related

to mental health and remote learning during the

COVID-19 pandemic. We explore several algorithms,

including BERT-based embeddings, Latent Dirichlet

Allocation (LDA), Latent Semantic Analysis (LSA),

and Non-negative Matrix Factorization (NMF). Addi-

tionally, our study incorporates dimensionality reduc-

tion using UMAP and clustering with HDBSCAN to

discover underlying topics in the survey data.

We now turn our attention to the evaluation of

topic modeling techniques. In supervised learning,

techniques are evaluated by comparing the predic-

tions against a known ground truth, but in unsuper-

vised learning, such ground truth is often absent, mak-

ing evaluation challenging without human judgment.

Although various metrics are used to evaluate topic

modeling and clustering methods, performance can

vary widely across techniques and data types (Doogan

and Buntine, 2021; Harrando et al., 2021), and the va-

lidity of fully automated evaluations without human

judgment has been questioned (Hoyle et al., 2021).

Some clustering/topic modelling techniques re-

quire as input the number of clusters/topics, while for

others, the ‘optimal’ number emerges from the data.

For the former, metrics like coherence scores (Abdel-

razek et al., 2023; O’Callaghan et al., 2015), can help

determine the optimal number of topics, but these also

need human judgment (Doogan and Buntine, 2021).

In our research, we combined coherence scores with

human evaluation.

While the usefulness of BERT-based approaches

for topic modelling has been shown for different types

of education-related data, there has only been one

study using a BERT-based approach on student feed-

Comparative Analysis of Topic Modelling Approaches on Student Feedback

227

back from open-ended questions and this study did

not include a comparison with other topic modelling

approaches. Our study contributes to a better under-

standing of the usefulness of BERT by providing the

ﬁrst comparative study for this type of data.

3 EXPERIMENTAL SETUP

In this section, we describe the data collection and

preprocessing, as well as the process for topic mod-

elling for each of the four investigated approaches.

3.1 Data Collection

Data collection for this research involved conduct-

ing a survey among students at a UK university in

2022. The aim was to assess the inﬂuence of the

COVID-19 pandemic on students. The questionnaire

included four open-ended prompts designed to clar-

ify the particular difﬁculties students encountered re-

garding their mental well-being and remote learning

during the pandemic: ‘What challenges or issues re-

garding mental health did you face during the pan-

demic? What aspects, if any, did you struggle with?’;

‘Please share any other comments/ opinions/ solu-

tions about your mental health during the pandemic.’;

‘What challenges or issues regarding remote learn-

ing did you face during the pandemic? What as-

pects, if any, did you struggle with?’ and ‘Please

share any other comments/ opinions/ solutions about

remote learning during the pandemic.’

Ethical approval was obtained from the univer-

sity’s Ethics Committee before distribution. The sur-

vey was distributed using email lists speciﬁc to each

faculty, reaching out to a diverse group of students

from different academic disciplines such as social sci-

ences, humanities, business and law, and technology.

The involvement in the survey was voluntary and re-

spondents remained anonymous.

Responses from 340 participants included 696

submissions from the open-ended questions: 375 on

mental health and 321 on remote learning. The sam-

ple size for topic modeling consisted of all 696 textual

responses. We made this decision due to the preva-

lence of short answers and many students responding

selectively to some questions and not others.

3.2 Data Preprocessing

Data preprocessing was conducted to prepare the

textual data for analysis. Speciﬁcally, this pro-

cess included the elimination of stop words, such as

“the,” “is,” and “and”, which are common words that

provide little value in understanding the underlying

themes of the text. Special characters and numbers

not contributing to semantic analysis were also ﬁl-

tered out to reﬁne the dataset and improve the quality

of information fed into the topic modeling algorithm.

3.3 Topic Modeling Algorithms

Four topic modeling algorithms were utilized: a

BERT approach described below, Latent Dirichlet Al-

location (LDA), Latent Semantic Analysis (LSA) and

Non-negative Matrix Factorization (NMF). The im-

plementation was carried out using Google Colab, a

cloud-based environment integrated with Python.

The BERT-Based Topic Modelling Approach. The

following steps were applied: 1) Obtaining document

embeddings by utilizing the ‘paraphrase-MiniLM-

L6-v2’ pre-trained model; 2) UMAP (Uniform Mani-

fold Approximation and Projection) (McInnes et al.,

2018) was used to reduce the dimensionality of

the embeddings, improving visualization and clus-

tering; 3) Performing clustering using the Hierarchi-

cal Density-Based Spatial Clustering of Applications

with Noise (HDBSCAN) (McInnes et al., 2017) algo-

rithm to generate the topics; 4) Using visualizations

to analyze the 10 most frequent words per topic, re-

sponse distribution across topics, and the dendrogram

from the clustering algorithm; 5) Conducting qual-

itative analysis to validate topics by examining re-

sponses assigned to each topic.

Latent Dirichlet Allocation (LDA). The following

steps were applied: 1) Using the Gensim library, a

dictionary and a document-term matrix were created

to represent the term frequency; 2) Applied LDA to

the document-term matrix to infer the underlying top-

ics and their word distributions; 3) Analyzing the re-

sulting topics by examining the most probable words

associated with each topic; 4) Conducting qualita-

tive analysis to validate the topics by reviewing docu-

ments assigned to each topic.

LSA (Latent Semantic Analysis). The following

steps were applied to derive the topics using LSA:

1) Creating a term-document matrix representing the

frequency of terms in documents. 2) Applying SVD

to the term-document matrix to decompose it into

three matrices: a term-concept matrix, a diagonal ma-

trix of singular values, and a concept-document ma-

trix. 3) Analyzing the resulting concept vectors to

identify latent semantic topics. 4) Conducting quali-

tative analysis to validate the topics by reviewing doc-

uments associated with each concept.

NMF (Non-Negative Matrix Factorization). The

following steps were applied: 1) Vectorizing the

preprocessed text data into a term-document matrix,

KDIR 2024 - 16th International Conference on Knowledge Discovery and Information Retrieval

228

where each row represents a document and each col-

umn represents a term. 2) Applying NMF to factor-

ize the term-document matrix into two matrices rep-

resenting topics and document-topic distributions. 3)

Analyzing the resulting topics by examining the most

prominent terms associated with each topic. 4) Con-

ducting qualitative analysis to validate the topics by

reviewing documents assigned to each topic.

4 RESULTS

As the BERT-based approach uses the HDBSCAN al-

gorithm, the optimal number of topics emerges from

the data; in our case, this was 13. LDA, LSA and

NMF require a number of topics as an input. For these

methods, to identify the optimal number of topics,

as mentioned in Section 2, we chose the coherence

score (Abdelrazek et al., 2023), which aggregates the

coherence of each topic, measured as the semantic

similarity between top words in the topic, in combi-

nation with human judgment. The highest coherence

scores were obtained for 13 topics with LDA, 12 top-

ics with LSA, and 16 topics with NMF, and our quali-

tative evaluation showed that for each method the top-

ics were relevant and distinct from each other.

We conducted a deeper qualitative assessment of

topics from all four algorithms and found the BERT-

based approach and NMF yielded the most interest-

ing results. Due to space constraints, we present de-

tailed results for these methods and summarize LDA

and LSA results for comparison in the next section.

The topics that resulted from the BERT-based ap-

proach are presented in Table 1. We grouped the top-

ics into themes, analyzed in the following paragraphs.

As anticipated, we see that the subjects are ar-

ranged in relation to the two elements—mental health

and remote learning—that were highlighted in the

open-ended questions. Out of the thirteen topics,

three (0 and 4-5) are related to mental health, two are

related to both (1 and 6) and eight topics (2-3 and 7-

12) are related to distant learning.

The application of the BERT-based modeling ap-

proach to mental health allows differentiation be-

tween several aspects, including anxiety (Topic 0), so-

cial isolation and loneliness (Topic 4), and the generic

impact of the epidemic on mental health (Topic 5).

It is interesting to note that a more comprehen-

sive picture of remote learning emerges, covering a

wide range of topics, from the more general ones

like the university experience in general (Topic 3) and

the impact of the pandemic on the university experi-

ence (Topic 8), to the more specialised ones like con-

centration problems (Topic 2), internet connectivity

(Topic 7), virtual communication (Topic 9), lecture

formats (Topic 10), the experience of remote learning

across various modules and courses (Topic 11), and

the value of in-person communication (Topic 12).

Aspects of both remote learning and mental health

are included in Topics 1 and 6. In Topic 1, motivation

is discussed as a practical requirement for participat-

ing in remote learning, as well as a crucial component

of mental health. The only positive topic is Topic 6,

which describes the methods respondents use to pre-

serve their mental health and academic motivation.

Table 1 displayed the number of textual instances

per topic in the second column, with a relatively large

variation. Topic 12 (face-to-face communication) has

the fewest instances (14), while Topic 5 (the pan-

demic’s effects on mental health) has the highest (88).

There are parallels between Topics 5 and 8, which

discuss how the pandemic has affected mental health

(Topic 5) and remote learning (Topic 8), respectively.

Topics 10 and 11 share commonalities as well, as they

both deal with challenges related to remote learning.

The variations between the two topics highlight ex-

periences related to lectures and teaching sessions in

Topic 10 and broader experiences related to remote

learning at the module or course level in Topic 11.

As mentioned in Section 3.1, responses to men-

tal health (375) outnumbered those to distant learning

(321). The fact that there are three topics about mental

health and eight about remote learning suggests that

while there are more different experiences with re-

mote learning, there is a greater homogeneity of expe-

riences with mental health. This further demonstrates

the capacity of the BERT-based approach to discern

between elements with subtle variations.

Table 2 presents the topics resulting from applying

NMF. Similar to the BERT-based approach, the topics

cover mental health, remote learning, or both aspects.

In terms of mental health, speciﬁc issues such as

anxiety, eating disorders and depression are covered

in Topic 1, dealing with uncertainty in Topic 3, and

the generic impact of the pandemic on mental health

in Topic 8. Topic 10 is also more generic, covering

emotional well-being aspects, while Topic 11 is more

speciﬁcally about social isolation challenges.

The topics covering remote learning aspects vary

from more generic, about distance learning and the

use of online tools (Topics 2, 6, and 9), to more spe-

ciﬁc issues such as motivation to study (Topic 12) and

difﬁculties in grasping learning content (Topic 13).

Several topics cover both mental health and re-

mote learning aspects: time management (Topic 0),

motivational issues (Topics 4 and 15), lack of social

interaction in remote learning (Topic 5), the impact

of the pandemic on physical health (Topic 7), and the

Comparative Analysis of Topic Modelling Approaches on Student Feedback

229

Table 1: Topics extracted with the BERT-based approach; (Docs refers to the number of documents/responses for each topic).

No. Docs Topic Name Topic Description Keywords

0 34 Anxiety and

Depression

Increased anxiety and depression disorders,

leading to heightened awareness and impacts

on mental health.

anxiety, depression, disorders, in-

creased, eating, panic, depressed, still,

health, aware

1 34 Motivation Struggle to maintain motivation, resulting in

challenges in staying focused and productive.

motivation, motivated, stay, keep, stay-

ing, lack, work, hard, struggle

2 26 Easily Dis-

tracted

Experience difﬁculty concentrating due to var-

ious distractions, affecting productivity and fo-

cus.

distracted, easily, concentrate, focused,

couldnt, distractions, attention, skip,

work, focus

3 19 University Ex-

perience

Mixed experiences during university, including

success, failure, and uncertainty

university, year, felt, experience, well,

think, failed, cheated, second, uni

4 74 Loneliness and

Friendship

Loneliness and lack of social contact affecting

mental well-being and interaction.

friends, loneliness, lonely, social, see,

depression, lack, contact, isolation, able

5 88 Pandemic and

Mental Health

Heightened awareness of mental health issues

during the pandemic, affecting individuals and

communities globally.

pandemic, health, mental, people, so-

cial, covid, anxiety, family, made, mea-

sures

6 19 Daily Routine Recognizing the value of a consistent, positive

daily routine for better mental health.

daily, good, routine, mental, health,

work, home, day, weekly, sleep

7 16 Internet Con-

nection Issues

Frustration and challenges from unreliable in-

ternet, affecting academic and personal tasks.

internet, connection, bad, unreliable, is-

sues, lesson, poor, exams, could, found

8 27 Remote Learn-

ing

Adapting to remote learning challenges, in-

cluding online lectures and assignments.

pandemic, remote, learning, working,

lectures, really, time, away, home, im

9 19 Zoom Calls Adjusting to the challenges and discomfort as-

sociated with online video calls, especially in

educational and professional settings.

zoom, camera, calls, people, would,

anyone, lessons, interacting, comfort-

able, answer

10 50 Online Lec-

tures

Facing challenges with online lectures, includ-

ing slower learning and engagement issues.

lectures, lecturers, questions, online,

without, lecture, students, felt, slower

11 52 Remote Learn-

ing Experience

Reﬂecting on remote learning, its beneﬁts, and

drawbacks compared to traditional methods.

learning, remote, time, modules, learn,

teachers, students, like, lectures, course

12 14 Face-to-Face

Learning

Emphasizing the importance of face-to-face in-

teraction in learning environments for effective

communication and understanding.

face, union, learning, lower, guidance,

lecturers, communication, important,

facetoface, seeing

need for support during studies (Topic 14).

The distribution of responses per topic, unlike

BERT-based approaches, NMF has a more balanced

range, with the smallest topic having 25 responses

(Topic 3) and the largest 60 (Topic 14). Eleven of

the sixteen topics have between 40 and 50 responses.

5 COMPARISON AND

DISCUSSION

To compare the four algorithms, we selected four

themes that cover all the topics produced across all

four solutions: remote learning and challenges, men-

tal health and challenges, social issues and loneliness,

and motivation and physical health. The topic distri-

bution by theme is shown in Table 3, and Fig. 1 illus-

trates the theme proportions for each algorithm.

The BERT-based approach allocates the highest

percentage (61.54%) of its thematic content to Re-

mote Learning and Challenges, indicating its strong

emphasis on analyzing issues related to remote edu-

cation. Conversely, it allocates smaller proportions

Figure 1: Comparitive Analysis based on Themes.

to Mental Health and Challenges (23.08%), Social

Issues and Loneliness (7.69%), and Motivation and

Physical Health Issues (7.69%), suggesting a rela-

tively narrower focus on these domains.

LDA (Latent Dirichlet Allocation) has the high-

est percentage to Remote Learning and Challenges

(30.77%), with smaller proportions for Mental Health

and Challenges, Social Issues and Loneliness, and

Motivation and Physical Health, each at (23.08%).

KDIR 2024 - 16th International Conference on Knowledge Discovery and Information Retrieval

230

Table 2: Topics extracted using NMF.

No. Docs Topic Name Topic Description Keywords

0 59 Time Manage-

ment Struggles

Focuses on time management challenges

worsened by pandemic-related work-life dis-

ruptions

struggled, focus, working, time,

work, helped, day, home, pandemic,

lot

1 30 Mental Health

Challenges

Addresses severe mental health issues, includ-

ing heightened levels of depression and anxi-

ety due to societal pressures.

severe, reached, leaving, eating, so-

ciety, disorder, house, increased, de-

pression, anxiety

2 50 Remote Learn-

ing Preferences

Highlights difﬁculties adapting to remote

learning and a preference for traditional face-

to-face interactions.

difﬁculty, pace, better, lecture, pre-

ferred, tutor, prefer, remote, face,

learning

3 25 Coping with

Uncertainty

Discusses struggles with coping mechanisms

during times of uncertainty, leading to feelings

of loneliness and boredom.

uncertainty, email, change, coping,

boring, struggling, extremely, covid,

help, loneliness

4 49 Motivation

Struggles

Focuses on maintaining motivation for com-

pleting coursework, with challenges in main-

taining consistent effort.

module, getting, far, complete,

week, whilst, went, struggled, mo-

tivation, work

5 40 Lack of Social

Interaction

Explores the absence of social interactions in

learning environments, leading to feelings of

disconnection.

teacher, medium, unable, aspect,

make, seeing, talking, people, inter-

action, social

6 40 Challenges

with Distance

Learning

Addresses difﬁculties in maintaining engage-

ment and interaction in distance learning set-

tings.

contact, distance, interaction, long,

learning, student, issue, lecture, on-

line, lack

7 45 Impact on

Physical Health

Examines how disrupted routines and less ex-

ercise affected health during the pandemic.

low, exercise, issue, daily, pan-

demic, struggle, routine, good, men-

tal, health

8 43 Impact on

Mental Health

Examines worsening mental health from iso-

lation, academic stress, and future uncertainty.

worse, life, depression, parent, job,

caused, worried, stress, isolation,

feel

9 45 Online Learn-

ing Experience

Evaluate online learning tools like Zoom,

highlighting effectiveness and engagement is-

sues.

use, useful, attention, session, zoom,

know, people, lecture, online, class

10 50 Emotional

Well-being

Addresses emotional challenges during uni-

versity, such as stress, depression, and lone-

liness.

quite, stressed, teaching, university,

feel, year, depressed, lonely, like,

felt

11 40 Social Isolation

Challenges

Explores challenges in maintaining social

connections with family and friends due to

prolonged social isolation.

future, kept, knowing, member, so-

cialise, difﬁcult, person, able, fam-

ily, friend

12 35 Study Motiva-

tion

Discusses maintaining study motivation and

focus amid distractions and coursework de-

mands.

skill, studying, lesson, money,

course, focused, stay, staying,

motivated, hard

13 45 Understanding

Course Mate-

rial

Explores difﬁculties in grasping course mate-

rial, especially under the distractions and pres-

sures of lockdowns.

happened, understanding, lock-

down, grade, thing, losing, under-

stand, study, assignment, time

14 60 Academic and

Financial Chal-

lenges

Addresses challenges in academics and ﬁnan-

cial stability, highlighting the need for institu-

tional and peer support.

needed, course, poor, socialising,

people, ﬁnancial, harder, really, lec-

turer, support

15 40 Exam Prepara-

tion Challenges

Discusses challenges in preparing for exams

due to distractions and unreliable internet con-

nections.

exam, concentrate, difﬁcult, bad, es-

pecially, learn, distracted, connec-

tion, easily, internet

In contrast, LSA (Latent Semantic Analysis) pro-

duces a unique thematic distribution in comparison

with the other methods. It assigns a substantially

higher percentage (33.34%) to Physical Health Issues

and Motivation, indicating a strong emphasis on these

two areas. There is no difference between Social

issues and Loneliness (25%) and Remote Learning

and Challenges(25%). It does, however, give Mental

Health and Challenges a lower percentage (16.67%).

NMF allocated the highest percentage (31.25%) to

Remote Learning and Challenges, equal percentages

to Mental Health and Challenges, and Motivation and

Physical Health (23.08%), and the lowest percentage

(18.75%) to Social Issues and Loneliness.

Overall, NMF and LDA have the most balanced

distributions across the four themes and can capture

Comparative Analysis of Topic Modelling Approaches on Student Feedback

231

Table 3: Algorithmic Topics Distribution.

Algorithm Remote Learning and Chal-

lenges

Mental Health and

Challenges

Social Issues and Lone-

liness

Motivation and

Physical Health

BERT Topic 2, 3, 7, 8, 9, 10, 11, 12 Topic 0, 5, 6 Topic 4 Topic 1

LDA Topic 2, 8, 9, 12 Topic 0, 4, 6 Topic 3, 5, 10 Topic 1, 7, 11

LSA Topic 3, 5, 7 Topic 0, 8 Topic 1, 2, 9 Topic 4, 6, 10, 11

NMF Topic 0, 2, 6, 9, 13 Topic 1, 3, 8, 10 Topic 5, 11, 14 Topic 4, 7, 12, 15

at a good level of detail several distinct aspects. The

BERT-based approach, on the other hand, has a more

unbalanced distribution across the four themes but

can capture more ﬁne-grained issues related to remote

learning. In particular, three topics identiﬁed by the

BERT-based approach were not identiﬁed as separate

topics by any of the other algorithms: internet connec-

tion issues (Topic 7), online calls (Topic 9), and face-

to-face learning (Topic 12). By volume of responses,

these are also among the smallest topics, with 16, 19

and 14 responses, respectively. From this point of

view, the BERT-based approach may be better when

a more ﬁne-grained picture would be of interest.

For all approaches, we applied data preprocessing,

as outlined in Section 3.2. There is very little empir-

ical evidence concerning the use of textual data pre-

processing when pre-trained LLMs are used. We ap-

plied the BERT-based approach with no preprocess-

ing as well as the preprocessing mentioned in Sec-

tion 3.2 and found more coherent results when us-

ing preprocessing, hence, we reported the results with

preprocessing. This aligns with the view that prepro-

cessing should still be considered for LLMs expressed

in a recent review of text preprocessing (Chai, 2023).

6 CONCLUSION

This paper presents a comparative study using four

topic modeling methods: BERT, LDA, LSA, and

NMF, on student feedback in textual format about the

mental health and remote learning students’ experi-

ences during the COVID-19 pandemic.

This study sought to determine the effectiveness

of the BERT topic model compared to traditional ap-

proaches like NMF, LDA, and LSA. The results indi-

cated that BERT provided deeper insights into remote

learning challenges during the pandemic. While tra-

ditional methods produced similar results in mental

health, social issues, isolation, and motivation, BERT

showed clear advantages in topic understanding.

Our study found that all methods produced coher-

ent topics covering various aspects, but BERT and

NMF generated more interesting topics than LDA

and LSA. NMF had a balanced response distribution,

while BERT exhibited signiﬁcant variation.

The two primary limitations of our study are the

sample size and the post-epidemic data collection,

which may have inﬂuenced students’ recollections.

We gathered 696 textual instances from 340 partici-

pants. Despite this small sample, all algorithms pro-

duced coherent topics.

Among the four algorithms, the BERT-based ap-

proach was least affected by the small sample size

due to its extensive pre-training, which may explain

its ability to capture more nuanced topics. Our re-

search highlights the potential of BERT-based topic

modeling for educational data. In the future we will

explore alternative BERT models, like DeBERTa (He

et al., 2020), known for its effectiveness in textual

emotion recognition (Boitel et al., 2023), to capture

more emotionally nuanced experiences.

REFERENCES

Abdelrazek, A., Eid, Y., Gawish, E., Medhat, W., and Has-

san, A. (2023). Topic modeling algorithms and appli-

cations: A survey. Information Systems, 112:102131.

Abuzayed, A. and Al-Khalifa, H. (2021). BERT for arabic

topic modeling: An experimental study on BERTopic

technique. Procedia computer science, 189:191–194.

Bai, X. and Stede, M. (2023). A survey of current machine

learning approaches to student free-text evaluation for

intelligent tutoring. International Journal of Artiﬁcial

Intelligence in Education, 33(4):992–1030.

Berry, M. W., Mohamed, A., and Yap, B. W. (2019). Su-

pervised and unsupervised learning for data science.

Springer.

Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent

dirichlet allocation. Journal of Machine Learning Re-

search, 3(4-5):993–1022.

Boitel, E., Mohasseb, A., and Haig, E. (2023). A compar-

ative analysis of GPT-3 and BERT models for text-

based emotion recognition: performance, efﬁciency,

and robustness. In UK Workshop on Computational

Intelligence, pages 567–579. Springer.

Buenano-Fernandez, D., Gonzalez, M., Gil, D., and Luj

an-

Mora, S. (2020). Text mining of open-ended questions

in self-assessment of university teachers: An LDA

topic modeling approach. IEEE Access, 8:35318–

35330.

Chai, C. P. (2023). Comparison of text preprocessing meth-

ods. Natural Language Engineering, 29(3):509–553.

Cochran, K., Cohn, C., Hastings, P., Tomuro, N., and

Hughes, S. (2023). Using BERT to identify causal

KDIR 2024 - 16th International Conference on Knowledge Discovery and Information Retrieval

232

structure in students’ scientiﬁc explanations. Artiﬁcial

Intelligence in Education, pages 1–39.

Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer,

T. K., and Harshman, R. (1990). Indexing by latent

semantic analysis. Journal of the American Society

for Information Science, 41(6):391–407.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.

(2019). BERT: Pre-training of deep bidirectional

transformers for language understanding. In The Con-

ference of the North American Chapter of the Associa-

tion for Computational Linguistics: Human Language

Technologies, page 4171–4186.

Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y.,

Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al.

(2023). Parameter-efﬁcient ﬁne-tuning of large-scale

pre-trained language models. Nature Machine Intelli-

gence, 5(3):220–235.

Doogan, C. and Buntine, W. (2021). Topic Model or Topic

Twaddle? Re-evaluating Semantic Interpretability

Measures. In North American Association for Com-

putational Linguistics 2021, pages 3824–3848.

Egger, R. and Yu, J. (2022). A topic modelling compar-

ison between LDA, NMF, Top2Vec, and BERTopic

to demystify twitter posts. Frontiers in Sociology,

7:886498.

Grootendorst, M. (2022). BERTopic: Neural topic mod-

eling with a class-based TF-IDF procedure. arXiv

preprint arXiv:2203.05794.

Harrando, I., Lisena, P., and Troncy, R. (2021). Apples to

apples: A systematic evaluation of topic models. In

The International Conference on Recent Advances in

Natural Language Processing, pages 483–493.

He, P., Liu, X., Gao, J., and Chen, W. (2020). DeBERTa:

Decoding-enhanced BERT with disentangled atten-

tion. arXiv preprint arXiv:2006.03654.

Hoyle, A., Goel, P., Hian-Cheong, A., Peskov, D., Boyd-

Graber, J., and Resnik, P. (2021). Is Automated Topic

Model Evaluation Broken?: The Incoherence of Co-

herence. Advances in neural information processing

systems, 34:2018–2033.

Hujala, M., Knutas, A., Hynninen, T., and Arminen, H.

(2020). Improving the quality of teaching by utilis-

ing written student feedback: A streamlined process.

Computers & Education, 157:103965.

Lee, D. D. and Seung, H. S. (1999). Learning the parts of

objects by non-negative matrix factorization. Nature,

401(6755):788–791.

Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D.,

Chen, D., and Arora, S. (2023). Fine-tuning language

models with just forward passes. In Advances in Neu-

ral Information Processing Systems, volume 36, pages

53038–53075.

Masala, M., Ruseti, S., Dascalu, M., and Dobre, C. (2021).

Extracting and clustering main ideas from student

feedback using language models. In International

Conference on Artiﬁcial Intelligence in Education,

pages 282–292. Springer.

McInnes, L., Healy, J., Astels, S., et al. (2017). hdb-

scan: Hierarchical density based clustering. Journal

of Open Source Software, 2(11):205.

McInnes, L., Healy, J., and Melville, J. (2018).

UMAP: Uniform manifold approximation and pro-

jection for dimension reduction. arXiv preprint

arXiv:1802.03426.

uller, M., Salath

e, M., and Kummervold, P. E. (2023).

COVID-Twitter-BERT: A natural language process-

ing model to analyse COVID-19 content on Twitter.

Frontiers in artiﬁcial intelligence, 6:1023281.

Oliveira, G., Grenha Teixeira, J., Torres, A., and Morais,

C. (2021). An exploratory study on the emer-

gency remote education experience of higher educa-

tion students and teachers during the COVID-19 pan-

demic. British Journal of Educational Technology,

52(4):1357–1376.

O’Callaghan, D., Greene, D., Carthy, J., and Cunningham,

P. (2015). An analysis of the coherence of descriptors

in topic modeling. Expert Systems with Applications,

42(13):5645–5657.

Shariﬁan-Attar, V., De, S., Jabbari, S., Li, J., Moss, H.,

and Johnson, J. (2022). Analysing longitudinal social

science questionnaires: Topic modelling with BERT-

based embeddings. In 2022 IEEE International Con-

ference on Big Data (Big Data), pages 5558–5567.

Stevanovi

c, A., Bo

c, R., and Radovi

c, S. (2021). Higher

education students’ experiences and opinion about

distance learning during the Covid-19 pandemic.

Journal of Computer Assisted Learning, 37(6):1682–

1693.

Sun, J. and Yan, L. (2023). Using topic modeling to un-

derstand comments in student evaluations of teaching.

Discover Education, 2:1–12.

Sung, C., Dhamecha, T. I., and Mukhi, N. (2019). Improv-

ing short answer grading using transformer-based pre-

training. In International Conference on Artiﬁcial In-

telligence in Education, pages 469–481.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,

L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.

(2017). Attention is all you need. Advances in neural

information processing systems, 30.

Waheeb, S. A., Khan, N. A., and Shang, X. (2022). Topic

modelling and sentiment analysis of online educa-

tion in the COVID-19 era using social networks-based

datasets. Electronics, 11(5):715.

Wang, T., Lu, K., Chow, K. P., and Zhu, Q. (2020).

COVID-19 sensing: negative sentiment analysis on

social media in china via BERT model. IEEE Access,

8:138162–138169.

Xu, W. W., Tshimula, J. M., Dub

E., Graham, J. E.,

Greyson, D., MacDonald, N. E., and Meyer, S. B.

(2022). Unmasking the Twitter discourses on masks

during the COVID-19 pandemic: User cluster–based

BERT topic modeling approach. JMIR Infodemiology,

2(2):e41198.

Zhao, H., Phung, D., Huynh, V., Jin, Y., Du, L., and

Buntine, W. (2021). Topic modelling meets deep

neural networks: a survey. In The AAAI Inter-

national Joint Conference on Artiﬁcial Intelligence

2021, pages 4713–4720.

Comparative Analysis of Topic Modelling Approaches on Student Feedback

233