Investigating Collaborative Problem Solving Temporal Dynamics Using

Interactions Within a Digital Whiteboard

Hua Leong Fwa

School of Computing and Information Systems, Singapore Management University My, 80 Stamford Road, Singapore

Keywords:

Collaborative Problem Solving, Unsupervised, Machine Learning, Temporal, Digital Whiteboard, Education

Data Mining.

Abstract:

Collaborative Problem Solving, the resolution of complex problems with the collaboration of multiple peo-

ple pooling their knowledge, skills and effort is postulated as an essential 21st century skills for the future

workforce. Collaborative Problem Solving has been embraced in schools where both online and face-to-face

collaboration are afforded through the proliferation of educational technology tools. Assessing the amount

of collaboration that has taken place among the students has however been challenging. In this research, we

seek to identify the collaboration patterns of our students by mining the temporal sequence of their actions

logs captured within a digital whiteboard tool. With the use of Hidden Markov Model, we have uncovered

three collaboration states of students (Low Activity, Solitary Contributor, Cognitive Collaboration) from the

temporal sequences of their actions within the digital whiteboard. Contrary to common belief, the level of

collaboration was also found to have no inﬂuence on the quality of the ﬁnal artifact produced by a student

team. Collaborative behaviour was also discovered to persist within the team which suggests opportunities for

implementing interventions at an early phase of the learning activity for a longer-lasting team collaboration.

1 INTRODUCTION

The modern workspace is becoming increasingly

global. Modern technology has afforded businesses

to build global teams located in different parts of the

world to work together and navigate the business to-

wards continual growth. It is thus paramount that

modern and future workers learn to collaborate in

distributed and diverse settings for the attainment of

common business goals. To stay ahead of their com-

petition and achieve higher proﬁtability, businesses

realize the need to create new products and services

or uncover new innovative ways of doing things. The

discovery of new innovations however necessitates

the transcending of disciplinary boundaries as well

as the effective collaboration among experts from the

different disciplines. Collaborative Problem Solving

(CPS), a process where multiple people pool their

knowledge, skills and efforts to solve complex prob-

lems is thus postulated as an essential skill not only in

work but also in education (OECD, 2017). It has in

fact been identiﬁed as an essential 21st century skill

for the future workforce (Care et al., 2012; Graesser

https://orcid.org/0000-0002-4472-2481

et al., 2018).

CPS is conceptualized as a complex skill en-

compassing critical thinking, problem solving, deci-

sion making and collaboration (Care et al., 2016).

It has been embraced in many contexts including

schools (James and Johnston, 1996; Hennessy and

Murphy, 1999; Scoular and Care, 2020), online learn-

ing (Rosen et al., 2020) and military tasks (Swiecki

et al., 2020). It entails individuals working to-

gether responsively to solve a problem. Assessing

the amount of collaboration that has taken place is

however a challenging endeavour (Hao et al., 2017).

Within the educational context, students interact with

and inﬂuence each other in CPS. From the teacher’s

perspective, it is essential to have a gauge of the level

of collaboration taking place within a learning activ-

ity so that interventions to enhance collaboration can

be implemented if necessary. Some studies (Rabbany

et al., 2014) have since used interactions and partici-

pation of students in a collaborative setting as a mea-

sure of collaboration. In addition, the proliferation of

educational digital tools in teaching and learning has

also made it easier to capture and store students’ in-

teractions and actions performed within the tool itself.

These interaction logs in turn offer a treasure trove

Fwa, H.

Investigating Collaborative Problem Solving Temporal Dynamics Using Interactions Within a Digital Whiteboard.

DOI: 10.5220/0012011500003470

In Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023) - Volume 1, pages 367-373

ISBN: 978-989-758-641-5; ISSN: 2184-5026

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

367

for mining or uncovering the collaborative activities

of students.

Educational technology tools are prevalent in the

modern classroom and the use of some of these tools

are further accelerated during the Covid-19 pandemic

when many educational institutions had to transform

from synchronous to asynchronous form of teaching.

To ensure that the learning outcomes are not com-

promised when they pivot to online teaching, educa-

tors leveraged on tools such as Zoom for delivery of

lessons and Learning Management System (LMS) for

dissemination of teaching materials. Messaging plat-

forms are also used to maintain communications with,

foster collaboration and enhance engagement of their

students. The compendium of educational technol-

ogy tools helps in replicating to a certain extent, the

learning experience and engagement of face-to-face

teaching in an online environment.

Digital whiteboard is one such tool that affords

both teachers and students to collaborate and com-

plete tasks online without the need to meet face to

face. Digital whiteboards are cloud-based and pro-

vides a virtual space for teams to brainstorm and or-

ganize ideas. A physical whiteboard or pen and pa-

per have traditionally been used for groups of learners

to work on collaborative tasks constrained to a collo-

cated face-to-face setting though. In comparison, a

digital whiteboard offers the possibility for multiple

learners to work collaboratively and simultaneously

in an asynchronous fashion. With digital whiteboards,

learners are also not conﬁned by the physical limits of

a physical whiteboard or paper. With a digital white-

board, the produced artifact and action traces of learn-

ers are also automatically captured and stored.

Emerging research has investigated collaborative

behaviour in problem-solving tasks using the “coding

and counting” method where the interaction logs are

classiﬁed into category and various statistical mea-

sures are then computed (Andrews-Todd and Forsyth,

2020; Tausczik et al., 2014) for each category. The

computed statistical measures are then compared for

conclusions on the effects of the manipulated vari-

ables on group behaviour (Suthers, 2006). CPS is

however an interactive and temporal process where

the actions are produced as a series of inter-connected

steps. By aggregating counts over a ﬁxed period of

time, the “coding and counting” technique loses the

valuable temporal information embedded within the

interactions (Kapur, 2011).

In this study, we thus seek to identify the collab-

oration patterns of students by mining the temporal

sequence of their actions logs captured within a digi-

tal whiteboard tool.

2 RELATED WORK

In the study by Perera et al. (2008), the authors tasked

their students to work collaboratively to develop soft-

ware solutions for clients. The entire software de-

velopment process activities from ticket issuance and

tracking, documentation to source code version con-

trol were tracked through an open-source professional

software development tracking system. Clustering

was ﬁrst applied to identify patterns characterising the

behaviour of groups and individual students. Sequen-

tial pattern mining techniques were next applied to re-

veal temporal patterns or sequence of events charac-

terising the work of stronger versus weaker students

and groups. The results also indicated the value of

analysis on the resource and individual level rather

than just the group level. The authors postulated that

the results can be used to both educate the students

on aspects of group work and as a form of formative

feedback to the students to nudge them towards good

collaborative practices.

The research by Martinez et al. (2011) was mo-

tivated by the goal of mining large amounts of data

generated from learners’ interactions with interac-

tive tabletop. The authors captured the raw interac-

tion logs from 6 groups of 3 students and compacted

them into sequences of actions before clustering the

sequences based on edit distance between the clus-

ters. Their ﬁndings revealed that the high performing

groups tended to discuss their thoughts and work in

parallel with different objects while the low perform-

ing groups tended to work sequentially with objects.

More recently, Zhou et al. (2022) transcribed and

coded zoom video recordings of college students

solving exercises within an online physics game.

They applied a temporal analysis technique, Multi-

level Vector Autoregression (mIVAR) on transcribed

video sequences of 10-seconds windows for the in-

vestigation of CPS processes. The models were ap-

plied separately on successful and unsuccessful CPS

sequences. The ﬁndings revealed six temporal rela-

tionships common to both groups (successful and un-

successful CPS sequences), six unique to only suc-

cessful attempts and another eight unique to only un-

successful attempts.

The above three studies applied and demonstrated

the feasibility of temporal mining techniques for re-

vealing differences in collaboration patterns between

high and low performing groups in CPS. In the cur-

rent study, we continue this line of research by utiliz-

ing action logs from students collaborating on a digi-

tal whiteboard. The ﬁrst two studies discussed above

compiled the raw interactions logs into frequent se-

quences of coded actions before the application of

CSEDU 2023 - 15th International Conference on Computer Supported Education

368

clustering techniques to cluster the sequences into

groups for analysis of collaboration behaviour. The

sequences of coded actions are however speciﬁc to a

unique learning activity and would not be applicable

to a different learning activity. To illustrate, the action

sequences in the study by Martinez et al. (2011) com-

prises of frequently occurring actions such as mov-

ing, enlarging and shrinking of objects. The enlarg-

ing and shrinking of objects is however speciﬁc to a

tabletop scenario where the objects are ﬁrst displayed

in a minimized form due to space constraint on the

tabletop and may not recur in another learning sce-

nario. Thus, in our study, we investigate into the use

of more generalizable features that can apply to most

CPS scenarios.

This leads to our research question: Can we iden-

tify collaboration patterns of students from their se-

quence of actions working on a digital whiteboard

over a speciﬁed time-period in a CPS scenario using

generic features?

3 METHODOLOGY

The participant of this research comprises of 30 year-

one university students (by convenience sampling)

from the faculty of computer science and informa-

tion systems and the faculty of accountancy within

our university. They were part of the 2 classes of

42 students and 34 students undertaking the course of

data management taught by the author in 2 separate

semesters of academic year 2022. The data manage-

ment course spans a total duration of ﬁfteen weeks

and covers the fundamentals of relational database

which includes data modelling, data design (Entity-

Relationship diagram) and database implementation.

This research has been reviewed and approved by the

university’s institutional review board and all 30 stu-

dents have given written consent for their data to be

used in this research.

3.1 Digital Whiteboard

The digital whiteboarding software – Miro

(http://www.miro.com) was introduced to the

students in week 2 of the course. In each lesson, the

students were ﬁrst introduced to the concepts to be

covered in the class. Other than instructor elaboration

on some demonstration exercises, in-class exercises

were dispersed throughout the entire lesson. An

active learning approach was undertaken where

students worked on the in-class exercises on the

digital whiteboard collaboratively in groups of three.

The digital whiteboarding software, Miro facili-

tates discussions and interactions among groups with

a mix of remote and in class students. Miro offers

collaboration features such as chat and zoom inte-

gration and more importantly synchronous interaction

features where participants can see each other’s anno-

tations and drawings on a real time basis. For this

study, we only store and analyse the historical action

logs of the students within the digital whiteboard plat-

form. Although chat and zoom integration were of-

fered by the digital whiteboard, the voice recordings

and chat history were not used for this study.

The students were tasked with a group assignment

to be submitted by week 8 (which involves designing

an ER diagram for a provided scenario). The students

were required to form groups of maximum 3 students

to complete the assignment using Miro. The students

who participated in this research conducted their dis-

cussion both synchronously and asynchronously as

afforded by the online collaboration features of Miro.

Only the action logs for the assignment (and not the

in-class exercises) were stored and used for analysis.

The students were briefed on the aims and details of

the research and we only use the Miro canvases of

students whom have given informed consent to partic-

ipate in the study. This research was also submitted to

and approved by our university’s Institutional Review

Board (IRB).

The board history feature in Miro stores a list of

historical actions performed by the logged in user.

The logged information includes the user who per-

formed the action, the date and time of the action

and details on the speciﬁc action performed e.g. add,

edit and delete and the object acted on e.g. text, line,

shape, sticky note e.t.c. Miro offers a Representa-

tional State Transfer (Fielding and Taylor, 2002) Ap-

plication Programming Interface (REST API) inter-

face for developers to extend on the current capabili-

ties of Miro but unfortunately does not provide for the

retrieval of board histories through REST API. Thus,

we developed our own Python program to web scrap

(Mitchell, 2018) the board histories of the 10 groups’

canvases. The scrapped logs are stored as comma de-

limited ﬁles.

The raw logs contain details of the user actions

such as the user’s name, date and time of action and

the action type i.e. add, edit or delete. We then ag-

gregated the action details into sessions of 10 minutes

in duration. From the action logs, a session with 10

minutes duration would have accumulated adequate

number of actions with a higher possibility of other

members of the team acting on the whiteboard canvas

within the session and thus the decision on breaking

into 10 minutes session. The action logs were not

continuously in time dimension and there could be

Investigating Collaborative Problem Solving Temporal Dynamics Using Interactions Within a Digital Whiteboard

369

long breaks (up to a few days) between consecutive

actions. Actions that had a prior long break were con-

sidered as a new session of actions. This also explains

why duration of some sessions may not be exactly 10

minutes.

We next computed the number of pauses, num-

ber of unique users, number of actions and the length

of duration for each session. These constitute the

features that were passed into our machine learning

model for prediction. To reiterate, these features were

chosen as they are generic across different learning

scenarios. We deﬁne pauses as breaks between con-

secutive actions that are more than a minute but less

than 10 minutes in duration. Pauses between ac-

tions are posited as periods of cognitive processing

in a number of research (O’Brien, 2006; Lacruz et al.,

2012; Shrestha et al., 2022). We also consider pauses

as a time period where students stop to think about ei-

ther the past actions or the next action to take which

would very probably lead on to better or ‘more con-

sidered’ future actions. The number of unique users

within a session signiﬁes the amount of collabora-

tion taking place. We consider that a session which

has a number of students acting on the same white-

board canvas would likely be evident of collaboration

and discussion taking place among the students. The

number of actions is the total count of the add, edit

and delete actions within a session while the duration

length is the length of each session in seconds.

3.2 Hidden Markov Model (HMM)

Taking temporality into account is important in the

modelling of collaboration behaviour among groups.

Comparing 2 groups of students collaborating to work

on a common problem, the trajectory of actions (per-

formed by the students) across the time domain would

likely differ for the 2 groups. The Hidden Markov

Model (Rabiner, 1989) lends itself well in the mod-

elling of the temporal sequences of actions for uncov-

ering collaboration sequences.

HMMs consist of stochastic Markov chains based

on a set of hidden states whose values cannot be di-

rectly observed with the relationship between a hid-

den state and the actual observations being modelled

with a probability distribution. HMMs adhere to the

Markov property which states that the state of a model

at time t is only dependent on the state of the model

at time t-1 and not on other prior states such that

P(S

t+1

) = P(S

t+1

, S

t−1

...S

) (1)

A HMM is described by the tuple {S, O, A, B, π}

where

N: Number of hidden states

S: {S

, S

, ..., S

}

M: Number of observation symbols

O = {O

, O

, ..., O

}

A = {a

i j

}, where a

i j

= P(S

t+1

, S

);i, j = 1, ..., N

B = {b

}, where b

(k)=P(O

)

The model parameters are valid probabilities that

must satisfy the following constraints:

∑

i j

= 1,

∑

(k) = 1 (2)

The probability of being in state i at time t

is given

by π = {π

}, where π

= P(S

)

HMMs can be used in unsupervised learning and

we adopted the unsupervised learning approach in this

research. This allows for the discovery of hidden

patterns without tedious human labelling as is done

in many other studies (Boussemart et al., 2009; Li

and Biswas, 2002; Trabelsi et al., 2013). In unsu-

pervised learning, the model parameters H = (A, B, π)

are learned by maximizing

∑

log(P(O

|H), the sum

of the posterior log-likelihoods of each training se-

quence O

using a form of expectation-maximization

(Dempster et al., 1977) called the Baum-Welch algo-

rithm (Baum et al., 1972).

We model the observations as Gaussian distribu-

tions as the features are continuous valued. We also

took the logarithms of both number of actions and du-

ration length before passing them as features into the

model. Models with a number of hidden states rang-

ing from 2 to 6 were trained multiple times with dif-

ferent randomly assigned initial parameters in order

to avoid convergence to a local optimum. The train-

ing was performed through the unsupervised Baum-

Welch learning technique. We used the Bayesian In-

formation Criterion (BIC) for determining the num-

ber of hidden states for the HMM. BIC allows for the

comparison of models with different number of hid-

den states, trained on the same underlying data. BIC

penalizes the likelihood of the model by a complexity

factor proportional to the number of parameters P in

the model and the number of training observations K.

BIC = −2 log(L (H)) + P log(K) (3)

Table 1: BIC values for the various number of hidden states.

No. of hidden states BIC

2 2075.14

3 1115.91

4 1089.71

5 365.75

6 291.65

From Table 1, although the BIC value for 6 hidden

CSEDU 2023 - 15th International Conference on Computer Supported Education

370

states is the lowest, we chose 3 hidden states model

instead as the results are more interpretable.

4 RESULTS

The mean values of the features learned by the chosen

3 hidden states HMM is shown in Table 2.

State 1 is characterized by low number of pauses,

one participating user with a low number of actions

within a short duration and thus we labelled it as the

“Low Activity” state.

State 2 is characterized by high number of pauses,

one participating user with a moderate number of ac-

tions within a moderate duration and thus the label of

“Solitary Contributor”.

State 3 has high number of pauses, high number of

participating unique users and high number of actions

within a long duration of time. This suggests a session

with multiple users performing actions on the canvas

but with more pauses or more deliberated actions. We

infer that the users may be discussing and acting after

cognitive processing of the impact of their actions and

thus the label “Cognitive collaboration”.

Table 3 shows the probability of transiting from

the source states in rows to the destination states in

columns. From the table, a user who is in “Low

Activity” state is likely to either stay in this state or

transit to “Solitary Contributor”. A user who is in

the “Solitary Contributor” state is more likely to stay

in the “Solitary Contributor” state and a user who is

in the “Cognitive Contributor” state would likely re-

main in this state. Across the different groups of stu-

dents, we then summed up the predicted hidden states

across the sessions to gauge the difference in collab-

oration patterns across the groups. The proportion of

states aggregated across all the sessions for the differ-

ent groups of students is shown below. Only groups

of students with more than 20 sessions are shown be-

low as we wanted to compare between groups which

were more active i.e. worked substantially more and

longer on their whiteboard canvas. Teams 1, 3, 4, 5

and 8 were identiﬁed as the more ‘active’ groups and a

comparison of the collaborative behaviour of the more

‘active’ groups is shown in Figure 1.

5 DISCUSSION

From the results, we have identiﬁed collaboration

states of students from their sequence of actions work-

ing on a digital whiteboard. As manifestations of stu-

dents’ interaction patterns, these states provided in-

sight into the collaboration behaviours of the students.

Speciﬁcally, we have inferred from the trained HMM,

the 3 states of collaboration – Low Activity, Soli-

tary Contributor and Cognitive Collaboration. The

“Low Activity” state denotes a session where students

are not acting on the whiteboard canvas while the

“Solitary Contributor” state denotes a session where

a single student is working on the whiteboard canvas.

The ideal session will be the “Cognitive Collabora-

tion” state where multiple students are working on the

whiteboard canvas and the actions are more deliber-

ated as hinted by the occurrence of pauses within the

session.

We further analysed the temporal evolution of stu-

dents’ collaboration patterns with the state transition

matrix. The state transition matrix in Table 2 indi-

cated that students in the “Solitary Contributor” ses-

sion tend to remain so and this applies for the “Cog-

nitive Contributor” state as well. We thus conclude

from the state transition matrix that a team who is

collaborative and encourages participation from their

members tend to stay collaborative while a team pos-

sibly with a dominant member who makes the most

contribution tend to continue demonstrating solitary

or non-collaborative behaviour. This signiﬁes that if

we can implement interventions to encourage the stu-

dents to collaborate at the initial stage of the project

work, then the collaboration will likely persist.

As shown in Fig 1, teams 1, 3 and 8 had propor-

tionately more sessions that were predicted as “SC”

states while teams 4 and 5 had proportionately more

sessions that were predicted as “CC”. This suggests

that teams 4 and 5 are more collaborative as com-

pared to the other teams. The whiteboard canvas as-

signments of the teams were assessed by a different

instructor and all 5 teams scored at least a B grade

for their assignment. This indicates that the extend

of collaboration within a team as predicted from their

members’ interaction actions on the whiteboard can-

vas seem to have no impact on their work quality.

We surmise that this may possibly be attributed to the

“SC” teams having cognitively capable member who

did most of the work.

6 CONCLUSION

We performed mining of students’ raw interactions

within a digital whiteboard using HMM in a CPS sce-

nario. The students were collaborating in groups of

3 to work on an entity-relationship diagram design

assignment. Our objective was to identify collabo-

ration patterns of students from their sequence of ac-

tions working on a digital whiteboard over a speciﬁed

time-period in a CPS scenario using generic features.

Investigating Collaborative Problem Solving Temporal Dynamics Using Interactions Within a Digital Whiteboard

371

Table 2: Mean values of the features learned by 3 hidden states HMM.

Hidden

State

No. of pauses No. of

unique

users

No. of actions Duration in secs Description

1 0.96 1.00 2.62 1.00 Low Activity

(LA)

2 3.29 1.01 8.59 227.80 Solitary Con-

tributor (SC)

3 4.11 2.37 11.79 353.59 Cognitive

Contributor

(CC)

Figure 1: Comparison between collaborative behaviour of more ”active” groups.

Table 3: Transition probabilities between the various states.

Hidden State LA SC CC

LA 0.38 0.47 0.15

SC 0.27 0.58 0.15

CC 0.17 0.27 0.56

From the raw interaction logs extracted from the dig-

ital whiteboard, we collated them into sessions with

each session comprising of generic features such as

number of pauses, number of unique users, number

of actions and the length of duration. These were then

passed into a HMM for unsupervised learning of the

hidden states and the transition probabilities between

the various states.

The results revealed 3 clusters of collaborative be-

haviours namely “Low Activity”, “Solitary Contribu-

tor” and “Cognitive Collaboration” and that early col-

laborative behaviour tend to persist. This suggests op-

portunities for implementing interventions at an early

phase of the learning activity for a longer-lasting col-

laboration among the team members. Comparing the

collaborative behaviour of “active” groups, we con-

clude that the extend of collaboration has no bearing

on the quality of their ﬁnal artifact though. For future

work, we would like to investigate the generalizabil-

ity of our work by extending it into a different CPS

scenario using digital whiteboards.

REFERENCES

Andrews-Todd, J. and Forsyth, C. M. (2020). Exploring so-

cial and cognitive dimensions of collaborative prob-

lem solving in an open online simulation-based task.

Computers in human behavior, 104:105759.

Baum, L. E. et al. (1972). An inequality and associated

maximization technique in statistical estimation for

probabilistic functions of markov processes. Inequal-

ities, 3(1):1–8.

Boussemart, Y., Las Fargeas, J., Cummings, M. L., and Roy,

N. (2009). Comparing learning techniques for hidden

markov models of human supervisory control behav-

ior. In AIAA Infotech@ Aerospace Conference and

AIAA Unmanned... Unlimited Conference, page 1842.

Care, E., Grifﬁn, P., and McGaw, B. (2012). Assessment

and teaching of 21st century skills. Springer.

Care, E., Scoular, C., and Grifﬁn, P. (2016). Assess-

ment of collaborative problem solving in education

environments. Applied Measurement in Education,

29(4):250–264.

Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).

Maximum likelihood from incomplete data via the em

CSEDU 2023 - 15th International Conference on Computer Supported Education

372

algorithm. Journal of the royal statistical society: se-

ries B (methodological), 39(1):1–22.

Fielding, R. T. and Taylor, R. N. (2002). Principled design

of the modern web architecture. ACM Transactions on

Internet Technology (TOIT), 2(2):115–150.

Graesser, A. C., Fiore, S. M., Greiff, S., Andrews-Todd, J.,

Foltz, P. W., and Hesse, F. W. (2018). Advancing the

science of collaborative problem solving. psychologi-

cal science in the public interest, 19(2):59–92.

Hao, J., Liu, L., von Davier, A. A., and Kyllonen, P. C.

(2017). Initial steps towards a standardized assess-

ment for collaborative problem solving (cps): Practi-

cal challenges and strategies. Innovative assessment

of collaboration, pages 135–156.

Hennessy, S. and Murphy, P. (1999). The potential for col-

laborative problem solving in design and technology.

International journal of technology and design educa-

tion, 9:1–36.

James, R. and Johnston, C. (1996). An evaluation of the

effectivess of collaborative problem-solving for learn-

ing economics. Technical report, The University of

Melbourne.

Kapur, M. (2011). Temporality matters: Advancing a

method for analyzing problem-solving processes in a

computer-supported collaborative environment. Inter-

national Journal of Computer-Supported Collabora-

tive Learning, 6:39–56.

Lacruz, I., Shreve, G. M., and Angelone, E. (2012). Average

pause ratio as an indicator of cognitive effort in post-

editing: A case study. In Workshop on Post-Editing

Technology and Practice.

Li, C. and Biswas, G. (2002). Applying the hidden markov

model methodology for unsupervised learning of tem-

poral data. International Journal of Knowledge Based

Intelligent Engineering Systems, 6(3):152–160.

Martinez, R., Yacef, K., Kay, J., Al-Qaraghuli, A., and

Kharrufa, A. (2011). Analysing frequent sequen-

tial patterns of collaborative learning activity around

an interactive tabletop. In Educational Data Mining

2011, pages 111–120. CEUR-WS.

Mitchell, R. (2018). Web scraping with Python: Collecting

more data from the modern web. ” O’Reilly Media,

Inc.”.

O’Brien, S. (2006). Pauses as indicators of cognitive ef-

fort in post-editing machine translation output. Across

languages and cultures, 7(1):1–21.

OECD (2017). Pisa 2015 collaborative problem solving

framework.

Perera, D., Kay, J., Koprinska, I., Yacef, K., and Za

ıane,

O. R. (2008). Clustering and sequential pattern

mining of online collaborative learning data. IEEE

Transactions on knowledge and Data Engineering,

21(6):759–772.

Rabbany, R., Elatia, S., Takaffoli, M., and Za

ıane, O. R.

(2014). Collaborative learning of students in online

discussion forums: A social network analysis per-

spective. Educational data mining: Applications and

trends, pages 441–466.

Rabiner, L. R. (1989). A tutorial on hidden markov models

and selected applications in speech recognition. Pro-

ceedings of the IEEE, 77(2):257–286.

Rosen, Y., Wolf, I., and Stoefﬂer, K. (2020). Fostering

collaborative problem solving skills in science: The

animalia project. Computers in Human Behavior,

104:105922.

Scoular, C. and Care, E. (2020). Monitoring patterns of

social and cognitive student behaviors in online col-

laborative problem solving assessments. Computers

in Human Behavior, 104:105874.

Shrestha, R., Leinonen, J., Zavgorodniaia, A., Hellas, A.,

and Edwards, J. (2022). Pausing while programming:

insights from keystroke analysis. In Proceedings of

the ACM/IEEE 44th International Conference on Soft-

ware Engineering: Software Engineering Education

and Training, pages 187–198.

Suthers, D. D. (2006). Technology affordances for intersub-

jective meaning making: A research agenda for cscl.

International Journal of Computer-supported collab-

orative learning, 1:315–337.

Swiecki, Z., Ruis, A. R., Farrell, C., and Shaffer, D. W.

(2020). Assessing individual contributions to collabo-

rative problem solving: a network analysis approach.

Computers in Human Behavior, 104:105876.

Tausczik, Y. R., Kittur, A., and Kraut, R. E. (2014). Col-

laborative problem solving: A study of mathoverﬂow.

In Proceedings of the 17th ACM conference on Com-

puter supported cooperative work & social comput-

ing, pages 355–367.

Trabelsi, D., Mohammed, S., Chamroukhi, F., Oukhellou,

L., and Amirat, Y. (2013). An unsupervised approach

for automatic activity recognition based on hidden

markov model regression. IEEE Transactions on au-

tomation science and engineering, 10(3):829–835.

Zhou, G., Moulder, R. G., Sun, C., and D’Mello, S. K.

(2022). Investigating temporal dynamics underly-

ing successful collaborative problem solving behav-

iors with multilevel vector autoregression. Interna-

tional Educational Data Mining Society.

Investigating Collaborative Problem Solving Temporal Dynamics Using Interactions Within a Digital Whiteboard

373