Proposal of Indicators for Measuring Collaborative Writing in a Digital

Learning Environment

Anis M. Haddouche

1 a

, Fahima Djelil

1 b

, Christian Hoffmann

2 c

, Nadine Mandran

2 d

and C

edric d’Ham

2 e

IMT Atlantique, Lab-STICC, UMR CNRS 6285, 29238 Brest, France

Universit

e Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

Keywords:

Indicators, Collaborative Writing, Computer-Supported Collaborative Learning (CSCL), Learning Analytics.

Abstract:

Collaborative Writing (CW) is a common activity in education, which is being enhanced by the use of digital

learning environments, leading to a growing research ﬁeld in Computer-Supported Collaborative Learning

(CSCL). In order to help teachers to monitor students CW, we propose two indicators that provide measures of

student contributions to a text writing, namely balance of contribution and co-writing. We also identiﬁed CW

strategies that are well deﬁned in the literature. Moreover, we conducted a questionnaire evaluation to verify

the interpretation of the indicators and the strategies by teachers in higher education context, using student

reports edited in a collaborative digital environment called LabNbook, during physics and chemistry courses

in undergraduate level. Results showed that teachers have a good interpretation of the indicators and strategies.

This work contributes to research insights in CW, and motivates future work to design meaningful learning

indicators.

1 INTRODUCTION

Providing students with group work is a common ac-

tivity in education. Assigning group work is largely

emphasized as a method that enables students to de-

velop collaborative skills (Sun et al., 2018). More-

over, this may be enhanced by the use of digital tech-

nologies, providing students with several tools such

as collaborative writing platforms (Zhang and Chen,

2022). This led to growing research in the ﬁeld of

Computer-Supported Collaborative Learning (CSCL)

(Chen et al., 2018). More particularly, Collaborative

Writing (CW) has gained an increasing research inter-

est in the last few years (Zhang et al., 2021). Students’

interactions can be efﬁciently captured and stored,

leading to a more ﬁne-grained analysis of the CW pro-

cess.

A large part of existing research aims at captur-

ing student collaboration dynamics using mixed ap-

https://orcid.org/0000-0002-5321-3988

https://orcid.org/0000-0001-8449-2062

https://orcid.org/0000-0002-0620-3621

https://orcid.org/0000-0002-8660-3827

https://orcid.org/0000-0002-7313-7097

proaches: analysis of digital logs and peer student in-

teractions such as exchanged messages or oral conver-

sations (Zhang et al., 2021). Despite prior abundant

works that characterize and assess collaboration dy-

namics in CW, there is a need for improved research

in this ﬁeld, in particular concerning measures and

metrics (Zhang et al., 2021). In this work, we con-

tribute with two new indicators that provide measures

for CW, namely Balance of Contribution reﬂecting

the extent to what students’ contributions are equili-

brated at a word level and Co-writing reﬂecting stu-

dents’ contributions at a sentence level in a text. We

base the indicator calculations on a variance metric

and explore the relationship between these indicators

and typical CW strategies. The deﬁnition of these in-

dicators is sourced from previous research that is not

detailed in this paper (Hoffmann et al., 2022). Our

research is anchored in a real-life context, using a

web-based learning environment, called LabNbook,

designed for supporting learners in the CW of sci-

entiﬁc documents (d’Ham et al., 2019). Moreover,

we evaluate the interpretation of the proposed indica-

tors and the deduced strategies by teachers in higher

education context. We use collaborative documents

edited during physics and chemistry courses in un-

Haddouche, A., Djelil, F., Hoffmann, C., Mandran, N. and d’Ham, C.

Proposal of Indicators for Measuring Collaborative Writing in a Digital Learning Environment.

DOI: 10.5220/0011757700003470

In Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023) - Volume 2, pages 495-502

ISBN: 978-989-758-641-5; ISSN: 2184-5026

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

495

dergraduate level. We address the following research

questions:

RQ 1.) What are the indicator metrics that allow to

measure students’ CW?

RQ 2.) How can we deduce CW strategies from these

indicators?

RQ 3.) To what extent these indicators and strategies

are interpretable by teachers? (How close is the rela-

tionship between these measures and their interpreta-

tion).

This paper is organized as follows. Section 2 pro-

vides a state of the art of existing research on CW.

Section 3 introduces the proposed indicators and the

co-writing strategies. Section 4 describes our research

method. Section 5 details the calculations of the

proposed indicators and their relationship to the co-

writing strategies. Results are discussed in Section 6,

conclusion and implications are derived in Section 7.

2 COLLABORATIVE WRITING

Collaborative Writing (CW) can refer to the produc-

tion of a text by two or more writers (co-authoring)

(Storch, 2013). More speciﬁcally, it is deﬁned as

a process involving substantive interactions between

learners sharing decision making and responsibilities

for a single produced document (Zhang and Chen,

2022).

There is a long-standing interest in CW and differ-

ent authors proposed early models and taxonomies of

CW (Posner and Baecker, 1992; Lowry et al., 2004;

Storch, 2013). With the fast-growing use of Online

Learning Environments (OLE) and the ability to an-

alyze data logs, several frameworks have been ap-

plied to examine behaviors, patterns and strategies of

CW in different educational domains (Onrubia and

Engel, 2009; Sundgren and Jaldemark, 2020; Olson

et al., 2017). For instance, in the domain of second

language (L2), a recent systematic literature review

(Zhang et al., 2021) has examined more than one hun-

dred studies revealing a strong research interest in the

ﬁeld of CW during the past decade.

One popular model in the literature that was pro-

posed, is a dyadic interaction model with two con-

structs, namely equality, reﬂecting the learner’s level

of contribution and control over the task, and mutual-

ity, reﬂecting the learner’s level of engagement with

each other’s contribution (Storch, 2013).

These two constructs also appear in less recent lit-

erature. For example, (Dillenbourg, 1999) compares

concepts from the ﬁeld of Human-Computer Collabo-

rative Learning Systems (HCCLS), where an artiﬁcial

agent collaborates with the human learner, and CSCL

systems, where the computer supports collaboration

between two human users. He argues that collabo-

ration implies negotiation and emphasizes on the de-

gree of symmetry in interactions between peers. He

deﬁnes Symmetry of action as the extent to which the

same range of actions is allowed to each agent (Dil-

lenbourg and Michael, 1996) (the agent may refer to

an artiﬁcial agent or a human peer). The term sym-

metry is borrowed from HCCLS domain to qualify an

equilibrated balance of control implied by collabora-

tion between the system and its user (Dillenbourg and

Michael, 1996).

Mutual reﬁnement is a second main characteris-

tic deﬁning collaboration through negotiation (Baker,

1994). This refers to speciﬁc strategies that exist for

achieving agreement in the interaction (each agent

successively reﬁnes the contribution of the other)

(Dillenbourg and Michael, 1996). This can be re-

ﬂected in ways of editing text written by others and

ways of coping with others editing one’s own text

(Larsen-Ledet and Korsgaard, 2019), or the degree

of engaging with each other’s ideas and each other’s

texts and providing scaffolding in producing joint

writing (Li and Zhu, 2016).

This led us to say that two key indicators may help

to characterize collaborative work. The ﬁrst one eval-

uates the symmetry in contributions, while the second

one measures the degree of interactions between stu-

dents.

3 INDICATORS AND

STRATEGIES

We deﬁne balance of contribution as a metric that in-

dicates how the student contributions are equal, well

balanced or imbalanced. It is aligned on the one pro-

posed by (Olson et al., 2017), called balance of par-

ticipation which is a team contribution measure that

reﬂects whether individuals’ contributions are equal

or imbalanced. As a balance metric, they considered

one minus the variance of team members proportions

in the collaborative work. The authors argued on vari-

ance as preferable over other ways of calculation such

as Gini coefﬁcient or Blau’s index, for its simplicity

and ease of interpretation. We based the calculation

of the balance of contribution indicator on a variance

metric, that provides a distribution between the au-

thors’ average contributions in terms of number of

words in a ﬁnal document.

Balance of contribution indicator measures a stu-

dent contribution at a word level, and co-writing in-

dicator measures a student contribution at a sentence

level. Indeed, balance of contribution measures only

CSEDU 2023 - 15th International Conference on Computer Supported Education

496

division of labor (number of words a student con-

tribute to the text), and co-writing goes beyond that by

measuring student co-construction of the text (modi-

fying and adding words in sentences written by others

in the text). The values of both indicators range be-

tween 0 and 1 (see section 5).

A possible exploitation of the indicators is the

identiﬁcation of CW strategies. We consider two

strategies among the ﬁve proposed by (Onrubia and

Engel, 2009), namely: 1) sequential summative text

construction, i.e., one group member presents a doc-

ument that constitutes an initial, partial or complete,

proposal for the task resolution and the rest of the par-

ticipants successively add their contributions to this

initial document, without modifying what has been

previously written, hence, systematically accepting

what is added by other co-authors; 2) sequential in-

tegrative text construction, i.e., one group member

presents a document that constitutes an initial, par-

tial or complete task proposal, and the other group

members successively contribute to this initial docu-

ment, proposing justiﬁed modiﬁcations or discussing

whether they agree with what has been previously

written or not.

We use the term summative to appoint to a strat-

egy where each student adds his text without modify-

ing the text of the others where the result is being a

juxtaposition of the individual contributions and, the

term integrative, to appoint to a strategy where one

student proposes an initial version and the other stu-

dents contribute successively making modiﬁcations to

the existing text (Hoffmann et al., 2022).

The ﬁrst is characterized by an explicit division

of work between the team members, and the second

by a co-construction of the text. Students may not

necessarily follow one strategy but a mix of them.

4 RESEARCH METHOD

Our objective is, on the one hand, to construct metrics

of students’ CW process, and on the other hand, to

verify that these metrics are intelligible for teachers.

To conduct our research, we choose the design based

research (Wang and Hannaﬁn, 2005) and the associ-

ated guides (Mandran et al., 2022). This method pro-

poses to build knowledge and associated tools in an

iterative way by integrating all the actors in the ﬁeld.

Each iteration consists of developing the tool and the

knowledge on the basis of the reality in the ﬁeld and

the opinion of the stakeholders.

4.1 Context

We use text documents produced by students on the

LabNbook platform

, which is a digital environment

for learning experimental sciences in secondary and

higher education (d’Ham et al., 2019). It provides

useful tools for writing collaborative scientiﬁc doc-

uments (text and equations, drawing, data processing,

etc.) and allows students to interact with each other

(team building, shared workspace, internal messag-

ing and chat, etc.). LabNbook operates in a locked

co-editing mode (Wang et al., 2017), i.e., students

can work simultaneously in the shared workspace but

each document, called LabDoc, composing it can be

edited only by one student at a time. For instance,

a collaborative report may be composed of different

sections, and each section corresponds to a LabDoc.

4.2 Data Collection

We use a questionnaire to measure to what extent the

strategies and indicators are interpretable by teachers

(RQ3). Our objective is to verify whether the deﬁni-

tions make sense to them.

We selected 12 LabDocs edited on the LabNbook

platform by groups of students during physics and

chemistry courses (undergraduate level) in Greno-

ble Alpes University (France). To distinguish stu-

dent contributions, we highlight in each LabDoc

with a color-code the text written by each student.

We choose LabDocs uniformly with different writing

strategies (integrative, summative, and mixed strate-

gies). Respondent teachers are a total of 15, situ-

ated in IMT Atlantique in Brest (France), from Com-

puter Science discipline and in Grenoble Alpes Uni-

versity in Grenoble (France), from Experimental Sci-

ences disciplines. After providing the teachers with

verbatim deﬁnitions of the strategies and of the indi-

cators, we ask them to indicate for each LabDoc:

(1) the strategy they perceive (Entirely Summative

(ES), Rather Summative (RS), Between Sum-

mative and Integrative (BSI), Rather Integrative

(RI), Entirely Integrative (EI) and I don’t know

(DN));

(2) an estimate of the level of each indicator (Low

(L), Medium (M) and High (H));

(3) a numerical value for each indicator between 0

and 1.

https://LabNbook.fr/

Proposal of Indicators for Measuring Collaborative Writing in a Digital Learning Environment

497

5 INDICATORS CONSTRUCTION

5.1 Text Sequences Matching Method

In order to study the evolution of a text document

and compare pairs of text sequences, we use a Se-

quence Matcher method which has its origins in an

algorithm published in the late 1980’s by Ratcliff and

Metzener under the name Gestalt Pattern Matching

(Ratcliff and Metzener, 1988)

To illustrate this method, consider a text co-

written by two students A and B. Student A writes the

ﬁrst version which is then modiﬁed by student B. In

order to qualify the evolution of the initial text to its

ﬁnal version, the approach consists ﬁrstly in ﬁnding

the longest, in terms of number of characters, con-

tiguous matching sequence (a set of words) that con-

tains no useless elements, such as blank lines or white

space. The same operation is then applied recursively

for the sequences to the left and to the right of this

longest contiguous matching sequence. Then, in or-

der to qualify changes in the text, each sequence is

tagged as follows: Equal (sequences are equal), Insert

(sequence is inserted), Delete (sequence is deleted),

Replace (sequence is replaced). For instance, let con-

sider the following pair of text sequences written se-

quentially by two students:

• Student A: “LabNbook is a digital platform used

by over 3500 students”

• Student B: “LabNbook is a platform used by more

than 3500 students in France”

Student B contributes to the text after student A by

inserting, deleting and modifying text. Table 1 gives

the correspondent tag for each altered sequence.

Table 1: Illustration of the tagging operation.

Tag Student A Student B

Equal “LabNbook is a” “LabNbook is a”

Delete “digital” “ ”

Equal “platform used by” “platform used by”

Replace “over” “more than”

Equal “3500 students” “3500 students”

Insert “ ” “in France”

5.2 Contribution Matrix

To quantify and track the evolution of the text, we use

a matrix that gives for each word, the level of con-

tribution of each of the students co-writing the text.

We call this matrix, a contribution matrix, since it

For implementation, we use the Python library Difﬂib

https://github.com/python/cpython/blob/main/Lib/difﬂib.py

gives the student levels of contributions to a text. In

this matrix, rows represent contributing students and

columns represent words constituting the ﬁnal text.

In order to provide a formal deﬁnition of this con-

tribution matrix, let consider a text composed of N

sentences, M words and co-written by K authors. Let

also x

i, j,l

∈ [0,1], be the contribution level of an author

i to a word l of a sentence j where i ∈ [1,K], j ∈ [1, N]

and l ∈ [1, n

]. Here, n

denotes the number of words

in a sentence j.

In this K × M matrix, the level of contribution of

an author is expressed by a score ranged between 0

and 1. It is set to 1 when the author writes entirely a

word, and to 0, when the author doesn’t contribute to

a word. The total authors’ contributions to a word is

equal to 1.

Since our scoring method is based on the sequence

matching method that identiﬁes the longest contigu-

ous matching sequence, deleted words do not result

in a score, and replaced words result in a score that

returns the ratio of contributions in terms of number

of characters.

In the previous example, 2 students contribute

to a text which is composed of 12 words (“LabN-

book is a platform used by more than 3500 students

in France”). Consequently, the contribution matrix

gives scores ranged in 2 rows and 12 columns, as fol-

lows



1 1 1 1 1 1 0.3 0.3 1 1 0 0

0 0 0 0 0 0 0.7 0.7 0 0 1 1



where the ﬁrst row gives the scores corresponding to

the contribution levels of the student A and the second

row those of the student B.

5.3 Balance of Contribution Indicator

The balance of contribution indicator measures stu-

dent’s contribution at a word level. More precisely, it

measures the degree to which students’ contributions

are equal (well-balanced) or not. The closer the value

of this indicator is to 1, the more students’ contribu-

tions are equal. At the opposite, the more it is closer to

0, the less the students’ contributions are equal. This

indicator is based on the variance in (3) of the stu-

dents’ average contributions, which reﬂects the dis-

tance between each student average contribution and

the mean of all students’ average contributions in (2).

Note that, if the students contribute in a well-balanced

way, then their scores are close to the mean and they

contribute quite equally. Moreover, in order to penal-

ize the case where the whole text is written by one

student, this variance is normalized by (4). We deﬁne

CSEDU 2023 - 15th International Conference on Computer Supported Education

498

the balance of contribution indicator as follows

e(X) = 1 −

K − 1

∑

i=1



i,,

−



(1)

where, for i = 1,..., K, x

i,,

is the average contribu-

tion of the student i to all words of the text. Note that,

∑

i=1

i,,

= 1 and the mean of all students’ average

contributions is

,,

∑

i=1

i,,

. (2)

Indeed, as we consider that students contribute in

a balanced way if their average contributions equals

1/K, we use in the construction of the balance of con-

tribution indicator the variance

∑

i=1



i,,

−



. (3)

However, in the case a student i writes alone the whole

text, only one x

i,,

equals 1. Then (3) becomes

K − 1

. (4)

Finally, as mentioned above, in order to penalize the

case where one student writes the whole text, we nor-

malize the variance (3) with (4), which gives

K − 1

∑

i=1



i,,

−



This dispersion reaches its minimum value 0 when

all students contribute equally (or in a balanced way)

to the text, that is, where the average contribution of

each student x

i,,

equals 1/K . It reaches its maxi-

mum value 1 when a student writes alone the whole

text. In this case, only the average contribution of the

student i, that is x

i,,

, equals 1. Therefore, in order to

make this measure easier to interpret for teachers, we

compute one minus this dispersion, which gives the

balance of contribution indicator in (1).

5.4 Co-writing Indicator

Similarly to the balance of contribution indicator,

which measures students’ contributions at a word

level, the co-writing indicator measures students’

contributions at a sentence level (6). Thus, this indi-

cator requires splitting the text into sentences. To this

end, we use a rule-based (heuristic) approach for sen-

tence segmentation, for its implementation simplicity

(Sadvilkar and Neumann, 2020). We use a sentence

boundary detection tool based on a set of rules called

Golden Rule Set

, hand-designed to determine sen-

tence boundaries, such as punctuation.

https://github.com/diasks2/pragmatic segmenter

When all sentences are written by a single student,

this indicator equals 0 and it equals 1 when all sen-

tences are co-written, in a balanced way, by all stu-

dents.

The co-writing indicator is based on the balance

of contribution indicator, but on a sentence level. Ac-

cording to the contribution matrix, we compute for

each student his average contribution in all sentences.

Thus, let x

i, j,

be, the average contribution of student

i to the sentence j. Then, we obtain a K × N matrix

of average contributions on a sentence level. For each

sentence, we compute the variance of all authors’ av-

erage contributions.

∑

i=1



i, j,

−



. (5)

Similarly to balance of contribution, we penalize

the case where only one student writes a sentence

alone, by computing the ratio between (5) and (4), and

we calculate one minus the resulting quantity which

gives the following dispersion measure

(X) = 1 −

K − 1

∑

i=1



i, j,

−



Therefore, the co-writing indicator is given by

c(X) =

∑

j=1

(X) where p

(6)

is the weight of the sentence j.

5.5 Indicators Property

The indicators present an inequality property. The co-

writing indicator c(X) in (6) is lower or equal than the

balance of contribution indicator e(X) in (1), i.e.,

c(X) − e(X ) =

K − 1

∑

i=1





i,,

−



− (7)

∑

j=1



i, j,

−





≤ 0 .

Indeed, equation (7) is non-positive as soon as

i,,

)

−

∑

l=1

i, j,

)

≤ 0 , (8)

we have

i,,

)

∑

j=1

i, j,

≤

∑

j=1

i, j,

)

. (9)

Thanks to (9), an upper bound for (8) is given by

i,,

)

−

∑

l=1

i, j,

)

≤

∑

l=1

− 1)(x

i, j,

)

which is non-positive since p

≤ 1 .

Proposal of Indicators for Measuring Collaborative Writing in a Digital Learning Environment

499

5.6 Collaborative Writing Strategies

As values of the indicators balance of contribution

and co-writing are continuous and ranged between 0

and 1, we can represent co-written text documents in

a two-dimensional (2D) plane plot (x-axis for balance

of contribution and y-axis for co-writing), see Figure

1. Moreover, as co-writing indicator is always lower

or equal than balance of contribution ((5.5)), the 2D

plane is reduced to a triangle representing the area of

potential plots for documents.

It is then possible to distinguish documents which

are written in summative strategies (ES, RS) or inte-

grative strategies (EI, RI) from documents written in a

mixed strategy (BSI), by considering their location in

the 2D plane. When the co-writing is low the strategy

is summative. Therefore, documents written in sum-

mative strategies are plotted near the x-axis. When

the strategies are integrative, the documents are plot-

ted near the 2D plane diagonal and the balance of con-

tribution and co-writing values are close. Documents

written in a mixed strategy (between summative and

integrative) are plotted in between.

Figure 1: A 2D plane illustrating the relationship between

the indicators and the CW strategies.

6 RESULTS AND DISCUSSION

After providing the mathematical demonstration of

the indicators (RQ1), and the illustration of the re-

lationship between the indicators and the strategies

(RQ2), the questionnaire results allow to evaluate the

teacher interpretations of the strategies and the indi-

cators (RQ3).

We compare the indicator values and levels esti-

mated by teachers to the values computed using for-

mulas (1) and (6). Indicator levels are set among Low

for indicator values in ]0,1/3], Medium for values in

[1/3,2/3[ and High for values in [2/3,1].

Teachers ﬁrst indicate the perceived strategies. Ta-

ble 2 compares the number of teacher responses re-

ceived for each LabDoc and for each strategy. We

report that 9/12 LabDocs are well classiﬁed with a

majority of good responses (8 to 14). LabDocs 1 and

6 received respectively 6 and 5 well classiﬁcations

while the majority of responses are spread-out on the

other strategies. This is probably due to the difﬁculty

for teachers to perceive a mixed strategy such as in

LabDoc 6. LabDoc 1 comprises mathematical for-

mulas, and this may inﬂuence the teacher perception.

For LabDoc 12, teachers answers are more spread out

comparatively to LabDocs 1 and 6, but with 5 good

answers and 1 answer as ”I don’t know”. This can be

explained by the fact that this LabDoc presents a very

low collaboration (mainly written by a single student,

a second contributor modiﬁed some words at its end)

leading to a misinterpretation of the strategy by teach-

ers.

Table 2: Teacher responses regarding the strategies they

perceived. In bold, LabDocs that are misclassiﬁed by the

majority of teachers. The wright strategy is ST.

LabDoc ST ES RS BSI RI EI DK

1 EI 0 2 4 3 6 0

2 ES 11 2 1 1 0 0

3 EI 0 1 1 1 12 0

4 BSI 0 1 8 5 1 0

5 EI 2 0 0 2 11 0

6 BSI 0 8 5 2 0 0

7 ES 12 1 0 0 2 0

8 RI 0 0 4 7 4 0

9 RS 0 8 5 1 0 1

10 ES 14 0 1 0 0 0

11 ES 14 1 0 0 0 0

12 EI 1 4 2 2 5 1

Secondly, teachers estimate for each LabDoc the

level and value of the two indicators. Table 3 provides

for each LabDoc the number of students having con-

tributed to its writing, the machine value and machine

level of each indicator, and the number of teacher re-

sponses for each indicator level.

We also calculate the distance between the ma-

chine value (MV) of an indicator θ ∈ [0,1] and teach-

ers’ estimate, using the Root Mean Standard Devia-

tion (RMSD) given by

RMSD =

∑

(

− θ)

where, for i = 1,...,n,

∈ [0,1] is a teacher’s esti-

mate and n is the number of teachers who respond to

the questionnaire.

Concerning indicator levels, (8/12) of LabDocs

are well classiﬁed by teachers regarding balance

of contribution and (10/12) regarding co-writing.

CSEDU 2023 - 15th International Conference on Computer Supported Education

500

Table 3: Comparing teacher perceptions and estimations of indicator levels and values with the machine values (MV) and

levels (ML). In bold, LabDocs for which a majority of answers are wrong. “NB” is the Number of Students (co-writers).

Balance of contribution Co-writing

LabDoc NB MV RMSD ML L M H MV RMSD ML L M H

1 2 0.49 0.32 M 0 7 8 0.41 0.42 M 0 4 11

2 2 0.87 0.19 H 0 6 9 0.11 0.15 L 13 1 1

3 2 0.84 0.24 H 1 2 12 0.76 0.28 H 1 0 14

4 2 0.99 0.21 H 0 1 14 0.41 0.22 M 3 9 3

5 3 0.49 0.36 M 15 0 0 0.44 0.35 M 12 1 2

6 2 0.99 0.19 H 0 1 14 0.50 0.2 M 7 7 1

7 2 0.51 0.3 M 13 2 0 0.07 0.09 L 14 1 0

8 2 0.99 0.3 H 0 7 8 0.74 0.21 H 0 6 9

9 4 0.48 0.27 M 13 2 0 0.17 0.12 L 13 2 0

10 3 0.92 0.39 H 1 11 3 0.09 0.1 L 13 2 0

11 2 0.29 0.19 L 15 0 0 0.04 0.04 L 15 0 0

12 2 0.14 0.07 L 15 0 0 0.09 0.09 L 14 1 0

RMSD take values in [0.07,0.39] regarding balance of

contribution and in [0.04,0.42] regarding co-writing.

We also report that, for both indicators, mostly

all miss-classiﬁed LabDocs have a value of RMSD

greater than 0.3.

This shows that when teachers succeed to perceive

the correct indicator level for a LabDoc, they estimate

well the indicator value. Moreover, regarding balance

of contribution, most LabDocs for which the indicator

levels are not correctly perceived, are those written by

more than two students. Regarding co-writing, levels

are not correctly perceived in LabDoc 1, this is due to

mathematical formulas in this LabDoc.

From these results, and as an answer for RQ3, we

can say that in some extent, the indicators and the

strategies are interpretable by teachers and succeed

to measure CW. Teacher misinterpretations are due to

the speciﬁcity of some LabDocs, comprising mathe-

matical formulas, or written in mixed strategies or of

a very low collaboration. In fact, teachers examined

manually the LabDocs, and these speciﬁc characteris-

tics led to misconceptions.

7 CONCLUSION AND

IMPLICATIONS

In this work we contribute to research in CW with

new indicators that provide measures of student con-

tributions in a text writing.

A ﬁrst contribution consists in two indicators that

allow to measure how student contributions are equal

(balance of contribution), and how students interact

with each other text contributions (co-writing).

A second contribution consists of identifying CW

strategies, namely summative and integrative strate-

gies that can be useful in pedagogical settings, for

example to distinguish between cooperative (summa-

tive) and collaborative (integrative) works (Onrubia

and Engel, 2009). We showed that these writing

strategies can be derived from the two proposed in-

dicators.

A third contribution is the evaluation of the in-

terpretation of the indicators and strategies, allow-

ing to compare teacher perceptions of the indicators

with machine calculations, and to verify the perceived

strategies. Results showed that teachers have a good

interpretation of the indicators and the strategies. In

fact, teachers were mostly able to estimate correctly

the indicators based on their textual deﬁnitions, as

well as the writing strategies. We conclude that the

indicators may help for monitoring student CW, and

characterizing the degree and strategies of collabora-

tive work.

This work is of scholarly and practical implica-

tions. It provides interesting insights on the design

of learning analytics tools that allow for a meaningful

reporting of CW dynamics between peers. This may

strengthen formative evaluation and provide learners

with quick feedback during their collaborative work

and reports co-writing.

This work is not without limitations. Our ap-

proach can produce misleading results when collabo-

rative documents are composed of mathematical for-

mulas. More advanced text mining methods can im-

prove formulas detection. Moreover, it would be rele-

vant to collect more data with teachers and students to

demonstrate the value of the proposed indicators, and

investigate better insights into CW and collaborative

work.

Future work is motivated to measure student cog-

nitive contributions, by considering the meaning of

sentences composing the text. Our approach can be

improved by considering semantic sentence detection

using machine learning models. This may better in-

form about the collaboration process and go beyond

Proposal of Indicators for Measuring Collaborative Writing in a Digital Learning Environment

501

contribution measures in terms of number of words

added or modiﬁed in a text. Ultimately, it would be

worthwhile to study how to enable teachers to take

actions from useful visualizations of indicators and

strategies, and evaluate their acceptability, usability

and utility.

ACKNOWLEDGMENTS

The project is co-ﬁnanced by the Brittany Region

within the research program SAD and the Finist

ere

Department Council within the research program

APRE. We thank all the teachers having participated

to the study.

REFERENCES

Baker, M. (1994). A model for negotiation in teaching-

learning dialogues. Journal of Interactive Learning

Research, 5(2):199.

Chen, J., Wang, M., Kirschner, P. A., and Tsai, C.-C.

(2018). The role of collaboration, computer use, learn-

ing environments, and supporting strategies in cscl:

A meta-analysis. Review of Educational Research,

88(6):799–843.

d’Ham, C., Wajeman, C., Girault, I., and Marzin-Janvier,

P. (2019). LabNbook, plateforme num

erique support

des p

edagogies actives et collaboratives en sciences

exp

erimentales. In Broisin, J., Sanchez, E., Yessad,

A., and Chenevotot, F., editors, EIAH 2019 : En-

vironnement Informatiques pour l’Apprentissage Hu-

main, Actes de la 9

eme Conf

erence sur les Environ-

nements Informatiques pour l’Apprentissage Humain,

pages 49–60, Paris, France.

Dillenbourg, P. (1999). What do you mean by collaborative

learning?, page 1–19. Elsevier, Oxford.

Dillenbourg, P. and Michael, B. (1996). Negotiation spaces

in human-computer collaborative learning. In Pro-

ceedings of the International Conference on Cooper-

ative Systems, page 12–14.

Hoffmann, C., Mandran, N., d’Ham, C., Rebaudo, S., and

Haddouche, M. A. (2022). Development of actionable

insights for regulating students’ collaborative writ-

ing of scientiﬁc texts. In Educating for a New Fu-

ture: Making Sense of Technology-Enhanced Learn-

ing Adoption, Lecture Notes in Computer Science,

page 534–541, Cham. Springer International Publish-

ing.

Larsen-Ledet, I. and Korsgaard, H. (2019). Territorial func-

tioning in collaborative writing. Computer Supported

Cooperative Work (CSCW), 28(3):391–433.

Li, M. and Zhu, W. (2016). Explaining dynamic interac-

tions in wiki-based collaborative writing. Language

Learning & Technology, 21(2):96–120.

Lowry, P. B., Curtis, A., and Lowry, M. R. (2004). Building

a taxonomy and nomenclature of collaborative writ-

ing to improve interdisciplinary research and prac-

tice. The Journal of Business Communication (1973),

41(1):66–99.

Mandran, N., Vermeulen, M., and Prior, E. (2022). Thedre’s

framework: Empowering phd candidates to efﬁciently

implement design-based research. Education and In-

formation Technologies, pages 1–24.

Olson, J. S., Wang, D., Olson, G. M., and Zhang, J. (2017).

How people write together now: Beginning the in-

vestigation with advanced undergraduates in a project

course. ACM Transactions on Computer-Human In-

teraction, 24(1):4:1–4:40.

Onrubia, J. and Engel, A. (2009). Strategies for collab-

orative writing and phases of knowledge construc-

tion in cscl environments. Computers & Education,

53(4):1256–1265.

Posner, I. R. and Baecker, R. M. (1992). How people write

together (groupware). In Proceedings of the Twenty-

Fifth Hawaii International Conference on System Sci-

ences, volume 4, page 127–138. IEEE.

Ratcliff, J. W. and Metzener, D. E. (1988). Pattern-

matching-the gestalt approach. Dr Dobbs Journal,

13(7):46.

Sadvilkar, N. and Neumann, M. (2020). PySBD: Pragmatic

sentence boundary disambiguation. In Proceedings

of Second Workshop for NLP Open Source Software

(NLP-OSS), pages 110–114, Online. Association for

Computational Linguistics.

Storch, N. (2013). Collaborative Writing in L2 Classrooms.

Multilingual Matters, Bristol, Blue Ridge Summit.

Sun, Z., Lin, C.-H., Wu, M., Zhou, J., and Luo, L. (2018).

A tale of two communication tools: Discussion-forum

and mobile instant-messaging apps in collaborative

learning. British Journal of Educational Technology,

49(2):248–261.

Sundgren, M. and Jaldemark, J. (2020). Visualizing online

collaborative writing strategies in higher education

group assignments. The International Journal of In-

formation and Learning Technology, 37(5):351–373.

Wang, D., Tan, H., and Lu, T. (2017). Why users do

not want to write together when they are writing

together: Users’ rationales for today’s collaborative

writing practices. Proceedings of the ACM on Human-

Computer Interaction, 1(CSCW):1–18.

Wang, F. and Hannaﬁn, M. J. (2005). Design-based

research and technology-enhanced learning environ-

ments. Educational technology research and develop-

ment, 53(4):5–23.

Zhang, M. and Chen, W. (2022). Assessing collaborative

writing in the digital age: An exploratory study. Jour-

nal of Second Language Writing, 57:100868.

Zhang, M., Gibbons, J., and Li, M. (2021). Computer-

mediated collaborative writing in l2 classrooms: A

systematic review. Journal of Second Language Writ-

ing, 54:100854.

CSEDU 2023 - 15th International Conference on Computer Supported Education

502