Analysing the Effectiveness of a Social Digital Repository for

Learning and Teaching: A Fuzzy Comprehensive Evaluation

Akrivi Krouska

, Christos Troussas

, Phivos Mylonas

and Cleo Sgouropoulou

Department of Informatics and Computer Engineering, University of West Attica, Egaleo, Greece

Keywords: Digital Repository, FCE, Fuzzy-Based Evaluation, Open Educational Resources, Social Networking, User

Experience.

Abstract: Since the beginning of the 21

century, Open Education has emerged as an important field in education. Open

Educational Resources (OERs) are closely related to it, which are hosted in Digital Repositories. OERs,

despite their global recognition and their growing number, are not yet established widely. Teachers face many

challenges when they want to use them, including the lack of pedagogical knowledge about their value, the

way to use them, produce them and integrate them into teaching process. The purpose of this research is to

strengthen the movement of OERs and to realise their full potential. To this end, a social digital repository

was developed, for promoting OERs in primary education. This platform aims to create an open and

interactive community of teachers, where through interaction, communication and collaboration, the teachers

will be educated on OERs. The effectiveness of this digital repository is assessed using the fuzzy

comprehensive evaluation model, in order to handle the subjective and imprecise information and better

interpret the results of the survey. The results are very encouraging regarding the adoption of this technology.

1 INTRODUCTION

The Open Educational Resources (OERs) constitute

the recently emerging concept in education, having

attracted the interest of researchers as well as the

recognition and support of international institutions,

governments and funders (Santos-Hermosa, Ferran-

Ferrer, & Abadal, 2017; Xie, Di Tosto, Chen &

Vongkulluksn, 2018). OERs can support the role of

education as an engine of social change (Barrueco &

Termens, 2021), creating knowledge societies and

contributing to the provision of quality, equitable,

open and participatory education. At the same time,

they enhance the academic freedom and professional

autonomy of teachers by expanding the range of

available educational materials (Admiraal, 2022).

Through providing access to a variety of resources,

information and practices, they contribute

significantly to improving education in all sectors and

promote Open Education (Santos-Hermosa, Ferran-

Ferrer, & Abadal, 2017; Admiraal, 2022).

https://orcid.org/0000-0002-8620-5255

https://orcid.org/0000-0002-9604-2015

https://orcid.org/0000-0002-6916-3129

https://orcid.org/0000-0001-8173-2622

OERs are high-quality educational resources that

teachers can use to prepare, improve or supplement

their teaching practice (Admiraal, 2022; Xie, Di

Tosto, Chen, & Vongkulluksn, 2018). The open

licenses that accompany OERs allow for their

modification, a process that fosters creativity and

shapes new content that can be used for personalized

instruction (Blomgren, 2018). Furthermore, OERs

can contribute to the achievement of effective

learning (Tang & Bao, 2020), as they stimulate

learners' interest in learning and increase satisfaction

from the learning experience (Chen, 2020).

Educators and learners spend many hours of

creating educational materials, searching, locating,

acquiring and reusing, with or without revisions

(Blomgren, 2018). OERs are a viable solution for

them to address the challenges of access, quality and

cost (Blomgren, 2018; Chen, 2020). However,

despite their global recognition and the growing

number of OERs, the levels of their use remain low

(Admiraal, 2022; Ossiannilsson et al., 2020; Schuwer

Krouska, A., Troussas, C., Mylonas, P. and Sgouropoulou, C.

Analysing the Effectiveness of a Social Digital Repositor y for Learning and Teaching: A Fuzzy Comprehensive Evaluation.

DOI: 10.5220/0012741800003687

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2024), pages 777-783

ISBN: 978-989-758-696-5; ISSN: 2184-4895

777

& Janssen, 2018) or may be hidden (Beaven, 2018),

meaning that teachers find resources online, receive

them from colleagues or already have resources in

their personal collections without being aware of

OERs (Admiraal, 2022). On the one hand, teachers

face many challenges when they want to use OERs,

and on the other hand, not enough attention has been

paid so far to their improvement and promotion

(Chen, 2020). In fact, according to Tang, Lin, & Qian

(2021), the levels of acceptance of OERs remain low

especially in primary and secondary education (K-12

education). Other reasons why teachers' adoption of

OERs is limited are their low quality (Huang, Tlili, et

al., 2020), their insufficient pedagogical value

(Abramovich & McBride, 2018) and the difficulty of

finding resources that are up-to-date and thematically

relevant to the subject they teach (Admiraal, 2022).

In order to overcome the above limitations, this

paper introduces a social digital repository for

promoting OERs in primary education. A digital

repository refers to an online storage system or

database designed to collect, manage, preserve, and

provide access to digital content and assets. Digital

repositories can host and provide access to OERs,

serving as platforms where educators and institutions

share educational materials openly. The developed

platform aims to motivate and facilitates teachers to

be actively engaged in using, creating and sharing

OERs. As such, this social digital repository provides

a user-friendly interface to search, develop and rate

content, as well as its social networking functionality

enables the communication and collaboration among

the repository community. For assessing its

effectiveness, a fuzzy-based evaluation process was

conducted. The fuzzy evaluation method is used in

order to deal the subjective and imprecise information

and better interpret the results of the survey.

2 SOCIAL DIGITAL

REPOSITORY OVERVIEW

The developed social digital repository belongs to the

category of Thematic Repositories, and specifically to

the subcategory of independent repositories, as it

hosts content related to a specific topic and is initiated

by the authors of this paper. It is aimed at primary

education teachers, hosting OERs that can be used to

prepare, improve or supplement their teaching

practice.

The repository’s OERs are characterized by

heterogeneity in terms of their educational and

technological characteristics on the basis of which

they are organized into categories. In particular, the

repository includes interactive exercises, quizzes,

crosswords, presentations, videos, images,

assessment tests, worksheets, etc. To facilitate

searching and selecting them, OERs have been

organized into six main categories according to the

grade of primary education to which they are

addressed and into subcategories based on the subject

in which they fall. In addition, for the categorization

of OERs, one or more tags, i.e., keywords or phrases,

have been added to them, which act as descriptive

elements (Troussas, Krouska, & Sgouropoulou,

2021). The open licenses that have been chosen to

accompany OERs are Creative Commons (CC)

licenses and, in particular, three types of licenses have

been utilized: Attribution – Non-Commercial – Share

Alike (CC BY-NC-SA), Attribution – Non-

Commercial – Share Alike 3.0 Greece (CC BY–NC–

SA 3.0 GR), and Public Domain Dedication (CC0).

Each OER is associated with metadata, i.e., a set

of data that identifies it. The metadata schema of the

repository is based on the Dublin Core metadata

schema, which is the responsibility of the Dublin

Core Metadata Initiative IEEE Learning Technology

Standards Committee. In particular, each resource is

accompanied by its title and a representative image

(thumbnail), as well as metadata that constitute either

general data or data related to its classification. The

general elements include a brief description of the

content, keywords, the date of creation, the author,

the source and the licence under which the OER is

distributed. The classification elements include the

subject area in which the resource belongs, the

class(es) to which it is addressed and the type of

resource. Each AEP can be rated by the repository

members by clicking on one of the five stars to

indicate its quality or can be commented.

The social networking feature of the repository

enables the members to the following actions. They

can exchange public/private messages with each

other, make friends, create groups and become

members of them. Furthermore, they can make status

updates (posts), publish and share OERs they have

developed or customized and receive feedback on

them. Finally, they can identify, rate, comment and

share opinions on OERs and share them on their

personal profiles. The members of the repository can

create and/or join groups based on their interests.

Groups are aggregations of members, posts and any

other user-generated content. The activity stream

records all kinds of activity, such as blog posts, new

friendships and blog comments. A central aspect of

the user experience is receiving notifications.

ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering

778

3 FUZZY COMPREHENSIVE

EVALUATION FRAMEWORK

In order to assess the usability of the developed social

digital repository, the fuzzy comprehensive

evaluation method was used. This model incorporates

fuzzy logic to handle uncertainty and imprecision in

the evaluation process. In particular, it provides a

fuzzy mapping process of each evaluation criterion,

i.e., content quality, usability, social engagement, and

educational effectiveness, to a set of linguistic

variables, i.e., “high”, “medium” and “low”.

Establishing these fuzzy sets allows a specific rating,

e.g. the average rate of 4.34 in a question on 5-point

scale, to be associated with both categories of “high”

and “medium” based on the degrees of membership.

As such, a better understanding of the questionnaire

feedback is obtained by transforming the quantitative

rating into qualitative one using fuzzy sets.

The steps of the fuzzy comprehensive evaluation

method are illustrated in Fig. 1.

Figure 1: Steps of Fuzzy Comprehensive Evaluation.

3.1 Define Evaluation Criteria

The evaluation criteria defined for assessing the

social digital repository were four, namely content

quality, usability, social engagement and educational

effectiveness. Each criterion was measured based on

4 indicators, as shown in Table 1. The selection of

these criteria was made because they cover all the

aspects of the social networking-based digital

repository for learning and teaching, contributing to a

comprehensive evaluation of its overall features and

capabilities.

The set of the selected n evaluation criteria can be

represented as a vector, where n = 4:

C = {c

, c

} = {content quality, usability,

social engagement, educational effectiveness}

Moreover, the indicators are represented as I =

, I

, …, I

Table 1: Evaluation Criteria and Indicators.

Criteria Indicators

Content

quality (CQ)

1. Is the content rating aligned with

its

ualit

2. Does the repository offer a diverse

ran

e of learnin

ects?

3. Do the metadata of the learning

objects correspond properly to

their content?

4. Is the content presented accurate

and up-to-date?

Usability (U) 5. Is the interface use

-friendl

6. Do you enjoy interacting with the

ositor

7. Can you easily find and access the

educational resources

ou need?

8. Do you like the way with which

the learnin

ects are

resented?

Social

engagement

(SE)

9. Do you enjoy the social interaction

feature provided, such as

commenting or group discussions?

10. Do you find the interactive features

of the social digital repository

engaging and helpful?

11. Do the social interaction features

facilitate the communication and

knowledge sharing among

repository community?

12. How responsive and supportive is

the repository community in

providing feedback or assistance

when needed?

Educational

effectiveness

(EE)

13. Does the repository contribute to

measurable learning outcomes?

14. Is the design of the learning objects

aligned with effective pedagogical

principles and instructional

strategies?

15. Is the content aligned with

educational objectives?

16. Do you find the repository an

effective tool for learning and

teachin

Step 1

• Define Evaluation Criteria

Step 2

• Establish Fuzzy Sets & Linguistic Variables

Step 3

• Determine Membership Functions

Step 4

• Define Fuzzy Rules

Step 5

• Determine Criteria Weights

Step 6

• Collect Data

Step 7

• Apply FCE

Step 8

• Report Results

Analysing the Effectiveness of a Social Digital Repository for Learning and Teaching: A Fuzzy Comprehensive Evaluation

779

3.2 Establish Fuzzy Sets and Linguistic

Variables

The indicators are aligned in the same m grade levels,

being represented as a vector, where m=3:

V = {v1, v2, v3} = {low, medium, high}

As such, the fuzzy set of each indicator has the

same linguistic variables, namely low, medium and

high. The evaluation process aims to provide a

mapping from I to V. For each criterion c

the fuzzy

mapping of its indicator I

to grade levels vector V is

represented by the vector:

Rik = {r

ik1

, r

ik2

, …, r

ikm

where r

ikt

indicates the fuzzy membership degree

of the indicator k of criterion i to the grade level t. For

example, if R

= {0, 0.3, 0.7}, it means that the

indicator “Is the content rating aligned with its

quality?” of the criterion “Content quality” has a

membership degree of 0.3 in the “medium” level and

0.7 in the “high” level.

As such, the fuzzy matrix of each criterion i is

represented as follows:

𝑅



=

𝑟



…𝑟



………

𝑟



…𝑟



, where k = 4 and m = 3

3.3 Determine Membership Functions

In this paper, each indicator is evaluated by the

participants using a 5-point scale. Therefore, a score

emerged from the average rating of all participant is

assigned to each indicator. The value of this score

ranges from 1 to 5. Then, the triangular membership

function is used to calculate the degree of each grade

level, as follows:

𝜇





𝑥



⎩

⎪

⎨

⎪

⎧

0, 𝑥1

𝑥1

, 1𝑥2

3𝑥

, 2𝑥3

0, 𝑥3

𝜇





𝑥



⎩

⎪

⎨

⎪

⎧

0, 𝑥2

𝑥2

, 2𝑥3

4𝑥

, 3𝑥4

0, 𝑥4

𝜇

ℎ𝑖𝑔ℎ



𝑥



⎩

⎪

⎨

⎪

⎧

0, 𝑥3

𝑥3

, 3𝑥4

5𝑥

, 4𝑥5

0, 𝑥5

Fig. 2 shows the triangular membership functions

scheme.

Figure 2: Triangular membership functions representation.

3.4 Define Fuzzy Rules

In this step, a set of fuzzy rules are established to

define the relationships between evaluation criteria

and overall evaluation of system’s effectiveness,

named Overall Effectiveness (OE). The fuzzy rules

are designed based on the authors’ knowledge as

expert on the field, regarding how each criterion

contribute to the overall evaluation.

A sample of the defined fuzzy rules is the

following.

 IF CQ = medium and U = high and SE =

medium and EE = medium THEN OE =

medium

 IF CQ = high and U = high and SE = medium

and EE = medium THEN OE = high

 IF CQ = medium and U = medium and SE =

low and EE = low THEN OE = low

 IF CQ = high and U = medium and SE =

medium and EE = medium THEN OE =

medium

 IF CQ = low and U = medium and SE = low

and EE = medium THEN OE = low

3.5 Determine Criteria Weights

Determining criteria weights is a crucial step, since

relative importance is assigned to different evaluation

criteria. These weights affect the decision-making in

the overall assessment process. Each criterion's

weight indicates its contribution to the final

ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering

780

evaluation, allowing for a more context-aware

analysis. The sum of the weights should be 1 to

maintain normalization.

The criteria weights can be determined through

various methods, including expert opinions, surveys,

or analytic hierarchy process (AHP). In this paper, the

weights are defined based on authors’ expertise. As

such, the weights of each indicator I

of each criterion

are the following:

= [0.325, 0.155, 0.295, 0.225]

= [0.255, 0.155, 0.305, 0.285]

= [0.175, 0.235, 0.315, 0.275]

= [0.175, 0.315, 0.235, 0.275]

4 EXPERIMENTAL WORK,

RESULTS & DISCUSSION

The developed social digital repository was used by

40 teachers at public primary schools in Greece,

during the school year of 2022-2023. Table 2

illustrates the demographical characteristics of the

participants.

Table 2: Demographic characteristics of 40 participants.

Characteristic Percentage

Gender Female 57.5%

Male 42.5%

Age <30 12.5%

30 – 40 42.5%

40 – 50 25%

>50 20%

Computer

literacy

High 70%

Medium 20%

Low 10%

Educational

level

Bachelor 27.5%

Master 67.5%

PhD 5%

The participants were interacting with the system,

utilizing the provided learning objects to their

teaching, commenting them, uploading their learning

objects, communicating with repository community

through private and public discussion rooms. At the

end of the school year, a 5-point Likert scale

questionnaire was delivered through Internet to the

participants, including the 16

indicators. All

participants corresponded positively to the process,

answering the questionnaire. Table 3 shows the

results of the survey.

Table 3: Results of survey.

Criterion Indicator

oint scale

Avg

1 2 3 4 5

I1 4 5 11 10 10 3.425

I2 2 2 7 11 18 4.025

I3 4 8 7 11 10 3.375

I4 3 6 11 11 9 3.425

I5 8 9 7 9 7 2.95

I6 4 7 8 10 11 3.425

I7 6 10 10 7 7 2.975

I8 3 3 9 9 16 3.8

I9 2 2 5 12 19 4.1

I10 2 3 6 14 15 3.925

I11 1 1 9 13 16 4.05

I12 4 6 11 9 10 3.375

I13 3 9 11 8 9 3.275

I14 4 9 8 9 10 3.3

I15 4 6 7 9 14 3.575

I16 5 3 10 13 9 3.45

Based on the results of the survey, the following

fuzzy matrixes are structured:

𝑅



= 

0.575

0.625

0.575

0.425

0.975

0.375

0.425



𝑅



= 

0.05

0.025

0.95

0.575

0.975

0.2

0.425

0.8



𝑅



= 

0.075

0.625

0.9

0.925

0.95

0.375



𝑅



= 

0.725

0.7

0.425

0.55

0.275

0.3

0.575

0.45



After calculating the above fuzzy sets, the fuzzy

rules and the weights are applied to estimate the degree

of each criterion. The fuzzy variables emerged from

the required calculations are described in Table 4.

The content quality is rated in medium and high

level. This suggests that the repository is perceived

positively in terms of representative content rating,

diversity of learning objects, metadata correspondence,

and the accuracy of presented content. The repository's

strength lies in providing accurate and diverse content,

aligning well with user expectations.

Analysing the Effectiveness of a Social Digital Repository for Learning and Teaching: A Fuzzy Comprehensive Evaluation

781

Table 4: Fuzzy evaluation results.

Criterion Indicator Linguistic Variables

based on Survey

Overall

Evaluation

CQ I1 Medium and High Medium

and High

I2 High

I3 Medium and High

I4 Medium and High

U I5 Low and Medium Medium

I6 Medium and High

I7 Low and Medium

I8 Medium and High

SE I9 High High

I10 Medium and High

I11 High

I12 Medium and High

EE I13 Medium and High Medium

and High

I14 Medium and High

I15 Medium and High

I16 Medium and High

The overall fuzzy result for usability is medium,

with variations in user-friendliness and ease of access

across different indicators. While interactions are

generally enjoyable, there are areas of improvement

identified, such as a less user-friendly interface and

challenges in easily accessing educational resources.

The repository may benefit from addressing specific

usability concerns to enhance the overall user

experience.

The high fuzzy result for social engagement

indicates that users find the social interaction features

enjoyable and engaging. Additionally, the repository

is perceived to facilitate effective communication and

knowledge sharing within the community. The strong

social engagement suggests that the repository is

successful in fostering a collaborative and interactive

environment.

The fuzzy result for educational effectiveness is

medium to high, reflecting positive perceptions

regarding the repository's contribution to measurable

learning outcomes, alignment with pedagogical

principles, and effectiveness as a learning and

teaching tool. The repository appears to be a valuable

resource for supporting educational objectives and

learning outcomes.

5 CONCLUSIONS

OERs and digital repositories play a crucial role in

modern education, offering various benefits that

contribute to the accessibility, flexibility, and quality

of learning. As such, this paper presents a social

digital repository for promoting OERs in primary

education. The paper aims to assess the effectiveness

of the digital repository regarding the content quality,

the usability, the social engagement and the

educational effectiveness, using the fuzzy

comprehensive evaluation model.

The fuzzy evaluation results show the social

digital repository's strengths and areas for

improvement. The evaluation across different criteria

and indicators highlights the multifaceted nature of

the repository's performance. The social digital

repository generally performs well across content

quality, usability, social engagement, and educational

effectiveness. The findings can guide further

enhancements to optimize user experience, content

quality, and educational impact.

Part of our future work is to improve the social

digital repository functionalities in order to increase

its effectiveness and user experience. Another future

plan is the enhancement of fuzzy comprehensive

evaluation model with the application of other

weighting techniques, the application of further

evaluation frameworks and the comparison of the

repositories with other ones.

REFERENCES

Abramovich, S., & McBride, M. (2018). Open education

resources and perceptions of financial value. The

Internet and Higher Education, 39, pp. 33-38. doi:

10.1016/j.iheduc.2018.06.002

Admiraal, W. (2022). Typology of educators using Open

Educational Resources for teaching. International

Journal on Studies in Education (IJonSE), 4(1), pp. 1-

23. doi: https://doi.org/10.46328/ijonse.60

Barrueco, J., & Termens, M. (2021). Digital preservation in

institutional repositories: a systematic literature review.

Digital Library Perspectives, 38(2), 161-174.

doi:10.1108/DLP-02-2021-0011

Beaven, T. (2018). ‘Dark reuse’: an empirical study of

teachers’ OER engagement. Open Praxis, 10(4), 377.

doi:10.5944/openpraxis.10.4.889

Blomgren, C. (2018). OER Awareness and Use: The

Affinity Between Higher Education and K-12.

International Review of Research in Open and

Distributed Learning, 9(2), pp. 55-70.

doi:10.19173/irrodl.v19i2.3431

Chen, H.-J. (2020). Developing an OER Website and

Analyzing Its Use. English Teaching & Learning, 44,

pp. 451-461. doi: https://doi.org/10.1007/s42321-020-

00067-x

Huang, R., Tlili, A., Chang, T.-W., Zhang, X., Nascimbeni,

F., & Burgos, D. (2020). Disrupted classes, undisrupted

learning during COVID-19 outbreak in China:

application of open educational practices and resources.

ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering

782

Smart Learning Environments, 7(19). doi:

https://doi.org/10.1186/s40561-020-00125-8

Ossiannilsson, E., Zhang, X., Wetzler, J., Gusmao, C.,

Aydin, C., Jhangiani, R., . . . Makoe, M. (2020). From

Open Educational Resources to Open Educational

Practices For resilient sustainable education. Distances

et Médiations des Savoirs. doi:10.4000/dms.5393

Santos-Hermosa, G., Ferran-Ferrer, N., & Abadal, E.

(2017). Repositories of Open Educational Resources:

An Assessment of Reuse and Educational Aspects.

International Review of Research in Open and Distance

Learning, 18(5), pp. 84-120. doi:10.19173/irrodl.v18

i5.3063

Schuwer, R., & Janssen, B. (2018). Adoption of Sharing

and Reuse of Open Resources by Educators in Higher

Education Institutions in the Netherlands: A Qualitative

Research of Practices, Motives, and Conditions.

International Review of Research in Open and

Distributed Learning, 19(3), pp. 151- 171. doi:

https://doi.org/10.19173/irrodl.v19i3.3390

Tang, H., & Bao, Y. (2020). Social Justice and K-12

Teachers’ Effective Use of OER: A Cross-Cultural

Comparison by Nations. Journal of Interactive Media in

Education, 1(9), pp. 1-13. doi:10.5334/jime.576

Tang, H., Lin, Y.-J., & Qian, Y. (2021). Improving K-12

Teachers’ Acceptance of Open Educational Resources

by Open Educational Practices: A Mixed Methods

Inquiry. Education Tech Research, 69, pp. 3209– 3232.

doi: https://doi.org/10.1007/s11423-021-10046-z

Troussas, C., Krouska, A., & Sgouropoulou, C. (2021).

Improving Learner-Computer Interaction through

Intelligent Learning Material Delivery Using

Instructional Design Modeling. Entropy, 23(6), p. 140.

doi: https://doi.org/10.3390/e23060668

Xie, K., Di Tosto, G., Chen, S.-B., & Vongkulluksn, V.

(2018). A systematic review of design and technology

components of educational digital resources.

Computers & Education, 127, pp. 90-106. doi:

https://doi.org/10.1016/j.compedu.2018.08.011

Analysing the Effectiveness of a Social Digital Repository for Learning and Teaching: A Fuzzy Comprehensive Evaluation

783