Digital Device and Mathematics: Multilevel vs Machine Learning Models

for Value-added Ranking in Italy

Donatella Papa

National Institute for the Evaluation of the Education and Training Educational System (INVALSI), Rome, Italy

Keywords:

Education, Mathematics, ICT, Multilevel, Machine Learning.

Abstract:

With the COVID-19 pandemic and the development of distance education programs, digital learning is popular

and strategic in many learning ﬁelds. The deployment of Information and Communications Technology and its

impact on both national and international learning programs are becoming increasingly signiﬁcant. This study

seeks to explore in the Italian context both the effectiveness of digital learning in Mathematics Education and

which features and how affect value-added at the classroom level. To explore Information and Communica-

tions Technology contribution and value-added scoring, the study takes into consideration the analytical power

of classical multilevel models concerning the predictive power of different types of machine learning models.

The study aims to investigate how Information and Communications Technology, and related concepts, im-

pact the Weighted Likelihood Estimates in Mathematics for students in the lower secondary school, using data

from the INVALSI of the school year 2017/2018. The main ﬁnding is that Personal Computer ownership at

home plays an important role in mathematical learning. Finally, a machine learning model incorporated in the

educational domain can be an interesting starting point for developing class-predictive policies.

1 INTRODUCTION

The idea of digital learning has recently caught the

attention of the general public to the gap created by

online learning throughout the COVID-19 pandemic

for lots of students around the world. Distance or on-

line learning needs careful exploration to develop an

overall read of the actions taken and to be undertaken

at the government and school levels. At the sunrise

of the digital age, it’s essential to know what pro-

portion and how digital learning is afﬁrmed in terms

of the possession and use of technologies, to avoid

new types of exclusion from the numerous areas of

information and knowledge society. The strategic ob-

jective formalized with the investments in hardware

and software made by Italy in recent years and in the

function of the EU2020 Lisbon Strategy, which in the

school began through the Digital School Plan with

particular attention to some regions of Southern Italy,

which had the aim of faster and wider dissemination

of the Information and Communications Technology

(ICT) among students at all school levels. If Europe is

moving towards the so-called gigabit society, starting

not by a chance from the places where the knowledge

of children and young people are formed to achieve

https://orcid.org/0000-0001-5261-3026

an increasingly interconnected society, Italy remains

in the latter places, after a small leap forward in 2019,

with the 25th place in 28th of the DESI 2020 rank-

ing (Commission, 2020). Exploring the involvement

of macro-territorial areas, moreover, is a key issue to

describing and understanding the emergence of differ-

entiated ICT practices. The Istat data from the “Mul-

tipurpose survey on households and ICT” 2018 un-

doubtedly has highlighted the existence of a differ-

ent acceleration in the digitization process that runs

through the country and with Italian families.

Figure 1: Family without the Internet at home by reason

(ISTAT, 2018).

In southern Italy, 41.6% of households say they

do not own a computer at home (compared to an av-

erage of about 30% in other areas of the country) and

Papa, D.

Digital Device and Mathematics: Multilevel vs Machine Learning Models for Value-added Ranking in Italy.

DOI: 10.5220/0011042700003182

In Proceedings of the 14th International Conference on Computer Supported Education (CSEDU 2022) - Volume 2, pages 171-178

ISBN: 978-989-758-562-3; ISSN: 2184-5026

171

only 14.1% have at least one computer available for

each component. Among the families who declared

that they did not have an Internet connection at home,

about 30% of cases reside in the South, declaring

that the absence is determined by economic reasons

in 10% of cases, while more than 60% declare that

they do not have an Internet connection because no

one would have been able to use it (in a greater per-

centage in the northeast) and about 20% because they

do not ﬁnd it useful or interesting.

The purpose of this study was to examine the hier-

archical relation between students, class, and the terri-

torial context about the school learning of Mathemat-

ics in the last grade of lower secondary school for Ital-

ian context regarding the use of technology devices

(please see below the research question Q1). An in-

depth study is needed to improve discussions on the

impact of digital technologies in the classroom and

at home on mathematics education in Italy. More-

over, the focus of the policy should also be on the op-

portunity to have access to digital resources provided

by ICT for learning and territorial development. At

the same time, the purpose is to evaluate the value

added by the classroom and the predictive potential

of machine learning models is explored, comparing

them with the traditional multilevel regression ap-

proach (please see below the research question Q2).

2 THE DIGITAL DIVIDE IN

EDUCATION

The literature suggests that having an Internet con-

nection and a Personal computer at home can have

positive effects on school performance (Wittwer and

Senkbeil, 2008). However, the more recent studies

of the impact of instructional computer use in school

settings have found mixed results and that vary across

countries (Eickelmann et al., 2012; Hu et al., 2018;

Carstens et al., 2021). Indeed, it is not the sim-

ple access to home or school to ICT that can inﬂu-

ence in positive terms school learning (which instead

would give negative outcomes) as much as the rela-

tive use and control that is provided. From this point

of view, inequalities in access to Personal comput-

ers and the Internet connection and their implications

have been known for some decades with the debated

term of the digital divide (NTIA, 1995; DiMaggio

et al., 2001). There are differences in their origin and

are attributable, at the same time, to those who ar-

gue that these lacks are linked to the structural nature

of the students, such as the socio-economic and cul-

tural status of the original family, and those who ar-

gue that the absences are subjective; hence, all these

facts are partly linked to the different ways in which

knowledge is processed and internalized by the differ-

ent subjects. Overcoming this dichotomy, in the per-

spective of the study of digital inequalities, the focus

is on the uses that are introduced and the skills that

can be exploited (DiMaggio et al., 2001; van Dijk,

2005). There is no doubt, that technology contributes

to a better distribution of knowledge. The complexity

of its use and fruition, as well as the resulting costs,

can intensify existing social inequalities, or, as the au-

thor reasons, large groups of misﬁts, people who are

ill-adapted to the information society. Among the in-

dividual variables, beyond gender and the migratory

background, the prevailing language spoken in the

family will be considered because of what van Dijk

also argued that for non-natives (p. 177): “The fu-

ture is considerably less bright for migrants and eth-

nic minorities with low education in a network society

dominated by natives and ethnic majorities. Usually,

they lack digital skills and, what is worse, they do not

speak or command the native or dominant language

sufﬁciently. So, they run the risk of missing out on

the technical and communicative skills required in a

network society. The major handicap is having in-

sufﬁcient command of the dominant language. The

only exception is to be able to speak and write in En-

glish”(van Dijk, 2005).

2.1 What Is the Relationship between

Digital Learning and Learning

Mathematics in the Classroom?

From the review of the international literature, about

the relationship between digital learning and learn-

ing in the classroom, the students have greater difﬁ-

culty using the computer equipment effectively when

teachers do not work in the classroom to develop

shared practices in the use of technology (Kozma,

2003; Balanskat et al., 2006; Burns, 2013; Drijvers,

2015; Bray and Tangney, 2017; Viberg et al., 2020).

In the present work, the exploration will start with

the creation of operationalized indicators from the fre-

quency of PC usage in the classroom with the super-

vision of the Mathematics teacher (INVALSI, 2017;

Rutkowski et al., 2013). The purpose is to distinguish

between established practices of PC usege and their

connotations (positive and/or negative) on individual

Mathematics learning and the relative inﬂuence on

classrooms/peers. For the cooperative learning, the

students tend to enjoy Mathematics and this fun mo-

tivates them to learn (Davidson, 1990). The impor-

tance of computer-supported learning is an emerging

branch of pedagogical sciences that deals with study-

ing how people can learn with the help of computers.

CSEDU 2022 - 14th International Conference on Computer Supported Education

172

2.1.1 Machine Learning Models in Education

As reported in the work ”Contrasting Classical and

Machine Learning Approaches in the Estimation

of Value-Added Scores in Large-Scale Educational

Data”(Levy et al., 2020), the computational sciences

and social sciences have been collaborating for some

time to achieve greater and better results in the ap-

proaches of machine learning in the educational ﬁeld,

because of the growth potential in a wide range of

areas of the society of supervised and unsupervised

machine learning models. The school and didactic

learning outcomes were evaluated with linear and/or

nonlinear regression methods, such as Support Vector

Machine, Random Forest, Extreme Gradient Boost-

ing, Neural Network, etc. Despite the wide vari-

ety of regression models, there is still no consensus

on which model is the ”best” (Papadogiannis et al.,

2020). Even with this, it may be possible to intervene

with institutions in advance to limit and/or manage

the phenomenon of students with a lesser chance of

effective learning. This includes their socio-economic

and cultural status, territorial and scholastic features,

thereby improving the school system’s effectiveness

in terms of student performance. There are several

possible applications of prediction data derived so far

for machine learning in the educational context (Pa-

padogiannis et al., 2020). Due to the implementa-

tion of policy actions in lower and upper secondary

schools, machine learning models will be developed:

to prevent students from leaving school too early; to

provide feedback to assist at-risk students; to differ-

entiate didactic planning at the classroom/peer level

in terms of mobile technology, etc.

3 RESEARCH PROBLEM &

OBJECTIVES

Due to the global economic crisis COVID-19 is ex-

periencing, the use of ICT and distance learning has

become a necessary teaching method. This implies

the application of new didactic strategies and peda-

gogical approaches to improve strategic skils. Fur-

thermore, it is just as important to make sure no one

is left behind in this digital competition and to explore

at the classroom context to understand which one of

the exogenous factors (from the scholastic) may con-

tribute to widening the gap. The educational process

is improved by exploring whether and to what extent

digital know-how developed at home and at school

have an equal impact on mathematical ability. It is

important to distinguish between existing practices

and connotations (positive and/or negative) concern-

ing individual mathematics learning, as well as its in-

ﬂuence on classes and peers, in collaboration activ-

ities with the mathematics teacher, supported by the

computer or in teacher continuing education. Specif-

ically, this work uses data from the National Large-

Scale Computer-Based Survey conducted by the Na-

tional Institute for the Evaluation of the Education

System (INVALSI) to assess the knowledge and skill

of eighth grade students in mathematics for the school

year (SY) 2017/2018. The following research ques-

tions are addressed in this study: Q1. Combined with

“exogen” features to the scholastic institution, for ex-

ample, gender, socio-economic and cultural status,

linguistic background and other contextual aspects,

how and how much ICT features contribute to the

value-added ranking? Q2. For the Italian context,

could be the analytic power of traditional multilevel

models relaunched by the predictive rule of different

types of machine learning approaches?

4 MATERIAL AND TOOLS

Although the National INVALSI test produces cen-

sus data, it also extracted a two-stage probabilistic

sample: in the ﬁrst stage, the schools are sampled

and in the second one, two classes for each school

selected from the previous stage (INVALSI, 2018).

The sample extracted for the SY 2017/2018 is 29

359 lower secondary school students, representative

of the general population of 567 986 students. It is

decided to proceed with the analysis of the sample

rather than the entire population for the need to an-

alyze the features of teachers, contained in the ques-

tionnaire for Mathematics teachers administered only

to sample classes participating in the National Sur-

veys (INVALSI, 2017). What’s more, it was neces-

sary to study to the s.y. 2017/2018, instead of the

most recent, because that teacher’s questionnaire was

not administered (neither to the population nor to the

sample). Which exogen variable is considered to eval-

uate value-added score is a long-term question. First

of all, it is necessary to point out that the dependent

variable and the independent variables adopted in this

paper have different nature, detection, and synthesis

procedures. The data is collected through different

sources: the school administration, teacher and the

student questionnaire and the standardized INVALSI

test. The independent variables about students are:

• the school career (here ”late enrolled student”);

• the measure of socio-economic and cultural status

(here ”ESCS”, corrected by the exclusion of the

two items of a computer and internet ownership at

home);

Digital Device and Mathematics: Multilevel vs Machine Learning Models for Value-added Ranking in Italy

173

• PC availability at home and Internet availability at

home (here ”PC/Internet avaiability at home”)

• the immigration background (here ”native”,

”I gen imm” and ”II gen imm”);

• the gender (male),

• the language is spoken at home (italian, here

”lang”),

• the math score in grade 7 (the dichotomized writ-

ten scoring in Mathematics at grade 7, where in-

sufﬁcient value is equal to 1 otherwise 0, here

”math score grade 7”).

For all these variables, macro variables have also

been prepared at the class level by aggregations (with

mean) of the subset of students that make up the

same class. By the teacher questionnaire, seven vari-

ables are administered for detecting the frequency of

ICT use by the Mathematics teacher in the classes,

such as Computer, multimedia interactive whiteboard

(MIW), educational and computer software, digital

camera, tablet, and smartphone. These variables

are synthesized through Principal Component Anal-

ysis (PCA). The ﬁrst component is responsible for

high scores in the variables related to teaching activ-

ities (the MIW, educational and computer software,

here ”PC use Classroom didactics”); while the sec-

ond one is linked to high scores in other activities

(use of digital camera, tablet and smartphone, here

”PC use Classroom other”). Other ten variables are

administered through the items of the Likert scale

for the updating activities in the last 2 SY (2016-17

and 2017-18) declared by the classroom Mathematics

teacher. Also, these variables are summarized through

PCA. Only one component is relevant for the present

study and considers the updating of teachers in the

last two school years for Didactics and the Integration

of Information Technologies in the teaching of Math-

ematics (here ”teacher update ICT”). Furthermore,

also a dummy variable related to the use in class of

peer activities is selected from the teacher question-

naire. Non-response data is more common with self-

administered questionnaires when speciﬁc questions

are left unanswered. In this study, albeit with a re-

duced incidence, these missing data occurred for data

such as administrative and/or teacher or student ques-

tionnaires. The missing data are concentrated among

the categorical variables and have a percentage of less

than 10%. In this regard, it was deemed best to use the

single imputation technique, which focuses on substi-

tuting each missing value using mode as a statistical

method.

4.1 Analysis of the WLE Distribution

The dependent variable considered is the mathemat-

ical Weighted Likelihood Estimates (WLE) score at

the INVALSI scale (with Mean = 200 and Standard

Deviation = 40). The estimate is based on the concur-

rent calibration of the INVALSI data from the Main

Study based on the Rasch model of measurement

(Rasch, 1960). As described in the INVALSI tech-

nical report (INVALSI, 2018), standardized weakly

parallel multiple test-forms were assembled from a

large item bank, developed by INVALSI and based

on the Rasch unidimensional model of measurement.

Taking into account the main international research

on mathematics education, the theoretical framework

is aligned with the National System of Evaluation

(SNV). Further details on grade 8 assessment de-

sign, item bank characteristics, test-form assembly,

and psychometric properties are reported in the cor-

responding chapter of the INVALSI technical report

(Desimoni, 2018). For the inferential models, the data

of the unweighted sample will be used. This is be-

cause the main goal of the machine learning model

is different from weighted analyses obtained with the

adjusted population estimates. Indeed, the machine

learning techniques were not developed to explain re-

lationships, but for predictive purposes.

5 RESULTS

This work seeks to identify which machine learning

regression model is closest to the results obtained by

the multi-level hierarchical model to improve the ed-

ucational system with more innovative and proactive

predictive, organizational, and institutional develop-

ments. For this evaluation, the ﬁrst step was either the

adaptation to the INVALSI data and the previously ex-

posed hypotheses of the procedure exposed by Levy

and colleagues (Levy et al., 2020) through the R

Project 4.1.0 software (with the ”lmer4”, ”caret”,

”nlme”, “kernlab”, ”nnet”, ”xgboost” libraries). Be-

fore moving on to the analysis of the results, it is nec-

essary to point out that in the far and the ﬁtting train-

ing we have opted for a reduction to 10 k-fold valida-

tion for each regression and tree model. As noted in

the literature, repeated cross-validation of ”k-folds”

has the advantage of improving the estimation of the

average performance of the model at the cost of adopt-

ing and evaluating many more models (James et al.,

2013). It is also recalled that for the implementa-

tion of machine learning models, intending to predict

Mathematics performance, this work has drawn on the

set of independent variables of the multilevel regres-

CSEDU 2022 - 14th International Conference on Computer Supported Education

174

sion model described above, preferring the simpli-

ﬁed multilevel regression model to random intercept.

From a ﬁrst analysis, as shown in Figure 2, the dif-

ferent models provide results very close to each other

with a better-predicted error, as well as for the mul-

tilevel model (abbreviated as hlm), for the Extreme

Gradient Boosting (boostin) and the Random Forest

(rf). The peculiarity of these last two ”tree” regression

models is precisely to consider exhaustively the pres-

ence of relationships between the variables belonging

to different hierarchical levels of a data matrix and

their breakdown of the variance in the terminal nodes

(e.g., classes), considering both the net effect on sta-

tistical units (ad.es. students) and the interactions

present in them. Among the models that have less re-

liable results in terms of R

are the polynomial (pol)

and linear regression (lm), the linear Support Vector

Machine (svmlin) and the Neural Network (nn).

Figure 2: R2 distribution by the statistical model.

Value-added (VA) is deﬁned as the difference be-

tween the expected and the actual performance. Each

VA score was computed based on average residuals

per classroom. The positive sign indicates that stu-

dents in each class have achieved a better-predicted

result in Mathematics (net of factors that cannot be

affected by a class), while a negative sign indicates

that a worse result has been obtained. The comparison

was made by classifying the added value scores into

three quartiles, calculated based on the average resid-

uals per classroom (the residuals represent precisely

what cannot be attributed to the personal, social, and

economic characteristics of the students and thus de-

pends on belonging to a given classroom). Based

on the classiﬁcation obtained by three quartiles, the

classes were divided into these categories of added

value: ”Needs improvement”, ”Neutral”, ”Highly ef-

fective”. It is apparent that the central class is the

one that does not have a positive or negative impact

on the students within; the classes with ”Highly Ef-

fective” labels are usually those that have achieved

results that are signiﬁcantly higher than those that

are on average obtained by students attending classes

with comparable characteristics (based on the socio-

demographic proﬁle and level of the previous school

year); the classes with ”Needs Improvement” labels

have achievement levels that are signiﬁcantly below

those achieved by students attending classes with sim-

ilar characteristics. As can be seen from Table 3, with

the multilevel classiﬁcation, the maximum disagree-

ment is 23% for Extreme Gradient Boosting Linear,

22% for Neural Network and 14% for RF.

Table 1: Crosstable of Multilevel model by Machine learn-

ing models VA classiﬁcations.

The results are shown highlighting the need for

additional exploration for decision tree algorithms.

In another word, the best models are the linear al-

gorithms, conﬁrming linearly separable data. The

largest difference between linear regression models

and multilevel model is that there is potentially a dif-

ferent intercept and a different slope coefﬁcient for

every level 2 variable for the classroom (Levy et al.,

2019). In other words, the students are clustered in

the classroom in multilevel analysis, giving more ac-

curacy to the results, while in linear regression models

it is not available.

5.0.1 Ensemble Methods vs. Multilevel Model

Going deeper into the analysis of the results for the

ensemble method (decision tree algorithms), the Fig-

ures 3 show the variables ordered by the models of

Extreme Gradient Boosting Linear and Random For-

est (the best in terms of adaptation to the model). The

differences in the ﬁrst positions derive from the de-

termination of the importance of the explanatory vari-

ables for which the ”variants” of the basic regressor

of the two models are produced. The Extreme Gra-

dient Boosting produces variants with a greater focus

on ”difﬁcult” examples and produces a reduction of

the forecast error in sequential terms, while the Ran-

dom Forest produces variants by introducing random-

ness into the tree construction process thanks to the

bootstrap aggregation meta-algorithm that guides the

Digital Device and Mathematics: Multilevel vs Machine Learning Models for Value-added Ranking in Italy

175

random choice predictors to use for each tree (James

et al., 2013).

Figure 3: Variable importance score by Extreme Gradient

Boosting, Random Forest and Multilevel models.

Which of these produces a more ”accurate” im-

portance of the variables is difﬁcult to pinpoint with

certainty (Breiman et al., 1984). However, the impor-

tance of each independent variable is calculated based

on the reduction of the ”impurity” compared to the av-

erage of all decision trees: it is nothing more than the

weighted average of each explanatory variable in the

creation of the node. As usual in machine learning

algorithms, the importance of the predictors is calcu-

lated with training that reduces the error propagated

in each subsequent training step and the important

values of the predictors vary from 1 to 0 (and trans-

formed into a percentage below). Focusing on the

analysis of variables with greater relative importance

in the training process, observing the jump in the Fig-

ures 3, the most important predictors are the aver-

age class ESCS, the class, the computer ownership

at home, the prevailing language is spoken at home

(Italian), the ESCS, the belonging to a class of South-

ern Italy, insufﬁcient scoring in grade 7 in Mathemat-

ics, both at individual and class level. The insufﬁcient

scoring in grade 7 is predominant, both at the indi-

vidual and class level because it is the student’s en-

try rating according to the evaluation of the teacher’s

performance. The effects of all other characteristics

diminish, even at the class level, indicating how the

favourable class context, in terms of prior preparation

in Mathematics, is crucial on individual outcomes of

the Standardized Invalsi test. As expected, obviously

along with the socio-economic and cultural status of

the household, computer equipment at home at the

individual level is important on individual outcomes

of the test, while one must scroll through the graphs

a bit to discern that of the class. Hence, ICT plays

an important role, speciﬁcally in the PC ownership

at home or to belonging to a class with a high av-

erage of PC ownership at home. What’s more, the

language spoken in the household is particularly in-

ﬂuential on outcomes across models, along with the

class size and collaborative classroom activities (for

the RF model). The different models also highlight

the lower importance of the immigrant background (I

and II generation immigrant) and Internet connection

at home, in addition to gender (male) and classroom

activities with a personal computer.

6 CONCLUSION AND

DISCUSSION

This paper examined the contributions of Information

and Communication Technology to mathematics ed-

ucation (Q1) and the process for analyzing the mul-

tilevel value-added current approach, which included

CSEDU 2022 - 14th International Conference on Computer Supported Education

176

several statistical improvements (Q2). It started by

giving a formal problem description of digital device

and its deﬁnition related to mathematics, some re-

lated literature was discussed in which the more re-

cent studies of the impact of instructional computer

use in school settings has found mixed results and

vary across countries (Eickelmann et al., 2012; Hu

et al., 2018; Carstens et al., 2021), while further stud-

ies on the Internet connection and a Personal com-

puter at home shown positive effects on school per-

formance (Wittwer and Senkbeil, 2008). It is not sim-

ply having access to an ICT at home or in school that

can inﬂuence in positive terms school learning (which

instead would provide negative outcomes) but rather

the relative ability to use and control it. From the

review of the international literature, about the rela-

tionship between digital learning and learning in the

classroom, the students have greater difﬁculty using

the computer equipment effectively when teachers do

not work in the classroom to develop shared prac-

tices in the use of technology (Kozma, 2003; Balan-

skat et al., 2006; Burns, 2013; Drijvers, 2015; Bray

and Tangney, 2017; Viberg et al., 2020). According

to the research question Q1, conﬁrmed from the ma-

chine learning models, the most important predictors

for mathematics education at the classroom level are

the ESCS, both at individual and classroom level, the

Personal Computer ownership at home, the language

is spoken at home (Italian), the belonging to a class-

room of Southern Italy (maybe in a pejorative con-

notation), the math score at grade 7, both at individ-

ual and class level. Here, Information and Commu-

nications Technology plays a part in the value-added

approach, speciﬁcally in terms of Personal Computer

ownership at home or belonging to a class with high

Personal Computer ownership at home. However,

the role of Information and Communication Tech-

nologies is a minor part compared to the ascribed

characteristics. Indeed, it can occasionally discour-

age students’ trouble and logical thinking if educa-

tion systems don’t borrow technology to meet their

tutoring needs. Therefore, it is essential to develop

a more articulated model in the Italian context, pos-

sibly taking cues from international efforts (Fraillon

et al., 2014), which discusses self-efﬁcacy and the

use of Information and Communication Technology

also in the classroom setting. In addition, the value-

added score analysis suggests that technology-based

professional development by teachers is not so es-

sential. However, for a future where technologies

are crucial, school principals cannot ignore their im-

portance (Karakose et al., 2021). The capability to

break ﬁne problems with group conditioning in the

school class tends to induce a disadvantage in the per-

formance of the individual test. The calculation test

score gap between ﬁrst-generation immigrants, na-

tives and second-generation emigrants has veriﬁed the

OECD framework (Pe

na-L

opez et al., 2017)) that’s

the language walls the key to explaining differences in

performance between these two groups of students, as

well as a major disadvantage in terms of implicit per-

ceptivity with access from home to a particular com-

puter. It remains to be explored how peer collabora-

tion activities are structured and how they interact in

the use of new information technologies. Recent stud-

ies show that the effectiveness of peer group activities

is strongly correlated with the strategies introduced

by the teacher to make it work, such as role assign-

ment, group contracts, anonymous peer assessments,

etc. (Chang and Brickman, 2018). Finally, the multi-

level hierarchical model is fully superimposable with

machine learning models that are recommended for

ranking added value. It should therefore be noted that

if a hierarchical model is not applicable, and there are

many other applications, the Extreme Gradient Boost-

ing models and the Random Forest could prove com-

plementary to hierarchical analysis to discover and

test complex relationships between variables in the

educative ﬁeld also exploiting their predictive poten-

tial. However, it must be pointed out that it is still

challenging, especially for concepts such as value-

added scores in classes, where the goal is still to in-

terpret and know in depth what are the factors that

inﬂuence the effectiveness of one class rather than an-

other, to use machine learning models without a con-

solidated conceptual scheme.

REFERENCES

Balanskat, A., Blamire, R., and Kefala, S. (2006). The ict

impact report. European Schoolnet, 1:1–71.

Bray, A. and Tangney, B. (2017). Technology usage in

mathematics education research–a systematic review

of recent trends. Computers & Education, 114:255–

273.

Breiman, L., Friedman, J., Olshen, R., and Stone, C. (1984).

Cart. Classiﬁcation and Regression Trees; Wadsworth

and Brooks/Cole: Monterey, CA, USA.

Burns, M. (2013). Success, failure or no signiﬁcant dif-

ference: Charting a course for successful educa-

tional technology integration. International Journal

of Emerging Technologies in Learning, 8(1).

Carstens, K. J., Mallon, J. M., Bataineh, M., and Al-

Bataineh, A. (2021). Effects of technology on stu-

dent learning. Turkish Online Journal of Educational

Technology-TOJET, 20(1):105–113.

Chang, Y. and Brickman, P. (2018). When group work

doesn’t work: Insights from students. CBE—Life Sci-

ences Education, 17(3):ar52.

Digital Device and Mathematics: Multilevel vs Machine Learning Models for Value-added Ranking in Italy

177

Commission, E. (2020). Shaping europe’s digital future.

Davidson, N. (1990). Cooperative Learning in Mathemat-

ics: A Handbook for Teachers. ERIC.

Desimoni, M. (2018). I livelli per la descrizione degli esiti

delle prove invalsi.

DiMaggio, P., Hargittai, E., Neuman, W. R., and Robinson,

J. P. (2001). Social implications of the internet. An-

nual review of sociology, 27(1):307–336.

Drijvers, P. (2015). Digital technology in mathematics edu-

cation: Why it works (or doesn’t). In Selected regular

lectures from the 12th international congress on math-

ematical education, pages 135–151. Springer.

Eickelmann, B., Drossel, K., Wendt, H., and Bos, W.

(2012). Ict-use in primary schools and children’s

mathematics achievement-a multi-level approach to

compare educational systems through an international

lens with timss data. In Joint AARE APERA Interna-

tional Conference, WERA focal meeting, Sydney, vol-

ume 2012.

Fraillon, J., Ainley, J., Schulz, W., Friedman, T., and Geb-

hardt, E. (2014). Preparing for life in a digital age:

The IEA International Computer and Information Lit-

eracy Study international report. Springer Nature.

Hu, X., Gong, Y., Lai, C., and Leung, F. K. (2018). The

relationship between ict and student literacy in math-

ematics, reading, and science across 44 countries: A

multilevel analysis. Computers & Education, 125:1–

13.

INVALSI (2017). Nota metodologica sul questionario in-

segnante revisione 2017 2018.

INVALSI (2018). Le prove computer based per la rile-

vazione nazionale degli apprendimenti invalsi 2018:

aspetti metodologici.

ISTAT (2018). Indagine multiscopo sulle famiglie, aspetti

della vita quotidiana.

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013).

An introduction to statistical learning, volume 112.

Springer.

Karakose, T., Yirci, R., and Papadakis, S. (2021). Exploring

the interrelationship between covid-19 phobia, work–

family conﬂict, family–work conﬂict, and life sat-

isfaction among school administrators for advancing

sustainable management. Sustainability, 13(15):8654.

Kozma, R. B. (2003). Technology and classroom practices:

An international study. Journal of research on tech-

nology in education, 36(1):1–14.

Levy, J., Brunner, M., Keller, U., and Fischbach, A.

(2019). Methodological issues in value-added mod-

eling: an international review from 26 countries. Ed-

ucational Assessment, Evaluation and Accountability,

31(3):257–287.

Levy, J., Mussack, D., Brunner, M., Keller, U., Cardoso-

Leite, P., and Fischbach, A. (2020). Contrasting clas-

sical and machine learning approaches in the estima-

tion of value-added scores in large-scale educational

data. Frontiers in psychology, 11:2190.

NTIA (1995). Falling through the net: A survey of the

”have nots” in rural and urban America. ERIC Clear-

inghouse.

Papadogiannis, I., Poulopoulos, V., and Wallace, M. (2020).

A critical review of data mining for education: what

has been done, what has been learnt and what remains

to be seen. International Journal of Educational Re-

search Review, 5(4):353–372.

na-L

opez, I. et al. (2017). Oecd digital economy outlook

2017. OECD Publishing.

Rasch, G. (1960). Studies in mathematical psychology: I.

probabilistic models for some intelligence and attain-

ment tests.

Rutkowski, D., Rutkowski, L., B

elanger, J., Knoll, S.,

Weatherby, K., and Prusinski, E. (2013). Teaching

and learning international survey talis 2013: Concep-

tual framework. OECD Publishing.

van Dijk, J. A. (2005). The deepening divide: Inequality in

the information society. Sage Publications.

Viberg, O., Gr

onlund,

A., and Andersson, A. (2020). Inte-

grating digital technology in mathematics education:

a swedish case study. Interactive Learning Environ-

ments, pages 1–12.

Wittwer, J. and Senkbeil, M. (2008). Is students’ computer

use at home related to their mathematical performance

at school? Computers & Education, 50(4):1558–

1571.

CSEDU 2022 - 14th International Conference on Computer Supported Education

178