Improvements in bASES21: 21

-Century Skills Assessment Model

to K12

Lúcia Helena Martins-Pacheco

, Leonardo Philippi Degering

, Fernanda Mioto

Christiane Annelise Gresse von Wangenheim

, Adriano Ferreti Borgato

and Giani Petri

Universidade Federal de Santa Catarina, Brazil

Universidade Federal de Santa Maria, Brazil

mioto.f@gmail.com, gpetri@inf.ufsm.br

Keywords: 21

-Century Skills, Assessment, Questionnaire, Computer Science Education.

Abstract: Digital age has been impacted worldwide economy, human communication, knowledge access, democracy

and days by day citizen lives. Consequently, several international entities make strategies for educational

system and to empower people to labour market. One strategy has been to define what skills are important

develop in 21

-century. To introduce educational systems approaches that aim to develop 21

-century skills

it is necessary encompass how to teach, how to learn, and how to assess this personal development. Some

practices have been indicated that teaching computing in basic education could act synergistically with 21

century skills learning. So that, bASES21 is an instrument created to assess such skills. The first version was

tested with a group of 148 high school students. Statistical studies pointed out the need of expand the sample

size. Then in a second study sample size reach 560 students. New statistical analyses were done, and it was

proposed several changes in first model. The instrument upgraded can assess such skills with 56 questions

instead of 82, that were originally taken. These results can be very useful to ensure to attain necessary

reliability and validity and to make possible popularization in using this instrument in K12 school setting.

1 INTRODUCTION

In the last two decades, human society has been coped

with new challenges, especially related to the way to

deal with knowledge. The enormous increase in

machine throughput and distributed processing,

massive data creation and also higher speed and

processing capacity of the electronic devices,

associated to great popularization of smartphones and

personal computers, are allowing, even for a common

citizen, to access, to create, to disseminate

information and knowledge easily and quickly

worldwide.

These great technological possibilities have

become globalization, in fact, a reality. Even facts in

remote places could have a fast impact worldwide.

https://orcid.org/0000-0003-3552-4421

https://orcid.org/0000-0002-2364-2423

https://orcid.org/0000-0002-5263-5097

https://orcid.org/0000-0002-6566-1606

https://orcid.org/0000-0001-6280-2525

https://orcid.org/0000-0002-9884-8151

This context has a strong influence on the economy,

in the workforce, in consumer profiles, in democratic

practices, and the global organization of countries.

This new scenery brought up several doubts,

especially about the future: What must be taught and

learned to our youth? (Harari, 2018) Which kind of

knowledge will be more useful? What must they

know to enter the formal work market? What must

they know to practice democracy and citizenship? For

example, Szikora and Ali (2018) called attention to

the various dangers connected to smart phones, the

importance of data and the privacy of the public,

which would develop security conscious thinking.

Concerning to digital inclusion of people, subjects

must develop skills to take advantage of these

benefits (Pischetola, 2019). The endeavour is to know

Martins-Pacheco, L., Degering, L., Mioto, F., von Wangenheim, C., Borgato, A. and Petri, G.

Improvements in bASES21: 21st-Century Skills Assessment Model to K12.

DOI: 10.5220/0009581702970307

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 1, pages 297-307

ISBN: 978-989-758-417-6

297

which skills are essential and indispensable and how

to teach and learn them. In this sense, since the last

decade of the 20th century, several international

entities have tried to recognize and to define such

skills (Delors et al., 1996).

Citizens must incorporate

the flexibility that new technology demands to engage

in social changes, democratic processes,

communication flows, and employment

opportunities. Given the rapid rate of change and the

influence of technology, employees need to develop

21st-century digital skills to cope and thrive in this

changing society (van Laar et al., 2017).



Ananiadou and Claro (2009) consider that young

people are already experiencing the new forms of

socialisation and social capital acquisition that ICT

developments are contributing to. Their education,

both at school and at home, needs to provide them

with the social values and attitudes as well as with the

constructive experiences that will allow them to

benefit from these opportunities and contribute

actively to these new spaces of social life (p. 5).

As an example of international initiatives, there

are Partnership for 21st Century Skills - P21

and

Assessment and Teaching of 21st Century Skills

Group

- ATC21S (Binkley et al, 2011). Several skills

are frequently considered for many authors, for

example, creativity, critical thinking, problem-

solving, communication, teamwork, collaboration,

ICT proficiency, flexibility, proactivity, social

responsibility, and so on.

This scenario brought several studies to try to

define these skills and some instruments aiming to

measure and assess such skills. In this sense, in this

article, we present bASES21, which is a self-

assessment instrument to measure 21

-century skills

focus in the K12 Brazilian context. At first, we show

some definitions of 21

-century skills and their

approach to computer science education.

2 21

-CENTURY SKILLS AND

COMPUTER SCIENCE

EDUCATION

Table 1 shows a summary of five approaches to 21

century skills and compares them (Care and Griffin,

2014). What is highlighted in grey shadow bold repre

sents a category of skills.

Another approach is Partnership for 21st Century

Skills – P21 – that also shows several details impor-

www.p21.org

www.atc.org

tant for assessment and measurement. The

Partnership for 21st Century Learning recognizes

that all learners need educational experiences in

school and beyond, from cradle to career, to build

knowledge and skills for success in a globally and

digitally interconnected world (P21, 2015 –

Framework Definitions).

OECD approach is concerning to a collaborative

problem-solving framework that also shows some

intersection with the other approaches. There has

been a marked shift from manufacturing to

information and knowledge services. Much of the

problem-solving work carried out in the world today

is performed by teams in an increasingly global and

computerised economy. However, even in

manufacturing, work is seldom conducted by

individuals working alone. Moreover, with the

greater availability of networked computers,

individuals are increasingly expected to work with

diverse teams spread across different locations using

collaborative technology

(OECD, 2013, p.3-4).

The approach Assessment and Teaching of 21st-

century skills group - ATC21S defined Binkley et al.

(2012) defined ten skills and identified into four

categories. Each skill is unfolded in detail in three

domains: knowledge, skills, attitudes/values/ethics.

These details are very important in identifying aspects

of assessment. ATC21S model for assessments of

21st-century skills, based on an analysis of

curriculum and assessment frameworks for 21st-

century skills developed around the world, identifies

ten important skills in four broad categories. This

model provides measurable descriptions of the skills,

considering knowledge, skills, and attitudes, values,

and ethics.

Gordon et al. (2009), sponsored by the European

Commission, created a significant document

concerned to lifelong learners defining keys

competences with connection with 21

-century skills.

The aim of the study is to provide a comparative

overview of policy and practice concerning the

development and implementation of key competences

in the education systems (…) of the European Union.

UNESCO approach (Delors et al., 1996)

is a

substantial document related to education for the 21

century which defined the four pillars of education as

learning to know, learning to do, learning to be and

learning to live together. There is a significant overlap

with 21

-century skills defined in other approaches.

It is possible to identify strong relations between

computer science (CS) education and 21

-century

skills. Digital competences encompass ICT and

information proficiency, collaborative work,

teamwork, and so on. These issues are central in

CSEDU 2020 - 12th International Conference on Computer Supported Education

298

Table 1: Comparison of 21

-century skills approaches based on Care and Griffin (2014).

P21

(2015)

LEARNING AND

INNOVATION:

INFORMATION, MEDIA

AND TECHNOLOGY:

LIFE AND CAREER:

Creativity; Critical

thinking; Problem

solving

Communication;

Collaboration

Literacy: Information;

Media; ICT

Flexibility and

Adaptability; Initiative and

Self-direction; Social and

Cross-cultural Skills;

Productivity and

Accountability; Leadership

and Responsibility.

OECD

(2015)

INTERACT IN

HETEROGENEOUS

GROUPS:

USE TOOLS

INTERACTIVELY:

ACT AUTONOMOUSLY:

Relate Well to Others;

Co-operate; Work in

Teams, Manage and

Solve Conflicts

Interactively: Use

Language, Symbols and

Texts; Use Knowledge,

Information and

Technology

Act within the Big Picture;

Form and Conduct Life

Plans and Personal

Projects; Defend and Assert

Rights, Interests, Limits

and Needs.

ATC21S WAYS OF THINKING: WAYS TO WORK: TOOLS TO WORK: LIVING IN THE WORLD:

(Binkley et

al., 2012)

Creativity and

Innovation; Critical

Thinking; Problem

Solving; Decision

Making; Learning to

Learn; Metacognition

Communication;

Collaboration

Information and ICT

Literacy

Citizenship (local/global);

Life and Career;

Responsability

(social/personal); Including

Cultural

(awareness/competence)

European

Commission

LEARNING TO LEARN:

(Gordon et

al., 2009)

Communication in

Mother tongue and

Foreign Languages

Mathematical, Science,

Technology and Digital

Competences

Social and Civic

Competences; Initiative and

Entrepreneurship; Cultural

Awareness and Expression

UNESCO

(Delors et

al., 1996)

LEARNING TO KNOW LEARNING TO DO LEARNING TO DO

LEARNING TO BE AND TO

LIVE TOGETHER

teaching and learning computer science. For example,

Gordon et al. (2009) defined: Digital competence

involves the confident and critical use of Information

Society Technology (IST) for work, leisure, and

communication. It is underpinned by basic skills in

ICT: the use of computers to retrieve, assess, store,

produce, present and exchange information, and to

communicate and participate in collaborative

networks via the Internet (p.45).

It is important to distinguish the difference

between skills and competence. According to

Ananiadou and Claro (2009), skill is the ability to

perform tasks and solve problems, while a

competence is the ability to apply learning outcomes

adequately in a defined context (education, work,

personal, or professional development (p. 8). For

them, competence is not limited to cognitive elements

(involving the use of. theory, concepts, or tacit

knowledge); it also encompasses functional aspects

(involving technical skills) as well as interpersonal

attributes (e.g., social or organizational skills) and

ethical values (p.8). Consequently, this broad vision

of competence clarifies that people that are involved

with computer science education, teaching or

learning, or even as a user or developer or designer

must be engaged in incorporated 21

-century skills.

Computer science and technological development

impact society all around the world, and the

workforce trained must have an ethical commitment

to their decisions.

We dare assert that 21

-century skills approaches

are especially effect of the digital age, approached

Improvements in bASES21: 21st-Century Skills Assessment Model to K12

299

people and provided new ways of communication and

knowledge access, practically available for everyone.

Practices particularly involved 21

-century skills as

creativity, innovation, critical thinking, problem-

solving, communication and collaboration (P21,

2017) are very useful for computer science students

and professionals. In this sense, van Laar et al. (2017)

identified key 21

-century skills or digital skills

dimensions resulting in a framework of seven core

skills: technical, information management,

communication, collaboration, creativity, critical

thinking and problem solving, and five contextual

skills: ethical awareness, cultural awareness,

flexibility, self-direction and lifelong learning.

Therefore, these issues encourage this present study

aiming to contribute to the development of 21

century skills in the K12 Brazilian educational

context.

3 bASES21 ASSESSMENT

MODEL

The model bASES21 (Assessing 21

-Century Skills),

began to be developed by Mioto (2018). After tests in

the first version, Degering (2019) expanded the

sample size and performed new statistical tests. Both

authors were under the supervision of the fourth one.

They developed this research as part of the bachelor

thesis.

Both studies began searching by state of the art by

means of a systematic mapping of literature (SML)

according to Petersen et al. (2015) methodology.

Mioto (2018) found eight models related to

assessment of 21

-century skills (Chai et al., 2015;

Rosen, 2015; Siddiq et al., 2017; Claro et al., 2012;

Rosen and Tager, 2014; Aesaert et al., 2014; Lau and

Yuen, 2014; Susnea and Vasiliu, 2016). These

models only assess one skill or a cluster of skills but

not all skills, as were defined by Binkley et al. (2011)

or by P21 (P21, 2015).

Mioto (2018), based on SML findings, to create

an instrument for 21

-century skills assessment. To

assess this instrument, she considered the

methodologies proposed by Basili et al. (1994),

Kasunic (2005), Beecham et al. (2005), Wohlin et al.,

The instrument was a questionnaire, written

originally in Brazilian Portuguese, composed of 82

questions, divided into 13 categories, which is shown

in Table 2. Each question was defined based on the

2012, DeVellis (2016) and Trochim and Donnely

https://repositorio.ufsc.br/handle/123456789/187851

(2008).

The instrument was a questionnaire, written

originally in Brazilian Portuguese, composed of 82

questions, divided into 13 categories, which is shown

in Table 2. Each question was defined based on the

showed references beside it. Binkley et al. (2011) and

P21 (P21,2015) supported a great number of

questions. This instrument is a self-assessment

questionnaire. Four points Likert scale is used with

these options: Totally Agree; Agree; Disagree;

Totally Disagree. It was designed for K12 Brazilian

students as target public. The model can be applied in

non-experimental studies using one-shot post-test

designs. Also, it can be used in quasi-experimental

studies by means of pre-test/post-test designs.

The assessment of the instrument consisted of the

application to 148 students in middle schools. After

that, statistical analysis was done based on the

collected data. The complete study is available in

Brazilian Portuguese

language.

Degering (2019) followed Mioto (2018) aim to

increase the size of the sample and improve the

quality of the instrument assessment. He used the

same methodology and redone the SML also

problem-solving, according to Petersen et al. (2015).

He found 13 models, eight covered by the first SML

added to 5 new ones (Ball et al., 2016; Osman et al.,

2010; Van Laar et al., 2018; Cevik, Senturk, 2019;

Mioto et al., 2019). The own bASES21 appeared and

was selected. Details of this SML are showed in

Mioto (2018) and Degering (2019). Spite of the

increasing of assessment 21

-century skills models

found in second SML, only Mioto et al. (2019)

proposed to assess a set bigger than others using as

main foundation Binkley et al. (2011).

The findings of the two SMLs showed that skills

related to ICT use are more frequent among models.

Also, there is no consensus concerning the 21

century skills classification. The instrument more

usual is a self-assessment questionnaire using a Likert

scale. These found instruments were statistically

assessed for validation. They presented good

statistical results or need to increase the sample size

for more consistent conclusions. Statistical tests in the

model proposed by Mioto (2018), in spite of a good

internal consistency among items, indicate

inconclusive validity (Ross, 2006). So, Degering

(2019) applied bASES21 to other groups extending

sample size. It is explained ahead in the sequence of

this document.

4 DATA ANALYSIS

Table 3 synthesizes the data collection. We highlight

CSEDU 2020 - 12th International Conference on Computer Supported Education

300

Table 2: bASES21 questionnaire version 1.0 - based on Mioto (2018) – free translate to English.

I - CREATIVITY AND INNOVATION:

1 I invent/imagine many things that do not yet exist. (Chai et al., 2015); (Petway et al., 2016); (Susnea and Vasiliu,

2016)

2 My ideas are useful (Chai et al., 2015)

3 I can solve problems in different ways. (Chai et al., 2015); (Kang et al., 2010)

4 I am a curious person. (Susnea and Vasiliu, 2016)

5 I am not embarrassed to talk about my ideas. (Susnea and Vasiliu, 2016)

6 I learn with my errors or when my ideas go wrong. (Binkley et al., 2011)

7 I need to im

rove m

ideas.

(

P21, 2015

)

II - CRITICAL THINKING, PROBLEM-SOLVING AND DECISION MAKING:

8 I compare different opinions/ideas to see which is the better. (Chai et al., 2015); (Binkley et al., 2011)

9 I make decisions according to the information that I have. (Binkley et al., 2011)

10 I like to ask and to answer questions to learn something new. (Kang et al., 2010); (Binkley et al., 2011); (IFL, 2015)

11 I listen to the ideas of my friends and consider them when I form my opinion. (Binkley et al., 2011)

12 I try to understand a problem before trying to solve it. (O’Neil and Schacter, 1997)

13 I choose and organize the material that I need when I am going to do something (homework, works, studies, etc.).

(O’Neil and Schacter, 1997)

14 I ask myself if I am doing my tasks well at school. (O’Neil and Schacter, 1997)

15 I make an effort when I do my school tasks. (O’Neil and Schacter, 1997)

16 I can ex

lain m

inions and decisions.

(

Chai et al., 2015

)

III - LEARN TO LEARN AND METACOGNITION:

17 I plan how to study (which tasks I am going to do in which days/time, etc.). (Chai et al., 2015))

18 If I have difficulty in a subject of a course, I spend more time studying this subject. (Chai et al., 2015)

19 I believe that I can learn everything that I want. (Binkley et al., 2011)

20 I like to learn new things. (Binkley et al., 2011)

21 I can kee

concentrated for a lon

time.

(

Binkle

et al., 2011

))

IV - COMMUNICATION:

22 I listen attentively to understand what others are saying. (Binkley et al., 2011)

23 Other people understand what I say. (Binkley et al., 2011)

24 When I read a text, I understand what I am reading. (Binkley et al., 2011)

25 I am not embarrassed to talk to an audience. (Binkley et al., 2011)

26 I like to say and to listen to different opinions. (Binkley et al., 2011)

27 I can ar

ue well in a discussion.

(

Binkle

et al., 2011

)

V - COLLABORATION AND TEAMWORK:

28 I like to work together with my colleagues and to solve problems. (Chai et al., 2015); (Kyllonen, 2012); (Petway et

al., 2016); (Kang et al., 2010)

29 I can find time to help other people. (Kyllonen, 2012)

30 I like to be the group leader. (Kyllonen, 2012); (Kang et al., 2010)

31 I always do my part when I work in a group. (Kang et al., 2010)

32 I can create a sequence of tasks in group work (Binkley et al., 2011)

33 I like to be a good example for others. (Binkley et al., 2011)

34 I respect differences among people from other regions, countries, and religions. (Binkley et al., 2011)

35 I commit to doing the necessary tasks to achieve a goal in group work. (P21, 2015)

36 In group work, my colleagues usually agree with my ideas. (Kang et al., 2010)

37 I do not easily give up. (Kyllonen, 2012); (Susnea and Vasiliu, 2016)

38 I usuall

finish the thin

s that I start.

(

Duckworth et al., 2007

)

VI - INFORMATION PROFICIENCY:

39 I can find the necessary information to do a task/ to solve a problem. (Lau and Yuen, 2014); (Aesaert et al., 2014);

(Binkley et al., 2011); (Kang et al., 2010)

40 I analyze if a piece of information is truthful or not. (Lau and Yuen, 2014); (Aesaert et al., 2014); (P21, 2015); (Binkley

et al., 2011)

41 I can change my opinion, depending on how much I know about the issue. (Siddiq, Gochyyev and Wilson, 2017)

42 I can explain why I change my opinion. (Siddiq, Gochyyev and Wilson, 2017)

43 I can interpret graphics and tables. (Claro et al, 2012); (Binkley et al., 2011))

44 I regard as wrong copying, sharing, or changing (information, text, pictures, etc.) that belong to other people without

their permission. (P21, 2015)

45 When I stud

, I search for more information be

ond m

notebook notes, didactical material

ooks.

(

Kan

et al., 2010

)

VII - ICT PROFICIENCY:

46 When I study, I access the internet to find useful information. (Aesaert et al., 2014); (Chai et al., 2015); (Kang et al.,

2010); (Lau and Yuen, 2014); (CSTA, 2016)

47 I use instant message Apps (WhatsApp, Messenger, etc.). (Lau and Yuen, 2014)

48 I know how to create documents (doc, pdf, spreadsheets, etc.) or presentations in the computer. (Chai et al., 2015)

49 I can use electronic devices (computer, internet, cellphone, etc.) to do my tasks. (Siddiq, Gochyyev and Wilson,

2017)

50 I understand the importance of taking care of my personal information on the internet. (Wangenheim, Alves, Weber,

2017

)

Improvements in bASES21: 21st-Century Skills Assessment Model to K12

301

Table 2: bASES21 questionnaire version 1.0 - based on Mioto (2018) – free translate to English. (cont.)

VIII - COMPUTATIONAL PROFICIENCY:

51 I can create programs in computers (games, apps, etc.). (CSTA, 2016)

52 I can identify the most important parts of a computer. (CSTA, 2016)

53 I know the risk of using a simple password. (CSTA, 2016)

54 I know how computers communicate on the internet. (CSTA, 2016)

55 I know how to identify, to test, and to correct an error in a computer program. (Tsai, Wang and Hsu,

2018

)

IX - LOCAL AND GLOBAL CITIZENSHIP:

56 I have the right to give my opinion. (Binkley et al., 2011)

57 I pay attention to the news that appears in the media (TV, social networks, sites, etc.). (Binkley et al.,

2011)

58 I respect that people can express different cultures, religions, lifestyle, and opinions. (Binkley et al.,

2011); (P21, 2015))

59 I speak/understand well another language (Spanish, French, etc.) beyond English. (P21, 2015)

60 I can establish a good relationship with people with personalities or interests different from my own.

(Kang et al., 2010)

61 I am friendl

and kind with new collea

ues in the classroom.

(

Kan

et al., 2010

)

X - PERSONAL AND SOCIAL RESPONSIBILITY:

62 I can learn many things from other people. (Siddiq, Gochyyev and Wilson, 2017)

63 I can teach something to other people. (Siddiq, Gochyyev and Wilson, 2017)

64 I make an effort to, as much as possible, fulfill the promises that I make. (Kang et al., 2010)

65 I treat people as I would like to be treated. (IFL, 2015)

66 I admit my errors, and I apologize. (Kang et al., 2010); (Petway et al., 2016)

67 I know that

overnment decisions can affect me in different wa

(

P21, 2015

)

XI - LIFE AND CAREER:

68 I imagine where/in what I want to work when I grow up. (Binkley et al., 2011); (Kang et al., 2010))

69 I accept criticism even when I believe that I have done a good job. (Binkley et al., 2011))

70 I always do my homework. (Kyllonen, 2012); (Kang et al., 2010); (Petway et la., 2016))

71 When I get a low score in school, I try to understand the reason for this. (Kang et al., 2010)

72 I make a to-do list. ((Kang et al., 2010); (Kyllonen, 2012)

73 I can do my homework by myself. ((Duckworth et al., 2007); (Kyllonen, 2012)

74 I avoid as much as possible to talk or to use a cellphone during classes. (IFL, 2015)

75 I can adapt to the changes in my routine. (Binkley et al., 2011)

76 I can achieve the

oals that I create for m

self.

((

Binkle

et al., 2011

)

XII - HEALTH PROFICIENCY:

77 I understand what is necessary for a healthy life. (P21, 2015)

78 I know how to prevent common diseases. (P21, 2015)

79 I know how to take care not to catch a cold.

(

P21, 2015

)

XIII - ENVIRONMENTAL AWARENESS:

80 I know the causes of global warming. (P21, 2015)

81 I separate organic and recyclable trash. (P21, 2015)

82 I tr

not to take a lon

shower to save water.

(

P21, 2015

)

that the number of students in high school

encompasses 148 that came from the first study

(Mioto, 2018).

Mioto (2018) made a statistical analysis of the

data collected using a sample size equal to 148 high

school students. The consistency analyses base on

Cronbach alpha (Cronbach, 1951) shows a strong

consistency among questions with a range between

0.957 and 0.958. However, the validity of the

instrument was showed inconclusive by means of

polychoric correlation matrix analysis (Drasgow,

1986); (Cohen, 1988). Likely because the instrument

has a great number of questions. So. it pointed the

need for more studies and expansion of sample size.

Degering (2019) performed an analysis of the new

data collection. He excluded the sample of 166

undergraduate students to keep the goal of these

studies that is K12. Then it was calculated internal

consistency by means of Cronbach alpha coefficients

for each item. The result was that Cronbach alpha

coefficients were equal or above 0.93 that indicated a

very good consistency among items. Also, Cronbach

alpha of instrument was 0.95, and no item was above

this. This analysis means that no item must be

excluded to improve internal consistency (DeVellis,

2016). Instead of this second analysis has a sample

size bigger than the first one, the results were very

similar) in terms of internal consistency.

The calculations of correlations among items were

done aimed to find pieces of evidence concerning

convergent and discriminant validation (DeVellis,

2016) of the bASES21 instrument. The convergent

validity shows if items that must be related indeed are

related and discriminant validation shows if items that

CSEDU 2020 - 12th International Conference on Computer Supported Education

302

must be no related indeed are not related (Souza et al.,

2017). Through the polychoric correlation matrix and

Cohen coefficient analysis (Drasgow, 1986),

(Cohen,1988), it was possible to figure out that some

items must be included in a different category and

some could be excluded. Details of this analysis are

shown in Degering (2019). For example, items related

to Creativity and Innovation showed a low correlation

among them what could indicate that they do not

measure the same factor. On the other hand, the items

related to ICT proficiency showed good results

concerning the validity, which indicates that no

change is necessary.

Table 3: Synthesis of data collection.

SCHOOL DEGREE

TO 5

TO 9

HIGH SCHOOL

UNDERGRADUATE

Quantity 74 120 200 166

Sample size = 560 students

Age Range:8-19 years old

Average = 14.5 years Median = 15 years

Standard deviation = 2.71 years

In addition, it was performed a factorial analysis

by means of Cattell´s Scree Test (Cattell, 1966);

(Raîche et al., 2013). The results indicate that three or

four factors showed better results than 13. Therefore,

statistical analysis showed that the questionnaire

needs to be changed: some categories and items could

be excluded, and some items could be reorganized.

The summary of changes proposed by Degering

(2019) is shown in Table 4.

Degering (2019), based on exploratory factorial

analysis, proposed, instead of 13 categories (factors),

to regroup skills in four categories (factors): Learning

and Teamwork; Citizenships and Social

Responsibility; ICT Proficiency; Communication.

Also, 26 items were excluded due to low factorial

charge; or low relation to other items in the category

or because there is another similar item. Table 4

shows the changes proposed by Degering (2019), the

reorganization of items. We keep the original index

number of items to facilitate comparison. Discussion

http://agenciabrasil.ebc.com.br/educacao/noticia/2018-

09/mec-divulga-nesta-segunda-indice-de-qualidade-

do-ensino-basico (access 02042020)

and conclusion.

5 DISCUSSION AND

CONCLUSION

The Brazilian educational context of K12 is

challenging. The performance of the students in the

national examination is poor. Takahashi and Sadaña

(2018) consider that difficult insertion in the labor

market by the youth, are related to violence in society

and to poverty. Even government agencies

consider

that it is necessary to change public policies to

improve K12 education. In this context, initiatives

that promote student motivation and could aim to

prosperous future for youths are welcome. One

possibility is to take into account 21

-century skills in

special ICT and information proficiency. These

concerning is emphasized by several international

organization. To teach, to learn and to assess 21

century skills are important to make possible to

include these aspects educational system for social

development updating for this century. They are allies

of prosperity and citizenship because these skills

aggregate technical aspects, interpersonal aspects and

an ethical appeal in terms of behavior and attitudes.

Therefore, bASES21 could be an interesting

instrument to play a role in the Brazilian

K12Educational scenario. Mioto (2018) and

Degering (2019) followed a serious statistical

methodology that increases the reliability and validity

(Ross, 2006) of the instrument. New tests could bring

up new benefits in the instrument.

The improvements proposed in bASES21 allow

assessing with fewer questions. It tends to make an

easier answer in a shorter time. In addition, a self-

assessment instrument enables participants to get

awareness because it calls their attention to the skills.

It is important to have an appropriate instrument for

Brazilian reality. It could support educational

practices to promote 21

-century skills. Also, other

counties could adapt the instrument for their reality or

for opening new opportunities.

It is important to highlight that we did not find any

other instrument for assessing 21

-century skills

written in the Brazilian Portuguese language, so it

seems that it is very original. bASES21 could call the

attention of the educational system to approach the

development of 21

-century skills in K12. We believe

Our perspectives for future works are to perform new

tests in bASES21 and to publicize this study in the

Brazilian Portuguese language aiming to contribute to

bringing these issues as part K12 Brazil educational

policies.

Improvements in bASES21: 21st-Century Skills Assessment Model to K12

303

Table 4: bASES v2.0 according to suggestion of Degering (2019)

I - LEARNING AND TEAMWORK (21 questions)

10 I like to ask and to answer questions to learn something new. (Kang et al., 2010); (Binkley et al., 2011); (IFL, 2015)

12 I try to understand a problem before trying to solve it. (O'Neil and Schacter, 1997)

13 I choose and organize the material that I need when I am going to do something (Homeworks, works, studies, etc.).

(O'Neil and Schacter, 1997)

14 I ask myself if I am doing my tasks well at school. (O'Neil and Schacter, 1997)

15 I make an effort when I do my school tasks. (O'Neil and Schacter, 1997)

17 I plan how to study (which tasks I am going to do in which days/time, etc.). (Chai et al., 2015)

18 If I have difficulty in a subject of a course, I spend more time studying this subject. (Chai et al., 2015)

20 I like to learn new things. (Binkley et al., 2011)

21 I can keep concentrated for a long time. (Binkley et al., 2011)

22 I listen attentively to understand what others are saying. (Binkley et al., 2011)

31 I always do my part when I work in a group (Kang et al., 2010)

33 I like to be a good example for others. (Binkley et al., 2011)

35 I commit to doing the necessary tasks to achieve a goal in group work. (P21, 2015)

37 I do not easily give up. ((Kyllonen, 2012); (Susnea and Vasiliu, 2016))

38 I usually finish the things that I start. (Duckworth et al., 2007)

39 I can find the necessary information to do a task/ to solve a problem. (Lau and Yuen, 2014); (Aesaert et al., 2014);

(Binkley et al., 2011); (Kang et al., 2010))

70 I always do my homework. ((Kyllonen, 2012); (Kang et al., 2010); (Petway et al., 2016))

71 When I get a low score in school, I try to understand the reason for this. (Kang et al., 2010)

72 I make a to-do list. ((Kang et al., 2010); (Kyllonen, 2012))

74 I avoid as much as possible to talk or to use a cellphone during classes. (IFL, 2015)

76 I can achieve the goals that I create for myself. (Binkley et al., 2011)

II - CITIZENSHIPS AND SOCIAL RESPONSIBILITY (16 questions)

6 I learn with my errors or when my ideas go wrong. (Binkley et al., 2011)

26 I like to say and to listen to different opinions. (Binkley et al., 2011)

44 I regard as wrong copying, sharing, or changing (information, text, pictures, etc.) that belong to other people without

their permission. (P21, 2015)

56 I have the right to give my opinion. (Binkley et al., 2011)

58 I respect that people can express different cultures, religions, lifestyle, and opinions. (Binkley et al., 2011); (P21,

2015))

60 I can establish a good relationship with people with personalities or interests different from my own. (Kang et al.,

2010)

61 I am friendly and kind with new colleagues in the classroom. (Kang et al., 2010)

62 I can learn many things from other people. (Siddiq, Gochyyev and Wilson, 2017)

63 I can teach something to other people. (Siddiq, Gochyyev and Wilson, 2017)

64 I make an effort to, as much as possible, fulfill the promises that I make. (Kang et al., 2010)

65 I treat people as I would like to be treated. (IFL, 2015)

66 I admit my errors, and I apologize ((Kang et al., 2010); (Petway et al., 2016))

67 I know that government decisions can affect me in different ways. (P21, 2015)

77 I understand what is necessary for a healthy life. (P21, 2015)

78 I know how to prevent common diseases. (P21, 2015)

79 I know how to take care not to catch a cold. (P21, 2015)

Original instrument in the Brazilian Portuguese language

CSEDU 2020 - 12th International Conference on Computer Supported Education

304

Table 4: bASES v2.0 according to suggestion of Degering (2019). (cont.)

III - ICT PROFICIENCY (11 questions)

40 I analyze if a piece of information is truthful or not. (Lau and Yuen, 2014); (Aesaert et al., 2014); (P21, 2015);

(Binkley et al., 2011)

43 I can interpret graphics and tables. (Claro et al, 2012); (Binkley et al., 2011)

46 When I study, I access the internet to find useful information. (Aesaert et al., 2014); (Chai et al., 2015); (Kang et

al., 2010); (Lau and Yuen, 2014); (CSTA, 2016)

47 I use instant message Apps (WhatsApp, Messenger, etc.) (Lau and Yuen, 2014)

48 I know how to create documents (doc, pdf, spreadsheets, etc.) or presentations in the computer (Chai et al., 2015)

49 I can use electronic devices (computer, internet, cellphone, etc.) to do my tasks (Siddiq, Gochyyev and Wilson,

2017)

50 I understand the importance of taking care of my personal information on the internet (Wangenheim, Alves and

Weber, 2017)

52 I can identify the most important parts of a computer. (CSTA, 2016)

53 I know the risk of using a simple password. (CSTA, 2016)

54 I know how computers communicate on the internet. (CSTA, 2016)

55 I know how to identify, to test, and to correct an error in a computer program. (Tsai, Wang and Hsu, 2018)

IV - COMMUNICATION (8 questions)

5 I am not embarrassed to talk about my ideas. (Susnea and Vasiliu, 2016)

16 I can explain my opinions and decisions. (Chai et al., 2015)

23 Other people understand what I say. (Binkley et al., 2011)

24 When I read a text, I understand what I am reading. (Binkley et al., 2011)

27 I can argue well in a discussion. (Binklet et al., 2011)

36 In group work, my colleagues usually agree with my ideas. (Kang et al., 2010)

42 I can explain why I change my opinion. (Siddiq, Gochyyev and Wilson, 2017)

59 I speak/understand well another language (Spanish, French, etc.) beyond English. (P21, 2015)

ACKNOWLEDGMENTS

The authors would like to thank Renata Martins

Pacheco for her help with formatting and reviewing

the English version of the final text.

REFERENCES

Aesaert, K.; van Nijlen, D.; Vanderlinde, R.; and Braak, J.

2014. Direct measures of digital information processing

and communication skills in primary education: Using

item response theory for the development and

validation of an ICT competence scale. Computers &

Education, v. 76, p. 168-181.

Ananiadou, K. and Claro, M. 2009. “21st Century Skills

and Competences for New Millennium Learners in

OECD Countries”, OECD Education Working Papers,

No. 41, OECD Publishing.

http://dx.doi.org/10.1787/21852526114

Ball, A., Joyce D. H., Butcher, A. D.; 2016. Exploring 21st

Skills and Learning Environments for Middle School

Youth. International Journal of School Social Work, v.

Basili, V. R.; Caldeira, G.; Rombach, H. D.; 1994. Goal

Question Metric Paradigm. In: Encyclopedia of

Software Engineering, John Wiley & Sons.

Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A.;

2005. Using an Expert Panel to Validate a

Requirements Process Improvement Model. Journal of

Systems and Software, 76(3), 251-275. doi:

10.1016/j.jss.2004.06.00

Binkley, M., Ola, E., Herman, J. Raizen, S., Ripley, M.,

Rumble, M., 2011. Defining 21st Century Skills. In:

Assessment and teaching of 21st century skills. New

York: Springer, p. 17-66.

Care, E.; Griffin, P.; 2014. An approach to assessment of

collaborative problem solving. Research and Practice in

Technology Enhanced Learning Vol. 9, No. 3,367－

388 - Asia-Pacific Society for Computers in Education.

Cattell, R. B., 1966. The Scree Test for the Number of

Factors, Multivariate Behavioral Research, 1:2, 245-

276, DOI: 10.1207/s15327906mbr0102_10.

Cevik, M., Senturk, C., 2019. Multidimensional 21st

century skills scale: Validity and reliability study.

Cypriot Journal of Educational Sciences, v. 14, n. 1, p.

11-28.

Chai, C. S.; Deng, F.; Tsai, P.; Koh, J.H.L., Tsai, C., 2015.

Assessing multidimensional students’ perceptions of

twenty-first-century learning practices. Asia Pacific

Education Review, v. 16, n. 3, p. 389-398.

Improvements in bASES21: 21st-Century Skills Assessment Model to K12

305

Claro, M.; Preiss, D. Martin, E.S.; Jara, I.; Hinostroza, J. E.;

Valenzuela, S.; Cortes, F.; Nussbaum, M.; 2012.

Assessment of 21st century ICT skills in Chile: Test

design and results from high school level students.

Computers & Education, v. 59, n. 3, p. 1042-1053.

Cohen, J.; 1998. Statistical Power Analysis for the

Behavioral Sciences. New York: Routledge Academic.

Cronbach, L. J. (1951). Coefficient alpha and the internal

structure of tests. Psychometrika, 16, p. 297–334. doi:

10.1007/BF02310555

CSTA (The CSTA Standards Task Force), CSTA K–12

Computer Science Standards – Revised 2011, New

York: ACM, 2011.

CSTA (The CSTA Standards Task Force), [INTERIM]

CSTA-12 Computer Science Standards: Revised 2016.

New York: ACM, 2016.

Delors, J., Mufti, I. A., Amagi, I., Carneiro, R., Chung, F.,

Geremek, B., Gorham, W., Kornhauser, A., Manley,

M., Quero, M. P., Savané, M., Singh, K., Stavenhagen,

R., Suhr, M. W. ; Nanzhao, Z. ,1996. Learning: The

treasure within. Paris: UNESCO. Retrieved from

http://unesdoc.unesco.org/images/0010/001095/10959

0eo.pdf.

Degering, L. P., 2019. Avaliação do Modelo de Avaliação

das Habilidades do Século XXI no Contexto do Ensino

de Computação na Educação Básica. Florianópolis.

Trabalho de Conclusão de Curso (Graduação em

Sistemas de Informação) – Universidade Federal de

Santa Catarina. Available at:

https://repositorio.ufsc.br/bitstream/handle/123456789

/202452/Monografia%20-

%20Leonardo%20Philippi%20Degering.pdf?sequence

=1&isAllowed=y

DeVellis, R. F., 2016. Scale Development: Theory and

Applications. Beverly Hills: SAGE Publications.

Drasgow, F.; 1986. Polychoric and polyserial correlations.

In: Kotz, S., Johnson, N. L. Encyclopedia of Statistical

Sciences, New York: John Wiley, 68-74.

Duckworth, A. L., Peterson, C., Matthews, M. D., Kelly, D.

R.; 2007. Grit: perseverance and passion for long-term

goals. Journal of Personality and Social Psychology,

92(6), 1087- 1101. doi: 10.1037/0022-3514.92.6.1087.

Gordon, J.; Halasz, G.; Krawczyk, M.; Leney, T.; Michel,

A.; Pepper, D.; Putkiewicz, E.; Wisniewski, J., 2009.

Key competences in Europe: Opening doors for lifelong

learners across the school curriculum and teacher

education, CASE Network Reports, No. 87, ISBN 978-

83-7178-497-2.

Harari, Y. N. (2018). 21 lessons for the 21st century. First

edition. New York: Spiegel & Grau.

IFL (Institute for the Future of Learning) Assessing the

Learning that Matters Most. 2015. Available at:

https://static1.squarespace.com/static/52c8c03fe4b0da

03983675b4/t/5653bcaae4b0bd925c080e46/14483283

62680/IFL+Report_rev_11.18.15+%282%29.pdf

Kang, M., Heo, H., Jo, I., Shin, J.; Seo, J.; 2010. Developing

an Educational Performance Indicator for New

Millennium Learners. Journal of Research on

Technology in Education, 43(2). doi:

10.1080/15391523.2010.10782567.

Kasunic, M.; 2005. Designing an effective survey.

NwHandbook CMU/SEI-2005-HB-004, Software

Engineering Institute. Pittsburgh: Carnegie Mellon

University.

Kyllonen, P. C.; 2012. Measurement of 21st Century Skills

Within the Common Core State Standards. In:

Proceedings of Invitational Research Symposium on

Technology Enhanced Assessments, 7-8.

Lau, W. W. F., Yuen, A. H. K., 2014. Developing and

validating of a perceived ICT literacy scale for junior

secondary school students: Pedagogical and

educational contributions. Computers & Education, v.

78, p. 1-9.

Martins-Pacheco, L., von Wangenheim, C., & Alves, N.,

2019. Assessment of Computational Thinking in K-12

Context: Educational Practices, Limits and Possibilities

- A Systematic Mapping Study. Proceedings of the 11th

International Conference on Computer Supported

Education (pp. 292-303). CSEDU2019

Mioto, F., Wangenheim, C. A. G., Petri, G., 2019. Um

Modelo para a Autoavaliação de Habilidades do Século

XXI no Contexto do Ensino de Computação na

Educação Básica. Revista Brasileira de Informática na

Educação, v.27, p. 26-57.

Mioto, F., 2018. Desenvolvimento de um Modelo de

Avaliação de Habilidades do Século XXI no Contexto

do Ensino da Computação na Educação Básica.

Florianópolis. Trabalho de Conclusão de Curso

(Graduação em Sistemas de Informação) –

Universidade Federal de Santa Catarina.

OECD (Organization for Economic Co-operation and

Development), 2013-2015. Collaborative problem

solving framework. Paris: OECD Publishing.

O’Neil, H. F., Schacter, J. ; 1997. Testing Specifications for

Problem Solving Assessments. Relatório técnico, Los

Angeles: University of California. 463. Disponível em:

https://cresst.org/wp-content/uploads/TECH463.pdf

Osman, K., Soh, T. M. T., Arsad, M. N.; 2010.

Development and validation of the Malaysian 21st

century skills instrument (M-21CSI) for science

students. Procedia - Social and Behavioral Sciences, v.

9, p. 599-603.

P21 (Partnership for 21st Century Skills), Framework for

21st Century Learning Definitions, 2019.

http://www.battelleforkids.org/networks/p21/framewo

rks-resources.

P21 (Partnership for 21st Century Skills), P21 framework

definitions, 2015. Available at:

<http://www.p21.org/storage/documents/P21_Framew

ork_Definitions.pdf>.

P21 (Partnership for 21st Century Skills), Computer

Science: A playground for 21st century skills, 2017.

Disponível em <http://www.p21.org/news-

events/p21blog/2128-computer-science-a-playground-

for-21st-century-skills>.

Petersen, K., Vakkalanka, S., Kuzniarz, L., 2015.

Guidelines for conducting systematic mapping studies

in software engineering: An update. Information and

Software Technology, v. 64, p. 1-18.

CSEDU 2020 - 12th International Conference on Computer Supported Education

306

Petway, K. T., Rikoon, S. H., Brenneman, M. W., Burrus,

J., Roberts, R. D., 2016. Development of the Mission

Skills Assessment and Evidence of Its Reliability and

Internal Structure. doi:10.1002/ets2.12107.

https://www.ets.org/Media/Research/pdf/session5-

kyllonen-paper-tea2012.pdf.

Pischetola, M. 2019. Inclusão Digital e Educação: a nova

cultura da sala de aula. Ed. Vozes; Rio de Janeiro,

Brazil. ISBN 978-85-326-6196-8.

Raîche, G., Walls, T. A., Magis, D., Riopel, M. & Blais, J-

G, 2013. Non-Graphical Solutions for Cattell’s Scree

Test. Methodology 2013; Vol. 9(1):23–29. DOI:

10.1027/1614-2241/a000051

Rosen, Y.; 2015. Computer-based Assessment of

Collaborative Problem Solving: Exploring the

Feasibility of Human-to-Agent Approach. International

Journal of Artificial Intelligence in Education, v.25,

n.3, p. 380-406.

Rosen, Y., Tager, M.; 2014. Computer-based Performance

Assessment of Creativity Skills: A Pilot Study. In: Proc.

of International Association for Educational

Assessment Conference, Singapore.

Ross, J. A.; 2006.The reliability, validity, and utility of self-

assessment. Practical Assessment, Research &

Evaluation, v. 11, n. 10, p. 1-13.

Siddiq, F., Gochyyev, P., Wilson, M.; 2017. Learning in

Digital Networks - ICT Literacy: A novel assessment of

students’ 21st century skills. Computers & Education,

v. 109, p. 11-37.

Souza, A. C. D., Alexandre, N. M. C., Guirardello, E. D. B.,

2017. Psychometric properties in instruments

evaluation of reliability and validity. Epidemiologia e

Serviços de Saúde, v. 26, p. 649–659.

Susnea, I., Vasiliu, G.; 2016. A Fuzzy Logic Software Tool

and a New Scale for the Assessment of Creativity.

International Journal of Computers Communications &

Control, v. 11, n. 3, p. 441- 449.

Szikora, P. & Ali, B., 2018. Information Security in the 21st

Century - Smart Phones in Focus. 2018 IEEE 16th

International Symposium on Intelligent Systems and

Informatics (SISY). DOI: 000255-000260.

10.1109/SISY.2018.8524767.

Takahashi, F. and Saldaña, P.; 2018. Ensino de má

qualidade acentua desigualdade e violência no país.

Folha de São Paulo (Newspaper), 09.11.2018.

Available at: https://temas.folha.uol.com.br/e-agora-

brasil-educacao/introducao/ensino-de-ma-qualidade-

acentua-desigualdade-e-violencia-no-pais.shtml

Trochim, W. M. & Donnelly, J. P.; 2018. Research Methods

Knowledge Base. Mason: Atomic Dog Publishing.

Tsai, M., Wang, C., Hsu, P., 2018. Developing the

Computer Programming Self-Efficacy Scale for

Computer Literacy Education. Journal of Educational

Computing Research, 1-16. doi:

10.1177%2F0735633117746747.

van Laar, E.; van Deursen, A.J.A.M.; van Dijk, J. A.G.M.;

Haan, J.; 2017. The relation between 21st-century skills

and digital skills: A systematic literature review.

Computers in Human Behavior, V72, 577-588.

van Laar, E.; van Deursen, A.J.A.M.; van Dijk, J. A.G.M.;

Haan, J., 2018. 21st-century digital skills instrument

aimed at working professionals: Conceptual

development and empirical validation. Telematics and

Informatics, v. 35, p. 2184-2200.

Wangenheim, C. G., Alves, N. C., Weber, A. R.; 2017.

Resumo do K-12 Computer Science Standards (Versão

2017). Relatório Técnico do INCoD/INE/UFSC,

Florianópolis/SC

Wohlin, C.; Runeson, P.; Host, M.; Ohlsson, M.C.; Regnell,

B.; Wesslen, A.; 2012. Experimentation in Software

Engineering: An Introduction. Springer.

Improvements in bASES21: 21st-Century Skills Assessment Model to K12

307