Are 21st Century Skills Evaluated in Robotics Competitions?

The Case of First LEGO League Competition

Mireia Usart

, Despoina Schina

, Vanessa Esteve-Gonzalez

and Mercè Gisbert

Universitat Rovira I Virgili, Tarragona, Spain

Keywords: Educational Robotics, 21st Century Skills, FLL, Learning Assessment Tools, Core Values.

Abstract: Citizens of the 21st century need to acquire skills related to collaboration and project design in order to be

able to meet the current demands for the workplace. Challenge-based learning contexts such as FIRST LEGO

League (FLL) Competition connect students with professionals and enable them to solve real-world problems

and develop 21st century skills. Main aim of this study is to examine whether 21st century skills are evaluated

in the context of FIRST LEGO League (FLL) Competition within Core Values, Project and Robot Design

competition categories. Our study is based on an expert analysis of the 21st century skills as presented in the

FIRST LEGO League (FLL) evaluation rubrics. Results show that the main 21st century skills are displayed

at the FLL evaluation rubric, in particular communication and problem solving. This study could be helpful

to FLL experts and coaches and could enable them to better target 21st century skills in the teams’ training

and evaluation.

1 INTRODUCTION

The World Economic Forum (2016) projected that

from 2020, there will be a shift in employee skill

requirements. Most needed skills would be

(Complex) problem solving, Critical thinking,

Creativity, and Coordinating with others.

Most curricula in formal education in primary and

secondary school cover science and math subjects,

however, less effort is applied in developing

creativity and problem solving (Pellegrino and

Hilton, 2012) - skills that are usually identified as 21st

century skills (Binkley et al., 2010; Voogt and Pareja

Roblin, 2012). Formal learning alone is not sufficient

to anticipate these evolutions in society since it takes

too long to set it up (Baert, 2000). In order to reinforce

the development of students’ skills, new learning

methods need to be implemented in both formal and

informal educational contexts. As Trilling and Fadel

(2009) points out “new learning methods should

include inquiry, design and collaborative learning

projects”.

https://orcid.org/0000-0003-4372-9312

https://orcid.org/0000-0002-6691-9170

https://orcid.org/0000-0001-5909-1099

https://orcid.org/0000-0002-8330-1495

During the last decade, results from different

studies in the field of educational robotics (ER)

highlight that primary and secondary school students’

engagement with ER has the potential to transmit key

skills for future citizens and equip them with

important for the 21st century skills (Eguchi, 2013;

Afari and Khine, 2017). ER is popular among

students of all ages however, it is mainly being

applied in informal learning contexts (Alimisis,

2013). Robotics competitions is an informal

educational context that offers a lot of opportunities

for learning and proposes new learning methods.

In our research, the main aim is to explore whether

21st century skills are evaluated in the context of First

Lego League (FLL) competition. We will study the

21st century skills (as defined in the most mentioned

frameworks) in comparison with the categories and

skill areas of the FLL evaluation rubric.

Usart, M., Schina, D., Esteve-Gonzalez, V. and Gisbert, M.

Are 21st Century Skills Evaluated in Robotics Competitions? The Case of First LEGO League Competition.

DOI: 10.5220/0007757404450452

In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), pages 445-452

ISBN: 978-989-758-367-4

445

2 21st CENTURY SKILLS

Puncreobutr (2016) states that the 21st century

context requires a new set of competences and skills

such as critical thinking, creativity and innovation,

cross-cultural understanding, information and media

literacy, career and learning skills. This new set of

skills goes far beyond the 20th century skills

addressed in formal education (Wisniewski, 2010),

based on the 3 R’s of Reading, wRiting, and

aRithmetic and focusing on the ‘‘lower levels of

Bloom’s taxonomy—knowledge, comprehension and

application’’. According to Trilling and Fadel (2009),

21st century skills are those skills required to succeed

in learning, working, and living.

Given the highly informal nature of 21st century

skills (Kickmeier-Rust and Dietrich 2012) and the

fact that researchers and educational projects have

developed different definitions of the 21st century

skills (Voogt and Pareja Roblin, 2012), there is yet no

commonly accepted definition nor list of skills

(Romero et al., 2015). However, the existing

frameworks are largely consistent and generally

focused on skills related to communication,

collaboration and problem solving.

Our aim is to better understand the evaluation of

21st century skills in the context of robotics

competitions and in particular in the FLL

Competition. To achieve that, we will study the 21st

skills century skills as reported by Romero et al.

(2015), in the context of Game-based Learning Their

research was based on the Voogt and Pareja Roblin

(2012) meta-analysis of the six following

frameworks: , P21 (2007), EnGauge 21st century

skills (Metiri Group and North Central Regional

Educational Laboratory, 2003), Assessment and

teaching of 21st century skills (ATCS), National

Educational Technology Standards and International

Society for Technology in Education (NETS/ISTE)

framework, competences for new millennium

learners by Organization for Economic Cooperation

and Development (OECD, 2005), and Center for

Social and Economic Research (Gordon et al., 2009).

In this research, the focus is on the eight most

mentioned skills of the 21st century skills

frameworks, as stated by Romero et al. (2015),

(Figure 1).

From the perspective of skills development and

evaluation, some authors consider that 21st century

skills should be instructed early from primary school

levels and be addressed both in formal and in informal

learning settings (DeJarnette, 2012). However, as

mentioned above, formal learning alone could not be

sufficient, since it takes too long to set it up (Baert,

2000). In this regard, according to Voogt et al. (2013),

there is substantial potential to acquire 21st century

skills in informal learning settings. Furthermore, the

adoption of active learning methodologies seems to

encourage self-regulated learning and help students

develop these skills (Bell and Kozlowski, 2008).

Figure 1: 21st century skills mentioned in six related

frameworks (adapted from Romero et al., 2015).

Active learning enhances engagement in learning

tasks and offers significant added value to provide

learners with the abilities to cope with new

challenges, solve problems of real life, and adapt to

changes in technology and knowledge (Edens, 2000).

When developing, assessing and evaluating 21st

century skills, two main areas can be defined

(Pellegrino and Hilto, 2012): Interpersonal and

intrapersonal domains. These authors state that

research in formal education, e.g. English language

arts, illustrated the potential for developing both

intrapersonal and interpersonal domains of 21st

century skills, and its relation to motivational factors.

Nevertheless, the authors admit that these domains

are difficult to measure in formal instruction, as they

focus on domain-specific learning goals.

Furthermore, Ma and Williams (2013), studied

the development of 21st century skills in a robotics

competition (FLL) from learners’ standpoint. The

authors found that teamwork was one of the more

commented aspects when students were asked what

they have learned from the FLL process.

From this review on the development and

evaluation of 21st century skills in informal and

active educational contexts, we can see that a team-

based activity with robot design could assess most of

these skills.

2.1 21st Century Skills in Educational

Robotics

In the context of active and informal learning settings,

research on the 21st century skills is just beginning.

Our search for relevant literature was carried out with

a selective literature review, through the research

databases: Science Direct and EbscoHost. The two

CSEDU 2019 - 11th International Conference on Computer Supported Education

446

search engines were chosen because most of the

journals in the field of education can be found via

these engines. We used two keywords: “21st century

skills” and “robotics education” and publication years

2013-2019. From this process, 118 papers emerge.

After getting the results from the two databases, each

abstract and reference section were classified. All the

references on 21st century skills and robotics

education cited were also included and cross-

referenced with the results found in the first search.

From the database only 10 papers stand. Furthermore,

up to authors’ knowledge, no research has been

published in the particular topic of skills’ evaluation

in robotics education.

Most of the studies in the field use self-reported

questionnaires to measure students’ and teachers’

perceptions on the improvement of knowledge and

skills in programming, problem solving, teamwork,

project management, and communication (Petre and

Price, 2004; Nourbakhsh et al., 2005; Robinson,

2005). However, the following studies aimed to

measure skills’ improvement in the area of ER.

Barker and Ansorge (2007) measured how robotics

activities improved students’ achievement in science,

engineering and technology in a quasi-experimental

setting. Also, Williams et al. (2007) found that

students’ physics content knowledge improved after

a summer camp on robotics, however, it failed to

show improvement on skills such as scientific

inquiry. In our study, we are going to go a step further

and attempt to study which skills that are being

evaluated in the context of a robotics competition. To

further look into the skills developed in a

competition, first the educational benefits that ER

brings regarding the acquisition of 21st century skills

will be examined and then the specific context of

robotics competitions will be presented.

The skills that students can acquire in the area of

21st century framework through ER activities are

really important. First of all, according to Eguchi

(2013), collaboration skills, communication skills,

creative thinking, and critical thinking/problem-

solving (4Cs), all skills that are needed to be

successful as a 21st century citizen, can be targeted in

an ER course. According to Eguchi (2017),

“educational robotics is rich with opportunities to

integrate not only STEM but also many other

disciplines, including literacy, social studies, dance,

music, and art”. Eguchi’s view on ER’s potential to

be integrated across disciplines is in line with the

Partnership For 21st Century Skills P21 (2015), that

supports that it is important to “build understanding

across and among key subjects as well as 21st century

interdisciplinary themes”. Furthermore, Eguchi

(2017), underlines that in ER settings “students have

the opportunity to find ways to work together to foster

collaboration skills, express themselves using the

technological tool, problem-solve, and think critically

and innovatively”. The need to develop skills such as

critical thinking, problem solving, communication

and collaboration is also highlighted by P21 (2007).

2.2 21st Century Skills in Robotic

Competitions

Regarding the context of robotics competitions, goal-

oriented and project-based learning (PBL)

approaches to learning are employed (Eguchi, 2016).

Given the fact that PBL supports 21st century

learning goals, (Buck Institute for Education, 2019),

we could infer that robotics competitions can promote

21st century skills development. To be more precise,

in the context of First Lego League (FLL)

Competition, students perceive that they have

improved their learning about real-world

applications, problem solving, engagement,

communication, and the application of the

technology/engineering cycle (Chalmers, 2013) skills

that are related to the 21st century skills framework.

In particular, problem solving skills development is

highly addressed in the context of FLL Competition

(Chen, 2018; Varnado, 2015). In addition, it has also

been reported that through FLL Competition, the

participants develop their collaboration skills (Ma

and Williams, 2013). The learning outcomes are

significant in the context of the FLL competition:

participants are equipped with necessary skills

important for success in the 21st century.

2.2.1 Evaluation of the 21st Century Skills

in Robotics Competitions (FLL)

Ma and Williams (2013) studied the potential of FLL

for teaching 21st century skills. It was found that FLL

provided opportunities for acquiring many 21st

century skills, in particular systems thinking, decision

making, problem solving, teamwork, conflict

resolution, flexibility, perseverance, and self-

management. However, all these skills outlined

above, have been reported by FLL participants. Our

research aims at going one step further and

investigating whether 21st century skills are also

evaluated in the context of FLL.

2.2.2 Context: First Lego League

Competition

FIRST LEGO League Competition (FLL) is an inter-

Are 21st Century Skills Evaluated in Robotics Competitions? The Case of First LEGO League Competition

447

national ER competition addressed to students from 9

to 16 years old (FIRST LEGO League, 2019). The

competition takes place in a lot of countries around

the world and the participants are assessed by the

same evaluation criteria regardless of the country in

which they compete. FLL Competition always has a

theme, the theme from 2018 season was

“Hydrodynamics” (FIRST LEGO League). Teams

compete in 4 categories, Robot Design, Core Values,

Project and Robot Game. The content of each

category will be briefly presented below.

First of all, in the Robot Design category, the

teams are asked about the design of their robot in an

interview called the technical review (Kelly and

Daudelin, 2008). The judges look for well-designed

robots that can accomplish missions in consistent,

clever or unique ways (Kelly and Daudelin, 2008). As

far as Core Values category is concerned, the judges

make an evaluation based on how the team members

work together to accomplish a given task. Teams

show “Gracious professionalism” - meaning to act

gracefully and respectfully to teammates, other

teams, and visitors to the competition (FIRST LEGO

League, 2019). If a disagreement comes up, the

judges will expect the team members to listen to each

other and professionally resolve the disagreement.

The Project category is one of the non-robotic

categories of the FLL competition and resembles to a

science fair (Kelly and Daudelin, 2008). Based on the

theme of the competition, the teams carry out a

research, regarding a real-world and present their

research and solutions. Teams are urged to choose a

topic that has a direct impact to the community. The

teams are asked to present the original research, e.g.

personal interviews with scientists or information

discovered during experiments in the judging rooms

and in their pits. In the above-mentioned categories of

the competition, teams are evaluated based on the

respective evaluation sheets that will be presented in

the instruments section (3.2.).

3 OUR STUDY

The main aim of our study is to explore the evaluation

of 21st century skills in the context of the FLL

competition analyzing the FLL rubric. This research

is a first step in the study of 21st century skills in

comparison with the categories, skill areas and sub-

skill areas (items) of the FLL evaluation rubric.

The Research Question that will guide our work

is: Can 21st century skills be evaluated using the FLL

rubric?

3.1 Design

As previously mentioned, our main objective is to

study if 21st century skills are evaluated in the

competition by analyzing the evaluation rubrics used

by the judging committees. An interpretative

paradigm, with a qualitative methodology and an

expert evaluation technique has been chosen to

retrieve data.

Regarding the expert evaluation process, the 21st

century skills table (Figure 1) was compared and

contrasted with the evaluation rubrics for each

competition category by 4 experts. The 21st-century

skills table includes the following 8 skills:

communication, collaboration, social/cultural skills,

ICT literacy, creativity, critical thinking, problem-

solving and development of quality products. These 8

skills were juxtaposed and contrasted with FLL

evaluation rubrics in order to examine whether 21st

century skills are evaluated in the competition. The

experts examined carefully the evaluation rubrics and

their reference to 21st century skills. To solve the

issue of coding reliability, researchers worked

together and disagreements were solved after

discussion.

3.2 Instruments

The instruments used for data retrieval were the

official evaluation sheets as retrieved by the official

FIRST website https://firstinspires.org. There are

three evaluation rubrics corresponding to each of the

three judging committees Robot Design, Core Values

and Project. The evaluation of the Robot Game is not

based on a rubric related to assessment of skills, but

rather on a scoring system for accomplishing

missions. For this reason, Robot Game evaluation

will be excluded from this analysis. The three

evaluation sheets Robot Design, Core Values and

Project share the same structure and instructions.

Regarding structure, in each evaluation sheet, the

team is evaluated based on 9 sub-skills each

corresponding to 3 main skill areas. The judge is

instructed to evaluate the teams’ sub-skills by

choosing what best describes the team's

accomplishments among: “Beginning”,

“Developing” “Accomplished” and “Exemplary”.

Additionally, judges are asked to provide written

comments under each skill area as feedback for the

teams. The evaluation rubric for each judging

committee is further explained below:

In Robot Design, the teams are evaluated on the

following skill areas: mechanical design,

programming and strategy and innovation (Judging

CSEDU 2019 - 11th International Conference on Computer Supported Education

448

Rubrics, 2018). In mechanical design the teams are

evaluated based on the robot’s durability and

mechanical efficiency meaning the “Economic use of

parts and time; easy to repair and modify” (Judging

Rubrics, 2018). The second skill area is

Programming, and teams are evaluated in

programming quality, programming efficiency and

automation/navigation. The third skill area is strategy

and innovation and its evaluation is based on the

following sub-skills: design process, mission strategy

and innovation (Judging Rubrics, 2018).

The Core Values evaluation rubric contains the

main skill areas, inspiration, teamwork and gracious

professionalism (Judging Rubrics, 2018). Inspiration

is evaluated based on discovery, team spirit and

integration, while teamwork is evaluated based on

effectiveness, efficiency and “kids do the work”

referring to “Appropriate balance between team

responsibility and coach guidance” (Judging Rubrics,

2018). Gracious professionalism is evaluated based

on inclusion, respect and “coopertition” - spirit of

friendly competition and cooperation with others

(Judging Rubrics, 2018).

As far as Project category is concerned, the three

main skill areas are research, innovative solution and

solution development (Judging Rubrics, 2018). First,

research skill area is divided into the following sub-

skills: problem identification, sources of information

and problem analysis. The second skill area is

evaluated based on team solution, innovation and

solution development. Solution development refers to

the “Systematic process used to select, develop,

evaluate, test, and improve the solution” (Judging

Rubrics, 2018). Finally, presentation is assessed

based on sharing, creativity and presentation

effectiveness (Judging Rubrics, 2018). Based on the

sub-skills area analysis, it could be inferred that

Project category is related to Project Based Learning.

3.3 Data Analysis

To study the link between the rubric and the 21st

century skills, we had to take into account all three

levels of analysis of the Rubric, corresponding to

categories (Robot Design, Core Values and Project),

the skill areas (9) and sub-skills (27). The procedure

of the data analysis was based on an expert evaluation

of the table, where 4 experts at Education and

Robotics (two teachers and two researchers, one of

them with experience in FLL) classified all 27 sub-

skills of the evaluation rubric in 8 categories

corresponding to the 8 skills of 21st century

framework.

4 RESULTS

In this study there are two levels of analysis

corresponding to the Categories (3) and skill-areas (9)

of the FLL Rubric (Judging Rubrics, 2018). The 21st

century skills (Romero et al., 2015) were compared to

the two levels of analysis of the FLL rubric categories

and skill areas (Figure 2).

Figure 2: 21st-century skills distribution in the FLL Rubri.

Figure 3: FLL Evaluation matrix VS 21st-century skills.

Analyzing the results displayed in Figure 2, in the

Robot Design category, 21st century skills are

reported in the rubric 22 times. Strategy/Innovation

skill area refers 11 times to 21st century skills,

programming skill area 8 times, while Mechanical

Design 3 times. Project is the category measuring less

skills, they are in total 21. Finally, Core Values is the

category measuring more times the 21st century skills

with 29 entries. In particular, the skill area Inspiration

measures 21st century skills for 17 times, while

Gracious Professionalism, and Teamwork 6 times

each.

According to the results, the most evaluated 21st

century skills are communication and problem

Are 21st Century Skills Evaluated in Robotics Competitions? The Case of First LEGO League Competition

449

solving (Figure 3). Problem solving and

communication are both displayed 14 times in the

FLL rubric. Interestingly, the less evaluated skills are

critical-thinking and ICT literacy- they are measured

4 and 6 times respectively.

5 DISCUSSION

The main aim of this study was to explore the

evaluation of 21st century skills in the FLL

Competition, using the FLL evaluation rubric. Up to

our knowledge, there is no previous research regarding

the evaluation of the 21st century skills in this context.

Ma and Williams (2013) studied the case of FLL

Competition from the point of view of exploring its

potential for teaching 21st century skills. Continuing in

this line of research, our research results on the FLL

Competition encourage the development of the 21st

century skills as the evaluation sheets used by the

judging committees assess teams according to the 21st

century skills.

The four skills most often assessed in the FLL

evaluation rubric are presented and discussed in the

table 1.

Further discussing the results, on one hand, the

importance of the Communication and Social /

Cultural skills is highlighted. These skills are

evaluated in a number of FLL sub-skill areas, and

allow researchers and coaches of the Competition to

reflect on the importance of teamwork, respect and

sharing in the process of the competition. The

importance of teamwork in the FLL competition was

also pointed out by previous research in the field

(Petre and Price, 2004; Chalmers, 2013). According

to Chalmers, the FLL participants perceive the

importance of teamwork: they gain new ideas and/or

different perspectives when communicating with

other students. Our contribution to current literature

lies on the fact that teamwork and communication

skills are not only self-perceived skills by FLL

participants but, they are now traced in the official

FLL evaluation procedure.

On the other hand, Critical Thinking and ICT

literacy are the least evaluated skills in the rubric. The

first could be due to the fact that students between 9

and 16 years old still demand a high guidance from

coaches and teammates (Suthers and Hundhausen,

2003). Additionally, it could be inferred that in the

FLL evaluation rubrics, a greater emphasis is placed

on Problem Solving competence, rather than on

Critical Thinking. As also pointed out in previous

research (Chen, 2018; Varnado, 2015), problem

solving skills development is highly addressed in the

Table 1: The most addressed 21st century skills in the FLL

Evaluation Rubric.

21st century skills

FLL Categories and skill areas

evaluate (and can develop)

Communication

Project category, and

Presentation skill area in

particular evaluate this skill.

Also, Core Values category

(scoring higher in our study),

with Inspiration and Teamwork

to be the skill areas most closely

related to this 21st century skill.

Problem Solving

Programming and Strategy and

Innovation skill areas (from

Robot Design category) are the

areas that are most closely

related to problem solving.

Develop Quality

Products

Both Robot Design and Project

(Innovative Solution and

Inspiration skill areas) focus on

planning and managing useful

robots for real needs that are

tested during the competition.

Social/Cultural skills

As in the above-mentioned

Communication skill,

Social/Cultural skills are mainly

measured in Project and Core

Values categories.

context of FLL Competition. As far as ICT skills are

concerned, they are considered to be less frequently

present in the FLL evaluation rubric and this could be

due to the fact that FLL focuses on the development

of engineering and programming skills (Oppliger,

2002) rather than the development of ICT skills. In

accordance with this finding, Romero, Usart and Ott

(2015) stated that the 21st context requires a new set

of skills that go beyond the obvious information and

communication technology (ICT) literacy.

Our findings are of particular interest to the

coaches who will now be better aware of the skills

that need to be reinforced during FFL training.

Additionally, coaches could integrate suitable

activities to address their team members’ needs by

making amendments to the existent training plan.

6 CONCLUSIONS

This study addressed the topic of 21st century skills

evaluation in the context of FIRST LEGO League.

The results from our exploratory study enable us to

CSEDU 2019 - 11th International Conference on Computer Supported Education

450

draw the following conclusions:

21st century skills are indeed evaluated in the

FLL competition through the evaluation rubric

for each category (Robot Design, Core Values

and Project). The 21st century skills evaluated

in the competition are communication,

collaboration, social/ cultural skills, ICT

literacy, creativity, critical thinking, problem

solving, developing quality products.

The FLL evaluation rubric gives particular

emphasis on the evaluation of Communication

and Problem Solving skills.

The FLL rubric is a 21st century skills

evaluation instrument that FLL coaches and

experts should take into account when

delivering ER training sessions. In addition, the

skills outlined could be integrated in the

teaching of robotics in the formal curriculum.

Last but not least, the teaching of these skills

should be included into future teachers’ training

programs.

It must be underlined that this exploratory study

is a first step in the research of how 21st century skills

are reinforced and can be evaluated in the context of

ER. Further research -as part of a PhD study- will

include the integration and the evaluation of 21st

century skills in the teacher training for educational

robotics.

ACKNOWLEDGEMENTS

This project has received funding from the European

Union's Horizon 2020 research and innovation

programme under the Marie Skłodowska-Curie grant

agreement No. 713679 and from the Universitat

Rovira i Virgili (URV).

REFERENCES

Afari, E., Khine, M.S., 2017. Robotics as an Educational

Tool: Impact of Lego Mindstorms, International Journal

of Information and Education Technology.

http://www.ijiet.org/show-88-1038-1.html.

Alimisis, D., 2013. Educational Robotics: Open Questions

and New Challenges, Themes in Science & Technology

Education, pp. 63–71.

Baert, P., 2000. Social Theory m the Twentieth Century,

New York: University Press.

Barker, B.S., Ansorge, J., 2007. Robotics as means to

increase achievement scores in an informal learning

environment, Journal of Research on Technology in

Education, pp. 229–243.

Bell, B.S., Kozlowski, S.W.J., 2008. Active learning:

Effects of core training design elements on self-

regulatory processes, learning, and adaptability,

Journal of Applied Psychology, pp. 296–316.

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M.,

& Rumble, M., 2011. Defining 21st century skills, draft

white paper. Melbourne, Australia: University of

Melbourne.

Buck Institute for Education, 2019. What is Project Based

Learning (PBL)? http://bie.org/about/what_pbl.

Chalmers, C., 2013. Learning with FIRST LEGO League.

In Society for Information Technology and Teacher

Education (SITE) Conference. Association for the

Advancement of Computing in Education (AACE),

New Orleans, pp. 5118-5124.

Chen X., 2018. How does participation in FIRST LEGO

League robotics competition impact children’s

problem-solving process?. In Robotics in Education.

RiE 2018. Advances in Intelligent Systems and

Computing. Springer.

DeJarnette, N.K., 2012. America's children: Providing

early exposure to STEM initiatives, Journal of

Education.

Edens, K. M., 2000. Preparing problem solvers for the 21st

century through problem-based learning, College

Teaching, pp. 55–60.

Eguchi, A., 2013. Educational Robotics for Promoting 21st

Century Skills, Journal of Automation, Mobile Robotics

& Intelligent Systems.

Eguchi, A., 2016. RoboCupJunior for Promoting STEM

Education, 21st Century Skills, and Technological

Advancement through Robotics Competition, Robotics

and Autonomous Systems.

http://dx.doi.org/10.1016/j.robot.2015.05.013.

Eguchi A., 2017. Bringing Robotics in Classrooms,

Robotics in STEM Education, Springer.

FIRST LEGO League., 2019. FIRST LEGO League

website. Retrieved from https://www.firstinspires.org

Gordon, J., Hala´sz, G., Krawczyk, M., Leney, T., Michel,

A., Pepper, D., Wis´niewski, J., 2009. Key competences

in Europe: Opening doors for lifelong learners across

the school curriculum and teacher education. CASE

Network Reports, 87, 328.

Judging Rubrics., 2018. Retrieved from

https://www.firstinspires.org/resource-

library/fll/judging-rubrics

Kelly, J.F., Daudelin, J., 2008. First LEGO League. The

Unofficial Guide, No Starch Press.

Kickmeier-Rust, M. D., Dietrich, A., 2012. A domain

model for smart 21st century skills training in game-

based virtual worlds. In ICALT 2012 Proceedings.

Rome, pp. 680–681.

Ma, Y., Williams, D.C., 2013. The Potential of a First

LEGO League Robotics Program in Teaching 21st

Century Skills: An Exploratory Study, Journal of

Educational Technology Development and Exchange

century_skills_education_and_competitiveness_guide.

Metiri Group & North Central Regional Educational

Laboratory, 2003. EnGauge 21st century skills:

Literacy in the digital age, Chicago, IL: NCREL.

Are 21st Century Skills Evaluated in Robotics Competitions? The Case of First LEGO League Competition

451

Nourbakhsh, I. R., Crowley, K., Bhave, A., Hamner, E.,

Hsiu, T., Perez-Bergquist, A., Richards, S., Wilkinson,

K., 2005. The robotic autonomy mobile robotics

course: Robot design, curriculum design and

educational assessment, Autonomous Robots, 18(1),

103–127.

Oppliger, D., 2002. Using FIRST LEGO League to

Enhance Engineering Education and to Increase the

Pool of Future Engineering Students (work in

progress). In 32nd Annual Frontiers in Education.

Boston, pp. S4D-S4D.

Organization for Economic Cooperation and Development,

2005. The definition and selection of key competencies:

Executive summary. Paris, France: OECD. Retrieved

June 20, 2014, from

http://www.oecd.org/pisa/35070367.pdf

Partnership for 21st Century Skills, 2007. Framework for

21st century learning. Retrieved June 20, 2014, from

http://www.p21.org/about-us/p21-framework

Pellegrino, J. W., Hilton, M. L. (Eds.), Committee on

Defining Deeper Learning and 21st Century Skills,

Center for Education, Board on Testing and

Assessment, Division of Behavioral and Social

Sciences and Education & National Research Council.

(2012). Education for life and work: Developing

transferable knowledge and skills in the 21st century.

The National Academies Press. Retrieved from

http://www.p21.org/storage/documents/Presentations/

NRC_Report_Executive_Summary.pdf

Petre, M., Price, B., 2004. Using robotics to motivate ‘back

door’ learning, Education and Information

Technologies.

Puncreobutr, V., 2016. Education 4.0: New Challenge of

Learning, St. Theresa Journal of Humanities and Social

Sciences, pp. 92-97.

Romero, M., Usart, M., Ott, M., 2015. Can Serious Games

Contribute to Developing and Sustaining 21st Century

Skills? Games and Culture, pp. 148-177.

Robinson, M., 2005. Robotics-driven activities: Can they

improve middle school science learning? Bulletin of

Science Technology and Society, 25(1), 73–84.

Suthers, D., Hundhausen, C., 2003. An Experimental Study

of the Effects of Representational Guidance on

Collaborative Learning Processes, Journal of the

Learning Sciences, 12(2), pp. 183-218.

Trilling, B., Fadel, C., 2009. 21st century skills: Learning

for life in our times. San Francisco, CA: Jossey-Bass.

Varnado, Τ.Ε., 2005. The effects of a technological problem

solving activity on FIRST LEGO League participants’

problem solving style and performance. Virginia Tech.

Voogt, J., Erstad, O., Dede, C., Mishra, P., 2013.

Challenges to learning and schooling in the digital

networked world of the 21st century, Journal of

Computer Assisted Learning.

Voogt, J., Pareja Roblin, N., 2012. A comparative analysis

of international frameworks for 21st century

competences: Implications for national curriculum

policies, Journal of Curriculum Studies, 44, pp. 299–

321.

Williams, D.C., Ma, Y., Prejean, L., Ford, M. J., 2007.

Acquisition of physics content knowledge and scientific

inquiry skills in a robotics summer camp, Journal of

Research on Technology in Education, 40(2), 201–216.

Wisniewski, M. A., 2010. Leadership and the millennials:

Transforming today’s technological teens into

tomorrow’s leaders, Journal of Leadership Education,

9, 53–68.

World Economic Forum, 2016. The future of jobs.

Employment, skills and workforce strategy for the

fourth industrial revolution. Retrieved from:

http://www3.weforum.org/docs/WEF_Future_of_Jobs.

pdf

CSEDU 2019 - 11th International Conference on Computer Supported Education

452