Empowering Teachers’ Self Assessment of Their Own Practice on

Student’s Oral Skills

EVALOE Decision Support System

Marta Gràcia

, Josep Casanovas

2,4

, Maria-Ribera Sancho

3,4

Jordi Casanovas

1,2

and

Marta Cuatrecasas

Department of Cognition, Development and Educational Psychology, Institute of Research in Education,

Universitat de Barcelona, Spain

Department of Statistics and Operations Research, Universitat Politècnica de Catalunya, Spain

Department of Service and Information Systems Engineering, Universitat Politècnica de Catalunya, Spain

Barcelona Supercomputing Center, Barcelona, Spain

Keywords: Teachers, Self-Assessing, Decision Support System, Oral Skills.

Abstract: School is an essential context that contributes to children’s communicative competence. Decision Support

Systems (DSS) are tools that can help teachers to assess their own practice and to make decisions to improve

it (Gregg, 2009). Our aim is the construction, implementation and validation of a digital Decision Support

System (EVALOE-DSS), conceived as a teacher empowerment tool for self-assessment and decision making

about their teaching practice concerning students’ oral skills. Participants are 4 teachers and their groups of

students of 4 diverse schools. The research procedure includes: 1) design and development of a DSS for

different type of schools; 2) researcher assessment of one classroom observation using EVALOE-DSS; 3)

weekly teacher self-assessment of classroom observation and decision making using EVALOE-DSS; 4)

analysis of self-assessments, decision making and changes in teaching practices; 5) teacher-researcher

assessment of the helpfulness of DSS and introduction of changes. EVALOE-DSS is a multimedia tool

consisting of 30 items that assess 5 dimensions: teaching management, instructional design, communicative

functions and teacher strategies, students’ communicative functions, and students’ management. It includes,

also, a brief description of each item and a variety of resources to help teachers to make decisions and

introduce the actions in their teaching practice.

1 INTRODUCTION

Although computers have been integrated into areas

related to educational management in many countries

since the 1970s, the integration of Decision Support

Systems (DSS) is still very recent and represents a

major challenge. The majority of published works

focus on the development of systems to improve

general aspects of school organization (Sadahiro and

Sadahiro, 2012) or systems to improve student

performance (Xu and Wang, 2006). Few works focus

on the construction of systems aimed at improving the

practice of teachers, with the ultimate goal of

contributing to the development of students'

competences. This paper reports our research in the

application of a first version of a DSS in the teaching

and learning of oral language by a teacher of early

childhood education.

1.1 Communicative Competence at

School

Catalan education law which develops the

organization of curricular teaching and development

in all educational stages defines the school as a space

of communication. A space where children can learn

to talk by talking and interacting with their mates and

with competent adults, in the classrooms, when in

these classes the conversation is the focus. On the

other hand, these regulations establish the ordering of

primary education and places great emphasis on

affirming that communicative and linguistic

competence should be the basis of all learning. This

development is the responsibility of all the areas of

the curriculum. Thus, educational legislation makes

clear that oral language plays a major role in the

272

Gràcia, M., Casanovas, J., Sancho, M., Casanovas, J. and Cuatrecasas, M.

Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills.

DOI: 10.5220/0006785902720277

In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 272-277

ISBN: 978-989-758-291-2

development of school activities: children must learn

to speak, listen, expose and dialogue.

The conceptual framework that we have recently

presented, Gràcia, Galván-Bovaira, and Sánchez-

Cano (2017), descriptively reviews seven

perspectives of study that address the issue,

identifying strategies and activities that prioritizes to

teach and learn using oral language in classroom. As

a result of the study we point out that teaching

strategies are diverse and the most frequent activities

are oral presentations and debates. We provide an

overview of the current situation and state that

programming and defining the objectives is required

for teaching-learning the systematized oral language.

It should be noted that, among the seven perspectives,

the Conversational Methodology (CM) is at the basis

of our research.

1.2 Assessment Scale of Oral Language

Teaching: EVALOE

Given the great importance of language interactions

inside the school context and the need for teachers to

be aware of the pragmatics of language, the Oral

Language Assessment Scale in the School context -

EVALOE (Gràcia et al., 2015a) was constructed with

the aim of providing an instrument for specialists in

language development (speech therapists,

counsellors, educational psychologists ...). This

framework have to be able to assess the skills and

strategies used by teachers to promote the

development of oral competence in the classroom.

EVALOE is an instrument that allows the

specialists to explore the interaction between the

teacher and the students in the classroom, both the

teacher's performance and its consequences in the

students' performance, elaborated from a socio-

pragmatic and eco-functional perspective of the

acquisition of language. It consists of two parts. The

first is an observation scale with a total of 30 items

grouped into three areas (Context and Management of

communication, Instructional Design and

Communicative Functions and Strategies). The

second part includes a set of questions to develop a

semi-structured interview with teachers, in order to

evaluate the aspects included in the first part in more

depth and taking as a reference their teaching

practice.

The methodological proposal that is at the base of

the construction of EVALOE is the Conversational

Methodology (CM). CM understands the classrooms

as communicative spaces in which teaching and

reflection on oral language is an end in itself, as well

as an instrument to help students learn contents

related to all subjects (Gràcia et al., 2015b).

1.3 The Use of DSS in Education

At present, the figure of a professional who reflects

and / or questions his practice emerges as part of the

competences of the curricula in any discipline and

especially in education (Mauri et al. 2016; Susinos

and Rada, 2016). The discussion with other

professionals (speech therapists, educators,

psychologists ...) and in a further stage individually,

allows teachers to generate knowledge about their

practice, which promotes a better understanding of it,

at the same time that transforms it and also transforms

its environment (Farrel, 2007).

In order to establish new ways to solve the

difficulties in this complex process, the use of

technology and DSS is considered a new challenge in

the world of education. In recent years DSS have

increasing relevance as a complementary instrument

in the traditional advice provided to professionals,

aimed at empowering them to define and clarify to

take more accurate decisions and to consider the

consequences that each of them entails (Eom and

Kim, 2006).

As indicated at the beginning of this section, the

use of DSS in the field of education is still in an initial

phase. Recent research points out advantages and

potentialities in its use, both for the planning of

processes of content teaching and learning (Kalay and

Chen, 2002), as to envisage decisions related to

structural and organizational changes in the school

(Sadahiro and Sadahiro, 2012). However, there are

already some studies highlighting the possibility to

use this kind of tools to help in the decision-making

process in the field of special education (Gregg, 2009)

or in virtual learning contexts (Xu and Wang, 2006).

This article brings to light the usefulness of DSSs

in the educational field, and, more precisely, the

intention of empowering teachers and helping them in

their decision-making regarding their educational

practice, their discursive strategies, and

communication management as well as the adaptation

to the environment, then contributing to the

development of the linguistic competence of

kindergarten and primary school students.

In this sense, the aim of the study is the

construction, implementation and validation of a

digital Decision Support System (EVALOE-DSS),

conceived as a teacher empowerment tool for self-

assessment and decision making about their teaching

practice concerning students' oral skills.

Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills

273

2 METHODS

2.1 Participants

In this pilot phase of the study, the participants are 4

teachers and their groups of students. Two of them are

kindergarten level teachers and two are primary level

teachers. Four diverse schools participated in the

study: two inclusive schools, one international school

and one bilingual intermodal school (oral-signed

language). All teachers had more than 5 years of

professional experience in the involved institutions.

2.2 Procedure

The research procedure includes: 1) design and

development of a DSS for different type of schools;

2) researcher assessment of one classroom

observation using EVALOE-DSS; 3) weekly teacher

self-assessment of classroom observation and

decision making using EVALOE-DSS for 6 months;

4) analysis of self-assessments, decision making and

changes in teaching practices; 5) teacher-researcher

evaluation of the helpfulness of DSS and introduction

of changes.

At present, the first and second phases have been

developed and the third is in progress. Regarding the

first phase, researchers linked to the field of the

psychology of education and researchers of the area

of computer science have worked together for 7

months to transform the EVALOE instrument into a

DSS for teachers. The principal criteria from the

psychology perspective that have been introduced at

the time are the following: 1) to transform an external

evaluation into a self-assessment tool; 2) to stablish

the differences between the items complexity

degrees; 3) to simplify the way teachers have to

structure and deliver their answer; 4) to transform the

second part of the original instrument, which initially

was an interview between the teacher and the

observer (external evaluator), in a set of supports for

the teacher to take more adapted and accurate

decisions; 5) to advise that the teacher is able to know

its level of resolution at the end of the self-

assessment, before decisions taking and the

introduction of the suggested changes. At the

technical level, the criteria have been the following:

1) to move from an external questionnaire system to

a self-developed solution; 2) to build a model which

conceptualises the needs of the DSS; 3) to design a

data model with the purpose of enabling an object-

oriented system; 4) to display different media based

support elements associated to each item, such as

videos or pictures describing the desired behaviour,

and to afford a user-friendly way of accessing this

support media; 5) to automate the process of choosing

which items will compound the teacher’s self-

assessment; 6) to automate the process of proposing

new elements to be introduced in forthcoming

classes.

Regarding the second phase, four classroom

observations of four different teachers have been

made later assessed by two researchers with the

EVALOE-DSS tool. In this case, researchers will

only use the tool with this assessment function during

the process of monitoring and controlling its

usefulness.

3 RESULTS

The implemented solution is an online platform

which integrates all the necessary features of user

management, questionnaire operation, help tools and

results viewing among others. It also has two main

engines: one responsible for generating the digital

questionnaires with the corresponding items, and the

other responsible for guiding teachers in their

decision on how they can improve their teaching

practice.

3.1 Use of the DSS

Teachers are invited to sign in and use the tool when

the expert registers their basic information. After

receiving an email and choosing their password they

will have access to the EVALOE-DSS, but they will

not be able to answer a self-assessment until the

expert had done a first evaluation of a class of the

teacher. This first evaluation fulfilled by the expert is

composed of the 30 items in the EVALOE scale,

whose answers will automatically generate the first

Table 1: Item examples of each dimension.

A.Instructional design

Pupils self-evaluate their communicative behavior

B.Teacher’s management

During the work session, the furniture, pupils and

teacher adapts to the characteristics of the activity

C.Pupils’ management

During conversation, discussion and/or debate activities

we adopt a network format

D. Communicative functions and teacher strategies

I expand the pupils’ sentences.

E. Pupils’ communicative functions

The pupils synthesize and/or extract conclusions

CSEDU 2018 - 10th International Conference on Computer Supported Education

274

self-assessment questionnaire to be completed by the

teacher. In Table 1 we present one item example of

each of the 5 dimensions in that these items are

grouped. This questionnaire generator can be

understood as an engine that follows a set of defined

criteria based on the whole history of responses and

adds or removes the appropriate items on the given

questionnaire.

At this point the first self-evaluation is available

for the teacher, in this case of N items resulting from

the questionnaire generator following the algorithm

of Figure 1. As mentioned before, every single item

has associated different media elements which will

guide the teacher through each answer. When the last

item is assessed, a summary is shown.

Figure 1: Algorithm which generates the first self-

evaluation resulting from first assessment.

Once this self-assessment is completed, the DSS

engine takes the necessary data to determine which 6

items may be the best to be introduced in the next

classes, so the teacher must choose 3 out of this 6.

Then, when the next class will have finalised (one

class each week), the teacher will do the next self-

assessment composed by the items determined by the

questionnaire generator. This process is summarised

in the Figure 2, showing the use case of a first

evaluation performed by the expert on the first class,

followed by three self-assessments to be answered by

the teacher after the classes.

Figure 2: Chronological use of EVALOE-DSS by a teacher

through 3 classes. In orange, the first evaluation completed

by the expert. In blue, the following self-evaluations of the

teacher with their recommended items.

In Figure 3 we present an example of how an item

of a self-evaluation is displayed, showing several

information of its scope or complexity, and other

controls such as the three levels which a teacher can

answer. The red-tick button corresponds to a score of

0 (I and or/ my pupils haven’t included/done this

action in my classroom), the yellow is 1 point (I and

/or my pupils have included/done this action in my

classroom with low frequency), and finally the green

one scores with 2 points (I and /or my pupils have

included/done this action in my classroom frequently)

the item.

Figure 3: Display of an item.

Every item has linked several media elements to

help teachers how to incorporate the action into their

teaching practice, such as video, audio or pictures. In

Figure 4 an example is presented.

Figure 4: Pop-up with help of an item.

Finally, when the self-evaluation is completed,

the teacher is expected to take the decision of

incorporating 3 out of 6 items into the following class.

This is done in the DSS screen presented in Figure 5.

Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills

275

Figure 5: Screen showing the 6 proposed items to be

potentially included in the following activity. The teacher

will select 3 of them.

3.2 Technical Architecture

The implementation of EVALOE-DSS is splitted into

two main components: An Application Programming

Interface (API) developed in Java using Spring

Framework, and a Single Page Application (SPA)

developed in Javascript using Vue JS Framework.

This architecture uncouples the user interface (SPA)

from the domain interface (API) as shown in Figure

6, so changes in how visual elements are displayed

can be easily done without affecting how the model

objects may communicate between them and vice

versa.

Figure 6: Architecture of the application. This uncoupling

allows a better maintenance, and an easier extendibility of

functionalities such as connectivity with 3rd party apps.

3.3 Assessments, Self-Assessments and

Decisions

As we have explained before, the second phase of the

research has been the assessment of one classroom

observation using EVALOE-DSS by the researcher

and the third phase has consisted on weekly teacher

self-assessment of classroom observation and

decision making using the tool. In Table 2 we present

the results regarding these assessments and self-

assessments. Scores show that teachers tend to score

higher than researchers, even taking into account the

number of items in each questionnaire. For example,

Teacher 1 scores 18 in a questionnaire of 16 items or

24 in a questionnaire of 18 items, while Researcher 1

scores 19 in the questionnaire of 30 items. Also, data

show that the number of items of the questionnaire

increases each week except in the case of Teacher 4.

Table 2: Scores of current assessments made by the four

teachers and two researchers.

Kindergarten

(Catalan)

Researcher 1

Teacher 1

Items

Score

Items

Score

Kindergarten

(English)

Researcher 1

Teacher 2

Items

Score

Items

Score

Primary Level

(Catalan)

Researcher 1

Teacher 3

Items

Score

Items

Score

Primary Level

(Bilingual)

Researcher 2

Teacher 4

Items

Score

Items

Score

4 CONCLUSIONS

Our preliminary results show that we have been able

to construct a DSS with the criteria that we

established after the analysis of the EVALOE as an

external assessment instrument in paper support

during a project successfully developed during 2012-

2015.

Secondly, we conclude that EVALOE-DSS on-

line platform has been used as an external instru-ment

by the researchers and as a self-assessment and DSS

tool by four teachers working in different type of

schools at different educational levels.

CSEDU 2018 - 10th International Conference on Computer Supported Education

276

Finally, at present researchers and teachers are

contributing to detect different types of problems or

limitations that it is necessary to check in order to

improve its usability.

ACKNOWLEDGEMENTS

We appreciate the collaboration and involvement of

the four participating teachers (Noèlia, Míriam,

Victòria and Pepita), as well as the facilities provided

by the schools (Tres Pins, Mare de Déu de

Montserrat, Francesc Aldea and Europa International

School) for the collection of data.

This work was supported by the Ministry of

Economy, Industry and Competitiveness (Spain)

[EDU2015-63616-P] and the European Union.

REFERENCES

Eom, S. and Kim, E. (2006). A survey of Decision Support

System Applications (1995-2001). The Journal of the

operational Research Society, 57(11), 1264-1278.

Farrell, T. (2007). Reflective languaje teaching. From

research to practice. London: The Tower Building.

Gràcia, M. (coord.), Galván-Bovaira, M. J., Sánchez-Cano,

M., Vega, F., Vilaseca, R., and Rivero, M. (2015a).

Evaluación de la enseñanza de la lengua oral. Escala

EVALOE, Barcelona, Graó.

Gràcia, M., Vega, F. and Galván-Bovaira, M. J. (2015b).

Developing and testing EVALOE: a tool for assessing

spoken language teaching and learning in the

classroom. Child Language Teaching and Therapy, 31

(3), 287-304.

Gràcia, M., Galván-Bovaira, M. J., and Sánchez-Cano, M.

(2017). Análisis de las líneas de investigación y

actuación en la enseñanza y el aprendizaje del lenguaje

oral en contexto escolar. Revista Española de

Lingüística Aplicada, 30 (1), 188-209.

Gregg, D. (2009). Developing a collective intelligence

application for special education. Decision Support

Systems, 47, 455-465.

Kalay, P. and Chen, D. (2002). Integrating a Decision

Support System into a School. Journal of Research on

Technology in Education, 34 (4), 435-452.

Mauri, T., Onrubia, J., Colomina, R., and Clarà, M. (2016).

Espacios de formación compartida escuela-universidad

en el practicum. Aula de Innovación Educativa, 523

(256), 12 - 16.

Sadahiro, Y. and Sadahiro, S. (2012). A decision support

method for school relocation planning. International

Journal of Urban Sciences, 16 (2), 125-141.

Susinos, T., and Sáiz, A. (2016). Los problemas

pedagógicos son mis aliados. El prácticum como un

proceso de reflexión e indagación colaborativa. Revista

de Investigación en Educación, 14 (1), 5-13.

Xu, D. and Wang, H. (2006). Intelligent agent supported

personalization for virtual learning environaments.

Decision Support Systems, 42, 825-843.

Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills

277