Empowering Teachers’ Self Assessment of Their Own Practice on
Student’s Oral Skills
EVALOE Decision Support System
Marta Gràcia
1
, Josep Casanovas
2,4
, Maria-Ribera Sancho
3,4
,
Jordi Casanovas
1,2
and
Marta Cuatrecasas
2
1
Department of Cognition, Development and Educational Psychology, Institute of Research in Education,
Universitat de Barcelona, Spain
2
Department of Statistics and Operations Research, Universitat Politècnica de Catalunya, Spain
3
Department of Service and Information Systems Engineering, Universitat Politècnica de Catalunya, Spain
4
Barcelona Supercomputing Center, Barcelona, Spain
Keywords: Teachers, Self-Assessing, Decision Support System, Oral Skills.
Abstract: School is an essential context that contributes to children’s communicative competence. Decision Support
Systems (DSS) are tools that can help teachers to assess their own practice and to make decisions to improve
it (Gregg, 2009). Our aim is the construction, implementation and validation of a digital Decision Support
System (EVALOE-DSS), conceived as a teacher empowerment tool for self-assessment and decision making
about their teaching practice concerning students’ oral skills. Participants are 4 teachers and their groups of
students of 4 diverse schools. The research procedure includes: 1) design and development of a DSS for
different type of schools; 2) researcher assessment of one classroom observation using EVALOE-DSS; 3)
weekly teacher self-assessment of classroom observation and decision making using EVALOE-DSS; 4)
analysis of self-assessments, decision making and changes in teaching practices; 5) teacher-researcher
assessment of the helpfulness of DSS and introduction of changes. EVALOE-DSS is a multimedia tool
consisting of 30 items that assess 5 dimensions: teaching management, instructional design, communicative
functions and teacher strategies, students’ communicative functions, and students’ management. It includes,
also, a brief description of each item and a variety of resources to help teachers to make decisions and
introduce the actions in their teaching practice.
1 INTRODUCTION
Although computers have been integrated into areas
related to educational management in many countries
since the 1970s, the integration of Decision Support
Systems (DSS) is still very recent and represents a
major challenge. The majority of published works
focus on the development of systems to improve
general aspects of school organization (Sadahiro and
Sadahiro, 2012) or systems to improve student
performance (Xu and Wang, 2006). Few works focus
on the construction of systems aimed at improving the
practice of teachers, with the ultimate goal of
contributing to the development of students'
competences. This paper reports our research in the
application of a first version of a DSS in the teaching
and learning of oral language by a teacher of early
childhood education.
1.1 Communicative Competence at
School
Catalan education law which develops the
organization of curricular teaching and development
in all educational stages defines the school as a space
of communication. A space where children can learn
to talk by talking and interacting with their mates and
with competent adults, in the classrooms, when in
these classes the conversation is the focus. On the
other hand, these regulations establish the ordering of
primary education and places great emphasis on
affirming that communicative and linguistic
competence should be the basis of all learning. This
development is the responsibility of all the areas of
the curriculum. Thus, educational legislation makes
clear that oral language plays a major role in the
272
Gràcia, M., Casanovas, J., Sancho, M., Casanovas, J. and Cuatrecasas, M.
Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills.
DOI: 10.5220/0006785902720277
In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 272-277
ISBN: 978-989-758-291-2
Copyright
c
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
development of school activities: children must learn
to speak, listen, expose and dialogue.
The conceptual framework that we have recently
presented, Gràcia, Galván-Bovaira, and Sánchez-
Cano (2017), descriptively reviews seven
perspectives of study that address the issue,
identifying strategies and activities that prioritizes to
teach and learn using oral language in classroom. As
a result of the study we point out that teaching
strategies are diverse and the most frequent activities
are oral presentations and debates. We provide an
overview of the current situation and state that
programming and defining the objectives is required
for teaching-learning the systematized oral language.
It should be noted that, among the seven perspectives,
the Conversational Methodology (CM) is at the basis
of our research.
1.2 Assessment Scale of Oral Language
Teaching: EVALOE
Given the great importance of language interactions
inside the school context and the need for teachers to
be aware of the pragmatics of language, the Oral
Language Assessment Scale in the School context -
EVALOE (Gràcia et al., 2015a) was constructed with
the aim of providing an instrument for specialists in
language development (speech therapists,
counsellors, educational psychologists ...). This
framework have to be able to assess the skills and
strategies used by teachers to promote the
development of oral competence in the classroom.
EVALOE is an instrument that allows the
specialists to explore the interaction between the
teacher and the students in the classroom, both the
teacher's performance and its consequences in the
students' performance, elaborated from a socio-
pragmatic and eco-functional perspective of the
acquisition of language. It consists of two parts. The
first is an observation scale with a total of 30 items
grouped into three areas (Context and Management of
communication, Instructional Design and
Communicative Functions and Strategies). The
second part includes a set of questions to develop a
semi-structured interview with teachers, in order to
evaluate the aspects included in the first part in more
depth and taking as a reference their teaching
practice.
The methodological proposal that is at the base of
the construction of EVALOE is the Conversational
Methodology (CM). CM understands the classrooms
as communicative spaces in which teaching and
reflection on oral language is an end in itself, as well
as an instrument to help students learn contents
related to all subjects (Gràcia et al., 2015b).
1.3 The Use of DSS in Education
At present, the figure of a professional who reflects
and / or questions his practice emerges as part of the
competences of the curricula in any discipline and
especially in education (Mauri et al. 2016; Susinos
and Rada, 2016). The discussion with other
professionals (speech therapists, educators,
psychologists ...) and in a further stage individually,
allows teachers to generate knowledge about their
practice, which promotes a better understanding of it,
at the same time that transforms it and also transforms
its environment (Farrel, 2007).
In order to establish new ways to solve the
difficulties in this complex process, the use of
technology and DSS is considered a new challenge in
the world of education. In recent years DSS have
increasing relevance as a complementary instrument
in the traditional advice provided to professionals,
aimed at empowering them to define and clarify to
take more accurate decisions and to consider the
consequences that each of them entails (Eom and
Kim, 2006).
As indicated at the beginning of this section, the
use of DSS in the field of education is still in an initial
phase. Recent research points out advantages and
potentialities in its use, both for the planning of
processes of content teaching and learning (Kalay and
Chen, 2002), as to envisage decisions related to
structural and organizational changes in the school
(Sadahiro and Sadahiro, 2012). However, there are
already some studies highlighting the possibility to
use this kind of tools to help in the decision-making
process in the field of special education (Gregg, 2009)
or in virtual learning contexts (Xu and Wang, 2006).
This article brings to light the usefulness of DSSs
in the educational field, and, more precisely, the
intention of empowering teachers and helping them in
their decision-making regarding their educational
practice, their discursive strategies, and
communication management as well as the adaptation
to the environment, then contributing to the
development of the linguistic competence of
kindergarten and primary school students.
In this sense, the aim of the study is the
construction, implementation and validation of a
digital Decision Support System (EVALOE-DSS),
conceived as a teacher empowerment tool for self-
assessment and decision making about their teaching
practice concerning students' oral skills.
Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills
273
2 METHODS
2.1 Participants
In this pilot phase of the study, the participants are 4
teachers and their groups of students. Two of them are
kindergarten level teachers and two are primary level
teachers. Four diverse schools participated in the
study: two inclusive schools, one international school
and one bilingual intermodal school (oral-signed
language). All teachers had more than 5 years of
professional experience in the involved institutions.
2.2 Procedure
The research procedure includes: 1) design and
development of a DSS for different type of schools;
2) researcher assessment of one classroom
observation using EVALOE-DSS; 3) weekly teacher
self-assessment of classroom observation and
decision making using EVALOE-DSS for 6 months;
4) analysis of self-assessments, decision making and
changes in teaching practices; 5) teacher-researcher
evaluation of the helpfulness of DSS and introduction
of changes.
At present, the first and second phases have been
developed and the third is in progress. Regarding the
first phase, researchers linked to the field of the
psychology of education and researchers of the area
of computer science have worked together for 7
months to transform the EVALOE instrument into a
DSS for teachers. The principal criteria from the
psychology perspective that have been introduced at
the time are the following: 1) to transform an external
evaluation into a self-assessment tool; 2) to stablish
the differences between the items complexity
degrees; 3) to simplify the way teachers have to
structure and deliver their answer; 4) to transform the
second part of the original instrument, which initially
was an interview between the teacher and the
observer (external evaluator), in a set of supports for
the teacher to take more adapted and accurate
decisions; 5) to advise that the teacher is able to know
its level of resolution at the end of the self-
assessment, before decisions taking and the
introduction of the suggested changes. At the
technical level, the criteria have been the following:
1) to move from an external questionnaire system to
a self-developed solution; 2) to build a model which
conceptualises the needs of the DSS; 3) to design a
data model with the purpose of enabling an object-
oriented system; 4) to display different media based
support elements associated to each item, such as
videos or pictures describing the desired behaviour,
and to afford a user-friendly way of accessing this
support media; 5) to automate the process of choosing
which items will compound the teacher’s self-
assessment; 6) to automate the process of proposing
new elements to be introduced in forthcoming
classes.
Regarding the second phase, four classroom
observations of four different teachers have been
made later assessed by two researchers with the
EVALOE-DSS tool. In this case, researchers will
only use the tool with this assessment function during
the process of monitoring and controlling its
usefulness.
3 RESULTS
The implemented solution is an online platform
which integrates all the necessary features of user
management, questionnaire operation, help tools and
results viewing among others. It also has two main
engines: one responsible for generating the digital
questionnaires with the corresponding items, and the
other responsible for guiding teachers in their
decision on how they can improve their teaching
practice.
3.1 Use of the DSS
Teachers are invited to sign in and use the tool when
the expert registers their basic information. After
receiving an email and choosing their password they
will have access to the EVALOE-DSS, but they will
not be able to answer a self-assessment until the
expert had done a first evaluation of a class of the
teacher. This first evaluation fulfilled by the expert is
composed of the 30 items in the EVALOE scale,
whose answers will automatically generate the first
Table 1: Item examples of each dimension.
A.Instructional design
Pupils self-evaluate their communicative behavior
B.Teacher’s management
During the work session, the furniture, pupils and
teacher adapts to the characteristics of the activity
C.Pupils’ management
During conversation, discussion and/or debate activities
we adopt a network format
D. Communicative functions and teacher strategies
I expand the pupils’ sentences.
E. Pupils’ communicative functions
The pupils synthesize and/or extract conclusions
CSEDU 2018 - 10th International Conference on Computer Supported Education
274
self-assessment questionnaire to be completed by the
teacher. In Table 1 we present one item example of
each of the 5 dimensions in that these items are
grouped. This questionnaire generator can be
understood as an engine that follows a set of defined
criteria based on the whole history of responses and
adds or removes the appropriate items on the given
questionnaire.
At this point the first self-evaluation is available
for the teacher, in this case of N items resulting from
the questionnaire generator following the algorithm
of Figure 1. As mentioned before, every single item
has associated different media elements which will
guide the teacher through each answer. When the last
item is assessed, a summary is shown.
Figure 1: Algorithm which generates the first self-
evaluation resulting from first assessment.
Once this self-assessment is completed, the DSS
engine takes the necessary data to determine which 6
items may be the best to be introduced in the next
classes, so the teacher must choose 3 out of this 6.
Then, when the next class will have finalised (one
class each week), the teacher will do the next self-
assessment composed by the items determined by the
questionnaire generator. This process is summarised
in the Figure 2, showing the use case of a first
evaluation performed by the expert on the first class,
followed by three self-assessments to be answered by
the teacher after the classes.
Figure 2: Chronological use of EVALOE-DSS by a teacher
through 3 classes. In orange, the first evaluation completed
by the expert. In blue, the following self-evaluations of the
teacher with their recommended items.
In Figure 3 we present an example of how an item
of a self-evaluation is displayed, showing several
information of its scope or complexity, and other
controls such as the three levels which a teacher can
answer. The red-tick button corresponds to a score of
0 (I and or/ my pupils haven’t included/done this
action in my classroom), the yellow is 1 point (I and
/or my pupils have included/done this action in my
classroom with low frequency), and finally the green
one scores with 2 points (I and /or my pupils have
included/done this action in my classroom frequently)
the item.
Figure 3: Display of an item.
Every item has linked several media elements to
help teachers how to incorporate the action into their
teaching practice, such as video, audio or pictures. In
Figure 4 an example is presented.
Figure 4: Pop-up with help of an item.
Finally, when the self-evaluation is completed,
the teacher is expected to take the decision of
incorporating 3 out of 6 items into the following class.
This is done in the DSS screen presented in Figure 5.
Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills
275
Figure 5: Screen showing the 6 proposed items to be
potentially included in the following activity. The teacher
will select 3 of them.
3.2 Technical Architecture
The implementation of EVALOE-DSS is splitted into
two main components: An Application Programming
Interface (API) developed in Java using Spring
Framework, and a Single Page Application (SPA)
developed in Javascript using Vue JS Framework.
This architecture uncouples the user interface (SPA)
from the domain interface (API) as shown in Figure
6, so changes in how visual elements are displayed
can be easily done without affecting how the model
objects may communicate between them and vice
versa.
Figure 6: Architecture of the application. This uncoupling
allows a better maintenance, and an easier extendibility of
functionalities such as connectivity with 3rd party apps.
3.3 Assessments, Self-Assessments and
Decisions
As we have explained before, the second phase of the
research has been the assessment of one classroom
observation using EVALOE-DSS by the researcher
and the third phase has consisted on weekly teacher
self-assessment of classroom observation and
decision making using the tool. In Table 2 we present
the results regarding these assessments and self-
assessments. Scores show that teachers tend to score
higher than researchers, even taking into account the
number of items in each questionnaire. For example,
Teacher 1 scores 18 in a questionnaire of 16 items or
24 in a questionnaire of 18 items, while Researcher 1
scores 19 in the questionnaire of 30 items. Also, data
show that the number of items of the questionnaire
increases each week except in the case of Teacher 4.
Table 2: Scores of current assessments made by the four
teachers and two researchers.
Kindergarten
(Catalan)
Researcher 1
Teacher 1
Items
Score
Items
Score
30
19
16
18
18
24
22
28
26
40
Kindergarten
(English)
Researcher 1
Teacher 2
Items
Score
Items
Score
30
18
16
20
21
33
Primary Level
(Catalan)
Researcher 1
Teacher 3
Items
Score
Items
Score
30
9
15
22
20
25
Primary Level
(Bilingual)
Researcher 2
Teacher 4
Items
Score
Items
Score
30
18
17
17
21
31
21
35
4 CONCLUSIONS
Our preliminary results show that we have been able
to construct a DSS with the criteria that we
established after the analysis of the EVALOE as an
external assessment instrument in paper support
during a project successfully developed during 2012-
2015.
Secondly, we conclude that EVALOE-DSS on-
line platform has been used as an external instru-ment
by the researchers and as a self-assessment and DSS
tool by four teachers working in different type of
schools at different educational levels.
CSEDU 2018 - 10th International Conference on Computer Supported Education
276
Finally, at present researchers and teachers are
contributing to detect different types of problems or
limitations that it is necessary to check in order to
improve its usability.
ACKNOWLEDGEMENTS
We appreciate the collaboration and involvement of
the four participating teachers (Noèlia, Míriam,
Victòria and Pepita), as well as the facilities provided
by the schools (Tres Pins, Mare de Déu de
Montserrat, Francesc Aldea and Europa International
School) for the collection of data.
This work was supported by the Ministry of
Economy, Industry and Competitiveness (Spain)
[EDU2015-63616-P] and the European Union.
REFERENCES
Eom, S. and Kim, E. (2006). A survey of Decision Support
System Applications (1995-2001). The Journal of the
operational Research Society, 57(11), 1264-1278.
Farrell, T. (2007). Reflective languaje teaching. From
research to practice. London: The Tower Building.
Gràcia, M. (coord.), Galván-Bovaira, M. J., Sánchez-Cano,
M., Vega, F., Vilaseca, R., and Rivero, M. (2015a).
Evaluación de la enseñanza de la lengua oral. Escala
EVALOE, Barcelona, Graó.
Gràcia, M., Vega, F. and Galván-Bovaira, M. J. (2015b).
Developing and testing EVALOE: a tool for assessing
spoken language teaching and learning in the
classroom. Child Language Teaching and Therapy, 31
(3), 287-304.
Gràcia, M., Galván-Bovaira, M. J., and Sánchez-Cano, M.
(2017). Análisis de las líneas de investigación y
actuación en la enseñanza y el aprendizaje del lenguaje
oral en contexto escolar. Revista Española de
Lingüística Aplicada, 30 (1), 188-209.
Gregg, D. (2009). Developing a collective intelligence
application for special education. Decision Support
Systems, 47, 455-465.
Kalay, P. and Chen, D. (2002). Integrating a Decision
Support System into a School. Journal of Research on
Technology in Education, 34 (4), 435-452.
Mauri, T., Onrubia, J., Colomina, R., and Clarà, M. (2016).
Espacios de formación compartida escuela-universidad
en el practicum. Aula de Innovación Educativa, 523
(256), 12 - 16.
Sadahiro, Y. and Sadahiro, S. (2012). A decision support
method for school relocation planning. International
Journal of Urban Sciences, 16 (2), 125-141.
Susinos, T., and Sáiz, A. (2016). Los problemas
pedagógicos son mis aliados. El prácticum como un
proceso de reflexión e indagación colaborativa. Revista
de Investigación en Educación, 14 (1), 5-13.
Xu, D. and Wang, H. (2006). Intelligent agent supported
personalization for virtual learning environaments.
Decision Support Systems, 42, 825-843.
Empowering Teachers’ Self Assessment of Their Own Practice on Student’s Oral Skills
277