Metacognitive Support in University Lectures

Provided via Mobile Devices

How to Help Students to Regulate Their Learning Process

during a 90-minute Class

Felix Kapp

, Iris Braun

, Hermann Körndle

and Alexander Schill

Chair of Learning and Instruction, Technische Universität Dresden, Dresden, Germany

Chair of Computer Networks, Technische Universität Dresden, Dresden, Germany

Keywords: Mobile Devices, Self-regulated Learning, University Lecture, Metacognitive Support.

Abstract: Even though classical lectures at universities are criticized for lacking interactivity and treating students like

passive receptors of information they are still very popular. Due to the big amount of students, interaction

between teacher and students is difficult to realize. Several projects address this problem by offering

technical solutions which aim at increasing the interactivity during classes or lectures – classic clicker-

systems as well as solutions in which students use their own smartphones, netbooks or tablet-PCs. Based on

research on self-regulated learning (SRL) processes we developed the already existing tools one step

further: instead of only providing questions we designed Auditorium Mobile Classroom Service (AMCS) –

a program which offers several possibilities to interact during a lecture. AMCS supports students to regulate

their own learning process during the lecture. Learning questions are one core element to support them. On

the basis of the results of the learning questions specific advices and hints are sent to the students’

smartphones or notebooks. The features increase the interactivity between the content and students and the

interaction in the lecture hall. In the present article the program AMCS is described. Furthermore we report

first experiences from a field test in a university lecture.

1 INTRODUCTION

Lectures are still an important form of teaching

courses at universities. They aim to expand students’

knowledge through the structured presentation of

expertise from a teacher. This form of teaching has

been criticized for offering too little interaction

between teachers and students. Learning as an

active, constructive and highly individual process

(Seel, 2003) is almost impossible in huge lectures.

As a consequence, students experience severe

difficulties – they do not manage to build adequate

mental models of the taught domain.

There are several approaches to increase the

interactivity in lectures. The spectrum ranges from

simple voting systems to the method of peer

instruction (Mazur, 1997). A large variety of

systems especially useful for implementing learning

questions in lectures are subsumed under the concept

“audience response systems” or “clickers”.

Audience response systems provide feedback to the

lecturer by giving the audience the possibility to

participate during the class by voting on questions.

By presenting questions during the class students get

more involved in the lecture and the lecturer in turn

gets some information about the audience’s

knowledge and attitudes. Almost all of these systems

work as follows: the lecturer defines a question

before starting the class; during the lecture the

question is presented on the screen and the students

are asked to answer via special technical devices

(clickers) or their smartphones; all answers are

aggregated and immediately pictured on the

presentation-screen. The lecturer can include the

answers from the audience into the lecture – provide

feedback to the audience or adapt the lecture to

special interests or needs. There are some studies

showing that audience response systems are capable

of increasing the interactivity in lectures and leading

to an improvement in academic achievement (e.g.

Mayer et al., 2009). A prerequisite for these positive

effects seem to be that the application is

accompanied with strategies that engage students in

deeper processing (Brady et al., 2013). For example,

194

Kapp F., Braun I., Körndle H. and Schill A..

Metacognitive Support in University Lectures Provided via Mobile Devices - How to Help Students to Regulate Their Learning Process during a

90-minute Class .

DOI: 10.5220/0004936901940199

In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 194-199

ISBN: 978-989-758-022-2

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

Lantz and Stawiski (2014) point out that obtaining

feedback after working on the questions is crucial

for an improvement in learning.

We took existing systems like SMILE (Weber &

Becker, 2013) as a starting point and combined first

experiences with the results of the research on self-

regulated learning and learning questions. On this

basis we extended the concept of classical audience

response systems. Our main goal is to support

students during large university lectures in achieving

their personal learning goals. On the basis of SRL

models we developed a system to provide interactive

learning questions, cognitive and metacognitive

prompts to students in university lectures. With

Auditorium Mobile Classroom Service (AMCS) the

lecturer designs in advance of the class learning

questions with feedback and messages with

additional information. These messages and learning

questions are delivered during the lecture in order to

facilitate successful regulation of the learning

process of each of the participants.

From a technical perspective we added one

direction of communication – in contrast to existing

systems AMCS does not only give the students the

possibility to vote during the lecture, the professor

gets the possibility to communicate to them during

the class as well. That way the lecture is designed as

an individual adaptive learning process. In the

following sections the core elements of AMCS are

described.

2 FEATURES OF AMCS

Models of self-regulated learning (e.g., Zimmerman,

2000) identify the requirements that must be met by

students at different points in the learning process.

Zimmerman (2000) assumes that the forethought

phase, the performance phase and the self-reflection

phase are recurrent at different levels during a

learning process. The goal orientation, attribution

style and individual differences in prior knowledge,

for example, have an impact on the forethought

phase and the planning of the learning process.

Depending on these variables students may differ in

preparing for university lectures. Planning and

preparing for the lecture is crucial for the successful

knowledge acquisition. During the performance

phase the diverse information needs to be processed.

This includes the use of pre-selected learning

strategies and the maintenance of motivation and

attention. In the self-reflection or evaluation phase,

learners should reflect on their learning process and

achievement and derive implications for future

learning activities. Processes during the performance

and the self-reflection phase are influenced by

individual differences as well. Depending on the

capability to concentrate, the personal goals and

interests learners master the demands of these two

phases differently. This results in different learning

outcomes. AMCS aims at supporting students in

self-regulated learning taking into account that

individual differences of the students play a decisive

role.

In the following section the features of AMCS

are presented. All instructional interventions are

delivered via mobile devices (netbooks, smart-

phones, tablets) during the lecture.

2.1 Interests / Personal Goals

At the beginning of the lecture students are asked for

their personal goals and interests. Why are they

attending the lecture? Are they interested in the topic

or focused on passing the exam? The goals must be

taken into account when supporting students in

regulation during the lecture. Therefore, the

information collected is used as a basis for

metacognitive prompts. Metacognitive prompts are

instructions that are sent to the mobile devices of the

students during the lecture. They contain

information which helps them to regulate their

personal learning process depending on their goals

and interests. Besides, students shall be encouraged

by this short survey at the beginning of the lecture,

to be clear about their goals and interests.

2.2 Learning Questions at the

Beginning, in the Middle and at the

End of the Lecture

Interactive learning questions are implemented to

support the learning process both on a cognitive and

a metacognitive level. Located at the beginning, in

the middle and at the end of the lecture they assist

students in an active engagement with the content.

Prerequisite for the effectiveness of learning

questions is the consideration of certain design rules.

Körndle, Narciss and Proske (2004) identified four

dimensions, which can be systematically

constructed: 1) format, 2) content, 3) cognitive

operations necessary to solve the question, and 4)

interactivity. Within AMCS the question format

multiple-choice is available. The interactivity goes

beyond already existing tools. AMCS allows

designers to implement a two-step feedback

algorithm defining specific feedback for any option.

In contrast to other audience response systems

MetacognitiveSupportinUniversityLecturesProvidedviaMobileDevices

-HowtoHelpStudentstoRegulateTheirLearningProcessduringa90-minuteClass

195

learners receive individual feedback on their mobile

devices. They can answer the learning questions

twice before the correct option is displayed.

Feedback contains information whether the answer

is correct or not and in case of an incorrect answer

hints on how to go on.

Learning questions at the beginning aim at

activating prior knowledge. In addition, the

requirements of the lecture are communicated

through the learning questions and the attention of

the students is guided to specific content. After half

of the lecture, the students can use the learning

questions to practice the recently learned concepts

and to get feedback on their level of knowledge

progress. At the end of the class learning questions

again aim at practicing relevant concepts and are

useful for the self-evaluation of the learning process.

As learners obtain feedback on their level of

knowledge they can draw conclusions for future

events - concerning the regulation of attention and

motivation as well as the application of learning

strategies. In contrast to already existing audience

response systems AMCS is accompanied with an

instructional concept which contains interactive

learning questions as a core element and

theoretically deduced how and when to implement

them in the lecture.

2.3 Metacognitive Prompts

During the lecture metacognitive prompts are sent

automatically to the students. They aim at

supporting the students in reaching their personal

learning goals. As they address regulation processes

on a more abstract level we named them

“metacognitive prompts”. The prompts are delivered

depending on personal goals and characteristics of

the student (e.g., learning goal orientation, exam

preparation or interest in the topic) and depending

on how they did in the learning questions.

In advance of the lecture the professor designs

messages containing helpful information and

prompts for different goals (e.g., exam preparation

vs. research interest) and different motivational

states (no interest in the topic at all vs. really curious

about the topic). At the beginning the students are

asked about their goals and motivation with the help

of a short questionnaire. Based on their answers they

get adaptive metacognitive prompts during the

lecture. An example of a metacognitive prompt,

which intents to help the students to adapt their

learning behaviour to their goal “passing the exam”

is the following:

"On the following slide the concept X is explained.

This concept is relevant for the exam. A question of

how it is raised repeatedly in the oral examination is,

for example: Why is it important to apply concept X

when starting the process?" Students are required to

select and process content, which is relevant for their

personal learning goals. At the same time the

personal goals can significantly differ within a group

of students attending the same lecture. Thus, it

results to be difficult for the lecturer to address all

the different goals in one session. As a result

students fail in selecting relevant information and

differentiate between important and marginal

content. The messages sent by AMCS containing

metacognitive prompts introduce adaptive support to

students in order to reach their personal learning

goals.

2.4 Cognitive Prompts - Individual

Adaptive Feedback during the

Lecture

Learning questions at the beginning of the lecture

and in the middle are not only interventional tools to

support students in knowledge construction. They

also deliver diagnostic information on the state of

knowledge acquisition of the students. This

information can be used to promote knowledge

building and further develop students’ mental

models.

Learning questions with several response options

offer the possibility to incorporate typical

misconceptions about concepts and theories. If the

mental model contains misconceptions and the

incorrect answer is chosen, the cognitive prompts

can support students in correcting the misconception

and overcome these obstacles. AMCS initiates

corrective processes by sending the student a

cognitive prompt at the moment the misconception

is explained by the lecturer.

If a student, for example, selects an incorrect

option to the first learning question at the beginning

of the lecture, then he gets the following message

when the lecturer is explaining slide number 13 of

his presentation:

"You have made a mistake in the first learning

question at the beginning. For some reason you

thought that concept X is the answer to the question.

What it really means is explained by Prof. Y on the

current slide."

The cognitive prompt should initiate behaviour

which leads to the correction of the misconception.

In order to do so it names the misconception and

draws the attention to the explanation of the concept

by the lecturer. These messages are referred to as

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

196

cognitive prompts as they directly address concrete

information and have the goal to stimulate

information processing of specific content. They

might also initiate regulation behaviour as the

consequence of a corrected misconception can be

changes in learning behaviour. Thus, the main

difference to metacognitive prompts is the level of

intended effects: cognitive prompts aim at

integrating or correcting specific content, whereas

metacognitive prompts put the focus on general

regulatory mechanisms such as the maintenance of

attention or the understanding of demands. It is also

possible to combine both types of prompts.

2.5 Providing Further Material to the

Students – Scripts, Links and

Additional Texts

AMCS offers the possibility to provide further

materials to the students. These include links, PDFs,

and slides of the presentation. The materials can be

chosen adaptively to the individual goals of the

students and/or to their learning behaviour. An

example of a message with further material for

students who are thinking about doing research or

writing their thesis in the field of the lecture is as

follows:

“You have indicated at the beginning of the

lecture that you are interested in writing a thesis on

this topic. The chair is doing research on the topic

which is presented on the current slide. You can find

possible research queries for a Bachelor thesis on the

subject under the following link: http:// .... “

2.6 Scripted Discussion – How to

Animate Students to Ask Questions

Which Are Helpful for Them

The sixth feature of AMCS applies during the time

slot, which is normally reserved for a discussion.

Both the auditorium and the lecturer exchange ideas

and questions at the end of the lecture. By sending

the students messages AMCS intends to initiate this

exchange and involve students who normally do not

participate in this interaction. In exceptional cases,

the discussion may even be staged. Pro and counter-

arguments could be distributed among the audience.

One example for a message with a request for a

comment that aims at starting the discussion is the

following: “Stand up right now and ask the

following question loudly into the room: What's the

practical use of this theory?” The goal of this feature

is to use the time reserved for discussion and

interaction between the lecturer and the students in

an optimal way.

3 PILOT STUDY

The AMCS prototype was tested in a 90-minute

lecture on psychology. The evaluation had mainly

three goals: we wanted to figure out if (1) the tool

works properly during the lecture (Does the tool

deliver messages and learning questions at the

correct moments etc.?). Furthermore we aimed at (2)

checking if the intervention is accepted by the

students (Do students appreciate the usage of mobile

devices with the reported features during the

lecture?). Finally we wanted to investigate (3)

whether AMCS is able to produce positive effects

concerning motivation, concentration and

achievement. We gained data to answer these

questions from log-file analyses, self-reports of the

students and achievement tests.

3.1 Technical Infrastructure

AMCS is based on a service-oriented system with

different client applications for students and

lecturers (see Figure 1). The students’ client enables

them to get prompts and questions during the class.

The interventions are delivered via inbound and

outbound messages on their smartphones or other

Internet-enabled devices (tablets, netbooks or

notebooks). They interact with the service via web

based application over an REST-API. This has the

advantage of platform independence over all device

classes.

Figure 1: system architecture of AMCS.

The responses of the students are stored in a

database on the Auditorium server. On this basis,

learners receive messages which are sent

automatically by the system. The timing of message

dispatch is dependent on the actual presentation of

the lecturer. Therefore it is necessary that the server

MetacognitiveSupportinUniversityLecturesProvidedviaMobileDevices

-HowtoHelpStudentstoRegulateTheirLearningProcessduringa90-minuteClass

197

communicates with the presentation device of the

lecturer while the class is taken place. This

synchronization with the presentation system (e.g.,

PowerPoint, Keynote or PDF) is done by the

lecturers’ app, which was implemented as native

Mac OS X application as well as a desktop

application for Windows. Within this application the

lecturer defines in advance of the class on which

presentation-slide which specific message will be

sent or which questions have to be answered.

During the class the professor can use the

lecturers’ view of the AMCS web application to see

how the students are doing. By observing the voting

results and answers in real-time lecturers can adapt

the pace of their presentation or address content

which was not understand yet.

3.2 Method

Thirty students (10 men, 20 women, mean age: 25.8

years, SD: 5.1 years) from a German university

participated in the field study. The sample size

differed between 22 (all evaluation data including

questionnaires and knowledge test) and 30 persons

(log-files during the lecture – delivery of messages

and learning questions) as some of the participants

did not fill out the post-questionnaires. Within the

sample smartphones (11), tablets (2), netbooks (9)

and notebooks (7) were used (missing information

for one participant).

The lecture was on the topic of self-regulated

learning, which is regular part of the curriculum in

the field of learning and instruction. Before starting

with the lecture every participant was requested to

answer the pre-questionnaire asking for interest in

the topic and motivation to attend the lecture.

Afterwards they received their personal login for the

AMCS platform and the class started. At the end of

the class participants answered the post-

questionnaire containing several measurements

about motivation, usability and a knowledge test.

3.3 Results

3.3.1 Technical Functionality

Nine persons reported that their devices work either

with iOS or with OS X, 10 devices were based on

windows operating systems and four on android.

Seven participants did not report on which basis

their devices work. The log-files from the data bank

revealed that the 26 users assessed created 206 logs

concerning the learning questions. In average 7.9

actions per person referring to the learning questions

were documented. There were 67 entries in the

database regarding the three questions at the

beginning of the lecture (concerning the goals and

interests of the students). As 26 users are registered

it is clear that not all of the participants answered all

questions. During the lecture 98 messages

containing either cognitive prompts, metacognitive

prompts, further information and material or

suggestions for the discussion at the end were send

to the mobile devices of the participants. The

average of messages sent was 3.8 per participant.

The comments in the post-questionnaire revealed

a number of technical problems which participants

experienced during the session. One student was not

able to connect and login into the system at all.

Further comments addressed a mixture between

languages in the user interface (2), that messages

should be high lightened in some way (2), that the

feedback algorithm can be improved (1), that there

were technical problems with the learning questions

at the end (1).

3.3.2 Acceptance

Participants were asked in the post-questionnaire if

they would recommend the program and would like

to work again with it. Twenty-one participants

answered the questionnaire of six items. The mean

value for the group is 3.76 (SD = .68) on a scale

from 1 “I do not agree” to 5 “I fully agree”. The item

asking if they consider the functionalities useful

(learning questions, messages and feedback to the

lecturer) was rated 4.19

(SD = .68). The usability criteria “conformity with

user expectations” (5 items; M = 5.8, SD = .93),

“suitability for the task” (5 items; M = 5.3,

SD = .86) and “self-description capability” (5 items;

M = 4.7, SD = 1.00) were rated positively. The scale

ranges from 1 “---“ to 7 “+++”.

The lecturer positively annotated that he could

use his normal presentation (based on PowerPoint)

and was able to see the results of the learning

questions and questionnaires in real-time.

3.3.3 Motivation and Knowledge

Twenty-two participants answered the questionnaire

on motivation, concentration and attention compared

to normal lectures. Scales ranged from 1 “I do not

agree” to 5 “I fully agree”. Students rather agreed

that their concentration

(M = 3.55; SD = .79), attention (M = 3.39; SD = .72)

and motivation (M = 4.09; SD = .71) was higher

with AMCS. There was one item asking for an

overall judgment on the lecture with AMCS

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

198

compared to normal lectures. Participants of the

field study rather agreed (M = 3.14, SD = 1.04) to

the statement “By using mobile devices in this

lecture I learned more than in normal lectures.”

Interests on self-regulated learning (n = 19; 7 items)

and motivation to study (n=21, 3 items) before and

after the lecture were assessed with questionnaires.

There were no significant changes from pre to post

(interest: t(18) = -.57, p > .5; motivation:

t(21) = -1.5, p = .15). Both interest (M

pre

= 3.03,

pre

= .68; M

post

= 3.08, SD

post

= .61; scale ranging

from 1 to 4) as motivation to study (M

pre

= 5.24,

pre

= .59; M

post

= 5.35, SD

post

= .54; scale ranging

from 1 to 6) remained on a high level. Twenty-two

students participated in the achievement test. Scores

ranged from zero to eight points (the test has 10

items – one point for each item was given). The

mean score was 3.96 (SD = 2.40).

4 CONCLUSIONS

Auditorium Mobile Classroom Service (AMCS)

provides an opportunity to support students during

university lectures. The six features aim at fostering

regulation and mastering demands of self-regulating

learning. The core elements of AMCS are derived

from empirical studies (e.g., Kapp, Proske, Narciss,

& Körndle, 2011) and theoretical considerations

based on models of self-regulation (e.g.

Zimmerman, 2000). The first test of the pilot is seen

as a demonstration of how learning questions,

cognitive and metacognitive prompts can be used in

university lectures in order to support students in

mastering the demands of this learning situation. Via

mobile devices, university lectures are made

adaptive – learning questions and individual prompts

are tailored to the personal goals and learning

processes of the students.

The interactivity is increased by interventions,

which animate students to engage in content

(learning questions) and by establishing a

communication channel (via the mobile devices of

the students), which allows the learning environment

to interact with the students (via predefined prompts

and messages by the lecturer).

The results of the pilot are of course limited and

do not go beyond the examination of requirements

necessary to generate learning effects. These

requirements are for example technical

functionalities and acceptance of the system and

self-reported attention, concentration, motivation

and achievement. The first evaluation suggests that

the minimum requirements are met. The intervention

was not perceived as distraction nor judged as

difficult to use during the lecture. The usability of

the system was rated as good and beside some

technical problems students would recommend

AMCS and further use it. First critical arguments

could be refuted: the distraction of the usage of

mobile devices during the lecture does not seem to

constrain learning and the need of extensive

computer literacy is not a requirement to use AMCS.

Nevertheless the data is not sufficient. In future

studies we want to test the system and its

components in large lectures and empirically

evaluate the effects of the single features.

REFERENCES

Brady, M., Seli, H., & Rosenthal, J., 2013. "Clickers" and

metacognition: A quasi-experimental comparative

study about metacognitive self-regulation and use of

electronic feedback devices. Computers & Education,

65(0), 56-63.

Kapp, F., Narciss, S., Körndle, H., Proske, A., 2011.

Interaktive Lernaufgaben als Erfolgsfaktor für E-

Learning. Zeitschrift für E-Learning, 6 (1), S.21-32.

Körndle, H., Narciss, S., Proske, A., 2004. Konstruktion

interaktiver Lernaufgaben für die universitäre Lehre.

In D. Carstensen & B. Barrios (Eds.), Campus 2004.

Kommen die digitalen Medien an den Hochschulen in

die Jahre? (pp. 57-67). Münster: Waxmann.

Lantz, M. E., & Stawiski, A. Effectiveness of clickers:

Effect of feedback and the timing of questions on

learning. Computers in Human Behavior, 31(0), 280-

286.

Mayer, R. E., Stull, A., DeLeeuw, K., Almeroth, K.,

Bimber, B., Chun, D., et al., 2009. Clickers in college

classrooms: Fostering learning with questioning

methods in large lecture classes. Contemporary

Educational Psychology, 34(1), 51-57.

Mazur, E., 1997. Peer Instruction: A User’s Manual.

Upper Saddle River: Prentice Hall.

Seel, N. M., 2003. Psychologie des Lernens (2

edition).

München: Ernst Reinhardt (UTB).

Weber, K., Becker, B., 2013. Formative Evaluation des

mobilen Classroom-Response-Systems SMILE. In C.

Bremer & D. Krömker (Hrsg.), E-Learning zwischen

Vision und Alltag, (S. 277-289). Münster: Waxmann.

Zimmerman, B. J., 2000. Attaining self-regulation: A

social cognitive perspective. In M. Boekaerts, P. R.

Pintrich & M. Zeidner (Eds.), Handbook of self-

regulation. (pp. 13-39). San Diego, CA US: Academic

Press.

MetacognitiveSupportinUniversityLecturesProvidedviaMobileDevices

-HowtoHelpStudentstoRegulateTheirLearningProcessduringa90-minuteClass

199