Synthesis of a Framework of Design Guidelines

for m-Learning Environments

A Study in a Tertiary Education Context in South Africa

Patricia Harpur and Ruth de Villiers

School of Computing, University of South Africa, Pretoria, South Africa

Keywords: Design and Development Strategy, Design Guidelines, Digital Devices, m-Learning Environment, Tertiary

Education, Virtual Learning Environment.

Abstract: Smartphones and tablets are ubiquitous in educational contexts, where students on-the-move expect access

to learning material via a range of digital devices in a mobile and transparent manner, whether on or off

campus. A successful m-learning experience can be facilitated by a mobile learning environment which is

efficient and effective, and that satisfies the users’ versatile needs. An ad hoc design and development

strategy that ignores design principles and guidelines, restricts the likelihood of successful m-learning

experiences. This study was implemented in a tertiary education context and aimed to establish – from dual

perspectives – a framework of design and development guidelines for m-learning environments. An initial

set of themes and guidelines was synthesized from a comprehensive literature study. Secondly, the

outcomes of a series of iterative evaluations of an m-learning application, Mobile Learning Research (m-

LR) were used to generate new themes and guidelines. The quantitative and qualitative findings of heuristic

evaluations by experts and questionnaire surveys administered to students, provided positive and negative

feedback that was converted to a set of practical guidelines. Jointly, the initial theoretical guidelines and the

subsequent empirical findings contributed to the synthesis of a comprehensive and cohesive set of design

guidelines for m-learning environments.

1 INTRODUCTION

Smartphones and tablet devices offer ubiquitous m-

learning opportunities for higher-education students

who study while on-the-move (Cochrane and

Bateman, 2010). Over and above educational

aspects, the perceived usability and user experience

(UX) of m-learning platforms depends to a large

extent on underlying design and development factors

(Oinas-Kukkonen and Kurkela, 2003). This study,

which proposes a framework of design guidelines

for the synthesis of m-learning environments, is a

secondary outcome of the iterative development and

evaluation of a real-world learning environment

called Mobile Learning Research (m-LR) (Harpur,

2013, de Villiers and Harpur, 2013). A first set of

design guidelines for m-learning was gleaned from

the literature and used to generate the first version of

m-LR. The purpose of this paper is not primarily to

report on the evaluation and redesign of m-LR

prototypes. However, the findings of the evaluations

formed a vital part of the design process and the

emergence of a second set of guidelines.

Ad hoc approaches that ignore design principles

in the development of m-learning, may have

unfortunate usability and UX implications that

designers and developers of the end-product could

regret. This study contributes to knowledge by

synthesising a framework of design guidelines that

incorporates theoretical guidelines based on

acknowledged literature sources, as well as practical

guidelines from the findings of empirical evaluation

studies. Participants’ responses identified problems

in m-LR and also provided positive feedback. The

composite set of guidelines, emerging from both

theory and research, offers a rich broad-based

collection of design guidelines that is transferable

and adaptable to various mobile- and tablet-based

learning situations. Through use and evaluation of

m-LR, the theoretical guidelines were affirmed by

use and new ones emerged.

Section 2 sets the context and outlines the

background of this research. The research design is

133

Harpur P. and de Villiers R..

Synthesis of a Framework of Design Guidelines for m-Learning Environments - A Study in a Tertiary Education Context in South Africa.

DOI: 10.5220/0004874501330143

In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 133-143

ISBN: 978-989-758-020-8

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

described in Section 3. Thereafter we view the

guidelines from three lenses: their emergence;

application; and evolution. Section 4 lists the initial

guidelines that emerged from a literature review,

while the impact of practically applying the

guidelines, is reported in Section 5. Section 6

presents further guidelines that evolved from

evaluations. The study is concluded in Section 7.

2 BACKGROUND

The primary researcher is a lecturer of

undergraduate students taking Software Engineering

at a private South African university. She observed

that some students seemed demotivated by

traditional face-to-face classroom education. In

addition, the limited access of certain students to the

Internet via PCs and laptops hindered effective

communication and collaboration on group projects.

However, the majority of students had smartphones

and/or tablet devices and were proficient in the use

of digital technology. They depended on these when

they returned home and contributed to group

projects from a distance, thus converting their mode

of study to blended learning. This suggested that a

technology-enhanced mobile learning solution might

enrich their learning experience, therefore the

researchers set out to custom-build an initial m-

learning prototype, version m-LR

, on a Moodle

platform, a distributed digital learning environment,

suited to m-learning requirements. Even though

distributed digital learning environments such as

Blackboard and Moodle provide support for

anytime- and anywhere-learning, pertinent design

issues include:

 The difficulty of integrating course material from

different sources;

 The unwieldy size of digital learning systems;

 Maintenance pressures due to continual addition of

learning content;

 A need to accommodate scalability and security

requirements;

 Network problems; and

 Limitations posed by the delivery of large

multimedia and course content (Li et al., 2008).

Guidelines were sought for producing and

improving the prototype to optimise the effect of

adaptations and to avoid an ad hoc design and

development strategy. Despite an extensive literature

study, no single appropriate design and development

methodology was found.

A decision was therefore taken to synthesize a

set of design guidelines in parallel with the

development of the m-learning environment. Ethical

clearance for the research was obtained from the

institution where it was conducted and from the

university where the primary researcher was

enrolled for postgraduate studies.

3 RESEARCH DESIGN

The research paradigm used to generate m-LR, was

design-based research. ‘Design science’ originated

from the Nobel prize winner, Herbert Simon (Simon,

1981). It led in turn to ‘design research’, which

investigates artificial phenomena and solves

complex problems by creating and evaluating man-

made products.

In the discipline of Information Systems,

research of this nature is termed ‘design science

research’ (DSR) (March and Smith, 1995, Peffers et

al., 2007), and in the educational technology milieu,

is called ‘design-based research’ (DBR) (Barab and

Squire, 2004, Amiel and Reeves, 2008, Anderson

and Shattuck, 2012). DBR is appropriate for

pragmatic contextual research in complex domains,

and is implemented by iterative cycles of empirical

studies. It has dual outcomes in the form of (i) useful

authentic products and (ii) theoretical contributions

that are transferable to other environments. In this

research, the practical outcome was m-LR and the

theoretical outcome was the set of design guidelines,

which are the contribution addressed by this paper.

3.1 Research Question

The following research question was thus posed:

What are appropriate guidelines to use for

the design and development of m-learning environments?

The process of answering the question is shown in

Figure 1, which depicts the iterative development

and evaluation processes of m-LR versions: m-LR

m-LR

, m-LR

, and a future final one, m-LR

This research reflects on these four versions and

four iterative studies: Evaluation Study 1, Evaluation

Study 2, Evaluation Study 3 and Evaluation Study 4.

3.2 Research Methods

Evaluations and Participants

The research was conducted over two and a half

years and entailed evaluations by two different

methods amongst two kinds of participants.

First, heuristic evaluations (HE’s) were

undertaken, in which between three and five

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

134

experienced evaluators, so-called ‘experts’, study a

system to identify problems and strengths (Nielsen

and Molich, 1990, Nielsen, 1992). Different

heuristic evaluators were hand-picked for each of the

four studies. All of them were experts in HCI or in

digital education or both, so-called 'double experts'.

The numbers of heuristic evaluators in the studies

were in line with Nielsen’s recommendations.

Second, questionnaire surveys were conducted

amongst end-users, i.e. using students as evaluators.

In the first two evaluation studies, samples of

experienced software engineering students were

purposively selected to participate, while in the third

and fourth studies, entire cohorts were used, i.e. the

participants were a population, not a sample.

Furthermore, Evaluation Studies 1 to 3 were

conducted on a suburban campus of the university,

while Evaluation Study 4 was done on two different

campuses of the institution – the suburban one, as

well as a campus in an urban area, which was less

affluent. This enriched the findings by administering

questionnaires to two varying cohorts.

The use of these research methods aimed to

achieve method triangulation (two different

methods) and data triangulation (across two different

campuses). The evaluations are described in detail in

Section 5.1.

Evaluation Procedure and Tasks

The iterative approach provided sequential

evaluations of the usability and user experience of

four different versions of the m-learning

environment.

Prior to conducting the evaluations, participants

– both expert evaluators and students – completed a

defined series of software engineering activities via

mobile devices. This familiarised them holistically

with the features of m-LR, enabling them to

effectively evaluate versions of the m-LR platform.

These activities included: secure login; exploration

of specific software engineering course content; a

brief review of a lesson, completion of a quiz

associated with the lesson; entry of a blog comment;

contribution to a forum discussion; a search for

particular terms in a glossary; participation in a

software engineering chat session; a contribution to

a wiki topic; and the viewing of online media. This

comprehensive exposure was crucial in helping them

contribute meaningfully to the development of new

guidelines. The resulting practical guidelines, which

reflect findings of the evaluation processes, are

presented in Section 6.

Research Instruments

The questionnaires were not adaptations of a

standard instrument. Instead the evaluation criteria

were custom-developed, based on five categories,

namely: general interface criteria; pedagogical

aspects; website specific criteria; factors specific to

m-learning; and user experience (UX) criteria. A

separate publication is in progress, focusing on the

generation of these criteria.

Constructs in the questionnaires were based on

these categories. There were evaluation statements

using a 5-point Likert scale, as well as open-ended

items for qualitative responses. The HE’s by experts

and questionnaire surveys amongst end-users thus

provided both quantitative and qualitative data. The

questionnaires and evaluation criteria are not

included in an appendix, because they are too long.

The quantitative data analysis and the qualitative

thematic analysis of free text identified usability and

UX problems as well as highlighting positive

factors. This led to a set of practical guidelines,

specific to the design of m-learning environments.

Figure 1: Development of m-LR through four versions and four evaluation studies.

SynthesisofaFrameworkofDesignGuidelinesform-LearningEnvironments-AStudyinaTertiaryEducationContextin

SouthAfrica

135

3.3 Derivation of Design Guidelines

As previously mentioned, the guidelines emerged

from two types of sources. Firstly, existing

guidelines were garnered from pertinent literature

and synthesized into an initial framework (Section

4), presented in Table 1, i.e. the literature served as

secondary data. Secondly, empirical research was

undertaken (Section 5) by evaluating versions of m-

LR as described in Section 3.1 and illustrated in

Figure 1. The findings of these evaluations (Section

6) were used as primary data to generate Table 3,

which extends and completes the framework.

4 INITIAL GUIDELINES

EMERGING FROM THE

LITERATURE

Literature sources address various challenges within

m-learning domains, including the issue of

appropriate design guidelines for m-learning

applications. The literature study underlying this

study was comprehensive, being both broad and

deep.

The work of many researchers contributed to the

formulation of the initial collection of guidelines.

Space constraints preclude detailed discussion of all

the sources consulted, but we specifically mention

contributions from certain acknowledged researchers

such as:

 Botha, van Greunen and Herselman (2010): mobile

learning interactions viewed from an HCI

standpoint;

 Ebner (2009): inclusion of Web 2.0 features such

as microblogging in mobile learning;

 Oinas-Kukkonen and Kurkela (2003): developing

successful mobile applications; and

 Sharp, Rogers, and Preece (2007): a focus on

interaction design.

The overall literature review resulted in the

structured synthesis of an initial set of design and

development guidelines, which are presented in

Table 1. This table plays an important role in this

paper, in that it is the first version of the framework

of design guidelines, namely the contribution that

emerged from theory. The initial framework

comprises eight categories of guidelines.

Within each category, key terms are italicised to

emphasise the contributions made by various

authors. The contributing authors are cited in

alignment in the third column.

Table 1: Design guidelines for m-learning environments – emerging from the literature.

Design and Development Guidelines Literature Sources

1 Strategy

 Provide interactivity via UCD

 Improve the environment by implementing iterative design

Göker and Myrhaug (2008)

 Involve experts in contributing to the design Bri, Garcia, Coll and Lloret (2009)

2 Mobile Specifications

 Provide accessible information to students whilst they are

moving to and from locations, around campus, in the

classroom, and between the outside world and the university

Oinas-Kukkonen and Kurkela (2003),

Sharma and Kitchens (2004)

 Focus more on content and m-learning than on technology Landers (2002), Pinkwart, Hoppe, Milrad

and Perez (2003)

 Link tasks to course content Parsons, Ryu and Cranshaw (2006)

 Support social networking and learning, seamlessly Landers (2002), Parsons, Ryu and

Cranshaw (2006), Sharma and Kitchens

(2004)

 Include mobile specifications with accessibility via all devices Low and O’Connell (2006)

 Aim for compatibility with a wide range of media Landers (2002), Parsons et al.(2006)

 Incorporate security and privacy features Naismith, Lonsdale, Vavoula and Sharples

(2004), Pinkwart et al. (2003)

3 User-centricity

 Involve end users in guiding the design of the interface, Sharp, Rogers, and Preece (2007)

 Consider users’ understanding of terminology and navigation

 Incorporate usefulness from a user’s perspective

 Allow customisation and adaptability for each user’s

preferences, needs and abilities

Oinas-Kukkonen and Kurkela (2003)

 Include features that enhance motivation Botha, van Greunen and Herselman (2010)

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

136

Table 1: Design guidelines for m-learning environments – emerging from the literature (Cont.).

Design and Development Guidelines Literature Sources

4 Ease of Use

 Focus on simplicity and flexibility Oinas-Kukkonen and Kurkela (2003),

Sharma and Kitchens (2004)

 Aim for easy assimilation on the part of the student

 Facilitate availability of important information

Bri et al. (2009)

 Present only the essential and consistent information

 Make provision for evaluation of usability

 Implement fluent navigation

Low and O’Connell (2006)

5 Content

 Include self-contained ‘chunks’ of educational material;

 Provide content in accessible and compact formats, presented

in multiple ways

Low and O’Connell (2006), Sharma and

Kitchens (2004)

 Provide facilities that accommodate communication and

collaboration within learning

Bri et al.(2009)

 Ground the content in teaching and learning Botha, van Greunen and Herselman

(2010), Bri et al.(2009), Cheung (2009)

6 Context

 Take cognizance of mobility levels, usage mode, time and place

of learning, budget, and network connectivity factors;

Botha et al. (2010), Oinas-Kukkonen and

Kurkela (2003)

 Plan for in-situ learning associated with new, individual and

team skills with social interaction;

Parsons et al. (2006)

 Incorporate a selection of screen and keyboard/touch options,

operating systems, device types, network configurations, and

student characteristics.

Botha et al. (2010)

7 VLEs

 Ensure that the environment reflects academic vision and offers

relevant curriculum content, providing training and support for

staff and students

Cheung (2009)

 Resolve copyright and intellectual property issues Levy (2003)

 Rapidly provide value in a natural way via mobile services Bri et al. (2009), Oinas-Kukkonen and

Kurkela (2003)

 Consider that digital technology has changed students’ views of

writing in the “old-fashioned” way. Current strategies differ

from “pencil and paper” lessons

Lai, Yang, Ho and Chant (2007), Pinkwart

et al. (2003)

 Offer uniform access to a variety of information sources, e.g.

websites, glossaries, reading material, relevant YouTube

videos, other student opinions

Bri et al. (2009), Pinkwart et al. (2003)

8 Web 2.0 Tools

 Extend the student’s classroom experience Ebner (2009)

 Include Web 2.0 features e.g. podcasts, blogs, microblogs,

wikis, and social networking sites (SNSs)

Ebner (2009), Ebner and Schiefner (2008),

Lockyer and Patterson (2008), Minocha

and Thomas (2007), Safran (2008)

 Emphasize the planning required for implementing social

networking applications within an online educational program

Lockyer and Patterson (2008)

 Facilitate communication and collaboration via synchronous

technologies (chat rooms) and asynchronous interactivity

(forums and e-mail)

Jones (2010), MacCallum and Kinshuk

(2008), Minocha and Thomas (2007)

5 EMPIRICAL STUDIES

5.1 Evaluation Studies

The guidelines in Table 1 were used in generating

four versions of a mobile learning environment: m-

, m-LR

, and m-LR

, with a view to a

future version, m-LR

. These versions can thus be

considered applications of the guidelines.

Figure 1 in Section 3 showed the four sequential

evaluation studies – Evaluation Studies 1, 2, 3, and 4

– that were respectively implemented on the four

successive versions of m-LR. Participants in these

studies were experts who served as heuristic

evaluators and students who completed

questionnaires. A brief description is now provided

of each study, explaining its context and purpose.

However, the findings reported in the next section,

Section 6, focus exclusively on findings of the final

two studies, Evaluation Study 3 and Evaluation

SynthesisofaFrameworkofDesignGuidelinesform-LearningEnvironments-AStudyinaTertiaryEducationContextin

SouthAfrica

137

Study 4, which impacted strongly on the evolution

of further guidelines.

Evaluation Study 1: As the study in 2010 that

evaluated the usability of the first version of m-LR, a

Moodle customised in an ad hoc manner, Study 1

uncovered usability challenges. Its purpose was to

evaluate m-LR

running on the researcher’s own

Blackberry 9700 smartphone device, and to produce

both quantitative and qualitative findings.

Participants were three experts for an HE and ten

students in a questionnaire survey, all drawn from a

single campus. Subsequent changes to m-LR

included redesign of:

 Course content, providing information in chunks

for viewing on mobile devices;

 Log-in features;

 Quantity of information per page;

 Formats of downloadable media and subject

matter;

 Contents of the glossary.

Adjustments to the m-LR

version led to m-LR

Evaluation Study 2: This small-scale evaluation of

m-LR

in 2011, was a pilot study for the major

Evaluation Study 3. Study 2 was implemented on a

single campus with only four students

(questionnaires) and one double expert (HE), As in

Study 1, each used the same Blackberry 9700

smartphone for their evaluation. Study 2 primarily

served to try out the research procedures, tasks,

instruments and evaluation processes, but did lead to

improvements in the:

 Privilege levels of the blog;

 Options offered by the glossary;

 Look-and-feel facets such as font styles, size and

colour;

 Open-ended requirements for the quiz.

The refinements resulted in m-LR

Evaluation Study 3: This study in 2011, was the

largest up to that point. Five HE experts, with their

own devices, and seventeen students from one

campus, using Blackberry 9700 smartphones,

evaluated m-LR

for usability and UX. Adjustments

and extensions resulted in the next version of m-LR,

namely m-LR

. Feedback called for new

functionalities or changes to:

 The help functionality;

 Built-in documentation;

 Links and navigation mechanisms;

 ‘Breadcrumbs’, and thus navigability;

 Compatibility of media such as video;

 The range of device types, which needed to be

broadened;

 File size, aiming to reduce buffering issues;

 Layout of goals and objectives of learning units;

 Offline reading capability, required by users;

 Access to social networking applications

A growing need for storage and collaborative

services via cloud technology such as Dropbox and

Google Drive became apparent. It was suggested

that the addition of these services would increase the

appeal of m-LR whilst facilitating communication

between teams of students doing collaborative

projects.

Evaluation Study 4: This final, and largest, 2012

study used m-LR

as input. Its feedback engendered

guidelines that, if applied, would produce a future

improved version, m-LR

. Five experts and 33

students from two different campuses completed

evaluations, proposing modifications to the:

 Application design;

 Specifications, as participants required

compatibility with several device types;

 The user experience, which could be enhanced

by improving ease of use.

Despite the high number of participating

evaluators, only a few problems were reported,

indicating the success of the iterative evaluation and

redesign process.

5.2 Aspects of m-LR

Figures 2 illustrates learning aspects of m-LR such

as: topics and lessons; activities – chats, forums,

glossaries, online self-assessment quizzes, resources

Figure 2: Homepage for the 2012 Software Engineering

course.

(content and media), and wikis. These features

provided team members with collaborative

opportunities, extending the classroom experience to

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

138

incorporate Internet links to course related websites.

Figure 3 presents a view of a student response to

a self-assessment quiz item, demonstrating

immediate feedback.

Figure 3: Self-assessment quiz with digital feedback.

Figure 4: Key software engineering terms stored

conveniently in a ‘mobile’ glossary – easily accessible via

mobile device.

A search for term ‘project manager’ in a software

engineering glossary is shown in Figure 4, where

‘Add a new entry’ indicates how students participate

in formulating the glossary.

5.3 Positive and Negative Feedback

Table 2 moves the overview away from the four

studies in Figure 1 and focuses on Evaluation

Studies 3 and 4, indicating the positive progress

made as m-LR evolved.

Table 2: Positive and negative feedback leading to the

synthesis of new guidelines.

Positives Negatives

New

guidelines

Evaluation

Study 3

(n=22)

74 87 25

Evaluation

Study 4

(n=38)

132 90 18

In Evaluation Study 3, the number of negative

comments (87) exceeded the positives (74). In

Evaluation Study 4, there were more positive

comments (132) than negatives (90). This

occurrence, despite the fact that there were almost

twice as many evaluators in Study 4, suggests that

the adjustments made after Study 3 were effective.

In particular, qualitative analysis showed that the

number of problems identified in the Ease of Use

category, decreased from 33% of the negative

feedback in Evaluation Study 3 to 26% of the

negative feedback in Evaluation Study 4.

Furthermore, two additional categories are

singled out, namely: Strategy and Mobile

Specifications. For Evaluation Study 3 and

Evaluation Study 4, the Strategy category

contributed 14% and 22% respectively to the total

number of problems. Problems reported in the

Mobile Specifications theme increased from 7%

(Evaluation Study 3) to 22% (Evaluation Study 4).

The increase in reported problems in these two

categories can be ascribed to three factors: a

dynamically changing digital environment; greater

technical acuity of the participants; and a more

techno-savvy attitude to mobile technology design

concepts.

As stated, the primary purpose of this paper is

not to serve as an evaluation study but to use

evaluation findings to derive generic design

guidelines. m-LR provided a platform requiring

evaluation, while the findings of the evaluations led,

in turn, new guidelines that are presented in the next

section.

SynthesisofaFrameworkofDesignGuidelinesform-LearningEnvironments-AStudyinaTertiaryEducationContextin

SouthAfrica

139

6 EVOLUTION OF NEW

GUIDELINES

An overview of four iterative evaluations was

presented in Figure 1. Positive factors that emerged

from these evaluations affirm the original guidelines,

while yet other positive aspects, spontaneously

articulated by participants, suggest implicit strengths

that should be made explicit by new guidelines.

Similarly, problems that emanated from the

evaluations, suggest further required guidelines. If

the resulting new guidelines, particularly those that

come from qualitative findings, are consolidated into

a framework along with the original eight

synthesised from the literature, the result would be

sixteen themes of guidelines.

These new aspects are now addressed in detail,

as emergence of the original set of guidelines is

reviewed, and the evolution of the additional sets,

particularly from Studies 3 and 4, is set out.

6.1 Emergence of the Original

Theoretical Guidelines

Table 1, which presented an initial framework of

theoretical guidelines for the design and

development of m-learning environments, comprised

eight themes, namely: 1 Strategy; 2 Mobile

specifications; 3 User-centricity; 4 Ease of use; 5

Content; 6 Context; 7 Virtual learning eight themes,

Table 3: Guidelines emerging from findings of Evaluation Studies 3 and 4.

Theme

Guidelines Theme Guidelines

9 Devices

 Use technology to enhance, rather than hinder,

learning experiences

 Ensure compatibility with a range of devices

 Support easy access

 Consider the limitations of input mechanisms

 Consider screen size when incorporating

features

13 Interactivity

 Enable ‘on-the-fly’ communication with

classmates

 Support visibility of other online students 

 Facilitate collaboration on group projects

 Allow sharing of information

 Include alternative forms of interactivity,

designed to suit user preferences

10 Assessment

 Provide opportunities for self-assessment

 Provide lecturer support for the correction of

errors in such self-assessment

 Include multiple choice questions

 Consider the possibility of short-answer

questions

 Locate quiz options with associated course

content

 Incorporate rapid test feedback to support

students doing revision while on the move

 Provide links to online material to facilitate

preparation of coursework assignments

 Align assessment exercises with examination

preparation

14Visual Factors

 Provide a simple and appealing layout

 Design a look and feel that is user-centric

 Implement suitable colour schemes

 Make effective use of white space

 Where possible and appropriate, enhance the

experience with suitable graphic content,

headings and font choices

 Strengthen the visual experience, possibly

including some animation

 When constructing a site, design for logical

page order 

 Create a professional learning environment

11 Efficiency

 Ensure speedy loading of site and pages

 Provide immediate responses for users

accessing features of the application

 Achieve fast navigation between links

 Aim for rapid content delivery

15 Innovation

 Offer the user an environment which is

perceived as new and novel

 Facilitate off-campus mobile learning

 Deliver course content in a creative and novel

digital manner

12 Navigability

 Facilitate easy and intuitive navigation

 Ensure visibility of links 

 Support browsing – anytime and anywhere

16 Satisfaction

 Create a match to user’s view of suitability

 Embrace the user’s sense of excitement

 Provide consistency with familiar applications

and interfaces to avoid user frustration

 Focus on easy readability

 Establish an easy-to-follow flow, starting with

an introduction and progressing logically to

module sections

 Build a learning environment which offers

enjoyable user experiences

 Ensure that functionalities operate correctly

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

140

namely: 1 Strategy; 2 Mobile specifications; 3 User-

centricity; 4 Ease of use; 5 Content; 6 Context; 7

Virtual learning environments; and 8 Web 2.0 tools.

These initial themes and guidelines were used in the

development of the first version, m-LR

, and also

contributed, along with findings of the series of

evaluations, to adjustments and improvements of

subsequent versions of the mobile learning

environment.

6.2 Guidelines from Evaluation Study 3

The evaluation of usability and UX of m-LR

Study 3 by 22 participants produced positive and

negative feedback, resulting in practical principles

and guidelines that led to the development of version

m-LR

The positive results of the evaluation served two

purposes. Firstly, they affirmed the utility of the

initial framework of theoretical guidelines and,

secondly, the positive response that emerged for

aspects of m-LR that had been designed intuitively,

showed the need to concretise and formalise certain

implicit guidelines. Negative feedback and concerns

indicated inadequacies in the design and highlighted

the need for further guidelines. Thematic analysis of

qualitative data, positive and negative, confirmed

existing guidelines, while other evaluation findings

were converted to five new categories of guidelines,

namely: 9 Devices; 10 Assessment; 11 Efficiency; 12

Navigability; and 13 Interactivity, which are

elaborated in Table 3.

6.3 Guidelines from Evaluation Study 4

The evaluation of m-LR

was a major study, with 38

participants, namely five experts and 33 students

from cohorts on two campuses. Due to the

improvements implemented to m-LR

after Study 3,

the number of negative issues decreased, while

positive feedback increased, as shown in Table 2.

Moreover, negativity previously indicated by some

initial themes namely: Mobile Specifications; User-

centricity; and Ease of Use declined, probably due to

the strength of the version m-LR

, that had resulted

from Evaluation Study 3.

Only three new categories of guidelines resulted

from the findings of Evaluation Study 4, namely; 14

Visual Factors; 15 Innovation; and 16 Satisfaction.

Finally, suggestions emerged from Study 4 for

improvements to m-LR

that would result in a future

version m-LR

6.4 Evolution of Guidelines

and Extension to the Framework

The new categories of themes and design guidelines,

introduced in Sections 6.2 and 6.3, are listed in

Table 3. The original themes that re-emerged are not

included, since they are already in Table 1. This

table, which is the contribution that emerged from

practical empirical research, continues the evolution

of the framework of design guidelines.

6.5 Final Set of Guidelines

An integration of Tables 1 and 3 would constitute

the final and comprehensive set of guidelines in

sixteen categories for the design and development of

mobile learning environments. Such a merged table

of final guidelines, though valuable, would be

repetitive and, for the sake of space, is explicitly

omitted from this paper.

7 CONCLUSIONS

This study addressed the following research

question:

Which guidelines should be included in a framework

for the design and development of m-learning

environments?

An aggregation of the themes and guidelines in

Tables 1 and 3, where theoretical sources from the

literature and practical empirical findings,

respectively, contributed to the synthesis of a final

set of guidelines, demonstrate that the research

question has been answered.

The goalposts moved over the duration of the

studies, and the target audiences became more

‘techno-savvy’! Although students were increasingly

satisfied with the successive versions, the later

feedback began to address sophisticated refinements.

It appeared that the users were expecting similar

functionality and features in a basic system designed

by an academic, to what is encountered in

commercial apps!

Rapid methods of evaluation, synthesis,

adaptation and evolution will depend on a

comprehensive and malleable set of design and

development guidelines for effectiveness.

Moreover, students who are mobile and on-the-

move expect fast and dynamic virtual learning

environments which demonstrate flexibility. The

evolution of the integrated multi-facetted set of

guidelines is in line with these requirements.

SynthesisofaFrameworkofDesignGuidelinesform-LearningEnvironments-AStudyinaTertiaryEducationContextin

SouthAfrica

141

The perspectives of a variety of evaluators

(content and application developers, students,

lecturers, administrators, e-learning experts)

contributed to participative evaluation findings that

led to iterations of redevelopment of a successful

mobile learning application. The empirical findings

of this study show that the varying expertise and

idiosyncrasies of participants provided a broad

spectrum of issues and positive contributions. A

design and development strategy should

accommodate the viewpoints from each of these user

groups.

Due to the dynamic and rapidly evolving nature

of mobile technology, the task of formulating a set

of guidelines is unlikely to be complete. Hence the

framework synthesized in Tables 1 and 3, begs

augmentation over timelines and application within

other tertiary education contexts. Moreover, it is

likely that different device types will be brought to

the classroom.

The application of the guidelines will evolve

further as capabilities and affordances of new

technologies are accommodated. Furthermore, it is

acknowledged that the current framework is lengthy

due to its comprehensive nature. Synthesis of the

categories into a tighter framework could facilitate

practical application. Finally in transferring the

application of the guidelines to other mobile

environments, they can be reduced and customized

to the context and content.

Whereas literature sources may provide an initial

foundational set of guidelines based on theoretical

underpinnings, this study demonstrates that

empirical findings based on participative user-

centric designs can extend and enrich a framework

of design guidelines.

REFERENCES

Amiel, T. & Reeves, T. C. 2008. Design-Based Research

and Educational Technology: Rethinking Technology

and the Research Agenda. Educational Technology &

Society, 11, 29-40.

Anderson, T. & Shattuck, J. 2012. Design-Based Research

A Decade of Progress in Education Research?

Educational Researcher, 41, 16-25.

Barab, S. & Squire, K. 2004. Design-based Research:

Putting a Stake in the Ground. The Journal of

Learning Sciences, 13, 1-14.

Botha, A., van Greunen, D. & Herselman, M. 2010. A

Mobile Human-Computer Interaction Perspective on

Mobile Learning. 6th Annual International

Conference on Computing and ICT Research.

Kampala, Uganda: Researchspace Software.

Bri, D., Garcia, M., Coll, H. & Lloret, J. 2009. A Study of

Virtual Learning Environments. WSEAS Transactions

on Advances in Engineering Education.

Cheung, K. S. 2009. A Framework for Developing E-

Learning Solutions. Fifth International Conference on

Semantics, Knowledge and Grid. IEEE Computer

Society.

Cochrane, T. & Bateman, R. 2010. Smartphones give you

wings: Pedagogical affordances of mobile Web 2.0.

Australasian Journal of Educational Technology, 26,

1-14.

de Villiers, M. R. & Harpur, P. A. 2013. Design-based

research – the educational technology variant of

design research: Illustrated by the design of an m-

learning environment. Annual Research Conference of

the South African Institute of Computer Scientists and

Information Technologists. East London, South

Africa: ACM International Conference Proceedings.

Ebner, M. 2009. Introducing live microblogging: how

single presentations can be enhanced by the mass.

Journal of research in innovative teaching, 2, 91-100.

Ebner, M. & Schiefner, M. 2008. Microblogging - more

than fun? In: SÁNCHEZ, I. A. & ISAÍAS, P. (eds.)

International Association for Development of the

Information Society (IADIS) Mobile Learning 2008

International Conference. Algarve, Portugal: Citeseer.

Göker, A. & Myrhaug, H. 2008. Evaluation of a mobile

information system in context. Information Processing

and Management (Online), 44. Available:

http://www.sciencedirect.com/science/article/B6VC8-

4PMJB64-1/2/d221a21b1981c20d6112e7b2d5431aef.

Harpur, P. A. 2013. Evaluation of Usability and User

Experience of an m-Learning Environment, Custom-

Designed for a Tertiary Educational Context.

Unpublished MSc dissertation, University of South

Africa.

Jones, P. 2010. Collaboration at a Distance: Using a Wiki

to Create a Collaborative Learning Environment for

Distance Education and On-Campus Students in a

Social Work Course. Journal of Teaching in Social

Work

(Online), 30. Available: http://0-

pdfserve.informaworld.com.oasis.unisa.ac.za/279896_

731386761_922215591.pdf [Accessed 6 June 2011].

Lai, C.-H., Yang, J.-C., Ho, C.-W. & Chant, T.-W. 2007.

Affordances of mobile technologies for experiential

learning: the interplay of technology and pedagogical

practices. Journal of Computer Assisted Learning, 23,

326-337.

Landers, P. 2002. From e-learning to m-learning (Online).

Ericsson. Available: http://learning.ericsson.net/

mlearning2/project_one/elearnmlearn.html, (Accessed

November 2012).

Levy, S. 2003. Six factors to consider when planning

online distance learning programs in higher education.

Online Journal of Distance Learning Administration, 6.

Li, Q., Lau, R. W. H., Shih, T. K. & Li, F. W. B. 2008.

Technology Supports for Distributed and

Collaborative Learning over the Internet. ACM

Transactions on Internet Technology, 8, Article 10.

Lockyer, L. & Patterson, J. 2008. Integrating social

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

142

networking technologies in education: a case study of

a formal learning environment. Eighth IEEE

International Conference on Advanced Learning

Technologies, 2008. Santander, Spain: IEEE

Computer Society.

Low, L. & O’Connell, M. 2006. Learner-Centric Design

of Digital Mobile Learning. Proceedings of the OLT

Conference. Brisbane, Australia.

MacCallum, K. & Kinshuk, K. 2008. Mobile technology

in collaboration: evaluation of a web-based discussion

board. International Journal Mobile Learning and

Organisation, 2, 318-328.

March, S. T. & Smith, G. F. 1995. Design and natural

science research on information technology. Decision

Support Systems, 15, 251-266.

Minocha, S. & Thomas, P. G. 2007. Collaborative

Learning in a Wiki Environment: Experiences from a

Software Engineering Course. New Review of

Hypermedia and Multimedia, 13, 187-209.

Naismith, L., Lonsdale, P., Vavoula, G. & Sharples, M.

2004. Literature Review in Mobile Technologies and

Learning. Available: http://citeseerx.ist.psu.edu/

viewdoc/download?doi=10.1.1.136.2203&rep=rep1&t

ype=pdf (Accessed 11 April 2011).

Nielsen, J. 1992. Finding Usability Problems Through

Heuristic Evaluation. CHI '92 Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems. ACM.

Nielsen, J. & Molich, R. 1990. Heuristic evaluation of user

interfaces. CHI '90 Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems.

Seattle, Washington: ACM.

Oinas-Kukkonen, H. & Kurkela, V. 2003. Developing

Successful Mobile Applications. IASTED

International Conference on Computer Science and

Technology. Cancun, Mexico: CSA Illumina.

Parsons, D., Ryu, H. & Cranshaw, M. 2006. A Study of

Design Requirements for Mobile Learning

Environments. Sixth International Conference on

Advanced Learning Technologies (ICALT '06).

Kerkrade, The Netherlands: IEEE Computer Society.

Peffers, K., Tuunanen, T., Rothenberger, M. A. &

Chatterjee, S. 2007. A design science research

methodology for information systems research. Journal

of Management Information Systems, 24, 45-77.

Pinkwart, N., Hoppe, H. U., Milrad, M. & Perez, J. 2003.

Educational Scenarios for Cooperative Use of Personal

Digital Assistants. Journal of Computer Assisted

Learning, 19, 383-391.

Safran, C. 2008. Blogging in Higher Education

Programming Lectures: An Empirical Study.

MindTrek '08. Tampere, Finland: ACM.

Sharma, S. K. & Kitchens, F. L. 2004. Web Services

Architecture for M-Learning. Electronic Journal on e-

Learning, 2.

Sharp, H., Rogers, Y. & Preece, J. 2007. Interaction

design: beyond human-computer interaction, USA,

John Wiley & Sons.

Simon, H. A. 1981. The sciences of the artificial,

Cambridge, MA, MIT Press.

SynthesisofaFrameworkofDesignGuidelinesform-LearningEnvironments-AStudyinaTertiaryEducationContextin

SouthAfrica

143