Virtual Laboratories

Use of Public Testbeds in Education

Niky Riga

, Vicraj Thomas

, Vasilis Maglaris

, Mary Grammatikou

and Evangelos Anifantis

BBN Technologies, Cambridge, Massachusetts, U.S.A.

School of Electrical and Computer Enginnering, National Technical University of Athens, Athens, Greece

Keywords:

Remote Labs, Computer Science Labs, Public Testbeds, Education, Computer Network Labs, e-Learning.

Abstract:

Experimentation is an invaluable part of learning in all sciences. However, building and maintaining lab-

oratories is expensive, time and space consuming. Moreover, in computer science advances in technology

can quickly make the infrastructure obsolete. In this paper we advocate the use of recently deployed public

testbeds as remote labs for computer science education. As an example we describe the successful use of the

GENI testbed in graduate and undergraduate courses and present a speciﬁc case study of GENI being used in

an undergraduate class on Network Management and Intelligent Networks.

1 INTRODUCTION

Experimentation is an invaluable part of learn-

ing (Bruner, 1961) in all sciences. Research has

shown that lab courses signiﬁcantly enhance learn-

ing (Freedman, 1997; Magin et al., 1986) and

help students develop problem-solving and critical-

thinking skills (American Chemical Society, 2014).

However, building and maintaining laboratories is of-

ten expensive, time and space consuming. In com-

puter science in particular that is a fast-evolving ﬁeld,

equipment gets outdated very quickly and requires

frequent costly upgrades.

At the same time there have been several efforts

around the globe to build and maintain publicly avail-

able testbeds (Berman et al., 2014; Peterson et al.,

2003; White et al., 2002; Mirkovic et al., 2010;

Fdida et al., 2011) to support scientiﬁc exploration

that scale beyond the capabilities of individual in-

stitutions. These testbeds, over the past few years,

have matured and transitioned from small prototype

deployments to large production virtual labs.

Although the primary purpose of these testbeds is

experimentation for research, their accessibility, scale

and ease of use makes them well suited for lab exper-

iments in computer science courses. While it is true

that remote labs cannot replace all computer science

labs (e.g. hardware labs), experience has shown these

testbeds work very well for a number of other com-

puter science classes.

Researchers have identiﬁed three different types

of laboratories for education and research: hands-on

labs, simulation labs and remote labs (Ma and Nick-

erson, 2006; M¨uller and Erbe, 2007). Hand-on labs

are co-located with its users and are intended for use

by users from the institution that owns the lab. Simu-

lation labs (also called virtual labs) simulate the labo-

ratory infrastructure on computers. Remote labs are

geographically separated from its users and are in-

tended to be a shared resource for users from differ-

ent institutions. Multiple studies have examined the

effectiveness of the different types of labs in student

understanding (Ma and Nickerson, 2006; Corter et al.,

2007). Although there is a component of the hands-

on labs that can not be easily substituted by simulated

or remote labs, studies indicate that in many cases re-

mote and hands-on labs are equivalent in terms of en-

hancing student conceptual understanding.

In this paper we advocate that the publicly acces-

sible research testbeds should be used as remote labs

for computer science education. We describe advan-

tages beyond the beneﬁts described in the above stud-

ies and urge educators around the world to take ad-

vantage of these valuable community resources and

incorporate them in their courses. As an example we

present how GENI (Berman et al., 2014), an infras-

tructure for computer networking and distributed sys-

tems research, is used as a remote lab for undergrad-

uate and graduate courses. Although GENI resources

are distributed across the United States, it is used by

instructors from aroundthe world and it supports mul-

tiple concurrent courses. We also include a case study

516

Riga N., Thomas V., Maglaris V., Grammatikou M. and Anifantis E..

Virtual Laboratories - Use of Public Testbeds in Education.

DOI: 10.5220/0005496105160521

In Proceedings of the 7th International Conference on Computer Supported Education (CSEDU-2015), pages 516-521

ISBN: 978-989-758-107-6

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

based on the use of GENI by the “Network Manage-

ment and Intelligent Networks” class at the National

Technical University of Athens.

2 RESEARCH TESTBEDS AS

REMOTE LABS FOR

EDUCATION

In this section we expand on our position that edu-

cators will reap great beneﬁts if they take advantage

of public testbeds being built for research, in their

classes. We advocate the use of these testbeds over

institution-speciﬁc hands-on labs and present a list of

beneﬁts for instructors and students.

We also describe a speciﬁc research testbed for

networking and distributed systems research, GENI,

and present why it has gained traction among educa-

tors as a remote lab in the past few years.

2.1 Educational Beneﬁts of Research

Testbeds

In most cases educators are better off using remote

labs over setting up their own hands-on labs. Hands-

on labs put a high demand on space, instructor time

and experimental infrastructure (Ma and Nickerson,

2006). The experimental infrastructure can be expen-

sive to acquire, set up and administer. Additionally,

advances in technology can quickly make the infras-

tructure obsolete.

An alternative to building local hands-on labs is

to use remote labs. While a few remote labs such as

(IBM News Release, 2007; Obstfeld et al., 2014) have

been built primarily for education many large testbeds

with substantial community support are being built

for research. Beneﬁts of leveraging these testbeds as

remote labs include:

• Accessibility. Research testbeds are designed for

use by researchers from around the globe. They

are available around the clock and accessible by

students from anywhere they have Internet con-

nectivity. Global accessibility also enables in-

novative teaching methodologies such as cross-

institutional student project teams.

• Access to unique or expensive resources. Re-

search testbeds often include resources and equip-

ment that would ordinarily not be affordable for

educational purposes. Students however can have

access to these resources if the testbed is used as

a remote lab.

• State-of-the-art resources. To facilitate cutting-

edge research, these testbeds are frequently up-

dated with state-of-the-art resources. This allows

students to be exposed to latest technology, with-

out the burden to the institution to maintain up-to-

date local labs.

• Number of resources. Many of the research

testbeds have orders of magnitude more resources

than is realistically possible in a typical hands-on

lab. This make it possible for instructors to as-

sign lab exercises that require more resources than

were previously practicable. This is particularly

beneﬁcial for classes in cloud computing and data

sciences.

• Community support and ease of use. Shared pub-

lic testbeds build a community of users that devel-

ops an maintain a variety of tools to enable easy

experimentation, and researchers spend more time

addressing research related problems than making

the testbed do what they want. Moreover educa-

tors using a research testbed as a remote lab, form

their own community that creates a pool of com-

mon and up-to-date resources (e.g. lab assign-

ments).

• Preparing students for research. Students who are

exposed to research testbeds during their course-

work are better prepared to use them for their re-

search.

2.2 GENI: A Laboratory for Research

and Education

GENI (Global Environmentfor Network Innovations)

is a laboratory for networking and distributed systems

research (Berman et al., 2014). It consists of com-

pute and networking resources distributed across the

United States. Researchers can reserve compute re-

sources and connect them in Layer 2 topologies that

are best suited to their experiments.

Although GENI is a research infrastructure, it has

been used extensively in graduate and undergradu-

ate classes as a remote lab by institutions around the

world ( Figure 1).

GENI is a sliced testbed i.e. multiple researchers

can run concurrent experiments in isolated slices of

the testbed, without functionally interfering with one

another (Figure 2). The sliceability of GENI makes

it an ideal platform for running classes since multi-

ple students can create virtual topologies on the same

infrastructure and multiple labs can run concurrently

without interfering with one another or with users

running experiments for their research.

VirtualLaboratories-UseofPublicTestbedsinEducation

517

Figure 1: Over 2000 students trained in GENI, in more than

80 courses.

Figure 2: Multiple users can concurrently use GENI in iso-

lated virtual topologies.

Other than its multi-user properties, GENI is a

great example of a research testbed that is suitable for

education because:

• It is deeply programmable and allows its users

to setup, modify, and study network protocols.

GENI is suitable for teaching a wide range of net-

working concepts. It has been used to teach the

basics of IP routing and TCP congestion manage-

ment as well as supported deployment of custom

routing algorithms.

• GENI includes unique resources that allows its

users to setup interesting experiments. For exam-

ple:

– Software Deﬁned Networking Resources.

GENI achieves deep network programmability

by deploying programmable network devices

(e.g. OpenFlow switches) in the core of the net-

work.

– WiMAX Base Stations. GENI has deployed

and virtualized several WiMAX base stations

that can be used remotely for wireless experi-

mentation.

• There is a community of instructors that

have developed and shared GENI-based course

ware (Marasevic et al., 2013; Grifﬁoen et al.,

2013; GENI Project Ofﬁce, 2014).

• GENI supports collaborative experimentation. In

the context of education this enables collaboration

between instructor and students and collaborative

team projects.

• GENI is designed to support cutting-edge net-

working research and thus has safe-guards in

place to protect the infrastructure from faulty ex-

periments. Additionally, since experimentation is

often a process of trial and error, GENI makes it

easy for a researcher to terminate an experiment

and start over. GENI is therefore a safe envi-

ronment for students to experiment and start over

when necessary, without the need for instructor or

administrator intervention.

• GENI employs a single sign-on mechanism that

provides easy access to students by simply us-

ing their institution login information. This is

achieved through InCommon (InCommon, 2014),

a US federation based on Shibboleth (Morgan

et al., 2004) authentication mechanism. Shibbo-

leth is very popular among educational institu-

tion around the world, making it straightforward

to provide access to international institutions not

members of InCommon.

GENI is not the only publicly available research

testbed. Over the past decade several testbeds have

been created to facilitate experimentation. Examples

of such testbeds include Emulab (White et al., 2002)

for networking research, Deter (Mirkovic et al., 2010)

for security research, ORBIT (Raychaudhuri et al.,

2005) for wireless experimentation, PlanetLab (Peter-

son et al., 2003) for distributed, peer-to-peer research,

OFELIA (Su et al., 2014) for programmable network

research, okeanos (Koukis and Louridas, 2013) and,

FIRE (Vandenberghe et al., 2013). Although these

testbeds are primarily intended to be used for scien-

tiﬁc exploration, they have been successfully used as

remote laboratories (Wong, 2012; PlanetLab, 2009).

Most of these testbeds are either federated or share

common APIs with GENI, forming a large ecosystem

at the disposal of educators.

3 CASE STUDY

The GENI infrastructure was used for two consecu-

tive years within the laboratory exercises that supple-

ment our 5th year undergraduate course on Network

Management Intelligent Networks at the Electrical &

Computer Engineering School of the National Tech-

nical University of Athens (NTUA), Greece (NET-

CSEDU2015-7thInternationalConferenceonComputerSupportedEducation

518

MODE, 2015) The main objectives of the lab exer-

cises were to provide students with hands-on expe-

rience regarding the reservation, interconnection and

usability of dispersed and heterogeneous virtualized

resources.

The students in the role of experimenters were

asked to create slices, design network topologies and

reserve GENI resources at several locations such as

Stanford University and the Georgia Institute of Tech-

nology. Moreover, students were called to explore

the networking capabilities provided by GENI infras-

tructure. For that, they deployed (i) GRE tunnels

to achieve connectivity among resources at differ-

ent locations and (ii) virtual machines running Open

vSwitch acting as Layer 2 software switches connect-

ing the hosts (VMs) in each island.

Overall, 72 users joined the GENI project which

was created for the networking class. Amongst them,

68 undergraduate students in 49 groups (comprised

of one or two students) were asked to create one slice

per group and run their own experiment following the

exercise instructions (Figure 3). For the completion

of the exercise assignment, 343 VMs were spawned,

98 local networks were created at different sites host-

ing GENI resources and 49 multi-domain networks

were formed. As a result, a signiﬁcant number of

simultaneous users deployed virtual slices across the

Atlantic, pushing GENI capabilities to higher levels

and indicating feasibility of sharing educational labs

across the globe.

Experimenters (students) were provided with the

GENI Experimenter’s guide along with step-by-step

instructions for slice creation via GENI-providedGUI

tools. Subsequently, they were called to document

their experience by ﬁlling in a questionnaire on the

following items:

1. Overall GENI service experience: 87% of the

students noted a good understanding the GENI

service functionality and 92% declared that they

were satisﬁed by it.

2. Resource creation/deletion: Approximately 74%

were satisﬁed with the reservation procedure.

This was highly dependent on students choice of

site where the resources were created. Also al-

though 83% of the students encountered difﬁcul-

ties the ﬁrst time they tried to setup their experi-

ment, 70% of them where able to overcome their

problem without help from the instructors and

start over.

3. Resources availability and accessibility: 35% had

difﬁculties in the process of resource reservation.

This might be partially attributed to students over-

subscribing for popular GENI resources.

It is worth noting that a staggeringly high percent-

age of students, 97.5%, declared that the integration

of remote labs (not just GENI) in education could ad-

vance signiﬁcantly the quality of their studies.

Moreover the instructors of the class found the

role-based membership of GENI projects very valu-

able. By assigning administrative privileges to the lab

instructors, it enabled remote assistance and debug-

ging from anywhere and anytime in response to stu-

dents inquiries.

4 BENEFITS IN A NUTSHELL

Leveraging public research testbeds, such as GENI,

in education yields multifold advantages for both stu-

dents and educators. Students have the opportunity

to work in a collaborative environment and join an

active worldwide community engaged in shaping the

Future Internet era. Besides the exercise assignments,

students experience in practice how dispersed and

heterogeneous compute resources can be remotely

reserved, controlled and orchestrated through open

source tools towards establishing on-demand net-

working experiments. In fact, in our study at NTUA,

92% of the students declared their eagerness for con-

tinuing using GENI for research purposes in the fu-

ture.

Educators using large scale research infrastruc-

ture with unique, otherwise unaffordable, resources

and user-friendly tools are able to provide students in

advanced networking courses with experiences they

would not otherwise have. The geographically dis-

persed remote labs provide also the opportunity to

surpass the geographical barriers by reproducing ex-

periments in different locations and offering to the

students a better understanding of how Internet works

in reality. For instance, students at NTUA used

testbeds to learn how an inter-autonomous system

routing policy can enforce global internet trafﬁc.

5 CONCLUSIONS

Technological advances in information technology

have radically changed not only our everyday lives

but have also transformed education and learning pos-

sibilities. They have enhanced the old-school hands-

on laboratories where students and infrastructure need

to be collocated, with simulated and remote labs (Ma

and Nickerson, 2006; M¨uller and Erbe, 2007).

Recently several publicly availbale testbeds for

computer science research have been built and made

available to the research community. In this paper we

VirtualLaboratories-UseofPublicTestbedsinEducation

519

Figure 3: Lab Experiment using GENI Infrastructure at NTUA.

argue that these testbeds should be adopted by educa-

tors as the platform of choice for developing course

modules for computer science cources. There are

great beneﬁts to the educators and the students not the

least of which are: no need to buy, build and maintain

infrastructure, access to unique resources, being part

of a community that actively develops and enhances

new modules.

Education is moving towards a model of continu-

ous, online learning where the norm will be for stu-

dents to be remote. We believe that we are going to

see an increase in use of remote and virtualized labs.

These large scale public testbeds are not only suitable

as remote labs for traditional classes but are also ideal

for online courses and as a platform for Massive Open

Online Courses (MOOC).

REFERENCES

American Chemical Society (2014). Importance of

hands-on laboratory science. http://www.acs.org/

content/acs/en/policy/publicpolicies/invest/computer

simulations.html. Public Policy Statement 2014-2017.

Berman, M., Chase, J. S., Landweber, L., Nakao, A., Ott,

M., Raychaudhuri, D., Ricci, R., and Seskar, I. (2014).

GENI: A federated testbed for innovative network ex-

periments. Computer Networks, 61(0):5 – 23. Special

issue on Future Internet Testbeds Part I.

Bruner, J. S. (1961). The act of discovery. Harvard Educa-

tional Review, 31(1):21–32.

Corter, J. E., Nickerson, J. V., Esche, S. K., Chassapis, C.,

Im, S., and Ma, J. (2007). Constructing reality: A

study of remote, hands-on, and simulated laboratories.

ACM Trans. Comput.-Hum. Interact., 14(2).

Fdida, S., Wilander, J., Friedman, T., Gavras, A., Navarro,

L., Boniface, M., MacKeith, S., Av´essta, S., and Potts,

M. (2011). FIRE Roadmap Report 1 Part II, Future

Internet Research and Experimentation (FIRE).

Freedman, M. P. (1997). Relationship among laboratory

instruction, attitude toward science, and achievement

in science knowledge. Journal of Research in Science

Teaching, 34(4):343–357.

GENI Project Ofﬁce (2014). Resources for instruc-

tors using GENI. http://groups.geni.net/geni/wiki/

GENIEducation.

Grifﬁoen, J., Fei, Z., Nasir, H., Wu, X., Reed, J., and Car-

penter, C. (2013). GENI-enabled programming exper-

iments for networking classes. In Proceedings of the

2013 Second GENI Research and Educational Exper-

iment Workshop, GREE ’13, pages 111–118, Wash-

ington, DC, USA. IEEE Computer Society.

IBM News Release (2007). Google and IBM announce uni-

versity initiative. http://www-03.ibm.com/press/us/

en/pressrelease/22414.wss.

InCommon (2014). Incommon. http://www.incommon.org.

Koukis, E. and Louridas, P. (2013). okeanos iaas. In EGI

Community Forum 2012 / EMI Second Technical Con-

ference, Proceedings of Science, Munich, Germany.

Ma, J. and Nickerson, J. V. (2006). Hands-on, simulated,

and remote laboratories: A comparative literature re-

view. ACM Computing Surveys, 38(3).

Magin, D. J., Churches, A. E., and Reizes, J. A. (1986). De-

sign and experimentation in undergraduate mechani-

cal engineering. In Proceedings of a Conference on

Teaching Engineering Designers.

Marasevic, J., Janak, J., Schulzrinne, H., and Zussman, G.

(2013). Wimax in the classroom: Designing a cellular

networking hands-on lab. In Proceedings of the 2013

Second GENI Research and Educational Experiment

Workshop, GREE ’13, pages 104–110, Washington,

DC, USA. IEEE Computer Society.

CSEDU2015-7thInternationalConferenceonComputerSupportedEducation

520

Mirkovic, J., Benzel, T., Faber, T., Braden, R., Wroclawski,

J., and Schwab, S. (2010). The deter project: Advanc-

ing the science of cyber security experimentation and

test. In Technologies for Homeland Security (HST),

2010 IEEE International Conference on, pages 1–7.

Morgan, R. L., Cantor, S., Carmody, S., Hoehn, W.,

and Klingenstein, K. (2004). Federated Security:

The Shibboleth Approach. EDUCAUSE Quarterly,

27(4):12–17.

M¨uller, D. and Erbe, H. (2007). Advances on Remote

Laboratories and E-Learning Experiences, chapter 2,

pages 35–59. Number 6 in Engineering. Deusto Pub-

licaciones.

NETMODE (2015). Courses offered by the netmode ntua

lab. http://www.netmode.ntua.gr/main/index.php?op-

tion=com

content& view=article&id=22&Itemid=36.

Obstfeld, J., Knight, S., Kern, E., Wang, Q. S., Bryan, T.,

and Bourque, D. (2014). VIRL: The virtual internet

routing lab. In Proceedings of the SIGCOMM 2014

Conference.

Peterson, L., Anderson, T., Culler, D., and Roscoe, T.

(2003). A blueprint for introducing disruptive technol-

ogy into the internet. SIGCOMM Comput. Commun.

Rev., 33(1):59–64.

PlanetLab (2009). Planetlab courseware. https://

www.planet-lab.org/courseware.

Raychaudhuri, D., Seskar, I., Ott, M., Ganu, S., Ramachan-

dran, K., Kremo, H., Siracusa, R., Liu, H., and Singh,

M. (2005). Overview of the ORBIT radio grid testbed

for evaluation of next-generation wireless network

protocols. In Wireless Communications and Network-

ing Conference, 2005 IEEE, volume 3, pages 1664–

1669 Vol. 3.

Su, M., Bergesio, L., Woesner, H., Rothe, T., Kpsel, A.,

Colle, D., Puype, B., Simeonidou, D., Nejabati, R.,

Channegowda, M., Kind, M., Dietz, T., Autenrieth,

A., Kotronis, V., Salvadori, E., Salsano, S., Krner, M.,

and Sharma, S. (2014). Design and implementation

of the OFELIA FP7 facility: The European OpenFlow

testbed. Computer Networks, 61(0):132 – 150. Special

issue on Future Internet Testbeds Part I.

Vandenberghe, W., Vermeulen, B., Demeester, P., Willner,

A., Papavassiliou, S., Gavras, A., Sioutis, M., Quereil-

hac, A., Al-Hazmi, Y., Lobillo, F., Schreiner, F., Ve-

layos, C., Vico-Oton, A., Androulidakis, G., Papa-

gianni, C., Ntofon, O., and Boniface, M. (2013). Ar-

chitecture for the heterogeneous federation of future

internet experimentation facilities. In Future Network

and Mobile Summit (FutureNetworkSummit), pages

1–11. IEEE.

White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad,

S., Newbold, M., Hibler, M., Barb, C., and Joglekar,

A. (2002). An integrated experimental environment

for distributed systems and networks. SIGOPS Oper.

Syst. Rev., 36(SI):255–270.

Wong, G. (2012). ProtoGENI and undergraduate courses.

http://dept.cs.williams.edu/∼jeannie/nsf-workshop/

slides/gary.pdf.

VirtualLaboratories-UseofPublicTestbedsinEducation

521