Implementation and Operation of User Deﬁned Network on IaaS Clouds

using Layer 3 Overlay

Ryo Nakamura, Yuji Sekiya and Hiroshi Esaki

The University of Tokyo, Bunkyoku, Tokyo, Japan

Keywords:

IaaS, Cloud Network, Layer 3 Overlay Network, LISP, VXLAN.

Abstract:

Server virtualization technology achieves ”Infrastructure as a Service (IaaS)” model services. Anyone can use

virtual server resource easily without preparing computer hardware, thus IaaS environments provides multiple

operational beneﬁts to users. Therefore, today various types of users, both of personal users and enterprises

users, deploy virtual resources on IaaS cloud. On the other hand, many enterprise users still have their own

environments and operate them by themselves (i.e, on-premise environments). When they migrate the services

from physical facilities to IaaS clouds, they have two problems. The ﬁrst problem is that services are bound

to IP addresses. The second problem is lack of ﬂexibility in network design and management on IaaS clouds.

To solve these problems, we propose an operation model to migrate on-premise services to IaaS clouds using

LISP and VXLAN. Using our proposed method, users can migrate IP addresses and services among IaaS

clouds. In this paper, we proposed new migration method of IP address among IaaS clouds and implemented

the method using LISP and VXLAN software on Linux systems. Then we evaluate performance of our system

and present a operation on actual IaaS clouds.

1 INTRODUCTION

Many virtual machines (VM) can work on sin-

gle physical machine by virtualization technologies.

Then, there are some service providers that provide

virtual machine environments that are called Infras-

tructure as a Service (IaaS) to users. IaaS providers

provide VM to users, and users pay fee depending on

resources of VM. Today, these IaaS services are used

by many consumers and enterprises. IaaS service of-

fers many operational beneﬁts such as reducing the

costs of facilities and fault tolerance. Hence, enter-

prises tend to adopt IaaS clouds for beneﬁts.

However, migrating existing services that are al-

ready operated in on-premise environment to IaaS

clouds. The ﬁrst problem is ﬂexibility of IP ad-

dresses. Almost services are bound to IP address,

e.g., www and IP address based access control (ACL).

IP addresses that are used for service servers can

be changed easily using the DNS system. However,

changing IP addresses that are embedded into conﬁg-

uration such as ACL requires expensive operational

costs. An IP address of IaaS provider is assigned to a

VM. Thus, if a user want to migrate their services to

IaaS cloud, IP addresses must be changed.

Besides, the layer 2 segment design and its con-

struction on IaaS cloud should have also ﬂexibility.

In on-premise environments, layer 2 network seg-

ments are separated by policies and usages. This

sort of network separations should be possible in IaaS

cloud. Several IaaS clouds and researches provide

functions to design ﬂexible layer 2 network (Amazon,

2012)(Benson et al., 2011)(Cabuk et al., 2007). How-

ever, this ﬂexibility is sometimes restrected by con-

tracts and basically design of IaaS clouds.

The second problem is redundancy. If a user mi-

grate their systems to IaaS cloud, the services are

not so redundant. A service provider of IaaS cloud

may construct their system highly redundant, how-

ever, there are some possibility of failures that de-

ploying services on single IaaS cloud rather than on

multiple IaaS clouds. Thus, the service which is mi-

grated from on-premise environment to IaaS clouds

must be constructed among several IaaS clouds.

In this paper, we present an approach that can iso-

late user networks from underlay IaaS clouds using

layer 3 overlay networks. By using layer 3 overlay,

user can deﬁne networks among several IaaS clouds

with their own IP address preﬁxes. In order to achieve

this isolation, we implemented Locator ID Separa-

tion Protocol and Virtual eXtensible LAN on Linux

system. Finally, we evaluated implementations and

present the operational model of our system.

366

Nakamura R., Sekiya Y. and Esaki H..

Implementation and Operation of User Deﬁned Network on IaaS Clouds using Layer 3 Overlay.

DOI: 10.5220/0004373803660369

In Proceedings of the 3rd International Conference on Cloud Computing and Services Science (CLOSER-2013), pages 366-369

ISBN: 978-989-8565-52-5

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

2 APPROACH

In this section, we describe the requirements to mi-

grate systems from on-premise environments to IaaS

clouds, and the overview of proposed approach.

2.1 Requirements

Requirements to achieve migration from on-premise

environments to IaaS clouds are shown below.

• Migrating IP address preﬁx from on-premise net-

works to IaaS clouds.

• Achieving ﬂexible layer 2 network design on IaaS

cloud.

• Constructing user networks among several IaaS

clouds.

In this research, to realize these requirements, we

introduce layer 3 overlay networks. If using speciﬁc

functions that depend on each IaaS cloud, migration

can be realized. However, on assumption that a user

network is constructed among several IaaS clouds,

the network can not depend on these speciﬁc func-

tions. Then, in our proposed method, layer 3 forms

boundary between user network and IaaS clouds. All

of functions to construct network are provided by

over layer 3. Figure 1 shows the overview of pro-

posed model. A user deﬁned network is isolated from

IaaS clouds through integration of LISP (Locator/ID

Separation Protocol) and VXLAN (Virtual eXtensible

LAN). The layer 3 topology in this system is deﬁned

by LISP that is IP over IP overlay protocol, and the

layer 2 segment is deﬁned by VXLAN that is ethernet

over IP overlay protocol.

LISP (Farinacci et al., 2012) manages two aspects

of the IP address, addressing and locator, in isola-

tion. LISP manages pair of the preﬁx (EID), and

the Routing Locator (RLOC) which accommodates

the EID preﬁx. Then, LISP overlay routing table

is constructed of this pair. By this approach, LISP

achieves multipoint to multipoint layer 3 overlay IP

routing. VXLAN (M.Mahalingam et al., 2011) is eth-

ernet over IP overlay architecture. VXLAN Tunnel

End Point (VTEP), is a gateway between VXLAN

network and non-VXLAN network. VTEPs encap-

sulate broadcast, unknown unicast and multicast eth-

ernet frame in IP multicast, and well-known unicast

in IP unicast. Thus, VTEPs construct overlay for-

warding database like ethernet switching. Therefore,

VXLAN constructs the Forwarding Information Base

for multipoint Ethernet over IP overlay network.

User Defined Network

IaaS Clouds

IP layer

LISP

VXLAN

Virtual

LISP Router

Virtual

LISP Router

Figure 1: Isolation of user deﬁned network and underlay

IaaS clouds using layer 3 overlay networks.

2.2 Design Overview of proposed

Approach

The proposed method achieves to isolate user net-

works from IaaS clouds. By this method, users can

design and construct networks using own address pre-

ﬁxes freely without changes to underlay IaaS clouds,

then users can migrate on-premise systems. LISP is

used to achieve address preﬁx migration. LISP real-

izes network mobility through separation of EID and

RLOC. Moreover, LISP avoids triangular routing be-

tween LISP sites by exchanging overlay routing table,

then, applying a LISP site to a IaaS cloud enables IP

address preﬁx migration and route optimization be-

tween IaaS clouds. Simultaneously, VXLAN is used

to deﬁne layer 2 segments on IaaS clouds. VXLAN

realize multi tenancy without changes to outer net-

works. VXLAN is originally assumed that it is used

into underlay IaaS systems to separate user networks.

However, By VMs directly terminate VXLAN, VM

users can design and construct layer 2 segments freely

without speciﬁc functions provided by IaaS clouds.

Figure 2 shows the overview of constructed sys-

tem. User deﬁned network is constructed among sev-

eral IaaS clouds. Layer 2 segments in each IaaS

cloud are constructed by VXLAN. Moreover, each

layer 2 segment is accommodated by LISP router. An

IP address preﬁx owned by user is associated with

a VXLAN segment. In IaaS clouds, an IP address

of IaaS provider is assigned to a VM. Then, A LISP

router uses IaaS provider’s address that is assigned as

a locator address, and uses the preﬁx that is assigned

to VXLAN interface as EID preﬁx. A LISP router ad-

vertises this preﬁx as EID preﬁx to the LISP overlay

network. Thus, user deﬁned network is constructed

among several IaaS clouds over layer 3 networks.

ImplementationandOperationofUserDefinedNetworkonIaaSCloudsusingLayer3Overlay

367

IaaS Cloud #2IaaS Cloud #1

Internet

on-premises

LISP Router (XTR)

Virtual Machine

VXLAN Segment

VNI 11

VNI 12

VNI 21

VNI 22

PXTR

Map Server

XTR XTR XTR XTR

Figure 2: The overview of proposed system.

2.3 Implementation and Operation

We implemented LISP and VXLAN in Linux user

space, and they can be conﬁgured through Vyatta.

In proposed system, LISP router including VXLAN

function is deployed as a virtual router into IaaS

clouds. Thus, we implemented them as vyatta exten-

sion. Vyatta, is open source routing suite, provides

CLI to conﬁgure many open source software for net-

working. Furthermore, vyatta is provided VM image.

By implementing LISP and VXLAN as vyatta exten-

sion, anyone can use proposed system in cloud envi-

ronments with some networking functions that vyatta

provides. These implementations, that are called lixy

and vxlan-vyatta, are published.

We deployed proposed system into the two actual

IaaS clouds. The ﬁrst IaaS cloud is WIDE Cloud

(WIDE Project, 2012) that is academic IaaS environ-

ment. WIDE Cloud is constructed from over 20 HVs

around Japan, and some HVs are located in foreign

countries. It is controlled by WIDE Cloud Controller

that is developed by WIDE project. All of HVs con-

nect to a large scale layer 2 segment that accommo-

dates VMs using vlan and vpn tunnels. And another

cloud is constructed independently at Keio University

using WIDE Cloud Controller too.

We deployed Vyatta with LISP and VXLAN ex-

tensions to these IaaS clouds. We extend LISP beta

network (Lisp, 2012) to these clouds using proposed

system. The wide-xtr that is located in WIDE back-

bone registers assigned preﬁx to the map server of

beta network periodically. Moreover, two XTRs that

are located in two IaaS clouds, register a part of as-

signed preﬁx and own address as a locator address

a map server that is located in WIDE backbone. In

this way, we separate assigned preﬁx into two small

preﬁxes that are used on two clouds. Moreover,

layer 2 segments in each cloud are constructed from

VXLAN. Thus, we use LISP beta network preﬁx

without any changes to underlay IaaS clouds using

proposed method.

3 PERFORMANCE EVALUATION

In our proposed method, all of trafﬁcs from VMs go

through layer 3 overlay. Layer 3 overlay causes de-

grading bandwidth because MTU is decreased due to

layer 3 encapsulation. Thus, this point is obvious bot-

tleneck of this system. This section shows the perfor-

mance evaluation result of LISP and VXLAN that we

implemented.

3.1 Performance Degradation due to

Fragmentation

First of all, we evaluated implementations to clarify

the performance degradation due to fragmentation.

Therefore, we tested the throughput of implementa-

tions changing message size of test trafﬁc that is UDP

payload length from 50 bytes to 1500 bytes with 50

bytes step on a physical node environment.

Figure 3 shows throughput of LISP and VXLAN

on physical nodes. With the result of LISP test,

when message size 1450 bytes message, throughput

was 952Mbps, and when the message size was 1500

bytes, throughput was 917Mbps. LISP encapsulates

IP packets in LISP header, so that, when message size

is over 1422 bytes, packets are fragmented by LISP

router. With this result, performance degradation due

to fragmentation is about 3.7%. With the result of

VXLAN test, when message size was 1400 bytes,

throughput was 924Mbps, and when message size

was over 1400 bytes, throughput degraded. VXLAN

encapsulates ether frames in VXLAN header, so that,

when message size is over 1408 bytes, packets are

fragmented by VXLAN node. With this result, per-

formance degradation due to fragmentation is about

25%.

3.2 Performance on Virtual

Environment

Implementations are assumed to be used as VM on

IaaS clouds, and so we evaluated performance on a

virtual environment. With the result of above experi-

ments, if the message size is over the threshold, per-

formance is degraded. Therefore, we evaluated the

performance on a virtual environment with the largest

message size that packets were not fragmented.

Figure 4 shows performances of LISP and

VXLAN on a virtual environment using kernel vir-

tual machine (KVM). The x axis means the direction

of test trafﬁc. The performance of LISP implementa-

tion is degraded to less than a half of the performance

of the physical node environment. In addition, the

performance of virtual node to physical node is 25%

CLOSER2013-3rdInternationalConferenceonCloudComputingandServicesScience

368

100

200

300

400

500

600

700

800

900

1000

0 200 400 600 800 1000 1200 1400

throughput [Mbps]

message size [Byte]

Throughput of LISP and VXLAN

LISP

VXLAN

Figure 3: UDP throughput measurement of LISP and

VXLAN implementations.

lower than physical to virtual. When packets come

from edge network of LISP router, router has to look

up LISP map table and encapsulates packets. There-

fore, this result shows that looking up LISP map table

and encapsulation overhead causes of 25% degrada-

tion of performance on a virtual environment. The

performance of VXLAN implementation is also de-

graded to less than a half of the performance of phys-

ical. In addition, virtual to physical case performance

is half of physical to virtual case. When the test trafﬁc

was transmitted from VM, vCPU usage on VM was

full.

200

400

600

800

1000

Physical to Physical Physical to Vritual Virtual to Physical

throughput [Mbps]

Throughput of LISP and VXLAN

LISP

VXLAN

Figure 4: UDP throughput of LISP with 1422 bytes mes-

sage, and UDP throughput of VXLAN with 1408bytes mes-

sage.

3.3 Consideration

With these results, performances of each implemen-

tation are over 900Mbps on a physical environment.

However, on the virtual environment, performances

are degraded to less than a half of the performance of

physical. Thus, using this implementation, enterprise

systems without large scale services like ofﬁce branch

systems can be migrated to IaaS cloud. However, per-

formancedegradation in a virtual environmentcan not

be ignored.

4 CONCLUSIONS

Also we deployed them in our actual IaaS cloud and

evaluated the implementations.The evaluation results

show that the implementations have some perfor-

mance degradation of network communications, how-

ever, it is acceptable degree of degradation for small

enterprise systems. Although current performance

of our systems is not feasible to accommodate large

scale systems that require high performance. More-

over, ﬂexible layer 2 design in IaaS clouds is achieved

by VXLAN. VXLAN requires IP multicast to under-

lay IaaS system and IP multicast does not work glob-

ally today. As a result, migratable layer 2 domain is

limited in a IaaS cloud. Therefore, our future works

are improving packet transfer performance on vir-

tual environments, and design a more ﬂexible method

for constructing layer 2 network among several IaaS

clouds.

REFERENCES

Amazon (2012). Amazon EC2. http://aws.amazon.com/

ec2.

Benson, T., Akella, A., Shaikh, A., and Sahu, S. (2011).

CloudNaaS: a cloud networking platform for enter-

prise applications. In Proceedings of the 2nd ACM

Symposium on Cloud Computing, SOCC ’11, pages

8:1–8:13, New York, NY, USA. ACM.

Cabuk, S., Dalton, C. I., Ramasamy, H., and Schunter, M.

(2007). Towards automated provisioning of secure

virtualized networks. In Proceedings of the 14th ACM

conference on Computer and communications secu-

rity, CCS ’07, pages 235–245, New York, NY, USA.

ACM.

Farinacci, D., Fuller, V., Meyer, D., and Lewis, D. (2012).

Locator/ID separation protocol (LISP) draft-ietf-lisp-

23. Internet Draft, IETF Network Working Group.

Lisp (2012). LISP beta network. http://www.lisp4.net/beta-

network.

M.Mahalingam, D.Dutt, K.Duda, P.Agarwal, Kreeger, L.,

Sridhar, T., M.Bursell, and C.Wright (2011). draft-

mahalingam-dutt-dcops-vxlan-00.txt. Internet Draft,

IETF.

WIDE Project (2012). WIDE Cloud. https://

wcc.wide.ad.jp/.

ImplementationandOperationofUserDefinedNetworkonIaaSCloudsusingLayer3Overlay

369