GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud

Computing Environments

avio Carvalho, Eduardo Roloff and Philippe O. A. Navaux

Informatics Institute, Federal University of Rio Grande do Sul, Brazil

Keywords:

Edge Computing, Fog Computing, Cloud Computing, Internet of Things, Distributed Computing.

Abstract:

The combination of Edge Computing devices and Cloud Computing resources brings the best of both worlds:

Data aggregation closer to the source and scalable resources to grow the network on demand. However, the

ability to leverage each time more powerful edge nodes to decentralize data processing and aggregation is

still a signiﬁcant challenge for both industry and academia. In this work, we extend the Garua platform to

analyze the impact of a model for data aggregation in a global scale smart grid application dataset. The

platform is extended to support global data aggregators that are placed nearly to the Edge nodes where data

is being collected. This way, it is possible to aggregate data not only at the edge of the network but also pre-

process data at nearby geographic areas, before sending data to be aggregated globally by global centralization

nodes. The results of this work show that the implemented testbed application, through the usage of edge node

aggregation, data aggregators geographically distributed and messaging windows, can achieve collection rates

above 400 million measurements per second.

1 INTRODUCTION

The ubiquity of the Internet of Things (IoT) unleashes

the potential of using innovations based on sensor

data to improve society’s overall quality of life. These

data proﬁles can be used to enable technologies such

as large-scale smart cities deployments, smart home

monitoring, smart industries and smart energy grids.

Devices that provide these kinds of capabilities are

now widespread through our cities and homes on de-

vices such as smartphones, medical devices, home ap-

pliances and street signals.

By 2025, researchers estimate that the IoT will

have a potential economic impact of $11 trillion per

year – which would be equivalent to about 11% of the

world economy. They also expect that one trillion IoT

devices will be deployed by 2025 (Buyya and Dast-

jerdi, 2016).

Novel technologies for mobile computing and

the Internet of Things (IoT) are shifting the focus of

its research and development to computing toward

dispersion. In this context, Edge Computing is one of

the most prominent areas, and it is a new paradigm

in which a high volume of computing and storage

resources – generally referred to as cloudlets, micro

datacenters or fog nodes – are placed at the Inter-

net’s edge close to mobile devices or sensors (Satya-

narayanan, 2017).

However, there is a large set of applications that

cannot tolerate the latency penalties of sending data

to be aggregated on the cloud and then waiting for

the response on edge nodes. Also, the act of send-

ing a potentially large number of small packets to the

cloud for data processing can saturate the network and

decrease the scalability of applications (Dastjerdi and

Buyya, 2016).

Smart grids will allow consumers to receive near

real-time feedback about household energy consump-

tion and price, allowing consumers to make informed

decisions about their spending. For energy produc-

ers, it will be possible to leverage home consumption

data to produce energy forecasts, enabling near real-

time actions and better scheduling of energy gener-

ation and distribution (Brown, 2008). In this way,

smart grids will save billions of dollars in the long

run, for consumers and the generators, as well as to

reduce the impact on the environment, according to

recent forecasts (Reuters, 2011).

This paper uses a realistic application use case and

dataset in order to further understand what is achiev-

able when certain computational tasks are moved

from cloud nodes to edge nodes, and the potential

Carvalho, O., Roloff, E. and Navaux, P.

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments.

DOI: 10.5220/0007734604370447

In Proceedings of the 9th International Conference on Cloud Computing and Services Science (CLOSER 2019), pages 437-447

ISBN: 978-989-758-365-0

437

beneﬁts of data aggregation and batching at the

edge of the network.

In order to improve over our previous work a new

communication layer was introduced, which is called

the aggregator layer. This layer has the primary goal

of pre-processing data in the same geographic loca-

tion of edge nodes. This way, it is possible to avoid

high-cost communication delays by pre-processing

data from multiple edge nodes in the same geographic

region, postponing the latency overhead of communi-

cating with potentially distant cloud nodes. Finally,

the scalability of the architecture is evaluated by de-

ploying the platform in a large scale deployment in a

public cloud environment.

Our results show, after including the aggregator

layer, the platform can achieve data collection rates

above 400 million measurements per seconds, using

15 geographic distributed datacenters on Microsoft

Azure platform, for a total of 1366 machines across

the globe. It also presents an eight times speedup im-

provement in throughput in comparison to the previ-

ous architecture, in a scenario using eight aggregators

in the same geographic region.

The rest of this paper is organized as follows. The

Section 2 presents and compares the related work with

our work. In Section 3 our mechanism is presented

and explained. Section 4 presents the results of our

experimental evaluation. Finally, the Section 5 con-

cludes this work and presents the next research direc-

tions.

2 RELATED WORK

The main works on the state-of-the-art of edge com-

puting focus on platforms and frameworks aiming to

provide scalable processing as close as possible to

the network border. These approaches are primarily

focused on providing the lowest possible latency re-

sults and better utilize the resources available on the

network. Multi-access Edge Computing (MECs) and

Cloudlets, which can be described as cloud-like de-

ployments at the network edge, are currently the pre-

dominant approaches to address these challenges.

The most prominent approaches and how they re-

late to this work are brieﬂy described below. At the

end of Section 2 there are detailed explanations of

how these works relate to the proposed solution on

the Section 3.

The state-of-the-art works include Femto-

Clouds (Habak et al., 2015), REPLISOM (Abdelwa-

hab et al., 2016), Cumulus (Gedawy et al., 2016),

CloudAware (Orsini et al., 2016), ParaDrop (Liu

et al., 2016), HomeCloud (Pan et al., 2016),

ENORM (Wang et al., 2017), RT-SANE (Singh

et al., 2017), EdgeIoT (Sun and Ansari, 2016) and

cloud provider based implementations (T

arneberg

et al., 2016), which are examples of applications that

explore computational ofﬂoading to nearby devices.

One important thing to perceive is that the ma-

jority of the related works either rely on ofﬂoading

computation to edges that are underutilized or ofﬂoad

processing to nearby network centralizers (modiﬁed

wireless network access points or specialized mobile

network base station hardware). EdgeIoT (Sun and

Ansari, 2016) stands apart on this aspect by explor-

ing computation to considerably more performance

nodes, by relying on virtual machines in a nearby mo-

bile base station.

Several works on the state-of-the-art either rely on

hardware speciﬁc tools or signiﬁcant modiﬁcations

on their underlining communication protocols. This

work, on the other hand, relies on standard tools and

protocols that add ﬂexibility and turn easier to port it

to other platforms.

Examples of applications which require signiﬁ-

cant changes in the underlining communication pro-

tocols or speciﬁc hardware are ParaDrop, a speciﬁc

edge computing framework implemented on WiFi

Access Points (APs) or other wireless gateways (such

as set-top boxes). It uses a lightweight virtualization

framework through which third-party developers can

create, deploy, and revoke their services in different

APs.

HomeCloud (Pan et al., 2016) focus on being an

open and efﬁcient new application delivery in edge

cloud, by integrating two complementary technolo-

gies: Network Function Virtualization (NFV) and

Software-Deﬁned Networking (SDN).

EdgeIoT, in its turn, is an architecture to handle

data stream at the mobile edge. The central idea con-

sists of fog nodes communicating with a Virtual Ma-

chine (VM) positioned at a nearby base station. On

the top of the fog nodes, the Software Deﬁned Net-

working (SDN) based cellular core is designed to fa-

cilitate the package forwarding among fog nodes.

Cumulus (Gedawy et al., 2016), Femto-

Clouds (Habak et al., 2015), CloudAware (Orsini

et al., 2016) and RT-SANE (Singh et al., 2017)

have a greater focus on task placement, providing

dynamic scheduling capabilities for operators and

tasks (such as tasks migration functionalities) which

are beyond the scope of this work. Cumulus (Gedawy

et al., 2016) provides a complete framework that

controls task distribution under a heterogeneous set

of devices on its cloudlet. FemtoClouds (Habak et al.,

2015) and CloudAware (Orsini et al., 2016) monitor

device usage on its cloudlets in order to improve

CLOSER 2019 - 9th International Conference on Cloud Computing and Services Science

438

Table 1: Research scope comparison of the state-of-the-art with the proposed work.

Name Cloud Edge Mobility Large Scale Hardware Agnostic

GaruaGeo (this work) • • • •

ENORM • • •

RT-SANE • • •

Tarneberg et al. • • •

HomeCloud • • •

CloudAware • • •

FemtoClouds • •

REPLISOM • •

Cumulus • • •

ParaDrop • •

EdgeIoT • •

device usage as part of its scheduling algorithms.

CloudAware (Orsini et al., 2016) also provides a

speciﬁc Application Programming Interface (API) to

improve the user experience of software developers.

RT-SANE (Singh et al., 2017) evaluates several

scheduling heuristics in comparison to a cloud-only

scenario.

Although the related works present multiple initia-

tives on Edge Computing towards computational of-

ﬂoading, only the work of T

arneberg et al. (T

arneberg

et al., 2016), ENORM (Wang et al., 2017) and RT-

SANE (Singh et al., 2017) explore the potential of

combining low latency edge nodes processing with

scalable and more powerful sets of commodity ma-

chines on public clouds. However, ENORM (Wang

et al., 2017) evaluation only considers a small set of

edge nodes communicating with a nearby Amazon

AWS cloud node in Dublin. RT-SANE (Singh et al.,

2017), in its turn, relies on a speciﬁc fog simulator to

obtain its results, which limits the scope in compar-

ison to a real-world evaluation. Finally, the work of

arneberg et al. (T

arneberg et al., 2016) directly of-

ﬂoad from the edges devices to the cloud infrastruc-

ture, which limits the scope of the work in relation to

data aggregation.

Apart from that, most works presented in this sec-

tion either rely on generated datasets or small datasets

(tenths of mobile devices) for its evaluations. On the

other hand, the present work uses a realistic dataset,

based on a real-world dataset from household energy

consumption in Germany (Ziekow and Jerzak, 2014).

In Table 1, we present a comprehensive descrip-

tion of the coverage of the state-of-the-art in compar-

ison with this work.

According to characterization, it is possible to per-

ceive that the proposed model (GaruaGeo) is the only

work that focuses on large scale hybrid (cloud and

edge) data processing.

This work also does not focuses on mobility issues

or hardware speciﬁc implementations such as several

works on the state-of-the-art.

The taxonomy used to group the state-of-the-art

works is based on recent surveys on fourth-generation

distributed stream processing systems, fog computing

and edge computing (de Assuncao et al., 2017) (At-

zori et al., 2010) (Mao et al., 2017) (Mahmud et al.,

2018).

3 ARCHITECTURE AND

IMPLEMENTATION

The architectural infrastructure of the testbed appli-

cation deployment can be described as a composi-

tion of four layers, as it is represented on Figure 1:

(1) Cloud layer, which executes long-running scal-

able jobs that can provide more powerful machines

for processing with the trade-off of greater latencies;

(2) Aggregator layer, which aggregates from multiple

edge nodes, usually placed on the same geographic

region as edge nodes, in order to aggregate all data

for a given geographic region before sending data to

the cloud layer, which is potentially placed in a distant

geographic region; (3) Edge layer, which is composed

of edge nodes that are used to pre-process and aggre-

gating sensor data before sending to the Aggregator

nodes; and the (4) Sensor layer, which is composed

of the sensors that communicate directly with Edge

layer nodes to receive actuation requests and provide

measurements to the network.

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

439

3.1 Cloud Layer

This layer is composed of virtual machines that exe-

cute the application in order to aggregate data to be

received from Edge layer nodes. The Cloud layer

should be composed of elements that can be able to

process data as it arrives. It can be instantiated by im-

plementing the same application logic from the layer

below, but instead, it should be conﬁgured to receive

data from multiple edge nodes.

This layer receives data from queues and ex-

changes through a message hub, so that the inputs

can be parallelized through multiple consumers. It

can also be conﬁgured to support clusters of machines

to execute transformations as distributed stream pro-

cessing jobs over these queues and exchanges.

In the evaluation work, the model is implemented

as an application running in a single node inside a

Linux VM at Microsoft Azure, which was chosen

due to its perceived beneﬁts over other cloud plat-

forms (Roloff et al., 2012) (Roloff et al., 2017). This

application was written in the Go programming lan-

guage and receives the processing request from the

layer below through GRPC communication frame-

work. The VM instance utilized was conﬁgured as

it is described in Table 2.

3.2 Aggregator Layer

The aggregator layer represents one or multiple in-

termediate layers of aggregation that could be poten-

tially used to mitigate the impact of latency between

data collection and the collection of global metrics

into the Cloud layer.

In previous works, it was possible to observe that

is possible to obtain signiﬁcant throughput gains by

aggregating data from sensors on edge nodes.

However, it was perceived that there was room for

improvement if it was possible to have an operator

in the architecture responsible for aggregate data for

speciﬁc regions before communicating with the cloud

layer. An example of this scenario is when multi-

ple edge nodes in Japan are transferring data to the

USA, each one of them paying the latency penalty of

Table 2: Cloud layer conﬁguration: Virtual machine type

and toolset description.

Parameter Description

Instance Type Basic A3 (4 cores, 7 GB RAM)

Operating System Ubuntu 16.04 LTS

Golang version 1.8

GRPC version 1.3.0-dev

Protocol Buffers version 3.2.0

VM VM VM

Cloud

Layer

Aggregator Aggregator

Aggregator

Layer

Edge Node Edge Node Edge Node

Edge

Layer

Sensor Sensor Sensor Sensor Sensor

Sensor

Layer

Figure 1: The architecture is composed by 4 layers: Cloud,

Aggregator, Edge and Sensor.

communicating with another continent. Instead, ag-

gregators could be placed in this location and aggre-

gate data from multiple edge nodes in Japan, before

transferring it to the USA.

In our evaluation, this layer was represented as a

single core machine, in order to potentially represent

less powerful machines, such as Raspberry Pi’s and

similar ARM processors which are well-suited for the

given scenario (and that were used for the edge layer

on previous works).

3.3 Edge Layer

The Edge layer is composed of a set of nodes with

transformation operators to apply over sensor mea-

surements one-at-time. Operators can be either trans-

formations over results, combinations with sets of

measurements received or mappings to machines on

the Cloud layer above.

On this layer, the application code will be ex-

pressed to deﬁne which computation will be done in-

side of edge nodes and which computation will be

managed by VMs on the Cloud Computing environ-

ment. The degree of control provided by this level

makes it possible to decrease the number of messages

sent to the Cloud. In this way, it is possible to de-

crease the amount of data that is sent to the Cloud.

Table 3: Aggregator layer conﬁguration: Virtual machine

type and toolset description.

Parameter Description

Instance Type Standard DS2 v2 (2 cores, 7 GB RAM)

Operating System Ubuntu 16.04 LTS

Golang version 1.8

GRPC version 1.3.0-dev

Protocol Buffers version 3.2.0

CLOSER 2019 - 9th International Conference on Cloud Computing and Services Science

440

Also, by processing certain amounts of data directly

on the edge nodes, the latency experienced by actu-

ator sensors is in the order of tenths of milliseconds

instead of a couple of seconds of cloud processing la-

tencies.

Actuator sensor logic can also be implemented on

this layer, in such a way that when a given condition

is matched by an edge node, it can trigger actuators

on the Sensor layer in order to act on external appli-

cations. For example, a consumer can conﬁgure its

smart grid energy meter to maintain the energy con-

sumption below a certain level during peak cost en-

ergy hours. In this way, the smart grid meter can turn

off certain machines when the average consumption

reaches a certain threshold.

In our previous works, edge layers were composed

by a set of Raspberry Pi Zero W machines. In this pa-

per, in order to stress the simulation in a globally dis-

tributed environment, these machines are simulated as

VMs on Microsoft Azure. These machines commu-

nicate with the sensor layer, which in our evaluation

will also be a simulated layer. The conﬁguration of

the edge nodes is the same as we have used for the

aggregator nodes, which is described in detail on Ta-

ble 4.

3.4 Sensor Layer

The Sensor layer is represented by a given set of sen-

sors that communicate with the Edge nodes. Ide-

ally, sensors should communicate with Edge Nodes

through their available input/output hardware inter-

connections or lightweight wireless connection such

as Bluetooth or LTE networks. However, in order to

limit the analysis scope of this work, the sensor net-

work dataset is previously loaded into edge nodes be-

fore the execution of tests.

Smart grid environments rely on speciﬁc meters

and plugs on households to collect data, which are

provided by the energy grid provider or standardized

to support only a set of accepted and veriﬁed plugs

and meters types. The data types generated by these

environments also need to respect a certain schema

to be shared, aggregated and analyzed by the energy

provider companies.

Table 4: Edge layer conﬁguration: Virtual machine type and

toolset description.

Parameter Description

Instance Type Standard DS1 v2 (1 cores, 3.5 GB RAM)

Operating System Ubuntu 16.04 LTS

Golang version 1.8

GRPC version 1.3.0-dev

Protocol Buffers version 3.2.0

The dataset used for the evaluation of this work

is based on the dataset provided by the 8th ACM

International Conference on Distributed Event-Based

Systems (DEBS). This conference provides competi-

tions with problems which are relevant to the indus-

try. In the year 2014, the conference challenge fo-

cus was on the ability of Complex Event Processing

(CEP) systems to apply on real-time predictions over

a signiﬁcant amount of sensor data. For this purpose,

household energy consumption measurements where

generated, based on simulations driven by real-world

energy consumption proﬁles, originating from smart

plugs deployed in households (Ziekow and Jerzak,

2014). For this challenge, a large number of smart

plugs has been deployed in households with data be-

ing collected roughly every second for each sensor in

each smart plug.

3.5 Communication Protocol

Although multiple protocols for communication in

IoT systems have been proposed in the recent years,

the protocols in use today are still being evaluated

and are subject of discussion and standardization ini-

tiatives, mainly due to advancements of internet pro-

tocols to support mobile and IoT applications. The

most widely adopted protocols in use today are, re-

spectively, MQTT (Banks and Gupta, 2014) and

CoAP (Bormann et al., 2012).

One of the most prominent proposals on this

area is the HTTP/2 protocol. The standard was ﬁn-

ished in 2005 and provides several improvements

over previous protocols, mainly due to the capability

of multiplexing data, avoiding handshake overhead

and their data compression capabilities (Belshe et al.,

2015) (Ruellan and Peon, 2015).

As an alternative to broker-centric communi-

cation protocols and synchronization costly proto-

cols such as REST (Richardson and Ruby, 2008),

Google Inc. has adopted a RPC protocol and ser-

vice discovery framework Stubby/Chubby (Burrows,

2006). The open source version of its tool is called

GRPC (Google, 2015), which relies on HTTP/2 in

order to avoid handshake overhead, and Protocol

Buffers (Gligori

c et al., 2011) to communicate us-

ing a binary method, which provides better data com-

paction by reducing the message size.

Due to the performance beneﬁts reported from the

usage of HTTP/2 protocols over standard HTTP, and

their ease of use for ﬂexible prototyping of distributed

applications, GRPC was used to build a reliable and

fast communication channel for all of the communi-

cation layers implemented on this work.

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

441

GRPC as a communication platform presents sev-

eral advantages over TCP only connections and com-

munication protocols such as REST. It has a simple

interface which hides conﬁguration complexity but is

still able to provide high-level features, such as long-

lived connections (to avoid unnecessary communica-

tion handshakes) and communication multiplexing in-

side a small number of channels (reducing the num-

ber of required connections open at a given point in

time). However, their usage is still subject of eval-

uation, mainly on networks with high package loss

percentages, which are a limiting factor not only for

HTTP/2 but also for AMQP based applications (Goel

et al., 2016) (Lee et al., 2013) (Chowdhury et al.,

2015) (Thangavel et al., 2014).

3.6 Measurement Algorithm

In order to proper extract insights from sensor data,

it was selected a broadly used approach to aggregate

and generate Short-Term Load Forecasting (STLF)

for smart grids. This algorithm is not only well-

known by the research community, but also provides

the potential to be applied at multiple aggregation lay-

ers.

Smart grids bring a rich set of tools to better con-

trol and balance energy supply and demand in near

real-time, as well as continuous visibility of consump-

tion patterns about energy generation and consump-

tion. This way, methods to extract knowledge from

near real-time observations are critical to the extrac-

tion of value from infrastructure investment.

In this scenario, Short-Term Load Forecasting

(STLF) describes the prediction of power consump-

tion levels in several time frames, from minutes and

hours up to a week ahead. It considers variables such

as date, temperature (including weather forecasts),

humidity, temperature-humidity index, wind-chill in-

dex and most importantly, historical load. Residential

versus commercial or industrial uses are rarely speci-

ﬁed.

Multiple approaches for time series modeling

for STLF have been developed over the last thirty

years. There methods (Kyriakides and Polycarpou,

2007) could be brieﬂy summarized into the following

groups:

• Regression models that represent electricity load

as a linear combination of multiple parameters

e.g. weather factors, day type and customer class.

• Methods based on linear time series includ-

ing those derived from Autoregressive Integrated

Moving Average (ARIMA) model and State-

Space Models (SSMs).

• Methods derived from SSMs that rely on a

ﬁltering-based (e.g., Kalman ﬁlters) techniques

and a characterization of dynamical systems.

• Nonlinear time series modeling through machine

learning methods such as non-linear regression.

According to the research of Shawkat Ali et

al. (Ali, 2013), the three most accurate models for

load prediction are respectively, Multiplayer Percep-

tron (MLP), Support Vector Machines and Least

Mean Squares. In this work, it is implemented an

approach which mixes MLP and ARIMA, bringing

together characteristics from both linear time series

methods and SSMs (Bylander and Rosen, 1997).

This approach was chosen not only due to the model

suitability to distributed architectures, but also due to

its similarity to the model proposed at the DEBS 2014

conference (Ziekow and Jerzak, 2014). This approach

is schematically described in Equation (1).

More speciﬁcally, the set of queries provide a

forecast of the load for: (1) each house, i.e., house-

based and (2) for each individual plug, i.e., plug-

based. The forecast for each house and plug is made

based on the current load of the connected plugs and

a plug speciﬁc prediction model.

L(s

i+2

) =

avgL(s

) + median(avgL(s

))

(1)

In the Equation (1), avgL(s

) represents the cur-

rent average load for the slice s

. The value of

avgL(s

), in case of plug-based prediction, is calcu-

lated as the average of all load values reported by

the given plug with timestamps ∈ s

. In case of a

house-based prediction the avgL(s

) is calculated as a

sum of average values for each plug within the house.

avgL(s

) is a set of average load value for all slices s

= s

i+2−n∗k

(2)

In the Equation (2), k is the number of slices in

a 24 hour period and n is a natural number with val-

ues between 1 and f loor(

i+2

). The value of avgL(s

)

is calculated analogously to avgL(s

) in case of plug-

based and house-based (sum of averages) variants.

4 EVALUATION

The main idea of adding aggregators to the platform

was that, by including a layer near to the edge nodes,

the latency experienced by the edge nodes would de-

crease and it would be possible to decide, on aggre-

gator nodes, when it was necessary to suffer the la-

tency penalties to communicate with potentially dis-

tant cloud nodes.

CLOSER 2019 - 9th International Conference on Cloud Computing and Services Science

442

After adding the new layer, some scenarios would

be required to be validated in order to verify that our

platform was able to provide its expected beneﬁts in

comparison to the previous version without aggrega-

tors.

Aggregators have the ability to receive, buffer,

pre-process and group messages before sending data

to the cloud layer nodes. This additional layer gives

the platform the ability to answer edge layer node

requests faster than the cloud layer nodes, which

could potentially be placed at other countries and

continents, providing slower responses due to the

greater latencies implied in communication between

machines in distant geographical regions.

4.1 Infrastructure Setup and Operators

Placement

Before the execution of the tests, it was essential to

setup a minimal set of tools to help on the automation

of the deployment process in multiple regions around

the globe.

Since we are using Microsoft Azure, it was possi-

ble to rely on ARM (Azure Resource Manager) tem-

plates to describe the infrastructure to setup. By using

ARM templates, it was possible to deﬁne generic tem-

plates to describe the number of nodes, size of virtual

machines, operating system version and default tools

that were important to have pre-installed on all ma-

chines.

The decision of which regions to use in our analy-

sis scenarios came from a previous latency measure-

ment evaluation done on previous works.

In this evaluation, it was analyzed the communi-

cation between machines placed in multiple regions

and one node placed on US West, which in this pa-

per was the region where the master node of the cloud

layer was placed. After this analysis was made, these

regions were classiﬁed regions into three categories:

baseline aggregator

Execution type

Throughput (QPS)

1 10 100 1000

Figure 2: Aggregator stress comparison analysis: 90 nodes

maximum throughput with and without an aggregation layer

(1 to 1000 message batch sizes).

Table 5: Region proﬁles: Description of the latency proﬁles

between Azure regions selected for evaluation

Region Low latency Medium latency High latency

westus2 •

brazilsouth •

ukwest •

southeastasia •

eastasia •

japaneast •

westeurope •

australiasoutheast •

northeurope •

centralus •

southindia •

canadacentral •

centralindia •

koreasouth •

francecentral •

Low latency, medium latency, and high latency. The

results with regions selected and their latency proﬁles

are displayed on Table 5.

4.2 Exploring the Impact of Adding

Aggregators into the Infrastructure

In this testbed evaluation, aggregators nodes are

placed in the same regions (datacenters) as the sim-

ulated edge nodes. Using this approach messages are

buffered into a nearby aggregator, where they can be

aggregated or pre-processed and sent to a centralized

node which is potentially in a distant region.

The ﬁrst experiment we have made was to evalu-

ate the impact of adding another moving piece to our

infrastructure. The rational behind it was to evalu-

ate how much it would impact the performance in a

stress scenario. In Figure 2 we evaluate a stress sce-

nario with a single master, before and after adding an

aggregator.

From this experiment, it was possible to perceive

that the performance improves for smaller groups of

messages, but in general, including an aggregator

does not impact the performance negatively in com-

parison to the previous architecture. In comparison

to the previous architecture, it is a worst-case eval-

uation scenario, since with an aggregator there is an

extra layer of communication overhead. Apart from

that, in this scenario the aggregator could not beneﬁt

from batching of multiple messages since batching,

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

443

1 2 4 8

50,000

1 · 10

Aggregators (1 to 8)

Throughput (QPS)

Figure 4: Aggregator groups evaluation: 40 edge nodes dis-

tributed between distinct groups of aggregators.

in this case, is done at edge node level. The potential

performance beneﬁts of using one or multiple aggre-

gators should be perceived only in the communica-

tion between aggregator and master nodes (where it

is possible to wait or batch multiple edge node mes-

sages before communicating with nodes on the cloud

layer).

4.3 Multiple Aggregators in a Given

Global Region

In our previous version of the architecture, which was

composed by a master node on the cloud layer and

edge nodes that received sensor data, it was possible

to validate that it the architecture was able to scale lin-

early up to 90 nodes (regarding throughput). In this

experiment, after the addition of our new processing

layer, it was collected data to understand if this be-

havior has not changed.

As it can be seen on Figure 3, regarding batches

of messages, it is still possible to perceive that the

architecture can scale linearly. However, regarding

the number of nodes, it is not possible to perceive

any performance gains when we increase the number

of nodes from 15 to 90. This behavior could be ex-

plained by the fact that our aggregators are now in the

same geographic region as our edge nodes.

In our previous analysis (which considered cloud

nodes and edges nodes only), cloud nodes were

placed in a region and edge nodes were placed in an-

other. In this analysis, from the standpoint of the mas-

ter node a stress scenario was never achieved (the up-

per bound of the system regarding throughput). How-

ever, in the current scenario, the only piece of the

architecture which is not in the same region is the

master node. Hence, the latency experienced by our

edge nodes is much lower, and the upper bound of the

system is experienced by all scenarios from 15 to 90

nodes.

This analysis has shown three crucial facts: 1) The

throughput from the standpoint of the edge nodes has

signiﬁcantly improved since aggregator nodes now

handle requests in the same geographic region with

lower latencies; 2) By adding aggregator nodes in the

same geographic regions as the edge nodes which col-

lect sensor data, we are now bounded by the com-

munication between the aggregator nodes and cloud

nodes (which will always be slower than the commu-

nication between edge nodes and aggregator nodes);

3) In order to evaluate the overall performance (re-

garding throughput and latency) of the system, it is

not possible anymore to aggregate the number of re-

quests made from edge nodes to their counterparts

(cloud nodes in the previous architecture or aggrega-

tor nodes in the current), but it is required to evaluate

the performance of the communication between ag-

gregator nodes and the cloud node.

4.4 Groups of Aggregators into a Single

Region

In this experiment, it was evaluated the impact of

adding multiple aggregators into the same local re-

gion. The analysis was made using VMs on the

uswest region of Azure with a ﬁxed number of edge

nodes (40 edge nodes) and a variable number of ag-

gregators (1 to 8 aggregators).

1 10 100 1000

Batches of messages (1 to 1000 messages)

Throughput (QPS)

15 30 45 60

Figure 3: Aggregator stress evaluation: Throughtput analysis from 15 to 90 nodes.

CLOSER 2019 - 9th International Conference on Cloud Computing and Services Science

444

Figure 5: Global scale deployment: One global master (red square) and 15 regions (blue dots, where each region contains 1

aggregator node and 90 edge nodes).

In order to execute this analysis, the number of

existent edge nodes was spread evenly between ag-

gregators. In Figure 4, it was possible to verify that

by adding 8 aggregators it was possible to achieve

exponential increase on throughput up to 8 aggrega-

tors, where the aggregator has achieved a throughput

of around 100k messages per second received from

the edge nodes.

Multiple aggregators perform better probably due

to the number of concurrent requests that one aggre-

gator can answer at the same time. Given a certain

amount of edge nodes, spreading their requests be-

tween multiple aggregator nodes decreases contention

level in each aggregator. This way, they can answer

more requests per second since the number of nodes

for each aggregator is more balanced. However, in-

troducing a more signiﬁcant amount of aggregators

will require that more elements pay the extra latency

needed to communicate with cloud layer nodes, but

these trade-offs were not explored in this scenario.

1000

8.2

8.4

8.6

Groups of regions (5 to 15 regions)

Throughput (QPS)

10 15

Figure 6: Global performance evaluation: 90 nodes per re-

gions, 1000 messages batches and variable number of re-

gions (5 to 15).

4.5 Multiple Region Edge Analysis

In this evaluation scenario, it was explored the maxi-

mum achievable performance from the standpoint of

data collection on edge nodes. In order to evaluate it,

we have deployed a large number of machines, into

multiple geographic regions, and collected data about

the aggregated data collection throughput of its edge

nodes.

Each scenario evaluated on these tests relies on

regions (datacenters) on Microsoft Azure across the

globe. In each region, there were placed 90 Edge

nodes and a single aggregator node. The analysis was

made based on data collected from at least 20 exe-

cutions for 5, 10 and 15 regions. In each execution

scenario, each edge node sends at least 100k energy

measurements to its respective aggregator nodes.

In Figure 5 it is possible to visualize the extension

of the largest execution. For this analysis it was used:

15 regions on Azure; one Global aggregator node on

the cloud layer; 15 aggregator nodes on the aggrega-

tor layer; 1350 Edge nodes, for a grand total of 1366

machines aggregating data across the globe. In this

experiment, it was possible to achieve data collection

rates above 400 million measurements per seconds on

the scenario with 15 aggregator nodes.

Another important aspect captured by this analy-

sis is the discrepancy in performance between distinct

regions. From the standpoint of the Edge nodes, it

was not possible to perceive signiﬁcant performance

discrepancies between regions for each one of the an-

alyzed regions, as it is displayed in Figure 7.

Linear scalability is also obtained as we increase

the batch factor on the number of messages which are

aggregated before being sent to the aggregator nodes.

Hence, it was possible to validate in this scenario that

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

445

1 10 100 1000

Batch sizes per region (1 to 1000)

Throughput (QPS)

brazilsouth japaneast southeastasia ukwest

westus

Figure 7: Global performance evaluation: 90 nodes per region, 5 regions and variable batch sizes (1 to 1000 messages).

the pattern analyzed in the scenario with a single re-

gion in Figure 3 holds signiﬁcant for multiple regions.

Finally, it is shown in Figure 6 a summary of

the highest throughput scenarios observed, which are

those that display the largest amount of messages per

batch evaluated. The results show that the perfor-

mance does not only increases linearly with batch

sizes, but also data collection rates, which increase

linearly as more regions are added globally. From

these tests, up to 15 regions (which use 90 nodes

and one aggregator per region), it was not possible to

achieve an inﬂection point where data collection rates

start to decrease.

5 CONCLUSION AND FUTURE

WORK

In this work, it was analyzed a model for workload

distribution and data aggregation using a large-scale

smart grid application dataset. The main contribution

was the addition of new processing layer, responsi-

ble for aggregating data from the geographic regions

they were placed before communicating with remote

cloud nodes. In summary, the application was able

to achieve higher throughput by leveraging process-

ing on edge nodes, batching techniques and data ag-

gregation to reduce the communication overhead with

distant nodes.

Our results show, after including the aggregator

layer, the platform can achieve data collection rates

above 400 million measurements per seconds. These

results were obtained leveraging 15 geographic dis-

tributed regions on the Microsoft Azure platform, for

a total of 1366 machines around the globe. In com-

parison to the previous architecture, the new archi-

tecture presents an eight times throughput improve-

ment in a scenario using 8 aggregators in the same

geographic region. The platform was able to scale lin-

early according to the number of messages on window

batches, as well as in relation to the number of nodes

and regions added to the platform, up to 90 nodes per

region.

The experiments show that the impact of window-

ing and aggregation on edge nodes is not negligible

and needs further investigation by the research com-

munity. Although it has similarities to data stream

processing research; the topic is still being initially

explored by researchers on the ﬁelds of the Internet

of Things, Fog Computing and Edge Computing.

The future works should focus on the exploration

of other scheduling, windowing and aggregation tech-

niques for edge processing.

Apart from that, a critical line of research would

be to explore how to evolve this testbed application

and its middleware into a generic framework for ap-

plications that need to distribute processing through

edge and cloud nodes. Finally, it would be essential to

explore other communication protocols to understand

their suitability to multiple scenarios based on hybrid

computations on cloud and edge environments.

ACKNOWLEDGMENTS

This research received partial funding from CYTED

for the RICAP Project. It has also received partial

funding from the EU H2020 Programme and from

MCTI/RNP Brazil under the HPC4E project, grant

agreement no. 689772.

Additional funding for this research was pro-

vided by FAPERGS in the context of the GreenCloud

Project.

Microsoft has provided cloud computing in-

stances on Microsoft Azure to the execution of the

experiments.

CLOSER 2019 - 9th International Conference on Cloud Computing and Services Science

446

REFERENCES

Abdelwahab, S. et al. (2016). Replisom: Disciplined tiny

memory replication for massive iot devices in lte edge

cloud. IEEE Internet of Things Journal.

Ali, A. B. M. S. (2013). Smart Grids: Opportunities, De-

velopments, and Trends. Springer.

Atzori, L., Iera, A., and Morabito, G. (2010). The internet of

things: A survey. Computer networks, 54(15):2787–

2805.

Banks, A. and Gupta, R. (2014). MQTT Version 3.1. 1.

OASIS standard.

Belshe, M. et al. (2015). Hypertext transfer protocol version

2 (HTTP/2). Internet Engineering Task Force (IETF)

- RFC-7540.

Bormann, C. et al. (2012). CoAP: An application protocol

for billions of tiny internet nodes. Internet Computing.

Brown, R. E. (2008). Impact of Smart Grid on Distribution

System Design. In Power and Energy Society General

Meeting-Conversion. IEEE.

Burrows, M. (2006). The Chubby lock service for loosely-

coupled distributed systems. In Proceedings of the 7th

symposium on Operating systems design and imple-

mentation. USENIX.

Buyya, R. and Dastjerdi, A. V. (2016). Internet of Things:

Principles and paradigms. Elsevier.

Bylander, T. and Rosen, B. (1997). A Perceptron-like On-

line Algorithm for Tracking the Median. In Neural

Networks, 1997., International Conference on. IEEE.

Chowdhury, S. A. et al. (2015). Is HTTP/2 more energy

efﬁcient than HTTP/1.1 for mobile users? PeerJ

PrePrints.

Dastjerdi, A. V. and Buyya, R. (2016). Fog Computing:

Helping the Internet of Things Realize Its Potential.

Computer.

de Assuncao, M. D., da Silva Veith, A., and Buyya, R.

(2017). Distributed data stream processing and edge

computing: A survey on resource elasticity and future

directions. In Journal of Network and Computer Ap-

plications. Elsevier.

Gedawy, H. et al. (2016). Cumulus: A distributed and ﬂex-

ible computing testbed for edge cloud computational

ofﬂoading. In Cloudiﬁcation of the Internet of Things

(CIoT). IEEE.

Gligori

c, N. et al. (2011). Performance evaluation of com-

pact binary XML representation for constrained de-

vices. In Distributed Computing in Sensor Systems

and Workshops, International Conference on. IEEE.

Goel, U. et al. (2016). HTTP/2 Performance in Cellular

Networks. In ACM MobiCom.

Google (2015). gRPC Motivation and Design Principles.

Habak, K. et al. (2015). Femto clouds: Leveraging mo-

bile devices to provide cloud service at the edge. In

Cloud Computing (CLOUD), IEEE 8th International

Conference on.

Kyriakides, E. and Polycarpou, M. (2007). Short Term

Electric Load Forecasting: A Tutorial. In Trends in

Neural Computation. Springer.

Lee, S. et al. (2013). Correlation analysis of MQTT loss

and delay according to QoS level. In Information Net-

working (ICOIN), International Conference on. IEEE.

Liu, P. et al. (2016). ParaDrop: Enabling Lightweight

Multi-tenancy at the Network’s Extreme Edge. In

Edge Computing (SEC), IEEE/ACM Symposium on.

Mahmud, R., Kotagiri, R., and Buyya, R. (2018). Fog com-

puting: A taxonomy, survey and future directions. In

Internet of everything, pages 103–130. Springer.

Mao, Y., You, C., Zhang, J., Huang, K., and Letaief, K. B.

(2017). A survey on mobile edge computing: The

communication perspective. IEEE Communications

Surveys & Tutorials, 19(4):2322–2358.

Orsini, G. et al. (2016). CloudAware: A Context-Adaptive

Middleware for Mobile Edge and Cloud Computing

Applications. In Foundations and Applications of

Self* Systems, IEEE International Workshops on.

Pan, J. et al. (2016). HomeCloud: An edge cloud framework

and testbed for new application delivery. In Telecom-

munications (ICT), 23rd International Conference on.

IEEE.

Reuters (2011). U.S. Smart Grid to Cost Billions, Save Tril-

lions.

Richardson, L. and Ruby, S. (2008). RESTful web services.

O’Reilly Media, Inc.

Roloff, E. et al. (2012). High Performance Computing in the

Cloud: Deployment, performance and cost efﬁciency.

In 4th International Conference on Cloud Computing

Technology and Science (CloudCom). IEEE.

Roloff, E. et al. (2017). HPC Application Performance and

Cost Efﬁciency in the Cloud. In 25th Euromicro In-

ternational Conference on Parallel, Distributed and

Network-based Processing (PDP).

Ruellan, H. and Peon, R. (2015). HPACK: Header Com-

pression for HTTP/2. Internet Engineering Task Force

(IETF) - RFC-7541.

Satyanarayanan, M. (2017). The Emergence of Edge Com-

puting. Computer.

Singh, A., Auluck, N., Rana, O., Jones, A., and Nepal, S.

(2017). Rt-sane: Real time security aware scheduling

on the network edge. In Proceedings of the10th Inter-

national Conference on Utility and Cloud Computing,

pages 131–140. ACM.

Sun, X. and Ansari, N. (2016). EdgeIoT: Mobile Edge

Computing for the Internet of Things. IEEE Commu-

nications Magazine.

arneberg, W., Chandrasekaran, V., and Humphrey, M.

(2016). Experiences creating a framework for smart

trafﬁc control using aws iot. In Proceedings of the 9th

International Conference on Utility and Cloud Com-

puting, pages 63–69. ACM.

Thangavel, D. et al. (2014). Performance evaluation of

MQTT and CoAP via a common middleware. In 9th

International Conference on Intelligent Sensors, Sen-

sor Networks and Information Processing (ISSNIP).

IEEE.

Wang, N., Varghese, B., Matthaiou, M., and Nikolopoulos,

D. S. (2017). Enorm: A framework for edge node

resource management. IEEE Transactions on Services

Computing.

Ziekow, H. and Jerzak, Z. (2014). The DEBS 2014 Grand

Challenge. In Proceedings of the 8th ACM DEBS Con-

ference, volume 14.

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

447