Orchestrating the Cognitive Internet of Things

Chung-Sheng Li

, Frederica Darema

, Verena Kantere

and Victor Chang

IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, U.S.A.

Air Force Office of Scientific Research, Arlington, VA, U.S.A.

Institute of Services Science, University of Geneva, Switzerland

School of Computing, Creative Technologies and Engineering, Leeds Beckett University, Leeds, U.K.

Keywords: Internet of Things, CIoT, Smarter Planet, Behavior Models, Cognitive Computing, World Models, Smart

Grid, DDDAS, Infosymbiotic Systems, Autonomy.

Abstract: The introduction of pervasive and ubiquitous instrumentation within Internet of Things (IoT) leads to

unprecedented real-time visibility of the power grid, traffic, transportation, water, oil & gas. Interconnecting

those distinct physical, people, and business worlds through ubiquitous instrumentation, even though still in

its embryonic stage, has the potential to create intelligent IoT solutions that are much greener, more

efficient, comfortable, and safer. An essential new direction to materialize this potential is to develop

comprehensive models of such systems dynamically interacting with the instrumentation in a feed-back

control loop. We describe here opportunities in applying cognitive computing on interconnected and

instrumented worlds (CIoT) and call out the system-of-systems trend on interconnecting these distinct but

interdependent worlds, and methods for advanced understanding, analysis, and real-time decision support

capabilities with the accuracy of full-scale models.

1 INTRODUCTION

The rapid adoption of Internet of Things (IoT),

together with unprecedented bandwidths and

computational power in high-end, mid-range and

instrumentation platforms and devices have already

produced ground-breaking real-time visibility (or

near real-time) access and transfer of information

about a system or device in both natural and

engineered systems, and in individual and industrial

environemnts, such as in the following examples:

• personal condition (wearable devices, and smart

phones),

• surrounding environment (bodycam),

• home (home security devices, appliances)

• power grid (eMeters, PMUs, other sensor and

actuator in the power distribution systems)

• traffic and transportation ( traffic sensors on cars,

busses, trains, roads, traffic lights, railroads,

aerial/UAVs, and congestion control devices)

• structural health monitoring (bridges, buildings,

vehicles, aerial platforms)

• water systems (distribution grids, asset mana-

gement and preventive maintenance; ambient

environments)

• oil & gas (intelligent oil field)

Interconnecting those distinct physical, people,

and business worlds (as shown in Fig. 1) through

ubiquitous instrumentation, even though still in its

embryonic stage, has the potential to unleash a

planet that is much greener, more efficient, more

comfortable, and safer. However, just a compedium

and deluge of instrumentation data is insufficient in

enabling these ultimate objectives. Cognitive

representations (a.k.a. models) of these distinct

physical, people, and business worlds are essential in

understanding the complexity of these systems-of-

systems worlds and their dynamics, and predicting

and controlling their evolution, and creating accurate

decision support capabilities when maneuvering

through uncertain environments and not known a

priori conditions. IoT combined with modeling is

referred to as Cognitive IoT (CIoT).

Rich multi-fidelity and multimodal modeling and

instrumentation are becoming key for enabling the

above referenced capabiltiues for physical world

systems (natural, engineered, and human systems).

Beyond present notions of CIoT (Wu 2014, Zaidi

2015), new capabilties derived through modeling

dynamically and synergistically integrated with

instrumentation in a feed-back control loop are

emerging (Darema 2000, 2005; Willcox 2014,

Li, C-S., Darema, F., Kantere, V. and Chang, V.

Orchestrating the Cognitive Internet of Things.

DOI: 10.5220/0005945700960101

In Proceedings of the International Conference on Internet of Things and Big Data (IoTBD 2016), pages 96-101

ISBN: 978-989-758-183-0

Bazilevs 2012, 2013; Celik 2011, Son 2010).

Furthermore, the trend towards higher fidelity, and

(semi-)autonomy through humans in the loop is

accelerating. Related methods and opportunities

include fusing multiple world models to extract

insights, capturing and using dynamic intelligent

interactions, and orchestrating these interdependent

models, information, processes, decisions, and

actions. In addition, robust IT systems are essential

for supporting the CIoT capabilities discussed

above.

Figure 1: CIoT solutions will require interconnected and

interdependent models representing the physical

environment, business & IT, and individual &

communities.

A cognitive Internet of Things (IoT) solution

really is a feedback control loop system (or system

of systems, since each individual component within

this system could be a system by itself). Figure 2

shows the system view of such a closed-loop CIoT

solution:

• Modeling & Orchestration Platform

• Data & Measurement Platform

• Control Platform

A CIoT solution inlcudes the real world itself –

whether it is for example a smart grid, a smart

building, a smart supply chain, or a smart water

system; and the instrumentation provides

mechanisms to capture information (of varying

levels of fidelity) from the observed world to

describe the real-world through models of the real

world. As discuss more below, cognitive

representations (or models) integrating data captured

from the instrumented world enable interpolation or

extrapolation of those areas where data could be

noisy, unavailable, or contaminated. And in other

cases, these models allow generation of the most

plausible hypotheses to explain the available

information.

From these models, the possible outcomes are

generated through simulation and/or predictive

analysis. Based on the what-if analysis conducted

by the models, a course of actions is then taken to

actuate the real world. This closed-loop system in

reality is an instance of the closed-loop control

system and is similar the MAPE loop of an

autonomic system, with the exception that we will

need to include the impacts from human

(individually or as a community) to the system.

2 MODELS

A CIoT solution requires optimal or near optimal

orchestration of the control flow and information

flow. (The music notes of the orchestration really

came from behavior models which dynamically

integrate real-world information). Consequently,

developing models at the behavior levels is

necessary in order to enable optimal orchestration of

both information and control flows. The behavioral

models are continuously updated by the input data

either to speed-up the execution by replacing parts

of the computation in the model with the actual data

or to impart additional information into the model as

it is quite often that the model does not accurately or

fully capture the system. The output of the behavior

model controls the instrumentation in order to either

refine data acquisition to improve the model

accuracy or actuates the controllers to effect

an action on the system or by the system

(DDDAS/Infosymbiotics paradigm – Darema 2000,

2005, 2006, 2010). The kinds of models of interest

Figure 2: Interconnected platforms provide data dynamic

capture & integration into models, orchestration of

behavioral models, and control for closed-loop prediction

& response.

Business & IT

World

(Enterprise Reality)

Human World

Individuals and

Communities

(Social-economic

Reality)

Physical World

(Physical Reality)

World

Models

Orchestrating the Cognitive Internet of Things

span numeric and non-numeric, agent-based and

graph models, as well as statistical models.

Examples are given in Section 3.

There are multiple abstraction levels of models

of the world. The most abstract level is at the

conceptual (theory or functional) level. Additional

details are available at the structural level.

Behavioral level models often capture the most

comprehensive aspects of the real world. Both at the

structural and the behavioral level, there may be

multiple levels of fidelity in describing the system at

hand. The evolution of the abstraction levels of the

model typically starts at the conceptual and/or

functional levels. There are quite a few examples

from various industries that demonstrate the gradual

evolution of model sophistication, and examples are

shown in Fig. 3.

During the development of Boeing 777,

substantial portion of its dynamic behavior was

entirely evaluated within a simulation environment

rather than going through numerous wind-tunnel

testing (Abarbanel 1996). In a later section, we

discuss new and more powerful methods (DDDAS-

based) that not only allow optimal design of aircraft,

but also use new modeling methods (discussed in

Section 2) to enable optimized operational

capabilities under dynamic conditions.

Figure 3: Examples of multiple abstraction levels of

models of the world.

Additional examples include the ability to

simulate the IBM z10 chip entirely within the

simulation environment in conjunction with virtual

bring up and processor-only exercise. The ability of

fully capturing the system at the behavior model

level enables the first tape out and bring up of the

IBM system z10 to be entirely successful (Lets

2009). Similarly, optimization in enterprise

processes at the behavior levels saves multi-billion

dollars annually through supply chain optimization

(Min 2002; Son 2010).

The evolution from conceptual/functional models

to behavior models in almost every domain in the

past has improved business outcome with

manageable complexity and uncertainty.

In general, the entire CIoT spectrum really

includes physical worlds, business and IT worlds,

and the human worlds, and can be further divided

into at least six domains: physical, embedded

(SCADA related), cyber, enterprise, community, and

individuals.

During the past few decades, cognitive models in

each of these silos are evolving from functional to

structural and now to behavioral. In the foreseeable

future, capturing and modeling the CIoT will happen

at multiple abstraction, multiple resolutions and

from multiple vantage points.

For example, in the enterprise domain, CBM

(component business model) (Chesbrough 2010) and

industry framework belong to the functional aspect.

Industry models (including data models, process

models, and service models) belong to the structural

levels. Customer and workforce logistics and the

enterprise risk models belong to the behavioral

levels.

Figure 4: Future CIoT solutions will require

interconnected and orchestrated measurements and models

across multiple domains.

In the cyber area, ITIL (Canon 2011) can be

viewed as belonging to the functional level while IT

configuration model belong to the structural level

and workload and network traffic belong to the

Level 1

Supply chain

scope

Level 2

Supply chain

configuration

Circuit

Schematic

VHDL, Netlist

Instruction

Set

Block Diagram

Logic

(digital)

Simulation

SPICE (analog)

Simulation

Supply Chain

Operational

Reference

Level 3

Process elements

and Performance

attributes

Supply

Chains

Optimization

IT Systems

Fully functional

IBM z10

Aerospace

Boeing 777 is

amon

first

Functional

Specification

3D Drawing

Structural

Model

Mechanical

Behavior

Model

Business and IT World

sical

Congestion Pricing

(Stockholm)

(Transportation

ÎPublic Safety,

Healthcare)

Risk Models:

Credit Risk,

Market Risk, …

Facebook,

Intelligent Utility

Network

(IUN, AMR, AMI)

eHealth

Weather

Model

Transport:

Traffic Flow

Models

Oil & Gas:

Reservoir

Models

Interconnection and Interdependency

Single Domain Ecosystem

Systemic Risk

Financial Market

Power Grid

Transportation

Pandemic

Surveillance

CBM

Org Chart

IT Config. Model

Evidence Based

Smarter City

(Malta)

(Water, Energy)

Smarter City

(Masdar)

(carbon, waste,

energy, water)

Current revenue

Opportunity and experience base

Crime Prevention (NYC)

(complaints, summons,

911, video surveillance,

RTCC

…

)

w/ Functional w/ Structural w/ Behavioral

Smarter Grid

(weather, people,

buildin

)

Smarter Cities (Dubuque)

(carbon, utility,

ttti)

Counter-party

Risk

Smarter Cit

(South Bend,

IN) (Drainage,

Wth)

IoTBD 2016 - International Conference on Internet of Things and Big Data

behavior level.

For the community side of the human world,

social networks (including LinkedIn and Facebook)

capture the structural level of human relationship.

Many of them evolve into capturing social or

community behavior in real time. From the

individual (personal) side, individual profile belongs

to the functional level while the purchase history

belongs to the behavior level.

The embedded system domain is related to

electric grid, transportation, dam, traffic lights, and

manufacturing where SCADA systems (Boyer,

2009) are often deployed. This area is transforming

itself at an extremely fast pace as increasingly more

of such systems are connected to each other as well

as to the internet, and through DDDAS-based

models.

By transforming from single domain into

ecosystem, as in Fig. 4, we could gain new insight

when analyzing the existing and future CIoT

solutions. Many of the existing CIoT solutions fall

into the category of single domain, and leveraging

only structural models for static analysis. There are

emerging opportunities – whether it is in the smart

grid (power grid) or smarter city areas – often

requires integrating more than one interdependent

domain at the behavioral levels as well as

DDDAS/Infosymbiotics based methods, of models

dynamically integrated with instrumentation. There

are advances and new capabilities that have been

demonstrated along these directions (per examples

in Section 3), and are likely ready to transition and

be exploited by industry over the next 2-5 years.

Figure 5: Smart Grid solutions continuously optimize the

expected outcome dynamic data driven behavior models.

3 CASE STUDIES

In this section, we will use the smart grid and smart

aerial platforms as examples to illustrate the

behavior-model based orchestration in DDDAS-

based CIoT solutions.

(A) Energy Related Applications

In a smart grid solution, intelligent utility networks

(IUN) provide capabilities for real-time prediction of

the onset of brown-outs or black-outs, and also

provide optimized dynamic load management. The

real-time instrumentation capability, often referred

to as automatic meter reading (AMR) or automatic

meter infrastructure (AMI), is based on

measurements made by voltage amplitude phasor

measurement unit (PMU) and other IoT sensors

dynamically integrated with statistical and agent-

based models. Dynamic load management within a

Smart Grid solution includes activating mitigating

actions prior to the onset of a brown-out or black-out

to ensure differentiated support for critical and high-

priority services vs. medium- and low-priority

services for multiple customers and multiple (and

possibly geographically dispersed) energy-sources

(including renewables - such as wind, solar, and

hydro, and energy storage which acts as a generation

source) (Celik 2013, 2015).

Other scenarios include incorporating weather

data and weather prediction models in dynamic data

driven behavior models of the power-grid to provide

continual optimization of load shedding during peak

demand period (such as during summer) or

restoration of the grid infrastructure after a weather

induced failure (integrated outage management).

Emerging scenario based on electric vehicles already

led to a new demand class and potentially a

generation source through the use of batteries.

Furthermore, demand response management reduces

demand during peak hours through incentives such

as dynamic pricing plan.

With such capabilities, the utility companies will

be able to provide much better assurance of the

business outcome for their customers. The bottom

line is to leverage the real-time visibility

(instrumentation) in order to build real-time

behavioral models so that the business can optimize

the expected outcome continuously.

Other Smart Grid related areas such as wind

farms pose new challenges and require new CIoT

capabilities. These CIoT capabilities include

optimized operation to mitigate effects of the wake

across stacked turbines (Perez 2015) and reducing

wear and tear or turbine rotors and prediction of an

adaptive repair schedule rather than a static one (all

turbines repaired periodically) (Ding 2006).

Accurate high resolution weather forecasts are

Usage

Intelligent

Utilit

Behavior

Models

Demand

Models

Real-Time

Visibility

Environment

al Models

Optimal plan &

schedule for

A common orchestration platform

optimizes outcomes by applying

behavior models to real-time

information.

Makin

decision choices to

Results

Model &

Anal

tics

Data &

Measur

Control

Orchestrating the Cognitive Internet of Things

central to predict potential storm severity and its

path. IBM Research’s Deep Thunder (Gallagher

2012) can provide high resolution forecasts for a 48

hour horizon for areas (in a given county) that are

most likely to have outage events (with some

uncertainty). This prediction can provide a basis for

planning the deployment of repair crews and trucks

in an anticipatory mode. An additional piece of

analysis would be to schedule work orders and

repair crews that would maximize the number of

customers that are brought back online with each

order. The ability to estimate the likelihood of

damage in different regions allows for predictive

planning in stationing crews for early repairs.

(B) Structural Health Monitoring, Energy

Efficiencies, and Decision Support

Aerial platforms (both civilian and military, human-

operated and UAVs, aerial and space-based) during

flight are subject to dynamic stresses accentuated by

turbulence-induced forces. Such stresses as well as

aging of materials can result in structural damage,

manifested as cracks, disbonding, delamination, or

waviness. All these conditions can cause disastrous

results with airplane crashing (as indeed have

happened, such as aileron detachment). Additional

sensor malfunction situations (such as pitot tubes

freezing) also result in catastrophic failures.

DDDAS-based modeling have shown advanced

capabilities: (1) detection of the onset of damage

(crack creation), (2) predict the propagation of the

damage and potential impact (Willcox 2014), (3)

application of time dependent control through

coordination of multiple actuators to mitigate the

propagation of the damage (Bazilevs 2012), and (4)

wing-level and aerial structure-wide assessment of

structure (5) through multi-fidelity models

dynamically driven by multiple levels of sensors to

assess platform health conditions in real-time, (6)

cognizant of environment (such as winds and wind-

induced turbulence) to plan and re-plan in real time

to optimize flight path and necessary maneuver to

fulfill mission (Willcox 2014, 2015). It was

demonstrated in (Varela 2013, 2014) by using

DDDAS-based methods to compensate for sensor

failures. In this case, the output from a continually

executing model of flight conditions is compared

against the actual measurements from pitot sensor.

The model can take over in case of abrupt

discrepancy with the measurement to allow time to

readjust and switch over to other sensor modalities .

4 SYSTEM IMPLICATIONS

CIoT solutions often have specialized requirements

on processing through models the data derived from

sensors and produce decisions and apply control

through actuators. In some applications the value of

data is highest when real-time or near real-time

response was possible. In most applications both

real-time and archival data are used as dynamic

inputs into the models (e.g. weather, flight-path,

etc). When multiple data sources are used as

dynamic inputs into the behavior models, the value

could be even higher as additional data can help to

make the model more accurate, speed-up the model,

reduce the uncertainty and contribute to the

improved accuracy for predicting future condition

and evolution of the system. In other applications,

after the initial interval when the data contributes

directly to the decision and proactive actions, the

value of the data monotonically declines as the data

can be potentially used for metering and billing,

auditing, and long term trend analysis. As a result,

system architectures optimized for CIoT solutions

need to accommodate latency requirements and

prioritize computation and communication resources

in order to maximize the value that can be derived

from the sensor data as well as the long term archive

requirements to facilitate long term trend analysis.

Video surveillance is at the forefront of these

requirements in terms of throughput (3.2-

26GB/day/stream). SCADA systems and health

monitoring systems have very stringent latency

requirements (on the order of microseconds to

milliseconds). High throughput and/or low latency

requirements often mandate moving some cognitive

capabilities to the edge of the CIoT solution even

though the primary analytic functions are still

carried out in the computing and data center(s).

5 ORCHESTRATION

Orchestrating a CIoT solution shown in Fig. 2

requires orchestrating interconnected platforms.

These platforms include those capturing the

information from the real world, dynamically

integrating the information (measurement data) from

the actual systems (the observed world) into the

world model (including behavior models), for

simulation and predictive analysis for what-if

scenarios, and using the decision model (subject to

the context and constraints) to render a set of

command and control instructions that will yield

IoTBD 2016 - International Conference on Internet of Things and Big Data

100

optimal expected outcomes. These command and

control instructions will then be executed by the

command and control mechanisms and applied to

the real world.

The modeling and analytic orchestration

platform (the brain of a CIoT solution), coupling

with the instrumentation (measurement and control).

The lifecycle of analytics can start with the raw data

coming from the real world, going through analytic

environments that may include analysis engines for

structured, unstructured, streaming, general

analytics, application/algorithmic specific analytics,

and dashboard/reporting tools, and in return

additional raw data are collected, fed into the model

to improve its accuracy or speed of the modeling.

It’s worth noting that when behavior models are

tightly coupled with the data specific analytic

environment, data modalities and very difficult to be

generalized, and more general levels of abstraction

of data and models are needed. Nevertheless, the

DDDAS/Infosymbiotics paradigm provides a clear

methodology of the value of dynamic integration of

models and data in a feedback control loop.

6 CONCLUSIONS

The introduction of pervasive and ubiquitous

instrumentation within a CIoT leads to

unprecedented real-time visibility of the power grid,

traffic, transportation, water, and oil & gas areas.

Interconnecting those distinct physical, people, and

business worlds through ubiquitous instrumentation,

even though still in its embryonic stage, has the

potential to unleash a planet that is much greener,

more efficient, more comfortable, and safer.

In this paper, we described some of the

opportunities after applying cognitive computing on

interconnected and instrumented worlds and call out

the system of systems trend on interconnecting these

distinct but interdependent worlds. It has become

increasingly crucial that cognitive representations of

these distinct worlds (a.k.a. models, dynamically

integrated with instrumentation) need to be created

as a pre-requisite so in order to assess the

complexity, maneuver through uncertain

environments and eventually achieve the predicted

outcome.

REFERENCES

Abarbanel, Bob, and William McNeely. "Flythru the

boeing 777." ACM SIGGRAPH 96 Visual

Proceedings: The art and interdisciplinary programs of

SIGGRAPH'96. ACM, 1996.

Bazilevs, Yuri et al, “Toward a computational steering

framework for large-scale composite structures based

on continually and dynamically injected sensor data”,

Procedia ComputerScience, 9 (2012) 1149--1158.

Boyer, Stuart A. SCADA: supervisory control and data

acquisition. International Society of Automation, 2009.

Cannon, David (2011). ITIL Service Strategy 2011

Edition. The Stationery Office.

Celik, Nurcin, et al. “DDDAMS-based Real-time

Assessment and Control of Electric-Microgrids,

ICCS2011 Proceedings (2011).

Chesbrough, Henry. "Business model innovation:

opportunities and barriers." Long range planning 43.2

(2010): 354-363.

Darema, F., “Dynamic Data Driven Applications Systems

(DDDAS)” NSF Workshop 2000, and 2006 and 2010

Workshopes (www.1dddas.org); “ Dynamic Data

Driven Applications Systems (DDDAS): New

Capabilities for Application Simulations and

Measurements”, ICCS05Proceedings (2005).

Darema, F. , Grid Computing and Beyond: The Context of

Dynamic Data Driven Applications Systems,

Proceedings of the IEEE, Special Issue on Grid

Computing, March 2005 (invited paper).

Ding, Yu, et al, “Dynamic Data-Driven Fault Diagnosis of

Wind Turbine Systems” ICCS2007 Proceedings

(2007).

Gallagher, Sean. "How IBM’s Deep Thunder delivers

hyper-local forecasts 3-1/2 days out." Ars Technica

(March 14, 2012).

Hoffmann, C. et al “DDDAS For Autonomic

Interconnected Systems: The National Energy

Infrastructure”, ICCS2007 Proceedings (2007).

Korobenko, A. Bazilevs, Y. et al“Structural Mechanics

Modeling and FSI Simulation of Wind Turbines”,

Mathematical Models and Methods in Applied

Sciences, 23 (2013) 249–272.

Letz, S., et al. "Functional verification of the IBM system

z10 processor chipset." (2009).

McCalley James, et al “Integrated Decision Algorithms for

Auto-Steered Electric Transmission System Asset

Management”, ICCS2007 Proceedings (2007).

Min, Hokey, and Gengui Zhou. "Supply chain modeling:

past, present and future." Computers & industrial

engineering 43.1 (2002): 231-249.

Son, Young Jun, et al, DDDAS-based Multi-fidelity

Simulation Framework for Supply Chain Systems, IIE

Transactions on Operations Engineering, (2010),

Willcox, Karen, et al “ Multifidelity DDDAS Methods

with Application to a Self-Aware Aerospace Vehicle,

ICCS2014 Proceedings (2014).

Wu, Qihui. et al, “Cognitive Internet of Things: A New

Paradigm beyond Connection”, Internet of Things

Journal, V.1. IEEE (2014).

Zaidi, Syed, et al, Cognitive Internet of Things: A Unified

Perspective, First International Internet of Things

Summit(2014).

Orchestrating the Cognitive Internet of Things

101