CONTROL OF NETWORKED SYSTEMS CONTAINING
MULTIPLE AGENTS
Jose B. Cruz Jr.
1,2
, Gregory Tangonan
2
,
Raymond R. Tan
3
,
Nathaniel Libatique
2
Fabian M. Dayrit
2
and Alvin B. Culaba
3
1
The Ohio State University, Columbus, U.S.A.
2
Ateneo De Manila University, Manila, Philippines
3
De La Salle University, Manila, Philippines
Keywords: Multi-agent Systems, Game Theory, Complex Systems, Networked Control Systems.
Abstract: This position paper illustrates the use of a natural framework for the modelling, analysis, and design of
engineering systems that involve two or more controllers, each of which has an associated objective
function. Such systems arise when ordinary single controller systems are networked through communication
links so that the information available to each controller may contain aspects of the other systems’ states and
the optimization of each objective function is no longer decoupled from each other. Single controller
optimization is no longer directly applicable. Appropriate to the study of such systems is the theory of
games that has been developing in mathematics, economics, and engineering for the past 60 years. There are
extensive applications in economics, but in engineering the applications are scarce. In recent years, there has
been great attention to global problems such as the negative environmental impact of energy use, and global
warming. These problems arise from complex systems with multiple controllers. Among the approaches for
dealing with the problems, there should be one on a total systems approach with a game theory base. A
natural framework for this is the subject of this policy paper.
1 INTRODUCTION
In this position paper we establish the benefits and
advantages of explicitly including multiple agents in
the modelling and control of networked engineering
systems, when in fact, multiple agents are present in
the application systems. The agents are not
necessarily cooperating in a team and not necessarily
antagonistic against each other, although in some
applications they might be cooperating as a team.
Some global problems with significant technological
components are (a) integration of renewable energy
sources (such as solar, geothermal, wind, hydro, and
biological) with the traditional fossil energy sources,
to reduce negative impact on the environment, (b)
recycling of wastewater to produce clean water, to
conserve scarce fresh water resources, (c) mitigating
damages due to disasters such as typhoons,
hurricanes, floods, and earthquakes. These national
and international problems are also examples of
complex systems. Complexity arises because of
large numbers of smaller systems that are networked
together, and total system behaviour is not easily
inferred from individual behaviours of the
component systems. These complex systems are
characterized by the presence of many stakeholders,
starting from the national government, to
provincial/state governments, private enterprises of
suppliers, industry associations, and large blocks of
consumers. The stakeholders generally have policies
that translate to actions affecting the system.
Notwithstanding announced plans to the contrary,
the complex systems typically evolve piece-meal,
and unexpected and undesirable effects are
addressed piece-meal. Finally, complex systems
have numerous time lags throughout and stability is
a crucial issue that could lead to a total collapse if
not addressed properly.
A specific scenario for a networked system is the
following: we can project a boom in ethanol
production in some countries, even exporting of
ethanol (after meeting local needs for ethanol), using
sugar cane. For the same country can also project a
return to the export of sugar as well. The sugar
industry is terribly inefficient and unkind to labour -
193
B. Cruz Jr. J., Tangonan G., Tan R., Libatique N., Dayrit F. and Culaba A. (2010).
CONTROL OF NETWORKED SYSTEMS CONTAINING MULTIPLE AGENTS.
In Proceedings of the 7th International Conference on Informatics in Control, Automation and Robotics, pages 193-198
DOI: 10.5220/0003002201930198
Copyright
c
SciTePress
six months of work and down for 6 months. The
ethanol industry needs sugar cane 12 months of the
year. The options for farmers have just changed -
what is their way of optimizing their land and/or
labour? The government needs to figure out how to
provide a good price of sugar for consumers, to
protect the growing ethanol market, and to
encourage investment by distillers moving into the
ethanol industry through incentives such as tax
breaks. This ecosystem is extremely rich in control
problems to consider. Various stakeholders or agents
need to understand various solution concepts and
different perspectives. A decision support system
that reflects the interactions of all the stakeholders in
the modelling and control strategies would be highly
useful.
The presence of multiple stakeholders in
complex systems can be studied through
mathematical game theory. Much of the literature is
for static systems with very few exceptions (Nash
1950, Nash 1951, von Neumann and Morganstern
1947, von Stackelberg 1952). Dynamics can be
studied using standard methods in mathematics and
engineering. Yet, there are no general computer-
based decision aids as tools for the design and
analysis of dynamic complex systems. To be sure
the challenge is not trivial and it requires significant
research effort.
The bulk of control theory pertains to dynamic
systems with a single controller (Bellman 1957,
Pontryagin, Boltyanskii, Gamkrelidze, Mischenko
1962). With the emergence of networked control
systems (Wang and Liu 2008) whereby previously
separate individual subsystems with their individual
controls are linked through communication
networks, the information available at each
subsystem through the links may generally contain
aspects of the other systems. If a subsystem
controller were to optimize a performance criterion
associated with that subsystem, the performance
criterion may contain variables pertaining to the
other subsystems because of the network links and a
dilemma of how to proceed is encountered.
There have been design approaches that ignore
the presence of the links in the network.
Subsequently the systems are analyzed to check
robustness against the neglected connections. In the
case of two interconnected control systems, worst
case scenarios have been assumed in the design, in
the sense that the controller of the other subsystem is
assumed to make the performance criterion of the
system as bad as possible while the controller of the
system maximizes the performance of the resulting
worst case. The theory of maxmin (or minmax)
optimization is applied. In many applications the
performance criteria of the two control systems are
not opposite of each other, so that the minmax
design is overly conservative. Furthermore, when
the system is dynamic and the controls involve
feedback, the structure of the feedback control of the
other system needs to be known, in order to perform
a correct minmax optimization with the correct total
system dynamics. Thus the minmax approach could
be problematic unless the other system including its
feedback structure is modelled properly.
When the agents are cooperating, the individual
performance criteria may be combined as a single
convex linear combination using the Pareto-optimal
concept, and once again the theory for single
controller systems may be used. In applications,
there is the additional task of choosing the weights
in the convex linear combination. A special case
arises when the performance criteria are identical
and the choice of weights is immaterial. This is the
team-optimal problem.
Unlike engineering systems, economic systems
are modelled, analyzed, and optimized using
multiple agents. In fact in the ideal case, there is an
infinite number of consumers and an infinite number
of suppliers interacting in a market. However, the
bulk of the literature is for static systems with very
few exceptions. The bulk of the literature in the field
of operations research with respect to multiple
agents is likewise for static systems.
2 NETWORKED STATIC
CONTROL SYTEMS
To focus on the effects of multiple agents in a
networked control system, let us initially consider
static systems to eliminate one dimension of
complexity induced by dynamics.
Furthermore, even the static area could be
utilized to great advantage in dealing with large
complex systems. Suppose we have two networked
control systems of producers of renewable energy.
The productions are modelled by
(1)
(2)
where
and
are quantities of renewable energy
produced by Companies 1 and 2 respectively, and
and
are resources (controls) used to produce
and
respectively. The functions
and
are
monotonically increasing so their unique inverses
exist. The renewable energy products are sold in a
ICINCO 2010 - 7th International Conference on Informatics in Control, Automation and Robotics
194
market and the price is determined from a supply
curve that relates price to total demand
=
+
(3)
and the total demand X is equal to the total supply
=
+
(4)
The parameters
and
are given and
> 0,
< 0. The costs for producing the renewable
energy products are
C1 =
(
)
(5)
C2 =
(
)
(6)
where
and
are nonlinear functions. Each
company wants to maximize its profit, which is
revenue minus cost. For Company 1 the profit is
1=
–
(
)
(7)
2=
–
(
)
(8)
It is not a simple matter for Company 1 to
maximize its profit Profit1 with respect to
because in Profit1 contains
, which is not under
its control. Similarly for Company 2, it is not a
simple matter to maximize Profit2 with respect to
because contains
.
This illustrates the intrinsic difference between a
single controller problem and a problem with
multiple controllers or multiple agents, such as the
example above where the two static control systems
are networked through the market mechanism where
their outputs are sold. From the point of view of
single controller theory, for example in the design of
, Company 1 may simply assume a value for
or
and proceed to maximize Profit1 with respect
to
. Except for some very lucky choice of
by
Company 1, when Company 2 chooses
using an
assumed value of
, Company 2 will obtain a value
of
different from the one assumed by Company
1, posing a dilemma for both companies.
Next let us consider each company’s worst-case
design whereby Company 1 assumes that Company
2 chooses u2 to minimize Profit1. Then Company 1
maximizes Profit1 resulting in its maxmin (or worst-
case) design. Similarly Company 2 may proceed to
calculate its worst-case design. When both apply
their worst-case designs, their resulting profits will
be generally higher but in any case no worse than
the worst-case profits they previously calculated.
The pair of separately calculated worst-case controls
will generally not lead to the worst case for either
company. Still, in general their designs would be
conservative because the companies are not trying to
destroy each other by making each other’s profit as
small as possible.
In the theory of games that applies to systems
with multiple agents, there are many solution
concepts that go beyond single controller
optimization. For example, one may consider the
Nash equilibrium concept (Nash 1950, Nash 1951)
whereby when (
,
) is a Nash solution pair
and Company 1 chooses a control u1 that is different
from
, but Company 2 still uses
, the
resulting profit of Company 1 can not be higher than
that when both use their Nash controls. There is also
a Stackelberg (von Stackelberg 1952) or Leader-
Follower solution concept whereby one subsystem
controller is dominant or more powerful than the
others. The leader’s control is announced in advance
and all other controllers know what the leader’s
control is before they choose their own controls. In
the case of Stackelberg games, it is of particular
interest to determine the role of the dominant player
in inducing desirable behaviour from low-level
players through incentives or disincentives. The
implications of such mechanisms are clearly evident
for situations in which, for example, it is desired to
determine government policies to facilitate
environmentally beneficial behaviour from the
private sector (e.g., Aviso et al 2010). The
Stackelberg hierarchy may have more than two
levels.
3 NETWORKED DYNAMIC
CONTROL SYSTEMS
A system that is more general than the class
considered in Section 2 is one where the individual
control systems are dynamic. If the systems are
discrete-time the individual models may be of the
form
1
) =
,
,
(9)
1, … , , 0, … , , where
is the state
vector of system, with dimension
;
is the
control vector of system, with dimension
, and
is discrete time with integer values from 0 to M.
The vector functions
are mappings from the
spaces of the arguments to the space of
and
,
, and are given. The network connections
may be modelled by an algebraic equation
,…,
0
(10)
CONTROL OF NETWORKED SYSTEMS CONTAINING MULTIPLE AGENTS
195
where may be a scalar or vector of a given
dimension. Associated with each system , may be a
scalar performance index or cost function
1
0
(( )) ((), ())
M
iii ii i
j
J
CxM Lx ju j
(11)
If
represents total cost for the entire horizon
from 0 to M, then
represents the
incremental cost at the terminal time, and
,
represents the incremental cost
during time . As in the static case, there will be a
dilemma in a simple dynamic optimization of
, in
Equation (11) with respect to the control vector
sequence because the control may be in
feedback form and even if only
is used for
feedback at time ,
is not independent of the
other system states because of the network coupling
modelled by Equation (10).
A more general effect of the network connection
represented by Equation (10) may be a change in the
individual control system model from Equation (9)
to Equation (12)
1
=
,…
,
(12)
1, … , , 0, … ,
and the constraint represented by Equation (10) may
remain.
For systems that are modelled as continuous-time
processes a typical description in state variable form
is given by the vector differential equation
dx
i
(t)/dt f
i
(x
1
(t),..., x
N
(t),u
i
(t))
(13)
where
is the state vector of system, with
dimension
;
is the control vector of system ,
with dimension
, and t is continuous time with
values in the interval [0,T], and T is a specified real
number. Instead of the cost function in Equation (11)
the continuous time version is an integral analogous
to the sum in Equation (11)
,
,
t)dt
(14)
Because of the links in the network there may be
a constraint as in Equation (10). The standard
dynamic optimization of the integral cost functional
with respect to the vector functions
) for t in
the interval [0,T] poses a dilemma because the
functional may depend on the states of the other
subsystems through the constraint in Equation (10).
In general, the direct application of dynamic
optimization for single controllers becomes a
problematic issue. The field of dynamic game theory
offers potential benefits in the design and analysis of
such systems (Isaacs 1955, Basar and Cruz 1982,
Basar and Olsder 1998, Starr and Ho 1969a, Starr
and Ho 1969b, Chen and Cruz 1972, Simaan and
Cruz 1973a-c, Cruz 1975, Castanon and Athans
1976). Macroeconomics has completely adopted
concepts from dynamic game theory. For multiple
agent dynamic engineering systems the application
of multi-agent models and equilibrium theories of
dynamic games would be beneficial also.
4 ILLUSTRATIVE EXAMPLE
In this section we consider a single, simplified
composite energy system with only one state
variable (x
t
) but two decision-makers, each with a
control variable. We consider only finite states of
zero, 1, and 2 and finite controls 0 and 1, and two
time stages. For each controller there will be an
associated incremental cost at each time stage and a
total cost for the two time stages. We will analyze
the network using some of the solution concepts in
dynamic game theory.
Table 1: State Transitions in Period 1. (Simaan and Cruz
1973b).
Controllers’ Decisions
(0, 0) (0, 1) (1, 0) (1, 1)
x
0
= 1 x
1
= 1 x
1
= 2 x
1
= 0 x
1
= 1
Controller 1 is assumed to be the upstream
agricultural sector of a biofuel supply chain, similar
to that considered in Cruz, Tan, Culaba, and
Balacillo 2009, while Controller 2 is the downstream
sector comprised of the biofuel processing sector. In
each time period, each controller is faced with the
option of expanding (u = 0) or maintaining (u = 1)
current production capacity. The composite system
is described by a trivalent state variable which
indicates upstream (agricultural) deficit (x = 0),
balanced production (x = 1), or upstream surplus (x
= 2). This is a biofuel supply chain interpretation of
the numerical example that appeared in Starr and Ho
1969 a,b and Simaan and Cruz 1973a,b. Tables 1
and 2, which are based on the game described in
Simaan and Cruz 1973b, show the possible state
transitions arising from decisions in this stylized
energy system. As each controller is faced with a
binary decision in a given time period, over the
entire planning horizon each will have four possible
decisions, namely, (0, 0), (0, 1), (1, 0) and (1, 1). For
open-loop control structure, i.e., the controls are
ICINCO 2010 - 7th International Conference on Informatics in Control, Automation and Robotics
196
functions of time (stage) only, each of the two
controllers have four possible decision sequences,
and the two-stage game may be expressed in bi-
matrix form as in Table 3, wherein the first and
second entries are the cumulative costs borne by the
two controllers over the entire time horizon. See
Simaan and Cruz 1973b for the incremental costs.
Table 2: State Transitions in Period 2 (Simaan and Cruz
1973b).
Controllers’ Decisions
(0, 0) (0, 1) (1, 0) (1, 1)
x
1
= 0 x
2
= 2 x
2
= 1 x
2
= 0 x
2
= 0
x
1
= 1 x
2
= 1 x
2
= 2 x
2
= 0 x
2
= 1
x
1
= 2 x
2
= 2 x
2
= 2 x
2
= 1 x
2
= 0
Table 3: Cumulative Cost for Open-Loop Bi-Matrix Game
(Simaan and Cruz, 1973b).
Controller 2
(0, 0) (0, 1) (1, 0) (1, 1)
Controller 1
(0, 0)
8, 8 11, 6** 10, -2 11, 0
(0, 1)
6, 4 12, 3 7, 2 12, 4
(1, 0)
5, 12 20, 15 5, 11 8, 9*
(1, 1)
6, 5*** 16, 7 3, 7 9, 6
*Nash equilibrium
**Open-loop Stackelberg equilibrium with Controller 2
as leader
***Open-loop Stackelberg equilibrium with Controller
1 as leader
If we assume that neither the upstream nor
downstream sectors of the energy supply chain
dominate the game, the system naturally tends
toward the Nash equilibrium as indicated in Table 3.
In this case, each decision maker identifies his
rational reaction, which is the response that
minimizes his cost for any possible action by the
other player. The Nash equilibrium is the
intersection of the rational reactions of the two
decision makers. They both commit to an open-loop
sequence at the beginning of the horizon. The
upstream sector maintains production capacity in the
first time period, and expands production in the
second period, while the downstream sector
maintains its capacity throughout. As a result, the
energy system is at a state of surplus farm
production at the end of the time horizon analyzed.
Note that this state is reached without any
centralized direction, and emerges purely from the
self-interested action of the two agents.
The energy system evolves differently if either
sector were dominant. For instance, if the
downstream (fuel processing) sector acted as the
leader, the system reaches the open-loop Stackelberg
equilibrium indicated in Table 3. In this scenario, the
dominant decision-maker selects his action so as to
minimize his cost, having anticipated that the
follower’s response is the latter’s rational reaction as
in the Nash case. It would commit in advance that it
would increase production capacity in the first
period, but maintain it in the second period. The
agricultural sector, the follower, would increase
production in both periods. The energy supply chain
thus also reaches a state of surplus upstream
production capacity (i.e., excess biofuel feedstock)
as in the Nash game, except that the cost burden of
the farmers would have increased while those of the
processors would have gone down. Note that the
leader’s Stackelberg solution can be no worse than
his corresponding Nash solution (Simaan and Cruz,
1973a,b).
A similar analysis can be made for the case
wherein the upstream sector dominates and acts as
leader. In this case, an alternative Stackelberg
solution is reached, as shown in Table 3, with the
supply chain ending in a state of deficit in upstream
production capacity. Note that in this case, both
controllers incur lower cumulative costs than they do
in either of the two previous scenarios. Thus, from
the system-level standpoint, this solution is superior
for the given transitions and payoffs.
For a closed loop structure the sectors have 16
decision choices that depend on time and the state,
see Simaan and Cruz (1973b). In particular, Simaan
and Cruz (1973b) showed through the examples that
the leader solutions violate Bellman’s principle of
optimality (Bellman, 1957). In economics, this
violation is known as time-inconsistency (Kydland
and Prescott 1977).
5 CONCLUSIONS
In this position paper we provide a discussion of the
need to use modelling and control methods that are
more appropriate than those for single controller
systems when we have a networked system of
control systems whereby the individual systems that
are networked have their individual controls and
individual objective functions. This need arises
because the network that may involve
communication links induces interaction and
complicates the choice of control strategies for the
various subsystems. There are methodologies that
could be applied now for multi-agent systems but
CONTROL OF NETWORKED SYSTEMS CONTAINING MULTIPLE AGENTS
197
there remains further need for research to address
issues such as estimation, adaptation, stability, and
robustness, to name a few. Global complex systems
such as reduction of the external costs of negative
environmental impacts of the use of various energy
sources, mitigation of natural disasters, and
consideration of global warming in technological
planning, are prime areas where these networked
control systems methods could be beneficially
applied.
REFERENCES
Aviso, K., Tan, R. R., Culaba, A. B., and Cruz, J. B., Jr.,
2010, Bi-Level Fuzzy Optimization Approach for
Water Exchange in Eco-Industrial Parks, Process
Safety and Environmental Protection, Vol. 88, pp. 31-
40.
Basar, Tamer and Cruz, Jose B., Jr., 1982, Concepts and
Methods in Multi-Person Coordination and Control,
Optimization and Control of Dynamic Operational
Research Models, Ed. S.G.Tzafestas. Amsterdam:
North-Holland, 455-462.
Basar, T. and Olsder, G. J., 1998, Dynamic
Noncooperative Game Theory. New York, NY:
Academic Press, revised ed.
Basar, T. and Selbuz, H., 1979, Closed-loop Stackelberg
strategies with applications in the optimal control of
multilevel systems, IEEE Trans. on Automatic Control,
vol. AC-24, no. 2, pp. 166-179.
Bellman, R., 1957, Dynamic Programming. Princeton, NJ:
Princeton University Press.
Castastanon, D. and Athans, M., 1976, On stochastic
dynamic Stackelberg strategies, Automatica, vol. 12,
pp. 177-183.
Chen, C. I. and Cruz, J. B., Jr., 1972, Stackelberg solution
for two person games with biased information
patterns, IEEE Trans. on Automatic Control, vol. AC-
17, pp. 791-797.
Cruz, J. B., Jr., Survey of Nash and Stackelberg
Equilibrium Strategies in Dynamic Games, Annals of
Economic and Social Measurement, Vol. 4, No. 4,
1975, pp. 339-344.
Cruz, J. B., Jr., 1978, Leader-follower strategies for
multilevel systems, IEEE Trans. on Automatic Control,
vol. AC-23, no. 2, pp. 244-255.
Cruz, J. B., Jr., Tan, R. R. Culaba, A. B., and Ballacillo,
J.A., 2009, A Dynamic Input-Output Model for
Nascent Bioenergy Supply Chains, Applied Energy,
Vol.86, Supplement 1, pp. 86-94.
Cruz, Jose B. Jr. and Tan, Xiaohuan, 2009, Dynamic
Noncooperative Game Models for Deregulated
Electricity Markets, Nova Publishers, New York NY.
Isaacs, R. P., 1955, Differential Games: a Mathematical
Theory with Applications to Warfare and Pursuit,
Control and Optimization. New York: John Wiley and
Sons, 1955.
Kydland, F. E. and Prescott, Edward C., Rules Rather
Than Discretion: The Inconsistency of Optimal Plans,
The Journal of Political Economy, Vol. 85 No. 3 (June
1977), pp. 473-492.
Nash, J., 1950 Equilibrium points in N-person games, in
Proc. of the National Academy of Sciences of the
United States of America, vol. 36, pp. 48-49.
Nash, J., 1951, Noncooperative games, Annals of
Mathematics, vol. 54, pp. 286-295.
Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R.
V.,and Mishchenko, E. F., 1962, The Mathematical
Theory of Optimal Processes. New York, NY:
Interscience Publishers, 1962.
Simaan, M. and Cruz, J. B., Jr., 1973, On the Stackelberg
strategy in nonzero-sum games
, Journal of
Optimization Theory and Application, vol. 11, no. 5,
pp. 533-555.
Simaan, M. and Cruz, J. B., Jr., 1973, Additional aspects
of the Stackelberg strategy in nonzero-sum games,
Journal of Optimization Theory and Application, vol.
11, no. 6, pp. 613-626.
Simaan, M. and Cruz, J. B., Jr., 1973, A Stackelberg
solution for games with many players, IEEE Trans. on
Automatic Control, vol. AC-18, no. 3, pp. 322-324.
Starr, A. W. and Ho, Y. C., 1969, Nonzero Sum
Differential Games, Journal of Optimization Theory
and Applications, Vol. 3, No. 3.
Starr, A. W. and Ho, Y. C., 1969, Further Properties of
Nonzero Sum Differential Games, Journal of
Optimization Theory and Applications, Vol. 3, No. 4.
von Neumann, J. and Morgenstern, O., 1947, Theory of
Games and Economic Behavior. Princeton, NJ:
Princeton University Press, 2nd ed.
von Stackelberg, H., 1952, The Theory of the Market
Economy. Oxford: Oxford University Press, English
translated ed., 1952.
Wang, Fei-yue and Liu, Derong, 2008, Networked Control
Systems: Theory and Applications, Springer.
ICINCO 2010 - 7th International Conference on Informatics in Control, Automation and Robotics
198
