the selected scenarios (Tr. 2), which corresponds to
the transformation of f′(x
i
, S′) to a single robustness
value, R(x
i
, S). This equates to an identity transform
in cases where only a single scenario is selected in
Tr. 2, as there is only a single transformed
performance value, which automatically becomes
the robustness value. However, in cases where there
are transformed performance values for multiple
scenarios, these have to be transformed into a single
value by means of calculating statistical moments of
these values, such as the mean, standard deviation,
skewness or kurtosis.
In relation to the performance value
transformation (Tr.1), which robustness metric is
most appropriate depends on whether the
performance value in question relates to the
satisfaction of a system constraint or not, and is
therefore a function of the properties of the system
under consideration. For example, if the system is
concerned with supplying water to a city, there is
generally a hard constraint in terms of supply having
to meet or exceeding demand, so that the city does
not run out of water (Beh et al., 2017). The system
performs satisfactorily if this demand is met and that
is the primary concern of the decision‐maker.
Alternatively, there might be a fixed budget for
stream restoration activities, which also provides a
constraint. In this case, a solution alternative
performs satisfactorily if its cost does not exceed the
budget. For the above examples, where performance
values correspond to determining whether
constraints have been met or not, satisficing metrics,
such as Starr's domain criterion, are most
appropriate.
In contrast, if the performance value in question
relates to optimizing system performance, metrics
that use the identity or regret transforms would be
most suitable. For example, for the water supply
security case mentioned above, the objective might
be to identify the cheapest solution alternative that
enables supply to satisfy demand. However, there
might also be concern in over‐investment in
expensive water supply infrastructure that is not
needed, in which case robustness metrics that apply
a regret transformation might be most appropriate,
as this would enable the degree of over‐ investment
to be minimized when applied to the cost
performance value. For the stream restoration
example, however, decision‐makers might simply be
interested in maximizing ecological response for the
given budget. In this case, robustness metrics that
use the identity transform might be most appropriate
when considering performance values related to
ecological response.
In relation to scenario subset selection (Tr.2),
which robustness metric is most appropriate depends
on a combination of the likely impact of system
failure and the degree of risk aversion of the
decision‐maker. In general, if the consequences of
system failure are more severe, the degree of risk‐
aversion adopted would be higher, resulting in the
selection of robustness metrics that consider
scenarios that are likely to have a more deleterious
impact on system performance. For example, in the
water supply security case, it is likely that robustness
metrics that consider more extreme scenarios would
be considered, as a city running out of water would
most likely have severe consequences. In contrast, as
the potential negative impacts for the stream
restoration example are arguably less severe,
robustness metrics that use a wider range or less
severe scenarios might be considered. However, this
also depends on the values and degree of risk
aversion of the decision maker. As far as the
robustness value calculation (Tr. 3) goes, this is only
applicable to metrics that consider more than one
scenario, as discussed previously, and relates to the
way performance values over the different scenarios
are summarized. For example, if there is interest in
the average performance of the system under
consideration over the different scenarios selected in
Tr.2, such as the average cost for the water supply
security example or the average ecological response
for the stream restoration example, a robustness
metric that sums or calculates the mean of these
values should be considered. However, decision‐
makers might also be interested in (1) the variability
of system performance (e.g., cost, ecological
response) over the selected scenarios, in which case
robustness metrics based on variance should be
used, (2) the degree to which the relative
performance of different decision alternatives is
different under more extreme scenarios, in which
case robustness metrics based on skewness should
be used, and/or (3) the degree of consistency in the
performance of different decision alternatives over
the scenarios considered, in which case robustness
metrics based on kurtosis should be used. As these
metrics are used to make decisions on outcomes, it is
important to obtain greater insight into the
conditions under which different robustness metrics
result in different decisions.
It is important to note that the relative ranking of
two decision alternatives (x
1
and x
2
), when assessed
using two robustness metrics (R
a
and R
b
), will be the
same, or stable, if the following three conditions
hold:
𝑅
𝑥
>𝑅
𝑥
and 𝑅
𝑥
>𝑅
𝑥
,
(1)