Which Tools Are Needed to Assist Audit Managers in Project

Portfolio Selection When Divergent Views Emerge?

Vivian Vivas

1,2

and Mónica Duarte Oliveira

Centre for Management Studies of Instituto Superior Técnico (CEG-IST), Universidade de Lisboa, Lisbon, Portugal

Ministry of Transparency and Comptroller General of the Union, Brasilia, Brazil

Keywords: Conflict Analysis, Multicriteria Decision Analysis, Portfolio Decision Analysis, Auditing.

Abstract: In Brazil, government programs differ in terms of territorial coverage, of sectorial action, and vary according

to regional contexts. Similarly to other public auditing organizations, the Ministry of Transparency and

Comptroller General of the Union (CGU) is responsible for monitoring the application of funds in these

government programs as a means to promote transparency and accountability in public spending. In order to

fulfil this mission, it is necessary to select the audit projects to be executed by the CGU teams. Several

multidisciplinary teams and stakeholders participate in this process of selecting auditing projects across

distinct management and public policies themes, with different opinions and views arising. The most

important challenges for CGU are to compare the audit projects submitted by these teams on a common and

transparent basis (under the presence of scarce resources) and to consider in the selection of projects the

divergent and often conflicting perspectives of those involved. In this study, we explore which tools from the

multicriteria portfolio decision analysis and negotiation literature can be used to inform a transparent and

negotiated selection of audit projects, as well as discuss which type of multicriteria portfolio decision analysis

models and features for negotiation should be combined in future research.

1 INTRODUCTION

The Ministry of Transparency and Comptroller

General of the Union (CGU) is responsible for

checking the application of public funds in

government programs as a means of promoting

transparency and accountability in public spending.

Decisions regarding which projects the CGU should

audit can be considered multicriteria resource

allocation problems, since they involve multiple

strategic objectives, the benefits are usually multi-

dimensional (related with the delivery of effective

and quality public policies, losses recovery,

transparency in public management, social

responsibility), and there limitations on the resources

available to dedicate to auditing. At a time when

public resources are particularly scarce, CGU needs

to decide on this allocation of resources in audit

projects. This allocation of resources is not always

straightforward as it is typical to have divergent and

conflicting views among those participating in the

auditing selection process, and a compromise solution

needs to be negotiated.

Thus, it is central for the CGU to decide: how to

select the audit projects to compose the Operational

Plan while considering the different views and

perspectives from CGU stakeholders? Selecting sub-

set of projects is a portfolio decision, and up to our

knowledge we have not identified literature

integrating negotiation with portfolio analysis, nor

negotiation literature associated with auditing

decision-making. This study aims to contribute to

these fields by digesting concepts and models from

the Multicriteria Decision Analysis (MCDA),

Portfolio Decision Analysis (PDA) and negotiation

literature, and reflecting upon how such concepts and

models can support the selection of projects in

auditing contexts.

In the next section we detail the results from a

review of MCDA, PDA and negotiation studies that

provides useful concepts and models to support the

selection of audit projects under the presence of

divergent views among those involved in the

selection process and under limited resources.

Departing from the results of the review, section 3

systematizes tools that can inform the selection of

audit projects, as well as identifies which type of

338

Vivas, V. and Oliveira, M.

Which Tools Are Needed to Assist Audit Managers in Project Portfolio Selection When Divergent Views Emerge?.

DOI: 10.5220/0007374003380345

In Proceedings of the 8th International Conference on Operations Research and Enterprise Systems (ICORES 2019), pages 338-345

ISBN: 978-989-758-352-0

models may be developed. Final conclusions and

remarks are drawn in section 4.

2 REVIEW OF STUDIES

Taking a constructivist view in MCDA (Lee, 2012;

Peterson, 2012) that involves research is performed

with the collaboration of participants/evaluators and

accepts that “a decision situation is, in general, an ill-

defined entity, unclear even to the actors involved in

the decision process” (Bana e Costa and Pirlot,

1997), our review of studies started by searching

studies that report multicriteria evaluation models and

tools to evaluate projects and that use some type of

negotiation instrument to deal with divergent

opinions. Our search protocol focused on

combinations of several keywords – negotiation,

group decisions, conflict analysis, disagreement,

MCDA, PDA, integrative negotiation, auditing – in

the data sources B-on; Web of Science;

ScienceDirect; SCITEPRESS Digital Library.

Surprisingly, a small number of studies filled these

search criteria. These studies typically make use of

multicriteria resource allocation and of multicriteria

portfolio decision analysis models and explicitly

make use of some type of negotiation or consensus

building mechanism, which are key features for

models to inform the selection of projects when there

are scarce resources and participatory processes are

endorsed. Before introducing these studies, we

present key concepts.

2.1 MCDA and PDA

Here, we begin by highlighting the elementary

principles of building decision support models that

are key for multicriteria modelling. From the

perspective of MCDA, one can say that evaluation

models for decision support require three interactive

phases: (i) the structuring of the decision-making

context; (ii) the construction of evaluation model; and

(iii) impact assessment and analysis (Bana e Costa

and Beinat, 2005). The structuring phase requires

understanding the problem and the decision context.

To achieve this, a representation in the form of a

hierarchical value structure commonly named as a

value tree – accepted and negotiated by all

stakeholders – is constructed (Keeney, 1992). This

tree should represent, in an organized way, the

dimensions of values and key-concerns that are

relevant to the evaluation process and according to

which the options/projects/actions will be evaluated.

In the evaluation phase, a mathematical evaluation

model (most commonly an additive value model),

through which options are evaluated, is constructed.

The impact assessment phase is sought to provide

those developing the model with the analysis of the

consequences of implementing each one of the

options considered, with model adjustment and

validation procedures being also used (Bana e Costa

and Beinat, 2005).

In turn, in case of limited resources we are dealing

with portfolio problems and Portfolio Decision

Analysis (PDA) is applicable. According to (Salo et

al., 2011) PDA means “a body of theory, methods,

and practice which seeks to help decision makers

make informed multiple selections from a discrete set

of alternatives through mathematical modelling that

accounts for relevant constraints, preferences, and

uncertainties”. When one comes across a real

situation of audit project selection to compose a

portfolio, we need mechanisms to evaluate these

projects in multiple dimensions and this has been

done by MCDA literature – more specifically by

multicriteria portfolio decision analysis literature –

applied to real situations, whose models have a

potential to help building multicriteria models (Bana

e Costa, 2001; Bana e Costa et al., 2001; Mateus et

al., 2017; Oliveira et al., 2012) in the auditing context.

Since these models are already consolidated in the

literature we suggest making use of built-on-purpose

additive value models, similar to those presented in

these references, to evaluate the audit projects. It is

worth mentioning that under the presence of

divergent views, the models should reflect the

different opinions of the decision-makers' groups (or

stakeholders, or those involved in evaluation) and

make use of well-designed participatory processes

(Phillips and Bana e Costa, 2007); and that when one

identifies non-additive cases, models should be

restructured so that additivity conditions are

respected (having additive models the advantage of

being transparent and better understood by evaluators

and stakeholders).

Seeking for applications of MCDA in conflict

analysis contexts, various approaches and techniques

for dealing with divergent views have been proposed.

Bana e Costa (2001) explored the use of MCDA to

support the search for less conflicting policy options.

The author highlighted that public resource allocation

often requires the management of conflicting

objectives of multiple policy actors at different spatial

levels. The mix of limited financial resources,

multiple and conflicting concerns, spatial variability

of policy impacts and several types of uncertainty in

the data available for policy evaluation, make this

process problematic. In this case, conflict analysis

Which Tools Are Needed to Assist Audit Managers in Project Portfolio Selection When Divergent Views Emerge?

339

was based on the spatial analysis of the results of the

value model. Bana e Costa et al. (2001) presented a

case study of conflict dissolution in the public sector

through identification of the fundamental points of

view characterising the different value systems of the

stakeholders. The authors brought a pre-negotiation

approach aiming to dissolve the conflict in an

enlarged frame. Impact assessment revealed the

conflicting nature of the alternatives. The authors

then engaged the planners in a decision-analysis

process oriented towards the generation of win-win

solutions – to dissolve the intrinsic value conflict.

Losa and Belton (2006) brought an exploratory

application of an integrated approach, combining

MCDA and conflict analysis. They have integrated

Drama Theory and MCDA to model the situation

using confrontation analysis with the following

elements: characters, actions, positions and fallbacks,

and preferences. The resolution of the conflict

consisted of detailed analysis of the characters

positions, threats and dilemmas, through a

multicriteria evaluation of the different futures. Also

worth mentioning is the use of the MACBETH

(Measuring Attractiveness through a Category Based

Evaluation TecHnique) approach, that only requires

qualitative judgments about differences in

attractiveness to build a multicriteria value model

(Bana e Costa et al., 2012), and has been used for

consensus generation. Specifically, two studies

reported the use of a “MACBETH Voting” decision

support system to promote compromise in model

building. “MACBETH Voting” is characterised by

using the MACBETH (intuitive) qualitative

questioning protocol together with voting procedures

that potentiate convergence of views in a decision

conferencing environment in which key stakeholders

physically meet (Bana e Costa et al., 2014; Mateus et

al., 2017).

Mateus et al. (2017) describe a real-world

application of MCDA and related Decision Support

Systems (M-MACBETH, MACBETH Voting, and

Web-MACBETH) to support the engagement and

participation of a group of key players. Two

alternative multicriteria aggregation schemes were

applied in order to assist the group in evaluating the

added value and doability of the proposed actions.

New measures and methods to analyse the dominance

relationships between the actions were proposed,

further assisting the group in the priority selection of

the most effective and doable actions.

Fasth et al. (2016) presented a method based on

disagreement constrained action selection in

participatory Portfolio Decision Analysis. They

investigated the stakeholders' disagreement with

regard to each action, and how portfolios can be

generated that elucidate how conflicting preferences

affect the portfolio composition. Their method for

participatory PDA consisted on: eliciting stakeholder

preferences; measuring stakeholder disagreement;

disagreement constrained portfolio generation; and

sensitivity analysis.

Salo (1995) developed an interactive approach for

the aggregation of group members' preference

judgements and presented joint preference

representation in the form of value trees that conveys

areas of conflict and disagreement. Vilkkumaa et al.

(2014) described a multicriteria portfolio modelling

for the development of shared action agendas in

group decision and negotiation. When seeking for

consensual compromise solutions, non-dominated

portfolios with a high acceptability index are viable

candidates because they contain topics that are in the

core or borderline for many group members.

2.2 Negotiation

Moving towards studies focusing mainly on the

negotiation process, as emphasized by Vetschera

(2013), negotiations are one means of resolving

conflicts. Negotiation depends exclusively on the

parties involved and on their attempt to reach an

agreement that is acceptable to all parties. It can be

seen from a prescriptive-descriptive perspective,

where rationality of the negotiator are supported by

prescriptive theories such as game theory and, on the

other hand, actual human behavior is considered

(Raiffa, 1982). Therefore, a negotiation can be seen

as a process at the group level, in which those

involved influence each other and try to converge

toward some point of agreement. The author points

out that negotiation processes can be based on

concessions, in which each party begins from a

desirable position and over time reduces its demand

until a point considered satisfactory for both parties

and an agreement is found. Or the parties can start

from a solution which is not attractive to either party,

and jointly look for improvements, as in single

negotiation text (SNT) type of negotiation (Raiffa,

1982).

Keeney (1992) also suggested procedures for

empathetic negotiation within a value focused

thinking frame: “view the situation from the

perspective of other stakeholders; structure his

values as much as possible; begin by identifying the

negative impacts of your desired alternative relative

to the status quo in terms of his values; follow their

implications through a mean-end objectives network

to the fundamental objectives of the stakeholder;

ICORES 2019 - 8th International Conference on Operations Research and Enterprise Systems

340

create modified alternatives that can at least improve

matters in terms of these objectives while maintaining

the key consequences desired by you”. In the end, the

goal is to create an alternative that both parties win.

In an integrative negotiation process, as explained

by Sarabando et al. (2013), successful strategies

include cooperation, information sharing and joint

resolution of problems. Mediation and arbitration are

particularly useful in integrative negotiation, since

they can help negotiators to identify potential areas of

improvement for both sides. A value-based

evaluation model allows each party to evaluate their

potential own proposals, proposals made by the other

party, and their BATNA (best alternative to a

negotiated agreement).

Filzmoser and Vetschera (2008) highlighted the

bargaining process that can be seen as a sequence of

offers and often, formal models of negotiation

processes based on theories such as game theory or

decision analysis focus on the exchange of offers.

Greenhalgh and Chapman (1998) showed that

information sharing could facilitate joint gain because

negotiators disclose and learn about the interests of

each party, providing integrative bargaining.

Górecka et al. (2016) presented an approach in the

verbal and holistic evaluation of the negotiation

template to evaluating negotiation offers when the

negotiator’s preferences are expressed verbally.

Present Measuring Attractiveness near Reference

Situations (MARS) approach, these authors

combined the algorithms of two multiple criteria

decision making methods: ZAPROS and

MACBETH. They also suggested a pre-negotiation

preparation, with a negotiation template, designed

and evaluated by means of the negotiation offer

scoring system. The problem of evaluating the

negotiation template from an individual negotiator’s

viewpoint is similar to a decision-making problem

with multiple criteria involved and negotiation offer

scoring system was modelling as a simple additive

weighting (SAW) method.

2.3 Key Aspects from Reviewed Studies

Table 1 summarizes key aspects from the reviewed

studies that may be specifically useful to the design of

inform and tools to assist the selection of auditing

projects in the presence of divergent opinions and of

resource constraints.

From the review of studies, we observe that no study

has focused on the negotiation process for conflict

resolution in auditing portfolio management context,

and that there is space for developing new models and

tools in the area of multicriteria PDA.

Table 1: Key aspects from reviewed studies.

Reference

Field of

Knowledge

Area of study

application

Study features with special relevance for a negotiated

selection of auditing projects

Keeney (1992)

MCDA

Negotiation

Conceptual

examples

Concepts for an empathetic negotiation. Structuring values in

mean-end objectives network.

Salo (1995)

MCDA

Group decisions

Marketing and

production

Joint preference representation and dominance concepts. Value

tree

Greenhalgh and

Chapman (1998)

Integrative bargaining

Negotiation tactic

laboratory study

Cohesive relationships encourage information-sharing and

discourage use of coercive tactics. Integrative bargaining.

Bana e Costa

(2001)

MCDA

Resource Allocation

Public sector

(road-links)

Conflict analysis based on the spatial analysis of the results of the

value model. Structuring multicriteria resource allocation model.

Bana e Costa et al.

(2001)

MCDA

Transport

planning

Value systems of the stakeholders. Pre-negotiation Conflict

dissolution through ‘win-win’ compromise solution.

Losa and Belton

(2006)

MCDA

Group decisions

Analysis of

conflicts in a

social service

Conflict analysis.

Integrated use of Drama Theory and MCDA.

Filzmoser and

Vetschera (2008)

Bargaining process

Offers

Electronic

negotiations

Develop a typology of bargaining steps for multi-issue

negotiations

Vetschera (2013)

Negotiation

Offers

Conceptual

examples

Survey of process models of negotiations. Concession-based

negotiation. Improvement-based negotiation

Sarabando et al.

(2013)

Integrative

negotiation

Conceptual

examples

Integrative negotiation. Value-based evaluation model.

BATNA (best alternative to a negotiated agreement)

Vilkkumaa et al.

(2014)

PDA

Group decisions

Agenda building

(wood products)

Interactive decision process. Group members’ preferences

synthesized into shared priorities for action topics.

Fasth et al. (2016)

PDA

Urban planning

Disagreement constrained portfolio generation.

Górecka et al.

(2016)

Negotiation offer

scoring system

Conceptual

examples

Pre-negotiation: negotiation template, designed and evaluated by

means of the negotiation offer scoring system.

Mateus et al. (2017)

MCDA

Group decisions

Brownfield

Evaluating the added value and doability of the actions. Application

of MCDA and related DSSs to support the engagement and

participation of a group of key players.

Which Tools Are Needed to Assist Audit Managers in Project Portfolio Selection When Divergent Views Emerge?

341

3 WHICH TYPE OF CONCEPTS

AND TOOLS ARE USEFUL FOR

SELECTING AUDITING

PROJECTS?

Departing from the auditing context, our proposal for

systematizing useful models and tools is organized in

the structure shown in Figure 1: we starting by

describing the building of multicriteria portfolio

decision analysis models through a socio-technical

approach; we then describe the tools to display

information concerning auditing projects or

portfolios; we then focus on participants’ interaction

for negotiation; and we finally focus upon on relevant

negotiation approaches that one can follow.

Figure 1: Components for systematizing models and tools

to be used in the selection of auditing projects.

3.1 Multicriteria Portfolio Modelling

To build a multicriteria portfolio model (by following

a socio-technical approach (Phillips and Bana e

Costa, 2007)), one can start by building the value

trees, which structure the fundamental

dimensions/objectives of the problem and the criteria

to assess benefits of audit projects, according to the

view of each group of stakeholders (Bana e Costa,

2001; Bana e Costa et al., 2004). At this point, the

MACBETH approach (Bana e Costa et al., 2012),

supported by the DSS M-MACBETH, is an available

approach to assist the construction of a model to

evaluate auditing projects (entailing an intuitive

protocols and being compatible with the use of voting

and negotiation procedures).

The selection of audit projects can inherently

make use of multicriteria PDA models, as shown by

Oliveira et al. (2012). Adapting for the auditing

context, stakeholders involved in the evaluation

process must evaluate each audit project j to be

included in the portfolio, according to the group

model. The performance 



of each audit project  on

each benefit criterion  can be measured by a level in

the respective descriptor with partial value 







, in

which we have 











  and













 . Under an additive

structure, which requires the respect for mutual

independence conditions, the value of the overall

benefit 



of the project , can be determined as:

















 

























 



    





(1)

Moreover, in the case of the information on the

costs of audit projects are not available, one can make

use of the doability concept (Bana e Costa et al.,

2014) as proxy for audit cost. Doability can represent

the extent to which each audit project can effectively

be implemented, considering the limited resources

and financial, legal, social, and other constraints that

are beyond CGU managers’ control (Mateus et al.,

2017). The analysis of the divergences will be pointed

out from the results of the evaluations of the audit

projects in terms of benefits and doability according

to each model.

In view of maximizing the value of Equation (1),

the following optimization problem must be solved,

considering each project  has 



 and cost 



and

 is the total of available resources (as 



 , if the

project  is included in the portfolio and 

otherwise):













(2)













 













   

(3)

Additional constraints can be considered.

On the social component of portfolio modelling,

to involve and model the preferences of stakeholders

and/or decision-makers, depending on the context,

several formats can be adopted in model building. For

instance, semi-structured interviews, web

questionnaires, Delphi processes (Bowling, 2009), as

well as decision conferences that involve a face-to-

face meeting with key players (stakeholders and

ICORES 2019 - 8th International Conference on Operations Research and Enterprise Systems

342

experts) and an impartial facilitator, typically with the

support of MCDA tools (Phillips and Bana e Costa,

2007) can be adopted. Participants need to participate

in several modelling tasks that include the structuring

of a value model and the building of partial value

functions and of weighting scales.

3.2 Information Basis

From the reviewed studies, there are a set of tools that

can help auditing stakeholders to analyse projects or

portfolios, and to portray the consequences of

differences in opinion for an informed negotiation.

These tools include:

 Info 1: Evaluation of Benefits and Doability,

separately: analysis/view of the benefits and

doability of each project according to a group or

groups of stakeholders (Bana e Costa et al., 2014).

 Info 2: Benefit x Doability Graph: trading

benefit off against doability should drive the selection

of the best actions (Mateus et al., 2017). This graph

allows stakeholders to perceive the expected impact

and cost of each audit project, in the view of each

party involved in the negotiation. We also suggested

make use of Strategic Matrix, with 4 distinct

quadrants, include the ‘pearls’ in the portfolio, and

negotiate the ‘oyster’ and ‘bread and butter’ (Bana e

Costa et al., 2014).

 Info 3: Benefit/Effort Ratio: prioritise the audit

projects by their value-for-effort, defined as the ratio

between the benefit and the effort scores (Bana e

Costa et al., 2014). This information is useful to rank

the list of audit projects, allowing to perceive, in the

view of each party, the order of projects with the best

benefit / effort ratio.

 Info 4: Portfolios According to the Views of

Each Group: from the definition of the available

budget, it is possible to define which audit projects

will compose the portfolios according to the models

of each stakeholders group, which means different

portfolios of projects may be obtained. For instance,

analysis of the differences and implications according

to each auditing stakeholder group will allow the

identification of divergences should focus in

preparation for discussion and negotiation between

groups.

 Info 5: Dominance: dominance analysis can be

useful in negotiation as the dominance criterion is a

natural starting point in selecting proposals

(Sarabando et al., 2013). When seeking for

consensual compromise solutions, the examination of

core index values makes it possible to analyse the

non-dominated portfolios and provide relevant inputs

for group deliberations about viable candidates (Salo,

1995; Vilkkumaa et al., 2014).

3.3 Interaction between Participants

for Negotiation

Assuming that we have two groups of stakeholders

with divergent views, we must model the different

views of each group and how the groups may interact.

Figure 2 summarizes key aspects from a project

selection process that can help defining the audit

projects, for instance to compose the Operational Plan

at CGU. Each stakeholder group has representatives

who will define the projects of their teams.

Considering a group A, with visions according to

model G1, and a group B, with vision according to

model G2. Individually, each leader of these groups

must evaluate their projects to enter the Operational

Plan. From the results, differences are raised and the

space for negotiation is defined. The groups will need

to negotiate to reach the compromise solution. We

would thus have general models, based on a common

and transparent basis, and at the same time, space to

include the knowledge of each stakeholder, for

instance, an expert in the field, in the final decision

project selection.

Figure 2: Illustration of the negotiation space for the audit

project selection process.

3.4 Negotiation Strategies

As emphasized by (Górecka et al., 2016), a pre-

negotiation phase should be defined to establish a

detailed “vision of the negotiation problem, the

parties involved and the context and, after analysing

them, define a negotiation strategy that would allow

the party to obtain the goals”. Thus, the initial

negotiation template would be composed of the

projects that compose the sets of divergence. The

Which Tools Are Needed to Assist Audit Managers in Project Portfolio Selection When Divergent Views Emerge?

343

actual negotiation can start by an initial stage, in

which two offers are submitted at the negotiation

table, one by each of the parties. At each negotiation

stage one can observe the scale of differences that

needs to be eliminated to achieve a compromise

between the parties and what their endeavours are in

achieving the current negotiation status. Moments of

reverse concessions should be identified and, by

analysing the structure of the offers being presented,

competing issues can be identified and addressed

within negotiation.

In case of an integrative negotiation is adopted, it

is essential to identify potential areas of improvement

for both sides, on the exchange of offers, as also their

BATNA (best alternative to a negotiated agreement).

Thus, parties have to share information to facilitate

joint gain.

At the end, by following structured, transparent

and informed decision processes, it is expected that a

compromise solution will be reached and a final

portfolio defined.

4 CONCLUSIONS

The CGU in Brazil faces the challenge of, under the

presence of scarce resources, executing auditing

projects across distinct management and public

policies themes, involving multidisciplinary teams

and stakeholders, with different opinions and views.

We have shown in this study that portfolio resource

allocation models can be built to reflect the opinions

of distinct decision-maker and stakeholder groups,

and simultaneously auditing projects can be analysed

on a common and transparent basis. Negotiation tools

can be used to systematize the confrontation of

situations of divergence and conflict, and to search

converges toward some point of agreement. To

implement that socio-technical processes (Phillips

and Bana e Costa, 2007) must be considered, in which

methods, techniques and tools for model building are

intrinsically combined with participatory processes

involving stakeholders, decision-makers and/or

experts. By combining multicriteria methodology

with negotiation tools and techniques, it is possible to

build multicriteria resource allocation tools that

support the negotiation process in a shape of an

informed negotiation framework.

Thus, in this study we show multicriteria PDA

concepts and tools couples with negotiation strategies

that can be used to inform a transparent and

negotiated selection of audit projects; and that there

is relevance and scope for developing and testing

such type of models to assist the evaluation and

negotiation of auditing projects to integrate the

Operational Plan at CGU. A real case study is

currently being developed through a methodology

inspired and combining many of the concepts

introduced in this paper.

REFERENCES

Bana e Costa, C.A., 2001. The use of multi-criteria decision

analysis to support the search for less conflicting policy

options in a multi-actor context: case study. J. Multi-

Criteria Decis. Anal. 10, 111–125.

Bana e Costa, C.A., Antão da Silva, P., Nunes Correia, F.,

2004. Multicriteria Evaluation of Flood Control

Measures: The Case of Ribeira do Livramento. Water

Resour. Manag. 18, 263–283.

Bana e Costa, C.A., Beinat, E., 2005. Model-structuring in

public decision-aiding, LSEOR 05.79. Operational

Research working papers, London.

Bana e Costa, C.A., De Corte, J.-M., Vansnick, J.-C., 2012.

MACBETH. Int. J. Inf. Technol. Decis. Mak. 11, N° 2,

359–387.

Bana e Costa, C.A., Lourenço, J.C., Oliveira, M.D., Bana e

Costa, J.C., 2014. A Socio-technical Approach for

Group Decision Support in Public Strategic Planning:

The Pernambuco PPA Case. Gr. Decis. Negot. 23, 5–

29.

Bana e Costa, C.A., Nunes Da Silva, F., Vansnick, J.-C.,

2001. Conflict dissolution in the public sector: A case-

study. Eur. J. Oper. Res. 130, 388–401.

Bana e Costa, C.A., Pirlot, M., 1997. Thoughts on the

Future of the Multicriteria Field: Basic Convictions

and Outline for a General Methodology. In: Clímaco J.

(Eds) Multicriteria Analysis. Springer, Berlin,

Heidelberg.

Bowling, A., 2009. Research Methods in Health:

Investigating health and health services, Third Edit. ed,

McGraw Hill. Open University Press, Berkshire.

Fasth, T., Larsson, A., Kalinina, M., 2016. Disagreement

Constrained Action Selection in Participatory Portfolio

Decision Analysis. Int. J. Innov. Manag. Technol. 7, 1–

Filzmoser, M., Vetschera, R., 2008. A classification of

bargaining steps and their impact on negotiation

outcomes. Gr. Decis. Negot. 17, 421–443.

Górecka, D., Roszkowska, E., Wachowicz, T., 2016. The

MARS Approach in the Verbal and Holistic Evaluation

of the Negotiation Template. Gr. Decis. Negot. 25,

1097–1136.

Greenhalgh, L., Chapman, D.I., 1998. Negotiator

Relationships: Construct Measurement, and

Demonstration of Their Impact on the Process and

Outcomes of Negotiation. Gr. Decis. Negot. 7, 465–

489.

Keeney, R.L., 1992. Value-focused thinking: A Path to

Creative Decisionmaking. Harvard University Press,

Cambridge.

ICORES 2019 - 8th International Conference on Operations Research and Enterprise Systems

344

Lee, C.J.G., 2012. Reconsidering Constructivism in

Qualitative Research. Educ. Philos. Theory 44, 403–

412.

Losa, F.B., Belton, V., 2006. Combining MCDA and

conflict analysis: An exploratory application of an

integrated approach. J. Oper. Res. Soc. 57, 510–525.

Mateus, R.J.G., Bana e Costa, J.C., Matos, P.V., 2017.

Supporting Multicriteria Group Decisions with

MACBETH Tools : Selection of Sustainable

Brownfield Redevelopment Actions. Gr. Decis. Negot.

26, 495–521.

Oliveira, M.D., Rodrigues, T.C., Bana e Costa, C.A., Brito

de Sá, A., 2012. Prioritizing health care interventions:

A multicriteria resource allocation model to inform the

choice of community care programmes. Advanced

Decision Making Methods Applied to Health Care,

International Series in Operations Research &

Management Science. Springer.

Peterson, T.E., 2012. Constructivist pedagogy and

symbolism: Vico, cassirer, piaget, bateson. Educ.

Philos. Theory 44, 878–891.

Phillips, L.D., Bana e Costa, C.A., 2007. Transparent

prioritisation, budgeting and resource allocation with

multi-criteria decision analysis and decision

conferencing. Ann. Oper. Res. Springer Sci. 154, 51–

68.

Raiffa, H., 1982. The Art and Science of Negotiation.

Harvard University Press, Cambridge.

Salo, A., 1995. Interactive decision aiding for group

decision support. Eur. J. Oper. Res. 84, 134–149.

Salo, A., Keisler, J., Morton, A., 2011. Portfolio Decision

Analysis: Improved Methods for Resource Allocation,

International Series in Operations Research &

Management Science. Springer, New York.

Sarabando, P., Dias, L.C., Vetschera, R., 2013. Mediation

with Incomplete Information: Approaches to Suggest

Potential Agreements. Gr. Decis. Negot. 22, 561–597.

Vetschera, R., 2013. Negotiation processes: an integrated

perspective. EURO J. Decis. Process. 1, 135–164.

Vilkkumaa, E., Salo, A., Liesiö, J., 2014. Multicriteria

Portfolio Modeling for the Development of Shared

Action Agendas. Gr. Decis. Negot. 23, 49–70.

Which Tools Are Needed to Assist Audit Managers in Project Portfolio Selection When Divergent Views Emerge?

345