Fostering Collaboration on Decision Processes

Jorge Augusto Pessatto Mondadori

and Juliana de Melo Bezerra

SENAI, Curitiba, Brazil

Computer Science Department, ITA, Sao Jose dos Campos, Brazil

Keywords: Framework, Collaboration, Decision Process, Industry Case.

Abstract: Due to intrinsic complexity and uncertainty, decision problems require involvement of stakeholders with

distinct backgrounds and points of view. Collaboration among stakeholders is then essential to identify the

problem and find solutions. We propose a framework with guidelines to aid decision-makers, together with

a facilitator, to structure and solve a problem collaboratively in a virtual environment. The framework points

out how collaboration takes place during the whole decision process, including phases to structure the

problem, to apply the multi-criteria decision analyses, and to explore the sensitivity analyses in order to

reach the final result. We conducted an empirical evaluation, where decision makers reported benefits of the

framework to engage stakeholders, to congregate ideas, and to reduce the duration of the decision process.

1 INTRODUCTION

Decision making is a very important topic discussed

in operation research, including areas as

transportation, scheduling, routing, supplier

selection, and alternatives ranking. If the problem

involves finding the best solution or optimizing the

actual process, it is needed to identify the problem

itself, generate correct criteria, and formulate a

model to achieve the desired results (Rönnqvist,

2010). Decision-making processes are complex by

their nature, since they usually involve uncertainty,

multiple criteria, conflict of interests and distinct

stakeholders (Saaty, 2008). By stakeholders, we

mean people involved and affected by the problem,

such as community, employees, company owners,

clients, suppliers, governmental groups, and even

society.

A decision-making process starts with a problem

that stakeholders need to solve. It is needed to

structure the problem in order to know the main

objective and the ways to accomplish such objective

(Montibeller et al., 2006). The first phase of a

decision-making process involves the application of

Problem Structuring Methods (PSMs) to

systematically map and structure a situation to face.

The second phase refers to the Multi-Criteria

Decision Analysis (MCDA). MCDA aims to

evaluate options taking into account distinct

decision-makers with in general conflicting

perceptions and goals (Goodwin and Wright, 2014).

In a group decision-making process, it is needed

the collaboration of distinct stakeholders. Sometimes

decision makers are people geographically dispersed

or with restricted agenda, which makes difficult to

gather them collocated. So, it is needed to conduct

the decision process in a remote way. Most of the

time, a neutral facilitator moderates the actions of

decision makers to organize ideas. The facilitator

then needs to aggregate information and give

continuous feedback to decision makers. This

practice takes time to be conducted, and sometimes

it unintentionally loses or hides information that can

be valuable to the group (Forman and Peniwati,

1998; Kamel and Davison, 1998; Ho, 2008; Angiz et

al., 2011). In this way, it is of interest a way to aid

online collaboration among decision makers and

facilitators when performing decision processes.

In this work, we propose a framework to support

online collaboration in executing decision processes.

Decision makers can identify criteria and

alternatives through PSMs, and to solve the problem

using a MCDA. The main benefit of our proposal is

the possibility of online collaboration during the

entire decision process, without the need of being

united in the same place or time. Other advantage is

the real time visualization of information provided

by decision makers, which allows them to engage

and reach the identification of criteria and

Mondadori, J. and Bezerra, J.

Fostering Collaboration on Decision Processes.

DOI: 10.5220/0006662105690576

In Proceedings of the 20th International Conference on Enterprise Information Systems (ICEIS 2018), pages 569-576

ISBN: 978-989-758-298-1

569

alternatives with better quality. There is also the

possibility of collaboration considering the

sensitivity analysis. It allows decision makers to

view other scenarios and not only the result of the

MCDA method, which is important to select the

final decision.

We present a flux with the main activities of a

decision process; we identify which activities are

prone to collaboration and explain how online

collaboration can be performed. We also present a

prototype developed according to the guidelines

established in our proposal. In order to evaluate the

framework, we make an empirical application,

where a group uses the prototype to identify the

main activities to increase energy efficiency in the

food and beverage industries.

Section 2 presents our background. Section 3

describes the proposed framework to support online

collaboration in decision problems. Section 4

describes an empirical study to evaluate the

framework, by using a prototype. Finally, conc-

lusions and future work are presented in Section 5.

2 BACKGROUND

In this section, we discuss about problem structuring

methods, multi-criteria decision analysis, and

sensitivity analysis. We explain the concept of

online collaboration and detail how it is used to

support decision-making. We also present related

work.

Problem Structuring Methods (PSMs) are usually

applied in fuzzy situations where it is hard to

identify reasons and goals intended to be achieved.

PSMs help decision makers to think in a systemic

way, not only on alternative focused thinking. PSMs

are mostly categorized as soft operational research

methods, involving dialog and scenario

identification. Some examples are Value Focused

Thinking - VFT (Keeney, 1992) and Value Focused

Brainstorming - VFB (Keeney, 2012), Soft System

Methodology - SSM (Neves et al., 2009), Strategic

Choice Approach – SCA, and Strategic Options

Development and Analysis - SODA (Mingers and

Rosenhead, 2004). These methods are based on

qualitative and diagrammatic modeling, in a way

that they allow the exploration of distinctive views

and the incentive of active participation by

stakeholders. The goal of these methods is not only

optimization; instead the goal is to explore the

problem scope, to identify the uncertainties and to

get commitment of the stakeholders. In this paper,

we choose Value Focused Thinking (VFT) and

Value Focused Brainstorming (VFB) as the PSMs.

The reason is that such methods emphasize the

stakeholders’ values. Such values are related to the

way of acting and thinking, socially and ethically.

Regarding the Multi-Criteria Decision Analysis

(MCDA), there are several techniques to support it.

These techniques do not always give the optimum

solution, but they help to evaluate alternatives to

reach the given objective. Some examples are

SMART (Goodwin and Wright, 2014), Analytic

Hierarchy Process (AHP) (Saaty, 2008), MAUT,

MACBETH, PROMETHEE, ELECTRE family,

TOPSIS, DEA and Goal Programming (Ishizaka and

Nemery, 2013). In this paper, we choose AHP as the

MCDA technique. AHP has been used in different

purposes, including optimization, planning

improvements in industrial plants, and selection of

best alternative among several ones with different

characteristics. AHP is widely used because it is

practical and well accepted by various industrial

sectors. As result, AHP provides an ordered list of

alternatives based on the used criteria (Vaidya, O.,

and Kumar, 2006; Steuer, 2003). In order to provide

better accuracy in possible different scenarios, it is

also important to make a sensitivity analysis, to

visualize other possibilities of criteria weighting

(Jalao et al., 2014). The final decision of the

decision process is usually made considering the

MCDA output, but decision makers must take the

last decision.

Online collaboration can be useful to mitigate

group interaction problems such as those related to

time, cost, distance and space. Gathering people

together can be complicated due to incompatible

agendas or logistic costs. The geographical distance

between people and the space needed to allocate

people together are also limiting problems to

interaction. Regarding online collaboration in

decision-making processes, Fiedler et al. (2014)

developed a tool called SeSAM e-democracy. It is

an online platform for modern parliamentary work to

solve problems. It allows online public and private

meetings, supporting discussions and documents’

development.

According to Siskos and Spyridakos (1999),

systems that support decision-making processes

consist basically of three components: a database, a

core model and a communication system. The

database manages data provided by decision makers,

storing and processing information. The core model

includes the structure and algorithm of the method

used to aid multi-criteria decision. The

communication system is designed to support

communication between the decision makers. Users

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

570

communicate and cooperate to insert information in

database, so the software can calculate the final

results. It exemplifies the collaboration among

decision makers in decision-making processes.

Collaboration in decision-making processes also

happens due to the existence of facilitation.

Santanen et al. (2004) explain that the facilitator,

during a decision process, can help in cases of

divergence (when group disagree on ideas),

convergence (by identifying opportunities to

improve ideas), organization (by understanding the

relationships among concepts), evaluation (by

understanding priorities toward goal achievement),

and consensus building (by identifying opportunities

to have less disagreement on courses of action).

Some approaches aim to support MCDA, for

instance Super Decisions (2016), WBMCDM (Al-

Azab and Ayu, 2010), 123AHP (2016), EasyMind

(2016), Criterium Decision Plus (2016), Expert

Choice (2016), MakeItRational AHP (2016),

Decision Lens (2016), RightChoice DSS (2016), and

Questfox (2016).

Super Decisions is a desktop-software that

allows decision makers to solve problems using

ANP (Analytic Network Process) and AHP methods

in a collocated way. It provides sensitivity analysis

at the end of the calculations, however does not

allow making a group aggregation and structuring

the problem. The WBMCDM (Web Based Multi-

Criteria Decision Making) system is an open source

solution for solving AHP. It is a simple web

interface that allows only one decision maker to

insert criteria and alternative limited by 5 and 3,

respectively. The WBMCDM is very similar to

123AHP, which in turn is but limited since it does

not allow group interactions or problem structuring.

EasyMind is a web base system that can work

offline on web browser. The system does not allow

group aggregation neither problem structuring. It has

a non-collaborative sensitivity analysis. Criterium

Decision Plus is a desktop-software to solve AHP

and SMART, but it does not allow group evaluation.

Expert Choice, MakeItRational, Decision Lens,

RightChoice DSS and Questfox are desktop or web

based platforms to aid multi-criteria decision-

making using AHP. However, they do not allow

problem structuring in their interface, and their

sensitivity analysis does not support collaboration.

Limitations are found in current approaches

mainly due to problems regarding collaboration. The

approaches regarding MCDA do not deal with

Problem Structuring phase in group decisions, so

that there is no available way to discuss criteria and

alternatives, which in turn restricts collaboration

among decision makers. As decision-making

processes in groups need to calculate the aggregation

of individual judgments (AIJ), it is important that the

system calculate using the correct method of

aggregation. Some approaches are limited by not

providing collaboration during AIJ, since they focus

only in aggregating preferences. Usually sensitivity

analysis is made by a system that calculates

priorities aiming to show other points of view. In

this phase, it is also desirable collaboration in order

to improve understanding of MCDA results by

decision makers. Our proposal aims to fill the

identified gaps for the purpose of fostering

collaboration in decision problems.

3 A FRAMEWORK TO SUPPORT

ONLINE COLLABORATION IN

DECISION PROBLEMS

In this section, we propose a framework to help the

collaborative execution of a decision process in an

online environment. The framework is composed by

activities. We explain each activity and detail where

collaboration occurs. The proposed framework is

presented as a workflow in Figure 1.

The framework has two main phases. The first

phase is problem structuring, when decision makers,

aided by the facilitator, acting as a group, use VFT

and VFB methods to identify the main decision goal,

the main objective, the distinct criteria to be

considered in decision, and the possible decision

alternatives. The second phase is the multi-criteria

decision analysis, where APH is used as the MCDA

technique. After the two main phases, the decision-

making process continues with sensitivity analysis.

Finally, decision makers discuss the final decision

considering the output of AHP calculation. In the

framework, the white boxes with solid contour are

activities that need someone (decision makers or

facilitator) to be performed. The white boxes with

dashed lines are activities that can be made

automatically without intervention of facilitator and

decision makers. The gray shapes represent the

activities where collaboration occurs. All activities

are described in the sections below.

3.1 “Understand the Problem” Activity

The first step to solve multi-criteria problem is to

understand the problem itself. The idea here is to

have a brainstorming among decision makers, but

moderated by the facilitator. To start the activity, the

Fostering Collaboration on Decision Processes

571

Figure 1: Decision process flow with collaboration.

facilitator creates the users and respective passwords

to access the framework. The facilitator also defines

how long the activity is active and verifies such

deadline continuously.

In our framework, we use Value Focused

Brainstorming (VFB) as a method to understand the

problem. The values may vary from one decision

case to another. For example, in a problem involving

energy efficiency, the conscious use of electricity

can be a value. Considering an employee

management problem, the ethical actions of a person

can be values. Each organization has its values

associated with its mission (the purpose of the

organization) and vision (how the organization

wants to be seen in the future). The values are the

main principles that guide individual actions. It is

important that the decision makers and the facilitator

focus on the real problem, guided by the main

objective, with the values of the organization and

society, ethical alternatives and valuable criteria to

achieve the identified main goal. More details about

VFB can be found in Keeney (2012).

The collaboration occurs in a way that each

decision maker can give his opinion on an identified

value added by others. Giving grades, increasing

information or simply explaining other point of view

for the given information. Each information

provided by a decision maker is available for other

decision makers, who can evaluate (for instance,

with grades from 1 to 5, where larger grades indicate

more relevance) and make further comments.

According to the facilitator experience, in the

subject considered in the decision process, it is also

recommended to allow the facilitator to collaborate

as a decision maker. When the defined time is over,

the facilitator closes the activity to structure the

hierarchy on the next activity.

3.2 “Structure the Problem” Activity

After VFB application, the facilitator must organize

the submitted ideas using the VFT hierarchy.

Usually it can be made by clustering information in

order to obtain multiple criteria. The criteria are the

mean objectives that must be accomplished to

achieve the main goal. The main goal is the

fundamental objective in a VFT approach. The

alternatives are the possibilities to attend criteria to

achieve main goal. More details about VFT can be

found in Keeney (1992).

The structured problem can be viewed as

hierarchies of objective, criteria and alternatives.

The organized structure information, displayed in a

hierarchy chart, makes the decision makers to

understand the criteria and alternatives involved in

the decision process. The decision makers can then

collaboratively validate this structure, giving their

points of view on different organization of decision

tree.

3.3 “Define Criteria and Alternatives”

Activity

Once the VFT hierarchy is completed, the facilitator

can insert in the framework the criteria and

alternatives to provide a way to decision makers

evaluate them. It is a layout transformation of

information, the mean objectives are now seen as

criteria, and the possible alternatives to achieve each

mean objective are the multi-criteria alternatives.

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

572

3.4 “Evaluate Criteria and

Alternatives” Activity

Since criteria and alternatives are identified, they are

submitted to decision makers’ evaluation. Each

decision maker individually evaluates criteria and

later alternatives in a pairwise way, which means

that he/she compares two items at time. It is

important that the decision makers think about the

best solution for all involved stakeholders,

sometimes making tradeoffs of their own opinion. It

is made using the fundamental scale provided by the

AHP method, from 1 (equal importance) to 9

(extremely more important).

The collaboration between facilitator and

decision makers tends to avoid a future

inconsistency after information consolidation. It is

done by the verification of completed evaluations,

requested by the facilitator to decision makers. The

idea is to mitigate incoherencies by sharing ideas,

opinions and tradeoffs. The facilitator also verifies if

the evaluation is completed.

3.5 “Consolidate Evaluations” Activity

Once all decision makers provide their information

comparing criteria and alternatives, the evaluations

must be consolidated in order to create an

aggregated pairwise comparison matrix. This matrix

uses the AIJ method, as all decision makers desire to

solve the structured problem thinking on the best

solution for all instead of choosing its priorities.

Judgments are unified by calculating geometric

mean of decision makers’ evaluation of criteria and

alternatives, considered by each criterion.

3.6 “Calculate Result Decision” and

“Consistency Verification”

Activities

The AHP method can calculate the consolidated

matrix in order to provide the priorities vectors of

criteria, alternatives and global priorities. Other

important information, provided by AHP, is the

consistency of the matrix. The consistency is a rated

value to check if the information makes sense

through given evaluation. Details of calculations

related to APH method can be found in Saaty

(2008).

The facilitator compares the output of

consistency calculation. If the consistency

verification fails, the criteria and alternatives must

pass through other pairwise comparison by those

decision makers that evaluations caused

inconsistency. It is described on the “Solve

Inconsistency” activity. If the succeeds, the decision

result and global priorities are presented to decision

makers.

3.7 “Solve Inconsistency” Activity

If the consistency check gives a negative result, the

inconsistency must be solved. The facilitator

requests decision makers to review their evaluations

pointing out the problem, opening the inconsistent

evaluation made by decision makers stored in

database.

Sometimes the problem occurs after aggregation

of judgments, representing conflicting ideas between

decision makers. The facilitator must identify where

it happened in order to avoid other problem after the

evaluation of criteria. It may be made through his

experience with the specific methodology, in order

to bring the information to the consistent values and

not to change the evaluation drastically.

3.8 “Present Decision Result” and

“Sensitivity Analysis” Activity

The sensitivity analysis receives the ordered

(ranked) result of AHP. A graphic presents

intersection points of turnover of an alternative

priority to another. It can be seen different scenarios

for other situations of weighting criteria and possibly

a completely different selection of alternatives.

Details of calculations related to sensitivity analysis

can be found in Jalao et al. (2014).

The main idea is that the facilitator can provide a

textual interpretation of the data, and open for

discussion, since the result of AHP only points the

best solution for evaluated criteria and alternatives.

It can be hard to extract important data for those not

familiar with the output of the sensitivity analysis, so

providing information such as “alternative 1

turnovers 2 above 40% of criteria ‘x’ weight” is very

helpful.

3.9 “Make Final Decision” Activity

With the sensitivity analysis and ordered alternatives

by AHP, decision makers can work on solutions in a

collaborative way. Sometimes the decision group

can be controlled by other factors such political

force and investment policies. Therefore, decision

makers must be able to choose or even select a

reduced group of alternatives to solve the problem.

In addition, the MCDA method only helps to

organize and rank alternatives based on previous

Fostering Collaboration on Decision Processes

573

evaluations. It aids the analysis of multi-criteria

problems, but the decision group makes the final

decision. They may also freely discuss positive and

negative point of ranked alternatives.

4 EVALUATION

In this section, an empirical application is provided

to evaluate the proposed framework by using a

prototype. We choose energy efficiency problem in

our study, since it is a relevant issue for the industry

in all fields of production, for instance supply chain,

manufacture and agro industrial sectors. Nowadays,

research has been conducted to find alternative

sources of energy to power the scattered grids all

around the globe, in a way to increase energy

efficiency.

4.1 Prototype

In order to support groups for using the framework

in multi-criteria decision processes, a prototype was

developed. It is a web application on ASP.NET

MVC 4. The ASP.NET is a framework developed

for creating web pages. It helps web designers to

create connections between database storage system

and the visualization part of a web page, using MVC

(Model, View, and Controller) model.

A database storage system called SQL

(Structured Query Language) Server is used to store

user data and information provided by problem

structuring and pairwise comparison. The algorithm

for calculations is written in C# language. The

visualization part uses JavaScript, CSS (Cascading

Style Sheet), HTML and ASP.NET Razor.

4.2 Design of the Empirical Application

We worked together with an institution in our

country, which has energy efficiency solution as an

important service in its portfolio. The institution also

has strong interfaces with food and beverage

industries in the region where it operates. The

objective of the empirical application was to identify

common characteristics in local industries and

prioritize services to be offered, considering a fast

impact on energy efficiency issues.

We had six participants in the empirical

application. One participant was a consultant from

the institution and he act as facilitator. Five others

participants were decision makers. From two distinct

food and beverage industries, we had two production

managers and two maintenance managers. The other

participant was a consultant of energy topics. The

decision makers worked together using the

developed prototype.

The effectiveness of the framework was

evaluated through a set of questions before and after

the usage of prototype. The decision makers

answered questions about their experience,

considering the easiness to understand a problem,

the time taken in a decision-making process, the

easiness of engaging people; and the easiness of

expressing themselves. They used the five-point

Likert scale: 1 (poor), 2 (fair), 3 (average), 4 (good),

and 5 (excellent).

4.3 Results

On the problem structuring activity, decision makers

provided information and evaluated it in order to

generate criteria and alternatives. The values that

guided decision makers on information were

proposed by the facilitator: Productivity; Process

efficiency increase; Competitiveness; Environment

preservation; Workers security; and Compliance

with regulation. The resultant hierarchy structure for

this particular problem drive the identification of

criteria and alternatives described below.

During the multi-criteria decision analysis

activity, the decision makers then identified five

criteria: Energy management (how the industry uses

the energy to manufacture consumer goods),

Knowledge (the expertise on energy efficiency),

Energy quality (the quality of electrical energy

provided by the electrical energy company),

Equipment (how efficient equipments are in a

production line), and Processes (how efficient a

process is when consuming energy). Regarding the

alternatives, decision makers chose services that can

be offered to industries, including: Optimization and

Layout, Diagnostic, Automation and Refrigeration,

Maintenance and Retrofit, Regulation Attendance,

and Education. Considering the AHP results and the

sensitivity analysis, decision makers agreed that

Education services are the most needed by their

industries, followed by Optimization and Layout

Changes services.

Table 1 shows the results of evaluations made by

decision makers before and after using the

framework.

Regarding the easiness of criteria identification,

the evaluation shows an increase of grades. It shows

that, using the framework, decision makers could

identify the multiple criteria involved through a

discussion oriented by value-focused brainstorming.

Comparing the evaluation of time taken in a

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

574

Table1: Framework evaluation driven from empirical

application.

Topic

Before

framework

After

framework

Easiness of criteria

identification

1.6

4.4

Short time taken in

decision-making

2.6

3.8

Easiness of expressing

themselves inside

group

3.2

4.4

Easiness of alternative

identification

3.4

4.2

Easiness of engaging

team in decision-

making

1.8

4.8

Potential to use the

framework on daily

decision-making

4.8

*NA means not applicable

decision-making process, the framework allowed

decision makers to work collaboratively in a shorter

time. The easiness of expressing themselves inside

the decision group also increased, since decision

makers could give their opinion and make their own

evaluations during the decision process. The

easiness of alternative identification also increased

using the online framework. A positive impact on

decision-making process was related to the easiness

of engaging decision makers in the process, since

they were able to work in despite of time or location

difficulties. Moreover, decision makers evaluated

positively the possibility to use the framework in

other decision processes that they participate.

Decision makers also cited the main advantages of

using the framework, as seen: fast information input,

possibility to evaluate each point of decision,

possibility to generate innumerous alternatives,

individual evaluation of criteria and alternatives,

structured visualization of decision process,

remotely engagement of the team, and reduction of

time taken for the decision-making process.

5 CONCLUSIONS

Working with multi-criteria decision problems is

complex since many uncertainties are involved. We

proposed a framework to foster collaboration in

multi-criteria decision processes, and so to help

reaching better and complete decisions take into

account distinct perspectives of stakeholders.

The proposed framework considers a

collaborative problem structuring method to make

all the criteria and alternatives clear and available

for discussion. Decision makers then collaborate to

identify criteria and alternatives to the decision

process. Decision makers complete pairwise

comparisons, mediated by the facilitator, as part as

the multi-criteria decision analysis. If any

inconsistency in result occurs, it is provided a way to

facilitator and decision makers re-evaluate the

pairwise comparisons. Considering the sensitivity

analysis of the problem, facilitator interprets the

information for decision makers. Therefore, decision

makers discuss the result encouraged by online

collaboration, making the final choice of a problem.

The framework is useful to identify involved

criteria and alternatives in decision-making process;

it aids to solve multi-criteria problems of ranking

several alternatives when multiple criteria cause

uncertainty; it helps to visualize different scenarios

when the problem is sensitive to criteria weight

changes; and it allows discussion among decision

makers to make the final decision. Problems that

have such characteristics are related to ranking of

alternatives, selection of suppliers, identification of

best delivery route, selection of industrial machinery

and many operational research problems.

The developed prototype assures the viability of

the framework. It provides a collaborative interface

where decision makers, together with facilitator,

may structure a problem, analyze the multi-criteria,

interpret the sensitivity analysis and make final

decision. All the phases are implemented through

the guidelines available on the online framework.

To evaluate the prototype and therefore the

framework, an empirical application in energy

efficiency was conducted. Decision makers

evaluated their experience with decision processes

before and after using the framework. They indicate

improvements on these processes when using the

framework. All aspects were considered better when

using the framework, as seen: easiness of criteria

identification, short time taken in decision-making,

easiness of expressing themselves inside group,

easiness of alternative identification, and easiness of

engaging team in decision-making.

As future work, we aim to conduct empirical

applications in teams with more decision makers.

We intend to evaluate the priority of services on

other industry sectors, not only focusing in food and

beverage industries. It is also possible to incorporate

other methods during both the problem structuring

phase and the multi-criteria decision analysis.

Fostering Collaboration on Decision Processes

575

REFERENCES

123AHP MCDA AHP. Available at <http://

www.123ahp.com/OMetodi.aspx.> Accessed on April

13th, 2016.

Al-Azab, F. G. M., and Ayu, M. A. (2010). Web Based

Multi Criteria Decision Making Using AHP Method.

In Information and Communication Technology for

the Muslim World, 1, A1-A12, 2010.

Angiz, L. M. Z., Mustafa, A., Ghani, N. A., and Kamil, A.

A. Group Decision via Usage of Analytic Hierarchy

Process and Preference Aggregation Method. Sains

Malaysiana, 41(3), 361-366, 2011.

Criterium Decision Plus. Available at

<http://www.infoharvest.com/ihroot/infoharv/products

.asp > Accessed on April 13th, 2016.

Decision Lens. Available at <http://decisionlens.com/

secret-sauce/analytics.> Accessed on April 13th, 2016

EasyMind Web. Available at <http://community.

easymind.info/page-105.htm > Accessed on April

13th, 2016.

ExpertChoice Decision Software. Available at

<http://expertchoice.com/about-us/our-decision-

making-methodology/> Accessed on April 13th, 2016.

Fiedler, G., Krautz, U., and Thalheim, B. SeSAM –

Support on profile, portfolio, and demand for (e-)

parliamentarians. In e-Society, 2004.

Forman, E., and Peniwati, K. Aggregating individual

judgments and priorities with the analytic hierarchy

process. European Journal of Operational Research,

108 (2), 165-169, 1998.

Goodwin, P., and Wright, G. Decision Analysis for

Management Judgement. Chichester: Wiley, 5th ed,

2014.

Ho, W. Integrated analytic hierarchy process and its

applications – A literature review. European Journal

of Operational Research, 186 (1), 211-228, 2008.

Ishizaka, A., and Nemery, P. Multi-criteria decision

analysis: methods and software. Chichester: Wiley,

2013.

Jalao, E., Wu, T., and Shunk, D. A stochastic AHP

decision making methodology for imprecise

preferences. Information Sciences, 270, 192-203,

2014.

Kamel, N., and Davison, R. Applying CSCW technology

to overcome traditional barriers in group interactions.

Information & Management, 34 (4), 209-219, 1998.

Keeney, R. Value Focused Thinking. London: Harvard

University Press, 1992.

Keeney, R. Value-Focused Brainstorming. Decision

Analysis, 9 (4), 303-313, 2012.

MakeItRational AHP. Available at <http://

makeitrational.com/analytic-hierarchy-process/ahp-

software.> Accessed on April 13th, 2016.

Montibeller, G., Gummer, H., and Tumidei, D. Combining

scenario planning and multi‐ criteria decision analysis

in practice. Journal of Multi

‐

Criteria Decision

Analysis, 14(1‐ 3), 5-20, 2006.

Neves, L., Dias, L., Antunes, C., and Martins, A.

Structuring an MCDA model using SSM: A case study

in energy efficiency. European Journal of Operational

Research, 199 (3), 834-845, 2009.

Mingers, J., and Rosenhead, J. Problem structuring

methods in action. European Journal of Operational

Research, 152 (3), 530-554, 2004.

QuestFox. Available at <http://www.questfox.com/

application-area.> Accessed on April 13th, 2016.

RightChoice DSS Software. Available at <http://

www.modeladvisor.com/general_use/ms_office_tools/

project_management/rightchoice_dss/rightchoice_dss.

htm.> Accessed on April 13th, 2016.

Rönnqvist, M. OR challenges and experiences from

solving industrial applications. International

Transactions in Operational Research, 19, 227-251,

2010.

Saaty, T. (2008). Decision making with the analytic

hierarchy process. International Journal of Services

Sciences, 1 (1), 83-98, 2008.

Santanen, E. L., Briggs, R. O., and Vreede, G. De. Causal

relationships in creative problem solving: comparing

facilitation interventions for Ideation. Journal of

Management Information Systems, 20 (4), 167-198,

2004.

Siskos, Y., and Spyridakos, A. Intelligent multicriteria

decision support: Overview and perspectives.

European Journal of Operational Research, 113 (2),

236-246, 1999.

Steuer, R. E. Multiple criteria decision making combined

with finance: A categorized bibliographic study.

European Journal of Operational Research, 150, 496–

515, 2003.

Super Decisions Software. Available at <http://

www.superdecisions.com/>. Accessed on April 15th,

2016.

Vaidya, O., and Kumar, S. Analytic hierarchy process: An

overview of applications. European Journal of

Operational Research, 169 (1), 1-29, 2006.

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

576