Assessing Information Security Risks of AMI

What Makes it so Difﬁcult?

Inger Anne Tøndel, Maria B. Line and Gorm Johansen

SINTEF ICT, Trondheim, Norway

Keywords:

Advanced Metering Infrastructure, Information Security, Risk Assessment.

Abstract:

A rich selection of methods for information security risk assessments exist, but few studies evaluate how such

methods are used, their perceived ease-of-use, and whether additional support is needed. Distribution system

operators (DSOs) ﬁnd it difﬁcult to perform information security risk assessments of Advanced Metering

Infrastructure (AMI). We have performed a case study in order to identify these difﬁculties and the reasons for

them. Our ﬁndings indicate that the risk assessment method in itself is not the main challenge. The difﬁculties

regard competence; more speciﬁcally, insight in possible information security threats and vulnerabilities, being

able to foresee consequences, and making educated guesses about probability. Improved guidelines can be a

valuable aid, but including information security experts as participants in the process is even more important.

1 INTRODUCTION

Risk assessments are an essential part of the overall

work on information security

in organisations. Dur-

ing a risk assessment process an organisation’s key as-

sets are identiﬁed, potential threats and vulnerabilities

are evaluated, and the probability and consequences

that the assets are harmed in potential incidents are

estimated. Knowledge of which assets that need to be

protected and the level of threat experienced is essen-

tial in order to achieve cost-efﬁcient security.

Several research papers propose new and im-

proved information security risk assessment methods,

but few provide systematic evaluations of such meth-

ods or compare methods based on empirical studies

(Sulaman et al., 2013). Limited information is avail-

able on how organisations perform risk assessments

and their key challenges. This paper contributes to

improved understanding of the challenges faced for

one speciﬁc type of system: Advanced Metering In-

frastructure (AMI)

For Distribution System Operators (DSOs), the in-

troduction of AMI represents a shift in technology to-

wards increased use and dependence on Information

and Communication Technology (ICT). This shift

provides us the opportunity to study challenges that

DSOs face when introducing new information secu-

Information security concerns preservation of conﬁ-

dentiality, integrity, and availability (ISO/IEC, 2005).

AMI is also commonly referred to as smart meters.

rity and privacy risks in their risk assessments. Regu-

lations (NVE, 2013) require DSOs to secure their sys-

tems, including communication infrastructures, from

unauthorized access. Each DSO is advised to perform

a risk assessment of AMI (Skapalen and Jonassen,

2013). Even though the DSOs are used to perform

risk assessments in several areas, we experience that

they ask for assistance and aid in performing risk as-

sessments of AMI. Several of these DSOs are small

organizations with limited resources to perform these

sorts of activities.

As part of the national research project DeVID

we have developed a guideline for information secu-

rity risk assessments of AMI that describes the high-

level process and provides support material (Line

et al., 2013). This guideline builds on existing na-

tional and international risk assessment guidelines

and standards, and in particular a general risk assess-

ment guideline provided by NVE (NVE, 2010). Main

additions to this national guideline include:

• Emphasising information security and privacy as

a topic worthy of special attention, possibly in a

separate risk assessment.

• Recommending that assets are identiﬁed prior to

the identiﬁcation of unwanted incidents.

• Providing support material in form of checklists

DeVID is partly funded by the Research Coun-

cil of Norway, grant no 217528, http://www.sintef

.no/Projectweb/DeVID/

Tøndel I., Line M. and Johansen G..

Assessing Information Security Risks of AMI - What Makes it so Difﬁcult?.

DOI: 10.5220/0005228900560063

In Proceedings of the 1st International Conference on Information Systems Security and Privacy (ICISSP-2015), pages 56-63

ISBN: 978-989-758-081-9

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

that provide an overview of relevant assets, inci-

dents and countermeasures when it comes to in-

formation security and privacy in AMI.

To evaluate our guideline, we have studied risk as-

sessments performed by Norwegian DSOs. Our study

was motivated by the following research questions:

• RQ 1: Why do DSOs state that assessing risks in

AMI is difﬁcult, i.e. what makes AMI so different

from other objects that DSOs are already assess-

ing the risks of?

• RQ 2: How well does our guideline support the

DSOs when assessing the risks of AMI?

This paper is structured as follows. Section 2

presents related work on methods for risk assess-

ment. The research method for our study is de-

scribed in Section 3, and ﬁndings from the documen-

tation study, interviews, and participant-observations

are summarised in Sections 4, 5, and 6, respectively.

Section 7 discusses the results, and Section 8 provides

concluding remarks.

2 RISK ASSESSMENTS

A large number of standards, guidelines and research

papers suggest different methods for risk assessments.

Though they have their differences, the methods tend

to include similar steps: characterisation of the sys-

tem, threat and vulnerability assessment, risk deter-

mination, control identiﬁcation, and evaluation and

implementation of controls (Fenz and Ekelhart, 2011;

ISO/IEC, 2011a). There is however limited empir-

ical evidence regarding how the available standards

and guidelines are used and what kind of support is

considered most important. In a study by Jourdan et

al. (Jourdan et al., 2010) 25% of information secu-

rity professionals stated that risk analyses were actu-

ally never, or at best rarely, performed. Shedden et al.

(Shedden et al., 2010) concluded that the risk assess-

ment standards are not easy to use, and that they are

actually difﬁcult to comprehend and understand.

In 1999, the United States General Accounting

Ofﬁce (GAO) performed a study of information se-

curity risk assessment methods in four organisations

(GAO, 1999). These organisations had organisation-

wide information security risk procedures that were

considered practical and useful, and they had used

these procedures for at least one year. The study iden-

tiﬁed critical success factors for efﬁcient and effective

implementation of information security risk assess-

ment programs:

• Obtain senior management support and involve-

ment

• Designate focal points

• Deﬁne procedures

• Involve business and technical experts

• Hold business units responsible

• Limit scope of individual assessments

• Document and maintain results

The main challenges identiﬁed were related to es-

timation of likelihood and cost of information secu-

rity risks. It was claimed that it is more challeng-

ing to reliably assess information security risks than

other types of risks. This has to do with limited data

available, as well as constantly changing risk factors.

This challenge has also been put forward by other re-

searchers (Fenz and Ekelhart, 2011; Cybenko, 2006;

Gerber and von Solms, 2005; Rhee et al., 2012), who

added that information is an intangible asset where

it is “extremely difﬁcult if not impossible to deter-

mine precise value” (Gerber and von Solms, 2005),

and that the current situation is that “many losses

are never discovered and others are never reported”

(Rhee et al., 2012). Although this lack of informa-

tion was considered a challenge, the organisations in

the GAO study did not believe this to preclude under-

standing and ranking of information security risks.

3 METHOD

To address the research questions, we performed

a case study (Yin, 2009), where information was

collected through documentation, interviews and

participant-observation. This choice of research

method was guided by our goal to improve under-

standing of why information security risk assessments

are considered difﬁcult (RQ1). We aimed at opin-

ions and experiences that can improve future im-

provements in guidelines and other support initiatives

(RQ2), rather than statistically signiﬁcant results. We

compared the results from two different types of risk

assessments:

A. Risk assessments of AMI performed by the DSOs

before our study started without the support of our

guideline (assessments A1-A3). Process leaders

were interviewed.

B. Risk assessments where we took part as process

leaders and brought expertise to the assessments

by extensive use of our guideline (assessments

B1-B2).

AssessingInformationSecurityRisksofAMI-WhatMakesitsoDifficult?

Table 1: An overview of the risk assessments.

Case DSO Status of as-

sessment

Process

leader

Material

received

Role of interviewee Other participants in the

process

A1 1 In progress Internal Documented

risk matrices

Proj.mgr AMI/ pro-

cess leader

Communications, IT, IT

manager, AMI, consultants

A2 2 Completed Consultant Final report and

risk matrices

Proj.mgr AMI Technical, specialists, com-

mercial; AMI and IT

A3 3 Completed Internal Final report and

risk matrices

Grid analyses/ pro-

cess leader and in-

fosec ofﬁcer

AMI, IT

B1 4 Completed We (na) (na) IT, IT security, automation

systems, AMI

B2 3 Completed We (na) (na) IT security, AMI owner,

AMI responsible, technical

experts

3.1 Development of Guideline

The Norwegian Water Resources and Energy Di-

rectorate (NVE) have provided recommendations on

general risk assessments for the Norwegian energy

domain (NVE, 2010). They have organised the

risk assessment activities in three phases: The plan-

ning phase (system characterisation, determining the

scope, the likelihood scale, and consequence dimen-

sions; organising the risk assessment phase, preparing

checklists), the risk assessment phase (threat and vul-

nerability assessments and risk determination, control

identiﬁcation activities; presentation of results in a

risk matrix), and the risk treatment phase (selecting

controls and following up on implementation).

As these recommendations are already well-

established in the energy domain, we aimed at ex-

tending these rather than developing some brand new

and completely different methods. Therefore, our

guideline (Line et al., 2013) follows the same struc-

ture as this existing guideline from NVE, with one

important addition: We recommend to start the risk

assessment phase by identifying information assets.

This is motivated by recommendations in recognised

risk management guidelines in the information secu-

rity ﬁeld, such as OCTAVE Allegro (Caralli et al.,

2007). Furthermore, we provide explanations to sev-

eral of the activities recommended by NVE, speciﬁ-

cally tailored to information security needs. Check-

lists present possible system characterizations (for

AMI), information assets, incident categories (based

on ISO/IEC 27035 (ISO/IEC, 2011b)) and speciﬁc

incidents, stakeholders, vulnerabilities (based on NI-

STIR 7628 (Group, 2010)), and countermeasures

(based on ISO/IEC 27001 (ISO/IEC, 2005)). Our

guideline introduces methods for information security

threat modelling as well, but these were not evaluated

in the case study.

Our guideline has been developed with the aim of

supporting a lightweight risk assessment process that

does not require an extensive amount of resources in

time or personnel, as several Norwegian DSOs are

small and have limited resources available for this

type of activities.

3.2 Case Study Context

An overview of the different cases studied is shown in

Table 1. The four participating DSOs were recruited

from the DeVID project consortium

. The scope of

the risk assessments A1-A3 were the overall project

of implementing AMI. This includes ﬁnancial, opera-

tional, and customer aspects, in addition to informa-

tion security and privacy aspects, which then consti-

tutes just a minor part of the complete analysis. As-

sessments B1-B2 considered information security and

privacy risks in the AMI.

Note that assessment A1 was in progress. They

had identiﬁed several incidents, but the identiﬁcation

of incidents were not completed, and further assess-

ments of the risk associated with the incidents were

in some cases missing. Though assessment A2 was

considered to be ﬁnished, the accompanying docu-

ments speciﬁcally stated that further work included

assessing likelihood and consequences of incidents,

suggesting that the provided risk values are initial val-

ues only.

For assessments B1 and B2, expertise within IT,

IT security, automation systems, and AMI, partici-

pated from each DSO (3, 4, c.f. Table 1). After agree-

ing on the scope of the assessment and ensuring a

common understanding of the system, the participants

engaged in brainstorming sessions to identify assets,

One DSO contributed to two different assessments; A3

and B2. They had already performed one risk assessment

without the use of our guideline, and we were used as pro-

cess leaders in a second risk assessment with a different

scope than the ﬁrst one.

ICISSP2015-1stInternationalConferenceonInformationSystemsSecurityandPrivacy

stakeholders and incidents. The group then assessed

the risks of the identiﬁed incidents and discussed rel-

evant controls for some of the high-risk incidents. For

B1, some relevant controls were added by the partici-

pating researchers after the risk assessment session.

Around-the-table evaluations concluded each work-

shop, where each participant reported his opinion on

what was successful, what was difﬁcult and/or did not

succeed during the day. Two researchers participated

in both assessments

, and results of the assessments

as well as participant evaluations were documented

by the researchers.

4 FINDINGS FROM THE

DOCUMENTATION STUDY

In this section we provide an overview of the risks

assessments A1-A3 with respect to incident types, as-

sets covered, estimated risk and the controls identi-

ﬁed. Note that in the tables that provide an overview

of the incident types and controls, the results from the

assessments B1-B2 are also included to enable com-

parisons.

4.1 Types of Incidents Covered and

Level of Detail

Table 2 provides an overview of the incidents iden-

tiﬁed in all assessments, categorised according to

ISO/IEC 27035. In general, risks from a broad range

of incident categories are included. However, the risk

categories ”Technical failure”, ”Technical attack” and

”Compromise of information” receive the most atten-

tion. Note that several of the incidents have been

placed in more than one category, as for example an

information compromise can have a technical cause

due to failure or an active attack.

Some of the incidents represent lack of routines,

or weaknesses in routines, that do not necessarily lead

to information security breaches in the short term,

as Lack of overview of AMI components and Lack

of control of collected meter values are examples of.

The DSOs are worried that the complexity of systems

might lead to a lack of competence and overview that

can lead to incidents on a long term.

Most analyses (A1-A3, B2) are performed at the

strategic stage of the AMI, hence the DSOs do not

know yet what the system will look like. The threats

are in general at a high level (e.g. someone will hack

One researcher leading the process (assessment B1:

MBL; assessment B2: IAT) and one documenting the re-

sults (GJ).

something, or someone gets access to communica-

tion).

4.2 Types of Assets Covered

In general, assets are not documented, although in

some cases speciﬁc assets are mentioned in descrip-

tions of incidents:

• Meter values providing information about power

consumption, and used as a basis for invoice (C,

I, A

)

• The meter itself (I, A)

• The communication network (A)

• The HES and other systems at the DSO (I, A)

• Breaker commands (I, A)

• Meter updates (A)

Incidents can however impact other assets than

those that are speciﬁcally mentioned. As an exam-

ple, inability to communicate with meters impacts the

availability of meter values, but it also impacts the

ability to send commands, collect status information

and to perform software updates. In several cases, the

incident is not related to any speciﬁc asset in particu-

lar, e.g. New technology - a lot of startup problems.

4.3 Estimated Risk

In A3, consequences of incidents are assessed due to

their impact on reliability of supply, privacy, economy

and reputation. Probabilities are also assessed due to

predeﬁned criteria. In A1 and A2 however, it is not

clear which criteria are used in the assessment of con-

sequences and probabilities.

In A2, few incidents related to information secu-

rity and privacy are assessed to be of high risk. These

have to do with insufﬁcient quality in the equipment

delivered by vendors. In A3 several incidents are con-

sidered high risk when it comes to economy and repu-

tation impacts. These have to do with lack of commu-

nication, meter errors, immature metering technology,

lack of control with collected meter values, privacy

breaches, IT system failure, and loss of control over

control systems. In A1, some of the incidents are as-

signed a consequence and probability rating, but none

of the incidents are categorised as low, medium, or

high risk or put into any form of risk matrix.

4.4 Controls Identiﬁed

There are large differences in which controls are in-

cluded and how they are presented in the different

C: conﬁdentiality, I: integrity, A: availability

AssessingInformationSecurityRisksofAMI-WhatMakesitsoDifficult?

Table 2: Types of incidents identiﬁed in the risk assessments.

Category A1 A2 A3 B1 B2 Examples of incidents covered

Natural

disaster

1 1 0 0 0 Equipment damage due to extreme weather conditions

Social un-

rest

1 0 0 0 0 Equipment destroyed due to war activities

Physical

damage

2 2 1 1 1 Substation break-in; Customer destroys meter by accident; Customer

vandalises meter

Infrastructure

failure

5 2 2 3 0 Instability in communication (e.g. in the GPRS network); Lack of com-

munication between smart meter and HES; Power stability problems

Radiation

distur-

bance

2 0 0 0 0 Power instability problems; EMP

Technical

failure

7 3 9 2 2 Existing system cannot handle increased data load; Encryption reduces

performance of solution; Software updates corrupted; Meter failure; Me-

ter lifetime shorter than expected

Malware 1 1 0 1 0 Customer infects the AMI system with malware, e.g. to manipulate own

energy consumption values

Technical

attack

7 4 3 3 2 Software updates corrupted; HES is compromised; Meter is compro-

mised; Competitor attack on product/system/data; Erroneous deactiva-

tion of power (breaker command) for several customers

Breach of

rule

1 1 0 2 2 Insiders get access to systems; Insiders manipulate own energy con-

sumption values

Compromise

of func-

tions

2 2 4 2 3 Insiders get access to the system; Vendor misuses access rights to ma-

nipulate energy consumption or increase their competitive advantage;

Criminals pretend to be meter installers

Compromise

of infor-

mation

6 3 2 8 3 Manipulation of information during communication; Eavesdropping;

Customers manipulate own meter values; Wrong price information to

customer; Documentation, maps, passwords, keys astray

Harmful

contents

0 0 0 0 1 Reactivation of power (breaker command) causes equipment failure or

ﬁre at customer

Other 0 2 0 2 1 Lack of info on suppliers’ security and privacy practices; Customer does

not accept collected meter values

risk assessments. A1 seem to have a balance between

procedural (organisation and personnel) and techni-

cal (physical measures and infrastructure) controls. In

A2, controls are identiﬁed for all critical incidents, al-

though few technical measures are identiﬁed and they

were not precise (e.g. Addressed by the IT depart-

ment). We did not have access to A3’s list of controls,

only the ﬁgures in their summary, which shows that

procedural controls are given priority.

Table 3 shows the number of procedural and tech-

nical controls identiﬁed in the ﬁve risk assessments

respectively. Due to different presentations of the

controls in the different assessments, the numbers

are not comparable. However, the relation between

procedural and technical controls are comparable for

each assessment.

Table 3: Types of controls identiﬁed.

Category A1 A2 A3 B1 B2

Procedures 32 58 4 8 21

Technical 58 8 1 12 18

5 EXPERIENCES AS REPORTED

IN THE INTERVIEWS

5.1 Sources of Information

The guideline from NVE (NVE, 2010) was men-

tioned by A1 and A2 as being used during the risk as-

sessment process as a support, inspiration, and knowl-

edge base. Further, two different publicly available

risk assessments was considered in A1 to be useful

support on on how to systemise the process. A2 re-

ferred to a seminar on information security in AMI.

All three respondents (A1, A2, A3) reported having

used a general risk assessment for the introduction of

AMI from Energi Norge AS (EnergiNorgeAS, 2012)

as a basis.

A1 reported that there is a lack of information on

actual experiences with information security incidents

related to AMI. They hired an external consultant to-

wards the end to get help with identifying risks within

the area of IT and IT security.

ICISSP2015-1stInternationalConferenceonInformationSystemsSecurityandPrivacy

The external consultants leading the process in A2

used an extensive checklist to make sure they included

all necessary aspects. This checklist was perceived as

comprehensive and sufﬁcient. A2 stated that several

guidelines and related reports were published in the

industry during their process. This was perceived as

confusing, as the recommendations were vague and

uncoordinated.

A3 successfully relied on the knowledge of the

well-experienced personnel participating in the pro-

cess. External information sources were not used.

However, they are familiar with recent risk assess-

ment reports made publicly available and the guide-

line from the authorities.

5.2 Challenges Regarding Information

Security and Privacy

A2 described the area of information security and

privacy as extremely technological and difﬁcult to

fully grasp. Specialists are needed, who understand

the technical aspects and all possibilities regarding

vulnerabilities, attacks, and potential consequences.

Identifying incidents are not too difﬁcult; e.g. Some-

one hacks into the system, Breach of conﬁdentiality,

but ﬁnding appropriate countermeasures is difﬁcult

without having competent specialists in the team.

A1 responded that identifying all possible inci-

dents felt like a challenge, partly due to the fact that

AMI is new and there might be aspects that are difﬁ-

cult to foresee. They see that cooperating with other

DSOs in exchanging experiences on incidents could

be useful. Also, A1 found it challenging to esti-

mate probabilities and possible consequences of inci-

dents. The worst consequences might have extremely

low probability for occurring, which makes it hard to

know whether they should be included or not. Statis-

tics from previous incidents are not available, and es-

timating what are the real and realistic risks is per-

ceived as quite hard. The possibilities seem inﬁnite.

5.3 Needs for Support

A1 would like to have a kind of recipe which guides

them through the whole risk assessment process.

They also wish for information on realistic probabil-

ities and possible consequences, not only unwanted

incidents. Knowledge and understanding of appro-

priate countermeasures are needed; as the DSO speci-

ﬁes system requirements for the suppliers and leave to

the suppliers to describe how they address the require-

ments, the DSO is left with the challenge of consider-

ing the appropriateness of the suppliers’ solutions.

A2, on the other side, did not feel the need for any

other support than what is already available. They

have an internal risk assessment method and they feel

satisﬁed with the work they did, as they have speciﬁed

all requirements regarding information security and

privacy. The most important thing is to include the

right people in the process, those who have the right

competence and knowledge in order to identify where

the risks are and which countermeasures should be

implemented.

A3 asked for necessary and basic templates for

risk assessments. Bullet lists suggesting countermea-

sures would be useful in that the DSOs could make

selections based on their own needs.

6 PARTICIPANT OBSERVATIONS

In the risk assessments, several of the incidents iden-

tiﬁed (see Table 2) were considered to be of high risk.

When it comes to reliability of supply, B1 consid-

ered virus infection, unauthorised access to breaker

functionality and external control over network func-

tions to be main risks. In addition to unauthorised

control of breakers and other main systems, B2 con-

sidered ﬁrmware errors in breakers and damage (e.g.

due to ﬁre) to main systems to be of high risk. B1 as-

sessed privacy consequences of incidents, and identi-

ﬁed virus infections, as well as access to meter values

for individuals to be of high risk in this respect. B2

assessed consequences regarding economy and rep-

utation, and identiﬁed considered incidents regarding

leakage of personal information, and also meter hack-

ing and meter value manipulation to be important in

this respect, in addition to the incidents that were es-

timated as high risk when it comes to reliability of

supply.

In both B1 and B2 a large set of information assets

were identiﬁed, but the cases did make similar pri-

orities regarding the importance of the assets; meter

value (meter ID and corresponding customer), power

switch (possibility to affect power supply), and en-

cryption keys (encryption keys and passwords) were

given highest priority.

In the risk assessment process, we encountered

challenges regarding system documentation. There

was a need for assumptions as it was almost impos-

sible to be sure of all details in the system. For B2,

this was particularly a challenge, as the risk assess-

ment considered a future implementation of AMI. In

general, it is not a problem to make assumptions as

long as all assumptions are well documented and ac-

tions are agreed upon in order to investigate further

details after the workshop.

AssessingInformationSecurityRisksofAMI-WhatMakesitsoDifficult?

Initially, we planned to spend one day on each risk

assessment process. However, B1 showed that two

half days could be more appropriate. It was found

to be difﬁcult to identify all system properties in ad-

vance. The discussions in the meeting revealed as-

pects that should have been better investigated or doc-

umented. Further, the ﬁve hours assigned for the as-

sessment was found not to be enough. Most people

would ﬁnd it easier to spend two half work-days than

one whole day due to other pressing tasks. Also, hav-

ing two half days would give room for processing

some of the information before completing the work.

The checklists were used towards the end of each

brainstorming session to make sure that we included

all relevant issues. We did not present them to the par-

ticipants before the brainstormings; this could have

reduced the number and variety of issues identiﬁed

during the creative process. The use of checklists did

result in some additions to the lists of both assets and

incidents.

In B2, risks per consequence dimension were dis-

cussed in smaller groups before risks were agreed on

in a plenary discussion. This was considered valu-

able for the results as differing opinions became vis-

ible; the participants had made different assessments

of the threats and thus the discussion could focus on

these differences.

The feedback in the end of the workshops were

in general positive. However, several participants in-

dicated that the method should have been presented

more clearly. Although we explained it at the begin-

ning of the workshop to make all participants famil-

iar with the intended work ﬂow, we should have put

more effort into explaining each step as the workshop

evolved. They found it somewhat confusing to follow

the steps during the process, although they considered

the process leader as key to what they perceived as a

successful session. The participants reported that as-

sessing a future system, where the technical details

are yet unknown, was challenging. Furthermore, both

analyses considered AMI and adjacent ICT systems,

which was perceived to be a too wide focus for experts

familiar with parts of the system only. At the same

time, participants considered this multidisciplinarity

to be a strength.

7 DISCUSSION

We started out with the aim to investigate why risk

assessments of AMI related to information security

was considered difﬁcult (RQ1) and to study whether

the guideline we had developed would provide sup-

port to DSOs in this respect (RQ2). The interviews

performed regarding the assessments A1-A3 point to

lack of experiences, statistics and examples as a main

challenge. As a consequence, estimation of probabil-

ities is difﬁcult. Information security is also consid-

ered extremely technological and different to grasp

for non-experts. The fact that AMI is currently not

implemented in many of these DSOs is also a chal-

lenge in the assessments.

It is not possible to overcome all these challenges

with a guideline. As AMI is currently not widespread,

limited statistics are available. However, a guide-

line may be able to aid with examples and also in-

crease understanding of information security. The

provided checklists on assets, incidents, vulnerabili-

ties and controls could contribute to this. When com-

paring the results of the assessments performed with

and without the use of our guideline, it is however not

possible to state that the quality has improved with

use of the guideline. There is no indication that use of

our guideline led to coverage of more incident types.

There are high variations in what risks are considered

to be most important, but this is the case also with the

support of the guideline. Feedback received from the

participants in the assessments B1-B2 were in gen-

eral positive. A focus on assets was considered useful.

Further, the use of checklists did result in additional

incidents and assets being identiﬁed in the meetings.

However, the positive feedback may also be due to

having an external facilitator rather than the content

of the guideline.

7.1 Systematic Approach and need for

Experts

A1-A3 show that the DSOs have tried to identify main

privacy and information security issues, but they seem

to lack systematic ways to approach this area, e.g. in

the form of asset identiﬁcation or the use of threat

categories. In A2 they used external documents and

a risk map to make sure all things are covered, and

felt conﬁdent as this support was provided by experi-

enced consultants. For the others, a more systematic

approach could potentially increase conﬁdence that

the most important threats have been covered. We re-

ceived positive feedback on our suggested focus on

assets.

Personnel who know the technical details of the

systems, and possible threats and their corresponding

potential consequences, are invaluable to an organi-

sation‘s risk assessment process. For non-experts it

is almost impossible to make realistic assumptions on

likelihood and consequences.

ICISSP2015-1stInternationalConferenceonInformationSystemsSecurityandPrivacy

7.2 Limitations

Our study of A1-A3 comprised a documentation

study and interviews, but we still do not know de-

tails about the discussions and the priorities made. We

might do the assessments injustice by making state-

ments about whether or not the important risks have

been identiﬁed and assessed.

Participant observation poses the risk of bias due

to the researcher’s active role in the process. As au-

thors of the guideline that was used as support, we

might use it differently than an independent process

leader, and we might be too supportive of our own

suggestions.

8 CONCLUDING REMARKS AND

FUTURE WORK

This case study shows that the DSOs need support for

their information security risk assessments, as they

experience challenges related to competence and un-

derstanding of information security issues. Check-

lists and a focus on assets may help in this respect.

However, more studies are needed in order to iden-

tify what type of support will signiﬁcantly ease the

performing of risk assessments by DSOs. We do not

have a sufﬁcient base for claiming that the use of our

guideline will increase the quality of risk assessments.

Feedback from the participants indicated that they ap-

preciated the process of identifying assets before con-

sidering threats and vulnerabilities, and they felt that

our checklists added value to the process as well. We

would like to stress that our guideline does not present

a new method, but the Norwegian energy industry

does not have traditions for including asset identiﬁca-

tion as part of their risk assessments, and this may be a

valuable approach when information security and pri-

vacy constitute the main focus for these assessments.

Irrespective of availability of guidelines or other

types of support material, it is still important that each

organisation perform their own assessments based on

their speciﬁc systems and priorities. Thus the compe-

tence of the participants in an assessment is likely to

be more important than any guideline support.

ACKNOWLEDGEMENT

The authors would like to thank the DSOs that partic-

ipated in our study.

REFERENCES

Caralli, R. A., Stevens, J. F., Young, L. R., and Wilson,

W. R. (2007). The OCTAVE Allegro Guidebook v1.0.

Software Engineering Institute.

Cybenko, G. (2006). Why Johnny Can’t Evaluate Security

Risk. IEEE Security & Privacy, 4(1):5.

EnergiNorgeAS (2012). Overordnet risiko-og

arbarhetsanalyse for innføring av AMS. PT-

1070549-RE-01.

Fenz, S. and Ekelhart, A. (2011). Veriﬁcation, Validation,

and Evaluation in Information Security Risk Manage-

ment. IEEE Security & Privacy, 9(2):58–65.

GAO (1999). Information Security Risk Assessment: Prac-

tices of Leading Organizations. United States General

Accounting Ofﬁce (GAO).

Gerber, M. and von Solms, R. (2005). Management of

risk in the information age. Computers & Security,

24(1):16 – 30.

Group, T. S. G. I. P. C. S. W. (2010). Guidelines for smart

grid cyber security.

ISO/IEC (2005). ISO/IEC 27001:2005 Information security

management systems - Requirements.

ISO/IEC (2011a). ISO/IEC 27005:2011 Information tech-

nology - Security techniques - Information security

risk management.

ISO/IEC (2011b). ISO/IEC 27035:2011 Information tech-

nology - Security techniques - Information security in-

cident management.

Jourdan, Z., Rainer, K., Marshall, T. E., and Ford, N.

(2010). An Investigation of Organizational Informa-

tion Security Risk Analysis. Journal of Service Sci-

ence, 3(2):33–42.

Line, M. B., Tøndel, I. A., Johansen, G. I., and Sæle, H.

(2013). Informasjonssikkerhet og personvern: Støtte

til risikoanalyse av AMS og tilgrensende systemer

(Norw.). Technical Report A24258, SINTEF. ISBN

978-8-214-053203.

NVE (2010). Veiledning i risiko- og s

arbarhetsanalyser for

kraftforsyningen (in Norwegian). Norwegian Water

Resources and Energy Directorate.

NVE (2013). FOR 1999-03-11 nr 301: Forskrift om m

aling,

avregning og samordnet opptreden ved kraftomset-

ning og fakturering av nettjenester.

Rhee, H.-S., Ryu, Y. U., and Kim, C.-T. (2012). Unreal-

istic optimism on information security management.

Computers & Security, 31(2):221 – 232.

Shedden, P., Ruighaver, A. B., and Ahmad, A. (2010). Risk

Management Standards - The Perception of Ease of

Use. Journal of Information System Security, 6(3):23–

41.

Skapalen, F. and Jonassen, B. (2013). Veileder til sikkerhet

i AMS (in Norw.). NVE.

Sulaman, S. M., Weyns, K., and H

ost, M. (2013). A review

of research on risk analysis methods for IT systems. In

Proceedings of the 17th International Conference on

Evaluation and Assessment in Software Engineering,

EASE ’13, pages 86–96, New York, NY, USA. ACM.

Yin, R. K. (2009). Case Study Research - Design and Meth-

ods, 4th ed., volume 5 of Applied Social Research

Methods. SAGE Publications.

AssessingInformationSecurityRisksofAMI-WhatMakesitsoDifficult?