Findability through Traceability

A Realistic Application of Candidate Trace Links?

Markus Borg

Dept. of Computer Science, Lund University, Lund, Sweden

Keywords:

Traceability, Impact Analysis, Information Seeking, Findability, Human Computer Interaction, Automation.

Abstract:

Since software development is of a dynamic nature, the impact analysis is an inevitable work task. Traceability

is known as one factor that supports this task, and several researchers have proposed traceability recovery tools

to propose trace links in an existing system. However, these semi-automatic tools have not yet proven useful

in industrial applications. Based on an established automation model, we analyzed the potential value of

such a tool. We based our analysis on a pilot case study of an impact analysis process in a safety-critical

development context, and argue that traceability recovery should be considered an investment in ﬁndability.

Moreover, several risks involved in an increased level of impact analysis automation are already plaguing the

state-of-practice work ﬂow. Consequently, deploying a traceability recovery tool involves a lower degree of

change than has previously been acknowledged.

1 INTRODUCTION

Change is an inherent characteristic of the evolu-

tion of large software systems. Consequently, impact

analysis, the process of determining possible effects

of proposed software changes, is an inevitable work

task. Conducting an impact analysis is often a labor-

intensive manual process, avoided unless absolutely

necessary (Bohner, 2002). However, in development

projects governed by safety regulations, impact anal-

ysis is a fundamental part of the development process,

necessary for safety certiﬁcation of products (IEC,

2003).

The impact analysis work task involves a high de-

gree of information seeking, an increasingly costly

activity among knowledge workers in general (Karr-

Wisniewski and Lu, 2010). Previous studies have

identiﬁed this issue also in software engineering

projects (Olsson, 2002; Sabaliauskaite et al., 2010).

As a large software development project constitutes

a complex information landscape (i.e., thousands of

artifacts such as requirements, source ﬁles, test cases

and user manuals), analysing change impact is a chal-

lenging task. Thus, an important aspect of a devel-

opment project is the ﬁndability it offers, deﬁned as

“the degree to which a system or environment sup-

ports navigation and retrieval” (Morville, 2005).

One way to support the information seeking is to

maintain traceability, deﬁned as “the degree to which

artifacts are related” (IEEE Computer Society, 1990).

It is widely recognized as an important factor for ef-

ﬁcient software development (Antoniol et al., 2002;

Domges and Pohl, 1998). However, maintaining trace

links in an evolving system is a tedious task. To

support this activity, several researchers have pro-

posed traceability recovery (i.e., proposing trace links

among existing artifacts) based on Information Re-

trieval (IR) approaches (Antoniol et al., 2002; Marcus

and Maletic, 2003). However, despite numerous re-

lated publications during the last decade, success sto-

ries in industrial settings are conspicuously few (Borg

et al., 2012). This makes us believe that the general

expectations on the approach are too high, and that the

tools should be considered from a new perspective.

The typical functionality of traceability recovery

tools is to present the user a ranked list of candidate

trace links, who then gets to vet the output (De Lu-

cia et al., 2012). Since the human still is expected

in the process, tools of this kind often claim to pro-

vide “semi-automatic” support. As automation is de-

ﬁned as “a device or system that accomplishes (par-

tially or fully) a function that was previously, or con-

ceivably could be, carried out (partially or fully) by a

human” (Parasuraman et al., 2000), one might won-

der to what level the tools actually automate human

work. This leads us to discuss ﬁndability around the

following research questions:

• RQ1: What type and level of automation do state-

173

Borg M..

Findability through Traceability - A Realistic Application of Candidate Trace Links?.

DOI: 10.5220/0004093501730181

In Proceedings of the 7th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE-2012), pages 173-181

ISBN: 978-989-8565-13-6

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

of-the-art (SoA) traceability recovery tools offer?

• RQ2: How would the introduction of a SoA trace-

ability recovery tool change the state-of-practice

(SoP) impact analysis work ﬂow?

To tackle these questions, we conducted a pilot

case study of an impact analysis process in a safety-

critical development context, and assessed how SoA

tool support could be applied. To structure our anal-

ysis, we applied the SWELL Automation Analysis

Framework (SAAF), developed as an extension to

an automation model initially proposed by Parasur-

aman et al. (Parasuraman et al., 2000). Based on our

ﬁndings, we argue that proposed traceability recovery

tools primarily should be considered as a step towards

improved ﬁndability, rather than as an attempt to gen-

erate a full set of traces.

This paper is organized as follows: Section 2

presents work related to IR-based traceability recov-

ery and automation analyses. Section 3 describes our

case study and SAAF. Section 4 reports the outcome

of our analysis, and ﬁnally Section 5 concludes and

outlines future work.

2 RELATED WORK

2.1 IR-based Traceability Recovery

Several researchers have proposed expressing trace-

ability recovery as an IR problem. Most developed

traceability recovery tools implement standard IR

techniques based on algebraic or probabilistic mod-

els (Antoniol et al., 2002; De Lucia et al., 2005; Mar-

cus and Maletic, 2003). In such tools, the answer to

a query is a ranked list of artifact suggestions, sorted

by the level of calculated similarity (algebraic mod-

els), or probability that they are related (probabilistic

models). The ranked list is analogous to the output of

web search engines and enterprise search tools. Con-

sequently, search results can be either relevant or non-

relevant to the information need of the speciﬁc user.

A number of traceability recovery tools were de-

veloped as plug-ins. Klock et al. have developed

Traceclipse, supporting trace link recovery and man-

agement within Eclipse (Klock et al., 2011). They

developed Traceclipse to be expandable, to simplify

meeting future feature requests and to easily sup-

port other IR models. The functionality of the plug-

in was initially evaluated, however only tool out-

put was considered rather than human-tool interac-

tion. Canfora and Cerulo developed Jimpa, another

traceability recovery plug-in for Eclipse (Canfora and

Cerulo, 2006). They implemented probabilistic re-

trieval to establish links between change requests and

source code, and evaluated the approach on three open

source systems. De Lucia et al. developed their own

Document Management Systen (DocMS), ADAMS,

and the IR-based traceability recovery plug-in Re-

Trace (De Lucia et al., 2005). Furthermore, they have

evaluated their plug-in in studies with student sub-

jects (De Lucia et al., 2009). Falessi et al. imple-

mented a plug-in, PROUD, to the industrial CASE

tool Enterprise Architect (Falessi and Briand, 2009),

and evaluated it in a controlled experiment with stu-

dents and in an industrial case study. We, on the

other hand, have proposed developing a traceability

recovery plug-in to HP Quality Center (Borg, 2011a;

Borg, 2011b). Developing plug-ins to tools already

deployed in industry enables in-vivo studies without

introducing additional external tools.

2.2 Automation Analysis

Several taxonomies and frameworks have been devel-

oped to support the analysis of automation. A com-

prehensive overview, however from the viewpoint of

manufacturing, was recently presented by Frohm et

al. (Frohm et al., 2008). In their work, they re-

port eight different deﬁnitions of “levels of automa-

tion”. Sheridan and Verplank developed a 10-level

taxonomy for automation levels in 1980 (Sheridan

and Verplank, 1978). Billings studied automation in

the context of air-trafﬁc controllers (Billings, 1997).

His work explicitly separated automation and human

functions, and deﬁned a continuum of management

modes from unassisted control to fully autonomous

operations. Parasuraman et al. developed a model

incorporating both the type and the level of automa-

tion (Parasuraman et al., 2000). We extend this model

by adding two preceding and one subsequent analy-

sis phases, described in detail in Section 4.3. The

model by Parasuraman et al. was previously ap-

plied in human-computer interaction research to ana-

lyze adaptive automation solutions for air-trafﬁc con-

trol (Clamann et al., 2002). However, to the best of

our knowledge, it has not been applied to analyze soft-

ware engineering tools.

Huffman Hayes et al. touched upon the automa-

tion questions for traceability recovery in a techni-

cal report (Huffman Hayes et al., 2006), in which

they base the discussions on their long experience of

hands-on traceability activities in industry. Further-

more, they have published several studies on how en-

gineers should work with semi-automatic tool sup-

port, including how humans interact with tools in

the traceability loop (Huffman Hayes and Dekhtyar,

ENASE2012-7thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

174

Figure 1: Overview of the automation analysis. The box

represents SAAF.

2005). However, the automation analysis is not the

central part of their publications, which motivated our

inquiry.

3 METHOD

To concretize our discussion on automation, we ap-

plied our automation analysis on a speciﬁc case in a

safety-critical development context. An overview of

SAAF, the framework used for the automation analy-

sis, is presented in Figure 1. It is based on the model

by Parasuraman et al. (Parasuraman et al., 2000) men-

tioned in Section 2.2. Our understanding of tool sup-

port for traceability recovery originates from an ex-

tensive literature review, and the outcome of this au-

tomation analysis is intended to guide our future tool

developing efforts.

3.1 Safety-critical Impact Analysis - A

Pilot Study

To better understand how traceability recovery can

support the impact analysis process, we developed an

initial model of the inherent information seeking ac-

tivity based on our industrial experiences. To vali-

date the model, we presented it to three software en-

gineers from the case company. We communicated

primarily via e-mail, and the respondents were se-

lected using convenience sampling. However, to im-

prove generalizability, we selected respondents repre-

senting three different development teams from two

different departments. Based on the feedback re-

ceived, we reﬁned the model to the version presented

in Section 4.2. Furthermore, we asked the respon-

dents about their views on risks involved in increasing

automation in the impact analysis process.

3.2 Phases of SAAF

The ﬁrst phase, Preliminaries, establishes the focus,

i.e., the scope of activities affected as well as effect

targets, for the automation effort within a context.

Also, the phase clariﬁes any assumptions taken and

what is included in the analysis. The three steps of

this phase describe: Context of the automation, Scope

of the automation, and Effect targets.

Automation change identiﬁcation, the second

phase of SAAF, describes pre- and post-automation

task ﬂows. This phase should specify which work

tasks are changed, added or removed as a conse-

quence of automation. The three steps of this phase

describe: Pre-automation work ﬂow, Post-automation

work ﬂow, and Changed/Added/Removed tasks.

Automation classiﬁcation, third phase, analyses

automation according to the model by Parasuraman

et al. (Parasuraman et al., 2000). The object of the

analysis is both the pre- and post-automation work

ﬂows. The two steps of this phase comprise analy-

sis of: Types of automation, and Levels of automation

(presented in Table 1 and Figure 3).

The ﬁnal phase, Automation impact analysis, es-

timates both direct and indirect effects of the in-

creased automation. Since automated solutions typ-

ically bring both positive and negative effects, under-

standing them prior to implementing any changes is

essential. Our analysis targets threats involved in the

actual automation, and also its cognitive side-effects.

The ﬁnal phase of SAAF also includes a break-even

analysis, a parametric assessment of beneﬁts, where

the parameter values are selected to equate costs and

beneﬁts. It is one of the methods that evolved to quan-

tify beneﬁts of information systems (Sassone, 1988).

Fixed costs and costs dependant on volume are com-

pared to determine the volume at which an automa-

tion investment results in neither a proﬁt nor a loss,

the so called breakpoint. We base discussions on the

feasibility of traceability recovery for impact analysis

on such an initial analysis, and report: Direct effects,

Indirect effects, and Existing evidence for effects.

4 RESULTS AND DISCUSSION

This section describes the results from applying

SAAF. Every phase is concluded by our ﬁnal assess-

ment, expressed using the levels presented in Table 1.

FindabilitythroughTraceability-ARealisticApplicationofCandidateTraceLinks?

175

Table 1: Summary of types and levels of automation (Parasuraman et al., 2000).

Type Level Description

Information 10 Autonomous, ignoring human.

acquisition 9 May inform human.

⇓ 8 Informs human if asked.

Information 7 Executes automatically, then informs.

analysis 6 Allows human time to veto.

⇓ 5 Executes if human approves.

Decision 4 Suggests one alternative.

selection 3 Narrows down selection to a few.

⇓ 2 Offers complete set of alternatives.

Action 1 No assistance at all.

implementation

4.1 Preliminaries

4.1.1 Description of the Context

The analyzed impact analysis process originates from

a large multinational company active in the power

and automation sector. The development context

is safety-critical embedded development in the do-

main of industrial control systems, governed by IEC

61511 (IEC, 2003). The number of developers is in

the magnitude of hundreds; a project has typically a

length of 12-18 months and follows an iterative stage-

gate project management model. The software is cer-

tiﬁed to a Safety Integrity Level (SIL) of 2 as de-

ﬁned by IEC 61508 (IEC, 2010), corresponding to

a risk reduction factor of 1,000,000-10,000,000 for

continuous operation. There are process requirements

on the maintenance of traceability information, es-

pecially between requirements and test cases. Both

requirements and test case descriptions are predomi-

nantly speciﬁed in English natural language text.

4.1.2 Description of the Work Task

As speciﬁed in IEC 61511 (IEC, 2003), impact of

proposed software changes should be analyzed be-

fore implementation. In the studied case, this pro-

cess is tightly integrated in the Defect Management

System (DefMS). The issues in the DefMS, i.e., de-

fect reports and change requests, are administered by

a change control board. The board distributes issues

to responsible teams for investigation. As part of the

investigation, developers are required to perform an

impact analysis, and report their results according to a

project speciﬁc template. The template, developed by

internal safety engineers and validated by an external

certifying agency, contains between 5-20 questions

depending on the SIL of the affected software compo-

nents. In the template, several questions explicitly ask

for trace links. The developer is required to specify

source code that will be modiﬁed (with a class-level

granularity), and also which related software artifacts

need to be updated to reﬂect the changes, e.g., re-

quirement speciﬁcations, design documentation, test

case descriptions, test scripts and user manuals. Fur-

thermore, the report should specify which high-level

system requirements cover the involved features, and

which test cases should be executed to verify that the

changes are correct once implemented in the system.

The test case selection should cover both developer-

centric functional testing, and system testing con-

ducted by the test organization.

4.1.3 Effect Targets

The intention of increased automation is to make the

impact analysis faster and more accurate. Also, one

engineer explained “It is important to reduce the num-

ber of mundane questions to save the effort for the

ones requiring thought and analytical abilities”.

4.2 Automation Change Identiﬁcation

(RQ2)

4.2.1 Pre-automation Task Flow

A major part of the impact analysis involves spec-

ifying trace links to related software artifacts. As

there rarely are any requirement traceability matrices

to consult, the tracing is mainly a poorly supported

information seeking activity. If the engineers do not

already know which artifacts (if any) are related to the

issue, or if they do not know where to ﬁnd the infor-

mation, they have to search or browse databases or

seek information from colleagues. When this infor-

mation need arise, a typical ﬁrst step is to search the

DefMS for already solved issues that are similar, as

presented in Figure 2. If no such issues are found,

one could search the DocMS for relevant project doc-

umentation, or ask a colleague for help as a last resort.

One engineer stated “I probably search for informa-

tion in project documentation more than I should, it is

ENASE2012-7thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

176

Figure 2: Overview of the SoP impact analysis process.

very time-consuming and rarely successful”.

4.2.2 Post-automation Task Flow

The idea of the traceability recovery tool is to support

the two steps of manual database searching by auto-

matically executing search queries. Without human

action, we envision the tool to search for both related

issues in the DefMS and related documentation in the

DocMS. Based on the textual content of the currently

analyzed issue, the tool predicts which software arti-

facts are the most likely to be related. As in the man-

ual work ﬂow, the last resort is to ask a colleague.

4.2.3 Changed/Added/Removed Tasks

From the perspective of the engineer, the work task

is slightly altered. An automatic search is conducted

and the resulting search hits, i.e., candidate trace

links, are presented. Thus, an added human subtask

is to assess the search hits. If the search result is

enough to satisfy the information need, the manual

DefMS and DocMS searching are removed subtasks.

On the other hand, if the engineer does not consider

the search results to provide enough information, the

engineer will as before have to seek information by

manual queries, or by asking a colleague.

4.3 Automation Classiﬁcation (RQ1)

4.3.1 Types of Automation

Table 1 shows Parasuraman et al.’s types of automa-

tion. The ﬁrst step, Information acquisition, refers to

operations supporting human sensory processes, sens-

ing and registration of input data. In the case of trace-

ability recovery, the required information is stored

digitally in databases. Software artifacts are typically

distributed in separate systems with poor interoper-

ability, a condition that applies also to the case we

study. Since the usefulness of an IR-based traceability

recovery tool is dependant on which software artifacts

it can access and index, plug-in solutions to existing

DefMS and DocMS have the advantage of being de-

ployed where the information actually resides.

Information analysis, the second step of an au-

tomation solution, deals with the cognitive func-

tions such as working memory and inferential pro-

cesses (Parasuraman et al., 2000). Regarding IR-

based traceability recovery tools, it is not meaningful

to distinguish between acquisition and analysis of in-

formation. As the information is accessed, it is also

analyzed. This includes the steps of the implemented

IR model such as preprocessing (stop word removal,

stemming etc.), feature extraction and weighting, and

extraction of language models. In the rest of this re-

port, we consider information acquisition and analy-

sis to be one single type of automation.

The third step, Decision selection, comprises the

augmentation and replacement of human decision op-

tions. Supporting decision selection is the backbone

of IR-based traceability recovery tools, since they are

intended to present candidate trace links. The tools

rank search results, and present them to the user. Fur-

thermore, a number of traceability recovery tools of-

fer the user both ﬁltering and highlighting.

Action implementation, the last automation type,

is deﬁned as the actual machine execution of the ac-

tion choice. For the IR-based traceability recovery

plug-in we envision, the action implementation is lim-

ited to correctly reporting the outcome in the impact

analysis template described in Section 4.1.2. Au-

tomating this step is meaningful since it would reduce

the risk of human input errors (which are known to

exist), e.g., incorrect use of document identiﬁers, and

copy/paste errors.

4.3.2 Levels of Automation

Table 1 shows the levels of automation according to

Parasuraman et al., from no support at all to a fully

autonomous solution. Based on our analysis of types

of automation, we studied the corresponding levels of

automation, and the risks involved in increasing them.

Regarding Information acquisition and analysis,

the SoP activity is to manually input search queries in

search tools offered by DefMSs and DocMSs. In such

tools, the human activity is at best supported by a set

of boolean search operations, e.g., AND, OR, NOT.

The SoA traceability recovery tools on the other hand,

automatically enters and executes search queries. The

human is not at all involved in this process, however

FindabilitythroughTraceability-ARealisticApplicationofCandidateTraceLinks?

177

error messages tend to appear if major failures occur.

Letting the tools access too much information is a se-

curity issue. In many cases, employees have different

access rights. Efforts to improve information access

in enterprise search solutions are often limited by pol-

icy decisions (Tolone et al., 2005).

The SoP activities corresponding to the automa-

tion type Decision support are mainly concerned with

vetting various search results. Search strings can be

reﬁned until the information need of the engineer

is satisﬁed. SoP search solutions typically return a

ranked list of documents and support features such

as sorting and ﬁltering. More advanced search solu-

tions also implement features as query expansion and

relevance feedback (Baeza-Yates and Ribeiro-Neto,

2011).

After automatic execution of the search queries,

SoA tracebility recovery tools do not differ from SoP

search solutions. In both cases, the goal is to limit the

search space of the engineers, and to at least give them

starting points for browsing to the information they

are seeking. Obviously, an ideal traceability recovery

tool would automatically make decisions with a bet-

ter judgement than a human engineer. Instead, since

search results include both relevant and non-relevant

results, increasing the automation to levels where the

human is not part of the process leads to both false

positives and missed trace links. Since the accuracy

of SoA traceability recovery is considered low, fully

removing the human involvement is currently not fea-

sible (Oliveto et al., 2007; De Lucia et al., 2012). IR

tools balance on the precision-recall trade-off (Baeza-

Yates and Ribeiro-Neto, 2011), and a fully automated

solution have to be designed with care. Missed links

threat underestimating change impact, which moti-

vates search tools offering a high recall. On the other

hand, false positives caused by low precision force en-

gineers to spend extra effort. For both cases, incorrect

effort estimations are consequences.

The last automation type in the sequence, the Ac-

tion implementation, is currently supported by a tem-

plate and corresponding guidelines developed by in-

ternal safety engineers. The outcome of the impact

analysis is reported manually, by entering free text in

a document. The tailored traceability recovery plug-

in we envision, enables simple drag-and-drop oper-

ations in a graphical user interface. Both manually

typing free-text information in a template and opera-

tions in a graphical interface introduce errors due to

the human factor, thus we do not consider this a ma-

jor risk. Although, efﬁcient UIs enable engineers to

input incorrect information faster. Consequently, an

increased level of automation of the last automation

type in the sequence might actually increase the num-

Figure 3: Types and levels of automation for IR-based

traceability recovery support in impact analysis. The ma-

jor risks involved in increasing the levels are from left to

right: contravening access policies, incorrect impact analy-

sis, increased number of human errors.

ber of human errors.

4.4 Automation Impact Analysis (RQ2)

4.4.1 Direct Effects

There are two main hypotheses of increasing the level

of automation by deploying a traceability recovery

plug-in. First, engineers will on average ﬁnish the im-

pact analysis work task faster. Second, the correctness

of the engineers’ impact analyses will be higher.

4.4.2 Indirect Effects

Besides risks already mentioned, there is a risk that

additional tool support, if it is well received, would

make engineers too conﬁdent in their output. Such

overconﬁdence might cause engineers, especially un-

der stress, to hastily accept tool output as ﬁnal an-

swers. Another risk of deploying efﬁcient search sup-

port is that communication between developers might

decrease, as people might rely more on tools.

4.4.3 Evidence of Effects

There are few evaluations of deploying IR-based

traceability recovery tools in complete software de-

velopment projects, and only one of them was con-

ducted in an industrial setting. Li et al. conducted a

case study on impact analysis in a ﬁve-people project

running in a Chinese company for 30 weeks (Li et al.,

2008). They concluded it to be a feasible approach,

and that it helped engineers ﬁnish the tracing tasks

faster. De Lucia et al. conducted another case study,

however in a university setting (De Lucia et al., 2009).

By studying seven student development projects, they

found that deploying their IR-based traceability re-

covery plug-in improved the maintenance of trace-

ENASE2012-7thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

178

ability information, as more trace links were discov-

ered.

Apart from case studies, several controlled exper-

iments have concluded that tools implementing IR-

based traceability recovery can be beneﬁcial. For sim-

ilar tasks related to establishing trace links, both Huff-

man Hayes et al. (Huffman Hayes et al., 2007) and

De Lucia et al. (De Lucia et al., 2009) concluded that

working with tool support is better than working man-

ually, and that it improves accuracy and/or efﬁciency.

De Lucia et al. also found that inexperienced devel-

opers beneﬁt most from such tool support (De Lu-

cia et al., 2009). On the contrary, an experiment by

Falessi et al. concluded that letting student subjects

work with their IR-traceability recovery tool did not

lead to any signiﬁcant advantages (Falessi and Briand,

2009). Also, in contrast to ﬁndings by De Lucia et al.

regarding impact of experience, they found the tool

support more useful when used by a requirements an-

alyst in an Italian company.

The usefulness of IR solutions are known to de-

pends on the context, users, and the task it is meant

to support, which is also indicated by the conclusions

from previous studies. Consequently, the applicabil-

ity of deploying a tailored plug-in in the safety-critical

context of our case study is uncertain. However, pri-

marily considering it as an investment in ﬁndability

(i.e., moving beyond manual keyword searching to

automatically suggesting search results), we expect

the information seeking to be more efﬁcient. Fig-

ure 4 presents a break-even analysis, assuming that

the tool support would make an engineer complete

an impact analysis faster. The main costs in an au-

tomated solution would be initial, as the tool mainte-

nance is expected to be negligible. Instead the costs

are mainly related to development and deployment of

a tool, and user training. The tracing costs would then

linearly increase depending on the number of tracing

tasks performed. For the manual tracing process, the

tracing cost is expected to decrease as the number of

tasks grow due to human learning effects. Thus, for

a traceability recovery plug-in to be a meaningful in-

vestment, we would expect two aspects from the de-

velopment context. First, the information landscape

must be challenging enough to make tracing a time-

consuming task (high slope of manual curve). Sec-

ond, the tracing task must be common enough to be

considered an issue (n high enough for curves to in-

tersect). Whether the studied case fulﬁls these aspects

requires further study.

Figure 4: Initial break-even analysis for automated impact

analysis.

5 CONCLUSIONS AND FUTURE

WORK

We have presented a structured analysis of how a

SoA IR-based traceability recovery tool could support

ﬁndability in an impact analysis work task. Also, we

argue that the tools should be judged as search tools

rather than as traceability miners, and thus should be

evaluated accordingly. The rest of our contribution is

threefold. First, we presented a model of the infor-

mation seeking activity involved in an impact analy-

sis work task in a safety-critical development context.

Second, we found that deploying a SoA traceability

recovery tool, with a limited risk, can increase the

level of automation, but not for the automation type

deﬁned as decision selection. Third, based on an ini-

tial break-even analysis, we claim that investing in au-

tomated traceability recovery only is worth the effort

if both (a) the information landscape is challenging,

and (b) the artifact tracing is a frequent work task.

Future work includes implementing an IR-based

traceability recovery plug-in to HP Quality Center

and evaluating it in an industrial context, based on

established methodology for user evaluations in in-

formation retrieval. Furthermore, a deeper case study

on impact analysis processes in safety-critical soft-

ware engineering should be conducted. Such a study

should be dominated by qualitative analysis, since

quantitative analyses have dominated the ﬁeld of IR-

based traceability recovery.

ACKNOWLEDGEMENTS

This study was done as part of the SWELL

course on

Automated Veriﬁcation and Validation where we have

swell.se

FindabilitythroughTraceability-ARealisticApplicationofCandidateTraceLinks?

179

studied and developed models for analyzing levels of

automation in V&V. Thanks go to Rickard Torkar,

ıd Assar, and Per Runeson for review comments.

REFERENCES

Antoniol, G., Canfora, G., Casazza, G., De Lucia, A., and

Merlo, E. (2002). Recovering traceability links be-

tween code and documentation. In Trans. on Software

Engineering, volume 28, pages 970–983.

Baeza-Yates, R. and Ribeiro-Neto, B. (2011). Modern In-

formation Retrieval: The Concepts and Technology

behind Search. Addison-Wesley.

Billings, C. (1997). Aviation Automation: The Search for a

Human-Centered Approach. Larrence Erlbaum Asso-

ciates, New Jersey.

Bohner, S. (2002). Software change impacts-an evolving

perspective. In Proceedings of the International Con-

ference on Software Maintenance, pages 263– 272.

Borg, M. (2011a). In vivo evaluation of large-scale IR-

based traceability recovery. In Proceedings of the

European Conference on Software Maintenance and

Reengineering, pages 365–368.

Borg, M. (2011b). IR-based traceability recovery as a plu-

gin - an industrial case study. In Fourth BCS-IRSG

Symposium on Future Directions in Information Ac-

cess.

Borg, M., Wnuk, K., and Pfahl, D. (2012). Industrial com-

parability of student artifacts in traceability recovery

research - an exploratory survey. In Proceedings of the

16th European Conference on Software Maintenance

and Reengineering.

Canfora, G. and Cerulo, L. (2006). Fine grained indexing

of software repositories to support impact analysis. In

Proceedings of the International Workshop on Mining

software repositories, pages 105–111.

Clamann, M., Wright, M., and Kaber, D. (2002). Compari-

son of performance effects of adaptive automation ap-

plied to various stages of human-machine system in-

formation processing. In Proc. of the Ann. Meeting of

the Human Factors and Ergonomics Soc., pages 342–

346.

De Lucia, A., Fasano, F., Oliveto, R., and Tortora, G.

(2005). ADAMS re-trace: A traceability recovery

tool. In Proc. of the 9th European Conference on Soft-

ware Maintenance and Reengineering, pages 32–41.

De Lucia, A., Marcus, A., Oliveto, R., and Poshyvanyk, D.

(2012). Information retrieval methods for automated

traceability recovery. In Cleland-Huang, J., Gotel, O.,

and Zisman, A., editors, Software and Systems Trace-

ability, pages 71–98. Springer, London.

De Lucia, A., Oliveto, R., and Tortora, G. (2009). Assess-

ing IR-based traceability recovery tools through con-

trolled experiments. Empirical Software Engineering,

14(1):57–92.

Domges, R. and Pohl, K. (1998). Adapting traceability

environments to project-speciﬁc needs. Communica-

tions of the ACM, 41(12):54–62.

Falessi, D. and Briand, L. (2009). The impact of automated

support for linking equivalent requirements based on

similarity measures. Technical report, Simula.

Frohm, J., Lindstrom, V., Winroth, M., and Stahre, M.

(2008). Levels of automation in manufacturing. Er-

gonomia, page 29.

Huffman Hayes, J. and Dekhtyar, A. (2005). Humans in

the traceability loop: can’t live with ’em, can’t live

without ’em. In Proceedings of the 3rd International

Workshop on Traceability in Emerging Forms of Soft-

ware Engineering, pages 20–23.

Huffman Hayes, J., Dekhtyar, A., and Sundaram, S. (2006).

Advances in dynamic generation of traceability links:

Two steps closer to full automation? Technical report,

University of Kentucky.

Huffman Hayes, J., Dekhtyar, A., Sundaram, S., Hol-

brook, A., Vadlamudi, S., and April, A. (2007). RE-

quirements TRacing on target (RETRO): improving

software maintenance through traceability recovery.

Innovations in Systems and Software Engineering,

3(3):193–202.

IEC (2003). IEC 61511-1 ed 1.0, safety instrumented sys-

tems for the process industry sector.

IEC (2010). IEC 61508 ed 2.0, Electri-

cal/Electronic/Programmable electronic safety-related

systems.

IEEE Computer Society (1990). 610.12-1990 IEEE stan-

dard glossary of software engineering terminology.

Technical report.

Karr-Wisniewski, P. and Lu, Y. (2010). When more is too

much: Operationalizing technology overload and ex-

ploring its impact on knowledge worker productivity.

Computers in Human Behavior, 26(5):1061–1072.

Klock, S., Gethers, M., Dit, B., and Poshyvanyk, D. (2011).

Traceclipse: an eclipse plug-in for traceability link re-

covery and management. In Proceeding of the 6th

International Workshop on Traceability in Emerging

Forms of Software Eengineering, pages 24–30.

Li, Y., Li, J., Yang, Y., and Li, M. (2008). Requirement-

centric traceability for change impact analysis: A case

study. In International Conference on Software Pro-

cess, pages 100–111.

Marcus, A. and Maletic, J. (2003). Recovering

documentation-to-source-code traceability links using

latent semantic indexing. In Proc. of the Int’l Confer-

ence on Software Engineering, pages 125–135.

Morville, P. (2005). Ambient Findability: What We Find

Changes Who We Become. O’Reilly Media.

Oliveto, R., Antoniol, G., Marcus, A., and Hayes, J. (2007).

Software artefact traceability: the Never-Ending chal-

lenge. In Software Maintenance, 2007. ICSM 2007.

IEEE International Conference on, pages 485–488.

Olsson, T. (2002). Software Information Management in

Requirements and Test Documentation. Licentiate

thesis, Lund University.

Parasuraman, R., Sheridan, T., and Wickens, C. (2000). A

model for types and levels of human interaction with

automation. Transactions on Systems, Man and Cy-

bernetics, 30(3):286–297.

ENASE2012-7thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

180

Sabaliauskaite, G., Loconsole, A., Engstrm, E., Unterkalm-

steiner, M., Regnell, B., Runeson, P., Gorschek, T.,

and Feldt, R. (2010). Challenges in aligning re-

quirements engineering and veriﬁcation in a Large-

Scale industrial context. In Requirements Engineer-

ing: Foundation for Software Quality, pages 128–142.

Sassone, P. (1988). Cost beneﬁt analysis of information sys-

tems: a survey of methodologies. In Proceedings of

the Conference on Ofﬁce Information Systems, pages

126–133.

Sheridan, T. and Verplank, W. (1978). Human and com-

puter control of undersea teleoperators. Technical,

MIT Man-Machine Systems Laboratory, Cambridge,

MA, United states.

Tolone, W., Ahn, G., Pai, T., and Hong, S. (2005). Ac-

cess control in collaborative systems. ACM Comput-

ing Surveys, 37(1):29–41.

FindabilitythroughTraceability-ARealisticApplicationofCandidateTraceLinks?

181