Expert Review of Taxonomy based Testing: A Testing Framework for

Medical Device Software

Hamsini Ketheswarasarma Rajaram

, John Loane

, Silvana Togneri MacMahon

and Fergal Mc Caffery

Regulated Software Research Centre, Dundalk Institute of Technology, Dundalk,

Ireland

School of Computing, Dublin City University, Dublin, Ireland

Keywords: The US FDA, Taxonomy, SW91, ISO/IEC/IEEE 29119, IEC 62304, Framework, Expert Review,

Software Testing, Recalls.

Abstract: This paper details the expert review of a framework developed to implement a novel testing approach called

taxonomy-based testing (TBT) for the medical device software domain. This framework proposes three

approaches to implement TBT and has been validated by experts from the software testing industry and the

medical device software domain. This paper details the results from the expert review. The expert review

focused on validating the three approaches to TBT, the benefits of TBT to medical device software

development, the accuracy of mappings of testing techniques from ISTQB and ISO/IEC/IEEE 29119-4:2015

to defects from a defect taxonomy, the integration of TBT into the standard test processes, ISTQB and

ISO/IEC/IEEE 29119-2:2013 and the structure of the framework. The contribution of this paper is to reveal

that (i) the framework is implementable in medical device software organisations that follow the IEC

62304:2006+A1:2015 software development process or that use standard test processes, (ii) using a defect

taxonomy could standardise the application of experience-based approaches to software testing and (iii)

considering potential defects before writing test cases could identify additional defects for test cases.

1 INTRODUCTION

This research proposed a testing approach called

taxonomy-based testing (TBT) to improve medical

device software (MDS) quality and to reduce adverse

events caused by MDS defects (Alemzadeh et al.

2013; Rajaram et al. 2020; Felderer and Beer 2013).

This research uses a defect taxonomy called SW91,

Classification of Defects in Health Software. SW91

is a standard for health software which has been

developed by the Association for the Advancement of

Medical Instrumentation (AAMI) in collaboration

with the US FDA (AAMI 2018). SW91 includes a

total of 186 defects from the planning phase to the

maintenance phase of a system. In TBT, the

requirements will be mapped into potential SW91

defects, and test cases will be written based on the

requirements and the mapped defects. Test cases will

https://orcid.org/0000-0002-3294-3906

https://orcid.org/0000-0002-9285-5019

https://orcid.org/0000-0003-0179-2436

https://orcid.org/0000-0002-0839-8362

be executed to verify whether the software complies

with the relevant requirements and does not contain

the mapped defects. By applying TBT, an

organisation can achieve benefits such as root cause

analysis, risk minimisation and early detection of

defects (Rajaram et al. 2020).

This research identified the need for a framework

detailing the TBT implementation process and the

alpha TBT (α – TBT) framework was developed to

meet this need (Rajaram(&) et al. 2019). The main

purpose of this framework was to implement TBT in

any MDS organisation with limited resources and

without the researcher's involvement with the

organisation's requirement and defect data.

The literature revealed that expert review is

important to identify the usefulness and applicability

of research. (Offenberger et al. 2019; Sjøberg et al.

2007; Dumas and Sorce 1995; MacMahon et al.

Rajaram, H., Loane, J., MacMahon, S. and Caffery, F.

Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software.

DOI: 10.5220/0010458503310339

In Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2021), pages 331-339

ISBN: 978-989-758-508-1

331

2014). This research used expert review to validate

whether the α – TBT framework was implementable

in MDS organisations. This will be followed by

implementing the framework in MDS organisations

to provide additional validation.

This paper details the expert review of the α –

TBT framework. Section 2 explains the α –TBT

framework and the validation points considered in the

expert review. Section 3 details the process followed

during this expert review. Section 4 details reviews of

the α – TBT framework. Section 5 details subsequent

changes considered in the next version of the

framework. Section 6 is future work. It details how this

research plans to implement TBT in MDS organisa-

tions to complete the validation process. Section 7

details the summary and conclusion of this paper.

2 ALPHA TBT FRAMEWORK

The α–TBT framework details what TBT is and the

following three approaches to TBT:

Approach A: TBT using all SW91 defects.

Approach B: TBT using testing technique mappings.

Approach C: TBT at different phases of software

development.

The purpose of providing three different

approaches is that the organisation can select an

appropriate approach in light of their resources and

testing practices. Also, the three approaches help to

narrow down the selection of SW91 defects from 186

and allow the implementation of TBT at any phase of

MDS development. The α – TBT framework details

the TBT implementation steps - review SW91

defects, plan-TBT, map requirements to SW91

defects, write test cases, execute tests and analyse

results. These steps have been integrated into the two

standard test processes: the ISO/IEC/IEEE 29119-

2:2013(ISO/IEC/IEEE 2013); and the test process of

the International Software Testing Qualifications

Board (ISTQB) (ISTQB 2010). These integrations

were published in (Rajaram(&) et al. 2019). The rest

of this section details the three approaches to TBT and

the points considered in the expert review of the α –

TBT framework.

Approach A - TBT using all SW91 Defects: In this

approach, the organisation has to search and map the

potential defects for each requirement to all SW91

defects. In this approach, the participants of TBT

must have a good understanding of all SW91 defects

and their classification.

Approach B - TBT using Testing Technique

Mappings: This approach uses testing techniques

from the ISO/IEC/IEEE 29119-4: 2015 and ISTQB

(Graham et al. 2006; ISO/IEC/IEEE 2015). These

testing techniques have been mapped into their

potential SW91 defects, which can be identified when

applying these techniques. The purpose of this

mapping is to reduce the number of SW91 defects that

need to be considered for each requirement.

Approach C - TBT at Different Phases of Software

Development: This approach allows an organisation

to narrow down the number of SW91 defects to those

specific to a particular software development phase.

Table 1 shows the number of SW91 defects from each

phase of software development that needs to be

considered when using TBT approach C.

Table 1: SW91 defects.

Parent level defects Number of defects

Planning (1) 2

Requirements (2) 15

Architecture and

Design (3)

Implementation (4) 106

Test (5) 18

Release Defects (6) 4

Maintenance (7) 13

2.1 Validation of the α – TBT

Framework

The α – TBT framework involves software testing,

MDS development, application of defect taxonomies,

and use of the ISO/IEC/IEEE 29119-2:2013 and

ISTQB test processes. The goal of this expert review

was to validate the following five points of the α –

TBT framework:

1. The three approaches to TBT.

2. The three approaches to TBT for MDS

development.

3. Testing technique mappings.

4. Standard test processes and TBT.

5. The structure of the α – TBT framework.

After the expert review, the TBT framework will

be implemented in a number of MDS organisations to

provide further validation. This validation plan is

presented in Section 6. The next section details the

process followed in this expert review and Section 3.3

details the reviews.

ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering

332

3 EXPERT REVIEW PROCESS

AND EXPERT PROFILES

The following steps have been followed in this expert

review: (1) Find relevant experts. (2) Develop

questionnaires and request permission. (3) Receive

and analyse reviews.

3.1 Find Relevant Experts

As detailed in Section 2.1 the α – TBT framework has

different parts to validate. It was not possible to find

individuals with expertise in all areas of the TBT

framework. Therefore, we looked to find one expert

for each point that we were looking to validate. There

was some overlap in expertise between experts. This

accounts for the difference in numbers of experts

validating each point. In order to validate the different

parts of the α – TBT framework, the following search

criteria have been used in expert selection:

• Expertise in software testing

• Expertise in defect taxonomy application

• Expertise in testing techniques application

• Expertise in MDS development

• Expertise in SW91 defects

• Expertise in the standard test processes such

as ISO/IEC/IEEE 29119-2:2013 and ISTQB.

Six experts were selected from authors who have

conducted taxonomy-supported testing and published

several papers, from the software testing industry and

from the MDS industry. Table 2 details the experts'

expertise considered for the framework validation.

Table 2: Experts.

Experts Expertise considered in this research

Expert 1 More than twenty years of experience in

software testing. Trainer for different levels of

ISTQB certification for more than fifteen

years. Leading active member of testing

standard family ISO/IEC/IEEE 29119.

Expert 2 More than fifteen years of experience in the

software development industry.

Internationally recognised researcher who has

conducted several research projects in

software industry including defect taxonomy-

supported testing by applying the ISTQB test

process.

Expert 3 Industrial testing consultant with knowledge

in the application of defect taxonomy-

supported testing in industrial projects along

with testing techniques and the ISTQB test

process. Thirty years' experience in test

management, test automation and quality

assurance.

Expert 4 Testing expertise with agile and the ISTQB

test process.

Over thirty years' experience in

the software development industry.

Expert 5 Medical device regulatory expert who works

at U.S. FDA for over ten years. Active and

leading member of the standard development,

AAMI SW91 "Classification of Defects in

Health Software".

Expert 6 Quality assurance manager for over ten years

at a leading multi-national software testing

organisation.

3.2 Develop Questionnaires and

Request Permission

As the framework is complex and had multiple

distinct parts to validate, three sets of open-ended

questionnaires (questionnaires A, B and C) were

developed. All three questionnaires had 80% overlap.

Table 3 details the questionnaires and experts who

received those questionnaires. Questionnaire B

included the same questions as Questionnaire A and

it added a set of questions focused on the testing

technique mappings. Questionnaire C included

questions from Questionnaire A, but it added

questions focused on finding the applicability of the

framework to the MDS domain and did not focus on

validating the point about standard test processes and

TBT.

In order to develop questionnaires, the question

design process detailed by Dawson was followed in

this study (Dawson 2009). Questions were developed

to validate different points of the α – TBT framework

detailed in Section 2.1. After developing questions, a

draft of the questions was reviewed by another

researcher to determine how well the questions

Table 3: Questionnaires, validation points and experts.

Questionnaires Validation points Experts

Questionnaire A Three approaches to TBT. Expert 1

Expert 2

Expert 4

Standard test processes and

TBT.

The structure of the α –

TBT framework.

Questionnaire B Three approaches to TBT. Expert 3

Expert 6

Testing technique

mappings.

Standard test processes and

TBT.

The structure of the α –

TBT framework.

Questionnaire C Three approaches to TBT

for MDS development.

Expert 5

The structure of the α –

TBT framework.

Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software

333

fulfilled their purpose and to ensure that they did not

contain any ambiguity.

Since expertise between experts was overlapped,

different numbers of experts were considered for

validating each point of α – TBT framework. Due to

this, an imbalance of experts to validate each point

was observed. As this research will further validate

the framework with MDS organisations, the

imbalance of experts for each point was not

considered as a major validity issue to this research.

Expert 5 was involved in the development of

SW91. This is considered a strength rather than a

limitation of this expert review. Their in-depth

knowledge of SW91 along with their medical device

software regulatory expertise means that they were an

ideal candidate to validate the framework.

After identifying the experts and developing the

questionnaires, emails were sent to the experts

requesting that they review the α – TBT framework.

The experts were also asked to participate in a focus

group interview to discuss the findings from their

reviews.

3.3 Receive and Analysis of Reviews

All experts agreed to provide written feedback and to

join a focus group interview or individual interview

after their initial reviews were analysed. Experts 1

and 3 participated in the focus group interview. The

other experts were unable to join at the requested

time. Additional interviews were held with Experts 5

and 6. Expert 4 was also contacted for further

clarification and he has provided this clarification via

email. The researcher analysed the reviews collected

from questionnaires, focus group interviews and

individual interviews. The next section presents the

analysis of the reviews.

4 EXPERT REVIEW OF THE

ALPHA TBT FRAMEWORK

The remainder of this section details the reviews on

the five points of α – TBT framework listed in Section

2.1.

4.1 Three Approaches to TBT

This part was validated by experts 2, 3, 4 and 6. This

part of the expert review focused on validating the

benefits of the three approaches to TBT in practice

and implementation of the three TBT approaches.

Expert 2 and 3 did not point out any difficulties

with the three approaches to TBT. This is significant,

as these two experts have experience in the

application of defect taxonomy-supported testing.

They prefer approach B, which uses testing

techniques. They said that this approach facilitates an

efficient design of test cases and it will be a good

testing strategy to consider when planning the testing

for projects which use testing techniques.

Expert 2 said that by adopting any of the

approaches, the implementation phase of a project

would have enhanced traceability by checking code

against requirements and mapped defects.

Expert 3 said that since TBT uses a defect

taxonomy from the MDS industry, it should include

defects which are familiar and that TBT could be

easily implemented into a MDS organisation.

Expert 4 said that the three TBT approaches could

be beneficial. He suspects only a minority of MDS

organisations will use it. Unless the adherence to

standards is made mandatory by regulatory bodies,

the overhead of application is too great for most

companies. He also said that TBT could be hard to

sell to an organisation due to the cost of the third step

of TBT, mapping requirements to SW91 defects.

Expert 6 said that defect taxonomies are often

used informally as an experienced-based technique

during test case generation. The TBT framework

could be seen as standardising, something that was

previously based primarily on experience. TBT can

be used as an additional review before or after the

creation of the initial test specification. During this

review using the lens of the mapped defect taxonomy,

the test cases could be reviewed for coverage.

However, he also suspected that TBT would be hard

to sell unless defect taxonomy analysis was part of the

existing test creation phase.

Experts 2 and 3 have provided positive feedback.

Experts 4 and 6 suspect that TBT is hard to sell to

organisations. Since both experts 4 and 6 have

provided feedback based on their expertise in non -

MDS, their feedback was discussed with a MDS

expert, expert 5. Expert 5 strongly disagreed with

their comments and said that while the "the overhead

of application is too great for most companies might

be legitimate", there is still no proper evidence to

show the cost or time of the mappings from TBT

implementation. Expert 5 said the benefits of TBT

such as early detection of defects, root cause analysis

have to be prioritised first and that the time needed to

conduct the mappings has to be identified through

industrial validation of this framework. The time

taken to conduct the mapping between requirements

and SW91 defects will be identified through

ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering

334

validation with MDS organisations. This is detailed

in Section 6.

MDS expert 5 did not agree with expert 4's

comment that the regulatory bodies should make

adherence to standards mandatory. Expert 5 said that

regulations aim to ensure safe and reliable software

development and to assist the MDS organisations in

developing software in the right way. MDS

organisations should not be governed by regulatory

bodies. They should understand the purpose and

benefits of regulations.

4.2 Review of the Three Approaches to

TBT for MDS Development

This part of the expert review has focused on finding

out the benefits of the three approaches to TBT for

MDS development. This part was validated by expert

Expert 5 provided a review of the three

approaches to TBT and their implementation to

different situations in a MDS organisation. Expert 5

said that approach C seems a good starting point that

will bring benefits to MDS development. The

selection of an approach has to be based on the

software development methodology. Existing

development methodologies at a MDS organisation

could determine which approach can be selected. For

example, test-driven development can use approach

B. Approach C is implementable in agile

development. However, approach A may also work

with an agile methodology where the focus is on all

the defects that match the current sprint. Expert 5

suggested detailing how TBT can be implemented at

a company that uses agile.

Expert 5 said that since IEC 62304:2006+A1:2015

is a process standard that does not include test criteria,

it is hard to tell whether or not the three approaches

comply with the processes of IEC

62304:2006+A1:2015. As there were no test criteria in

IEC 62304:2006+A1:2015 (Bujok et al. 2017), expert

5 suggested that the TBT framework should include

the following three direct links between IEC

62304:2006+A1:2015 and SW91 defects:

Link 1: The defect category Security (3.8) from

SW91 was developed with IEC

62304:2006+A1:2015 in mind. MDS organisations

can use this category and its definition to refer to

defects related to security.

Link 2: Segregation necessary for risk control (3.2)

is one defect category from SW91. This category

aligns with Section 5.3.5 of IEC

62304:2006+A1:2015 (IEC 2015).

Link 3: The framework suggests considering potential

defects of SW91 in each phase before that phase. This

can be linked with Section 5.1.12 from IEC

62304:2006+A1:2015. Section 5.1.12 details the

identification and avoidance of common software

defects.

The researcher extended the study on the link

between IEC 62304:2006+A1:2015 and SW91

defects and found the following fourth link:

Link 4: Section 5.1.6 of IEC 62304:2006+A1:2015

details that each life cycle activity has to include

verification. This can be linked with the TBT

framework, approach C which details how SW91

defects can be used in the verification at different

phases of MDS development.

The links above between SW91 and IEC

62304:2006+A1:2015 show that the application of

the α – TBT framework could help MDS

organisations to achieve regulatory compliance.

4.3 Review of Testing Technique

Mappings

Testing techniques from the ISTQB and the

ISO/IEC/IEEE 29119-4:2015 were mapped into

potential SW91 defects. These mappings are used in

TBT approach B. These mappings were validated by

expert 3 and expert 6.

This part of the validation focused on establishing

the accuracy of the mappings. Expert 3 said that the

"mappings in the excel sheets are understandable and

plausible". Expert 3 said that this mapping could help

testers to design test cases effectively and that it would

also significantly improve the design of test cases.

Expert 6 said that these mappings are good and

comprehensively cover the testing techniques in the

software testing industry. Expert 6 said that the

granularity of defects in the mappings is much finer

than what his organisation currently works with. For

example, the mapping of equivalence partitioning

testing technique includes defects such as Data

definition (4.1), Scalar data type (4.1.1) and Scalar

Data Operations (4.1.1.1). Expert 6 said that testers

would not log this granularity of defects to a root

cause in their defect management/software life cycle

application.

Since expert 6 is from non-MDS testing, this

comment was discussed with expert 5 to know about

the granularity of defects in MDS development.

Expert 5 said MDS organisations should consider root

causes and contributing factors to ensure quality via

mitigation of root causes.

Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software

335

4.4 Standard Test Processes and TBT

The six steps of the TBT implementation process are

integrated into the two standard test processes,

ISO/IEC/IEEE 29119-2:2013 and ISTQB. These

integrations were detailed and published in

(Rajaram(&) et al. 2019).

This part of the validation

focused on finding out:

• Whether or not the two standard test processes are

aligned correctly.

• Whether the six steps of TBT are correctly

integrated into the standard test processes.

• Whether TBT is implementable in an

organisation which follows a standard test

process.

This part was validated by experts 1, 2, 3 and 4.

Experts 1 and 3 said that both standard test processes

have different scopes and that the researcher had not

explained this during the alignment of both standard

test processes. The ISO/IEC/IEEE 29119-2:2013 test

process has three sub processes, from the organisation

level to the project level. The scope of the ISTQB test

process is project-level testing only. It cannot be

aligned with the organisational test process from

ISO/IEC/IEEE 29119-2:2013. The experts suggested

detailing the difference in scope in the framework.

Regarding the integration of TBT into the test

processes ISO/IEC/IEEE 29119-2:2013 and ISTQB,

all the experts provided positive comments on the

integration. They suggested that TBT is currently

integrated into the older version of the ISTQB test

process and should be integrated into the newer

version of the ISTQB test process.

Regarding the six steps of TBT, the experts

suggested that Step 1: Review SW91 defects, should

not be considered in the test processes. This step

should be considered as knowledge sharing and

conducted at the requirement engineering phase,

which is outside of the test process. If this step is

considered at the requirement engineering phase, this

will provide an opportunity to consider all potential

defects from all phases of software development. The

experts also suggested minor changes related to

terminologies and their explanations in both test

processes. The experts said that both test processes,

the ISTQB and the ISO/IEC/IEEE 29119-2:2013

process allow the selection of appropriate testing

techniques for a project. TBT is a new testing

technique and could be considered when a testing

team selects testing techniques for a project in an

organisation. TBT is implementable in an

organisation using either standard test process, the

ISTQB or the ISO/IEC/IEEE 29119-2:2013.

4.5 Review of the Structure of the

Framework

This part was validated by experts 1, 2, 3, 4 and MDS

expert 5. The purpose of this part of the validation is

to find out whether the framework is coherent. It also

aimed to assess if the framework was implementable

in MDS organisations.

Expert 1 and expert 3 said that the framework is

readable, understandable and well structured. The six

steps are explained very well. If the researcher

includes examples throughout the framework, it will

be clearer to the reader. Expert 2 and expert 4 said that

the α – TBT framework is clearly structured, although

they pointed out the challenges on the implementation

of TBT due to the time it will take to conduct the

mappings between requirements and SW91 defects.

Expert 5 said that by including identified pros and

cons for each approach, it would make the decision-

making easier for MDS organisations by showing that

possible failure points have been taken into

consideration.

In order to make the implementation of TBT

easier, they said that the researcher should perform

training on conducting mappings using sample

requirements and SW91 defects at the organisation

which agreed to implement it. The experts suggested

that by implementing TBT in an organisation,

efficiency can be measured and that this detail should

be included in the framework. They suggested giving

thought to automating this framework.

5 CHANGES RESULTING FROM

EXPERT REVIEW

The experts provided their reviews on the α–TBT

framework. This section summarises the changes

resulting from the expert review that will be built into

the next version of the framework, the β – TBT

framework.

The first part of the α–TBT framework validation

focused on the three approaches to TBT. No changes

to the α–TBT framework were suggested apart from

finding the time needed to conduct the mappings

between requirements and SW91 defects. This will be

identified through the validation of TBT approach B

in a MDS organisation and it will be included in the

β - TBT framework. This has been detailed in future

work, Section 6.

The second part of the α–TBT framework

validation focused on the three approaches to TBT for

MDS development. This was validated by expert 5,

ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering

336

who suggested detailing the direct links between IEC

62304:2006+A1:2015 and SW91. This suggestion

has been adopted and these links have been included

in the β - TBT framework. Expert 5 also suggested

including details on how TBT can be used with an

agile methodology. A document will be developed

detailing how to implement TBT when using an agile

methodology. This document will be added to the β -

TBT framework.

The third part of the α–TBT framework validation

focused on testing technique mappings. Testing

techniques from the ISTQB and the ISO/IEC/IEEE

29119-4:2015 were mapped into the potential SW91

defects and used in approach B of TBT. This part was

validated by experts 3 and 6, who provided positive

comments about the accuracy of the mappings.

Expert 6 said that the mappings contain much finer

defect details than they typically work with. Testers

would not log the finer detail of defects in their defect

management/software life cycle. This review was

discussed with expert 5, who said that MDS

organisations should consider finer detail of defects

to find root causes. Taking into account the statement

made by Expert 5 on finer defect details, the use of

finer defect details will be investigated through the

TBT approach B implementation with MDS

organisations. This has been detailed in future work,

Section 6.

The fourth part of the α–TBT framework

validation focused on standard test processes and

TBT. The experts said that TBT is implementable in

an organisation which follows the standard test

process, either ISO/IEC/IEEE 29119-2:2013 or

ISTQB. However, the experts suggested the

following changes, which have been adopted into the

β - TBT framework:

• Do not integrate the first step of TBT into both

standard test processes, ISTQB and

ISO/IEC/IEEE 29119-2:2013. Consider it at the

requirement gathering phase.

• Detail the scope of both test processes ISTQB,

ISO/IEC/IEEE 29119-2:2013 and where TBT can

fit.

• Integrate TBT into the new version of the ISTQB

test process and provide additional detail to the

phases of both test processes ISTQB and

ISO/IEC/IEEE 29119-2:2013.

The last part of the α–TBT framework validation

focused on the structure of the framework. All the

experts provided positive comments on the structure

of the α–TBT framework. However, the experts

suggested performing training on how to conduct

mappings between requirements and SW91 defects at

the organisation which agreed to implement TBT.

The experts suggested including the efficiency, pros

and cons of TBT approaches in the framework to

make the selection easier for the organisation.

This research will validate TBT approach B in an

MDS organisation. From this validation, the

efficiency, pros and cons of TBT approach B will be

investigated and includes in the β - TBT framework.

This has been detailed in future work, Section 6.

Table 4 summarises the changes suggested from the

expert review. Changes in bold have already been

adopted into the β – TBT framework. The rest of the

changes will be addressed through future work.

Table 4: Changes for building β – TBT framework.

Validation points Suggested changes

Three approaches

to TBT.

Investigate the time taken to conduct

the mappings.

Three approaches

to TBT and MDS

development.

Develop an approach for

implementing TBT in Agile

development.

Examine the direct links between

SW91 and IEC

62304:2006+A1:2015.

Testing technique

mappings.

Include the benefits of finer defects to

MDS development.

TBT and standard

test process.

Consider the first step of TBT at the

requirement gathering phase.

Provide the scope of both test

processes ISTQB and

ISO/IEC/IEEE 29119-2:2013 and

TBT.

Integrate TBT into the newer

version of ISTQB.

The structure of

the α – TBT

framework.

Detail the pros and cons of three

approaches to TBT.

Provide examples through the

framework.

6 FUTURE WORK

As detailed in Section 1, the expert review on the α –

TBT framework is one part of the validation. This

validation suggested including the following points in

the next version of the framework:

• Pros and cons of the three approaches to TBT.

• Time taken to conduct mappings between

requirements and SW91 defects.

• The efficiency of three approaches to TBT.

• Benefits of finer defects to MDS development.

In order to validate the points raised in the expert

reviews, a research collaboration was established

Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software

337

with an MDS organisation, Company B in the U.K.

The researcher presented the three approaches to

TBT. By considering resources and project time at

Company B, they agreed to implement TBT approach

B. During this implementation, five requirements

from a past release at Company B will be used to map

into SW91 defects using the testing technique

mappings. Based on the mappings, the researcher will

interview the test engineer at Company B to

investigate the points raised in the expert review.

7 CONCLUSION

This paper has detailed a part of the validation, expert

review of the α–TBT framework. Six experts from the

software testing industry and the MDS industry have

reviewed the α–TBT framework. The expert review

of α–TBT framework focused on validating

approaches to TBT, the benefits of TBT to MDS

development, the accuracy of mappings of testing

techniques from ISTQB and ISO/IEC/IEEE 29119-

4:2015 to defects from SW91, the integration of TBT

into the standard test processes such as ISTQB ,

ISO/IEC/IEEE 29119-2:2013 and the structure of the

framework.

Experts provided positive feedback on the three

approaches to TBT. The review identified that the

three approaches to TBT could be implemented in

different situations at MDS organisations.

Approaches A and C can fit into agile development.

Approach B can fit into test-driven development and

it enables the standardisation of the experience-based

application of defect taxonomies in software testing.

Since the α–TBT framework includes mappings of

testing techniques to SW91 defects, it will be

beneficial to consider potential defects before writing

test cases.

The experts said that the framework is readable

and well structured. It is implementable in MDS

organisations which use the

IEC62304:2006+A1:2015 process. It enables the

implementation of TBT into existing standard test

processes such as ISTQB and ISO/IEC/IEEE 29119-

2:2013. Experts said that the testing technique

mappings comprehensively cover the testing

techniques in the software testing industry and the

mappings are correct. These mappings would

significantly improve the design of test cases.

Experts suggested including additional details in

the next version of the framework such as (i) pros,

cons and efficiency of the three approaches to TBT,

(ii) time taken to conduct mappings between

requirements and SW91 defects and (iii) benefits of

reporting fine grained defects to MDS development.

These points will be investigated through validation

with MDS organisations. This will be included in the

next version of the framework.

ACKNOWLEDGEMENT

This work was supported with the financial support

of the Science Foundation Ireland grant 13/R.C./2094

and co-funded under the European Regional

Development Fund through the Southern & Eastern

Regional Operational Programme to Lero - the Irish

Software Research Centre (www.lero.ie).

REFERENCES

AAMI. (2018). American National Standard ANSI / AAMI

SW91 : 2018.

Alemzadeh, H., Iyer, R.K., Kalbarczyk, Z., Raman,

Jaishankar and Raman, Jai. (2013). Analysis of safety-

critical computer failures in medical devices. IEEE

Security and Privacy Magazine, 11(4), pp.14–26.

Black, R. (2008). Advanced Software Testing - Vol. 1. 2nd

ed. Santa Barbara: Rocky Nook Inc.

Bujok, A.B., MacMahon, S.T., Grant, P. and McCaffery, F.

(2017). Approach to the Development of a Medical

Device Software Quality Assurance Framework. In:

STV17 and INTUITEST 2017.

Chillarege, R., Bhandari, I.S., Chaar, J.K., Halliday, M.J.,

Ray, B.K. and Moebus, D.S. (1992). Orthogonal Defect

Classification-A Concept for In-Process

Measurements. IEEE Transactions on Software

Engineering, 18(11), pp.943–956.

Dawson, C. (2009). Introduction to Research Methods.

Dumas, J. and Sorce, J. (1995). Expert reviews: how many

experts is enough?. Proceedings of the Human Factors

and Ergonomics Society, 1(October 1995), pp.228–232.

Felderer, M. and Beer, A. (2013). Using defect taxonomies

to improve the maturity of the system test process:

Results from an industrial case study. In: SWQD 2013.

pp.125–146.

Graham, D., Veenendaal van, E., Evans, I. and Black, R.

(2006). Foundations of software testing; ISTQB

Certification. London: Cengage Learning Emea.

IEC. (2015). Medical device software — Software life-

cycle processes. Bs En 62304:2006 +a1:2015,

3(November 2008), p.88p.

ISO/IEC/IEEE. (2013). BSI Standards Publication

Software and systems engineering — Software testing

Part 2 : Test processes.

ISO/IEC/IEEE. (2015). BSI Standards Publication

Software and systems engineering — Software testing

Part 4 : Test techniques. , 2015.

ISTQB. (2010). Certified Tester Foundation Level

Syllabus.

ENASE 2021 - 16th International Conference on Evaluation of Novel Approaches to Software Engineering

338

Karahroudy, A.A. and Tabrizi, M.H.N. (1996). Software

Defect Taxonomy , Analysis and Overview. , 1996.

MacMahon, S.T., McCaffery, F. and Keenan, F. (2014).

The MedITNet assessment framework: development

and validation of a framework for improving risk

management of medical IT networks. Journal of

Software: Evolution and Process, 26(12), pp.1172–

1192.

Offenberger, S., Herman, G.L., Peterson, P., Sherman,

A.T., Golaszewski, E., Scheponik, T. and Oliva, L.

(2019). Initial Validation of the Cybersecurity Concept

Inventory: Pilot Testing and Expert Review.

Proceedings - Frontiers in Education Conference, FIE,

2019-Octob, pp.1–9.

Rajaram(&), H.K., Loane, J., MacMahon, S.T. and Fergal,

M. (2019). A Framework for Taxonomy Based Testing

Using Classification of Defects in Health Software-

SW91. Systems, Software and Services Process

Improvement, 2019, pp.606–618. Available from:

https://link.springer.com/chapter/10.1007/978-3-030-

28005-5_47.

Rajaram, H.K., Loane, J., MacMahon, S.T. and Caffery,

F.M. (2020). A Retrospective Study of Taxonomy

based Testing using Empirical Data from a Medical

Device Software Company. In: ICSOFT 2020.

Sjøberg, D.I.K., Dybå, T. and Jørgensen, M. (2007). The

future of empirical methods in software engineering

research. FoSE 2007: Future of Software Engineering,

2007, pp.358–378.

Expert Review of Taxonomy based Testing: A Testing Framework for Medical Device Software

339