Shaping User-Centered Health Innovation Through Assessment

Arthur Trognon

1,2,3,*

, David Servais

3,4

, Islem Habibi

2,3

, Robert Picard

, Thomas Lihoreau

3,5,6

Lionel Pazart

3,5,6

, Sylvia Pelayo

3,7,8

, Thierry Chevallier

3,6,8

, Kathryn Ernecq

3,9

, Anaïs Garin

3,10

Mathias Béjean

3,10

and Denis Abraham

1,3

Association Innov’Autonomie, Nancy, France

Clinicog, Nancy, France

Consortium DynSanté

Forum des Living Labs en Santé et en Autonomie (LLSA), Paris, France

Inserm CIC 1431, CHU Besançon, F-25000 Besançon, France

Tech4Health Network - FCRIN, France

Université de Lille, Lille, France

University Hospitals of Nîmes, France

Frog Part of Capgemini, Paris, France

Paris Est Créteil University, IRG, F-94010, France

Keywords: Maturity Evaluation, Technologies for Health, Medical Devices, Computational Psychology, Innovation.

Abstract: Historically, evaluation methods for innovative projects have focused mainly on technological development

aspects. However, recent research suggests that, in the context of consumption by the general public, the

decision parameters for acceptance seem to be based more on characteristics extrinsic to technological

maturity. In the present work, we present a model for the evaluation of innovative projects, the Concept

Maturity Level Santé France model, inspired by the CML model developed by the National Aeronautics and

Space Administration and specified in the context of MedTech project development, and placing co-design

with the end-user and its ecosystem on the same level of importance as the regulatory and technological

development aspects, and giving it a theoretical and fundamental basis.

1 INTRODUCTION

Traditionally, the evaluation of innovative systems

has focused on risk prevention in three challenging

areas: performance, schedule, and budget (Mankins,

2009). Such evaluation should be incorporated in

each step of the life cycle of new systems in order to

avoid products failure and anticipate technical risks

and needs.

In order to standardize the evaluation of these

aspects of research and development, and project

programming, a number of tools have been

developed, such as the Technology Readiness Levels

(TRLs) grid, which was developed by the National

Aeronautics and Space Administration (NASA) in the

early 1970s and completed in 1995 (Mankins, 1995).

It was initially developed to standardize the

assessment of the maturity of spaceflight projects

through a technology readiness assessment (TRA)

examining key concepts, technological needs, and

demonstrability, while taking into account economic

aspects, making it possible to establish an inventory

of risks as well as to produce a standard

understanding of technological status (Dawson,

2013). However, while this model and its direct

descendants such as the Concepts Maturity Levels

model (Wessen et al., 2013) seem relevant in the

evaluation of institutional projects, it appeared

limited in the evaluation of subsequent acceptability

if the device were placed in the hands of a wider

population (Salazar &Russi-Vigoya, 2021). These

authors have highlighted that the TRL scale does not

allow the reading of the ease of use, the satisfaction

of the final user, the human performance in the use of

the device, as it does not allow the reading of the

comprehensiveness of the program or the device as

well (See et al., 2019). Thus, the grid does not seem

exhaustive in an innovation approach centered on the

user and not the technology per se. It is also of interest

to mention that TRL scale is used largely in different

Trognon, A., Servais, D., Habibi, I., Picard, R., Lihoreau, T., Pazart, L., Pelayo, S., Chevallier, T., Ernecq, K., Garin, A., Béjean, M. and Abraham, D.

Shaping User-Centered Health Innovation Through Assessment.

DOI: 10.5220/0011925800003414

In Proceedings of the 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2023) - Volume 1: BIODEVICES, pages 229-242

ISBN: 978-989-758-631-6; ISSN: 2184-4305

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

229

fields, for instance by funders for tender of calls in

health research, asking candidates to estimate their

pre and post project TRL level, and justifying

consequently the improvement of maturity thanks to

the activities they wish to develop thanks to the

grants.

Indeed, it would seem that the technology

development factor represents only one aspect of the

decision-making process for the acceptability of a

device by a user (Claudy et al., 2015). Consumer

Behavioral Reasoning Theory predicts that the main

barriers to acceptability of innovative devices lie in a

trade-off between the use value and the functional

barriers perceived by the user (ibid.).

However, and as highlighted by (Claudy et al.,

2015), while the number of works investigating

resistance to innovation has increased significantly in

recent years, the majority of works are conceptual and

no operating system has yet been proposed,

particularly for the French and European MedTech

ecosystem.

In response to these challenges, several key

players in the French Medtech ecosystem have

undertaken an initiative codenamed "CML Santé

France", for Concept Maturity Levels, aiming to

establish a more inclusive, legible and structured

process for collaborative innovation processes in the

French Medtech ecosystem (Béjean et al., 2021, ANR

Dynsanté). This approach differs from other existing

more 'top-down' approaches (e.g. CIMIT) in that it is

highly participatory and community-based.

Mobilizing a national network, this endeavor brings

together, since 2017, partners covering the entire

Medtech value chain, from research labs to start-ups

and SMEs, including the Clinical Investigation

Centers in Technological Innovation (Inserm CIC-IT)

of the National Institute of Health and Medical

Research (INSERM) as well as new types of actors

called "Living Labs", the initiative having been

driven by the Forum of Living Labs in Health and

Autonomy (LLSA Forum).

So far the "CML Santé France" initiative has

resulted in the formalization of a vocabulary

associated with the design process of an innovative

dispositive, from the formulation of the initial idea

(CML1) to the post-industrialization follow-up of the

solution (CML9), passing through various

intermediate evaluation stages. This process provides

a methodological framework that integrates (i) the

definition of 9 levels of maturity, (ii) concrete actions

to structure the maturation activities for each level;

(iii) a mapping of the tools and skills needed to carry

them out. All of this was integrated into a

collaborative platform developed by the start-up

Agile Solutions. Consortium DynSanté, an ANR

program, was further constituted to further develop

and test the use of CML Santé France. Dynamically

integrating the CMLs Santé, the "CML Santé Forum"

platform has been used on concrete projects since

2019.

In the present work, we will present the "CML

Santé France" model, and the extent to which it

addresses the contemporary challenges of innovation

design in a user-centered approach in the context of

Health. Through the use of CMLs, we suggest a tool

aiming to participate in facing the very actual

demanding context around particularly medical

devices, that must anticipate the complex and

multidimensional regulatory, industrial, and clinical

evidence aspects as well.

2 THE IMPORTANCE OF NEEDS

AND USAGE VALUE

Innovations in technology are accelerating at a rapid

pace, in a way that the user can barely get used to a

technology, the following one that has just come out.

Innovation levels of the newly developed and highly

sophisticated systems do not reflect its degree of

acceptance by consumers admitting that innovative

products mean change for consumers, and resistance

to change is a common consumer response that must

be overcome before adoption can begin and that

consumers would instead prefer efficient and easy-to-

use systems that meet their needs (Laukkanen et al.,

2007). The application of self-concept to customer

behavior suggests that customers purchase products

and/or brands that are similar to their own self-

concept (Sirgy, 1982).

Hence the importance of integrating the concept

of user acceptance and satisfaction of needs in the

marketing approach of the product (Dunphy

&Herbig, 1995). In this direction, some

manufacturers rely on Kano model (KANO et al.,

1984) which has been widely adopted by industries

and consists of classifying and prioritizing customer

needs based on how they affect customer satisfaction.

(KANO et al., 1984). This customer survey-based

model (table 1) is able to illustrate the relationship

between product performance and customer

satisfaction in four types of product attributes: (1)

must-be attributes are expected by the customers and

they lead to extreme customer dissatisfaction if they

are absent or poorly satisfied, (2) one-dimensional

attributes are those for which better fulfillment leads

to linear increment of customer satisfaction, (3)

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

230

attractive attributes are usually unexpected by the

customers and can result in greater satisfaction if they

are available, and (4) indifferent attributes are those

that the customer is not interested in the level of their

performance (KANO et al., 1984; Xu et al., 2009).

Statistics showed that kano model-based marketing

strategies were positively influenced by the model

(Asian et al., 2019; Huang, 2017; Rotar&Kozar,

2017). On the other hand, the theory of reasoned

action suggests that consumers evaluate innovations

in regard to product attributes like relative advantage,

compatibility, and complexity, which have a strong

influence on their adoption decision (Claudy et al.,

2015b; Fishbein & Ajzen, 1977).

Before buying a product, customers have the

intention to consult different information sources

helping to decide which product to choose (Broilo et

al., 2016; Nici&Creutlein, 2017). To deal with the

increasing volume of information that may be false

and/or negatively influence customers, it may be

essential to include a usability evaluation in a

simulated environment. Such an approach reminds

the third attribute in Kano model product attributes

highlighting the importance of a pre-purchase study

englobing the product usage in a realistic

environment not only to stand out from competitors

(Joachim et al., 2018) but also to aid the consumer

decision-making (Broilo et al., 2016).

(Claudy et al., 2015b) analyzed the behavior of

users toward innovation and showed that there are 5

factors that manufacturers have not paid attention to

and which lead to user resistance to innovation. These

factors are subdivided into two categories: functional

and psychological barriers. Functional barriers

include usage, value and risk barriers that consumers

may associate with a new product or service whereas

psychological barriers are issues that consumers may

experience when innovations force them to change

existing beliefs or break with traditions and norms

(Antioco&Kleijnen, 2010; Claudy et al., 2015b).

Customers do evaluate both the reasons for and

against adoption, which can have a greater influence

on consumers' adoption decisions. This result reflects

the behavior of consumers in the market in a positive

or negative way. For example, in the first study

conducted by Claudy et al. (Claudy et al., 2015b)

reasons against adoption: high upfront costs,

perceived incompatibility with existing

infrastructure, and uncertainty regarding overall

performance; have a stronger influence on the

consumer adoption decision than reasons for

adoption: energy cost savings, environmental

benefits, and being independent from conventional

sources of energy like oil or gas; which influence

intentions only indirectly via attitudes (i.e. the

psychological tendencies that are expressed by

evaluating a particular entity with some degree of

favor or disfavor (Eagly and Chaiken, 1998)). These

results have helped managers to focus on overcoming

barriers to adoption, instead of over-emphasizing

reasons for adoption in order to improve the diffusion

of their product in the market. Unlike study 1, study

2 showed that reasons for adoption: saving money,

convenience and flexibility; have a stronger influence

on customer adoption decisions than reasons against

Table 1: Example of Kano items. (Xu et al., 2009).

Kano question Answer

Functional form of the question

(e.g., if the car has airbags, how do you feel?)

● I like it that way

● It must be that way

● I am neutral

● I can live with it that way

● I dislike it that way

Dysfunctional form of the question

(e.g., if the car does not have airbags, how do you feel?)

● I like it that way

● It must be that way

● I am neutral

● I can live with it that way

● I dislike it that way

Shaping User-Centered Health Innovation Through Assessment

231

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

232

adoption: availability and security; and that reasons

against adoption influence customer decisions only

indirectly via attitudes. Reasons for and against

adoption were elicited by a group of nearly equal

numbers of males and females, and it included

different age groups and educational levels (Claudy

et al., 2015b). These findings insist on integrating the

customer self-concept in the marketing strategies at

the pre-purchase phase to help businesses in

identifying the required needs they must fulfill

(basic), characteristics they should be competitive

with (i.e. performance) and the advantages making a

differential in the eyes of the customer (i.e.

excitement) (Tontini, 2007).

Specifically for medical devices we can also cites

the “IEC 62366 standard for usability engineering to

medical devices”, as well as the medical device

regulation (MDR) (EU) 2017/745 that cites explicitly

the importance of usability and users, to take into

account also for pre clinical and clinical evaluation.

3 THE FRENCH CML HEALTH

MODEL

The French CML Health (CMLH) model is an

iterative reading grid that decomposes the innovation

process into three interdependent axes: needs,

technology, and programmatic. It is therefore a direct

descendant of the original CML model, which

integrates these last two domains, by completing

it with an equivalent user-centered axis. It also

specifies the two original domains of technology

and programmatic in order to adapt them to the

French and European regulatory specificities in terms

of research methodology and data management

(Table 1).

3.1 Technological Maturity

An example of the milestones used to assign maturity

levels on the technology axis is shown in Table 2.

The first axis of the CML Santé France model comes

from the direct heritage of the TRL model mentioned

above. It assesses the development of technological

concepts, the management of the products that it

allows to obtain, as well as their ownership, by being

formalized on three axes: technological development,

data management, and intellectual property. The first

axis of technological development allows us to

gradually assess, on a scale of 1 to 9, the development

processes from the evaluation of the bibliographic

state of the art, through its critical functionality

simulations, to the management of the product life

cycle. The second axis permits us to appreciate the

way in which the project leaders will manage the data

resulting from their own devices, from the R&D data

(including bibliographic) to the protocols allowing

their protection as well as the automation of the

product life cycle data. Finally, its last axis of

intellectual property gives an insight about the

competitive study, from the monitoring of existing

patents to the management of infringements that

could emerge.

Table 2: Factorial structure of the French CMLH model

Domain Sub-domain

Needs

Usa

Marke

Clinicalproofs

Technology

Technicaldevelopmen

Data mana

emen

Intellectualpropert

Schedule

Pro

ect mana

emen

ulation

Fundin

Shaping User-Centered Health Innovation Through Assessment

233

Table 3: Example of the different maturity levels for each sub-area of the "Technological Maturity" domain.

CML1 CML2 CML3 CML4 CML5 CML6 CML7 CML8 CML9

Technological

development

State of

the art

Theorizing Functional

simulation

Software

demonstrator

Prototype

alpha

Technological

analysis for

improvement

Automation

of function

testing

Software bug

reports and

corrections

Product

lifecycle

management

Data

management

/ R&D data

collection

Software

data

structure

Cybersecurity Data

availability

Exploitation

of clinical

data

Data server

access

Implementation

of data

collection

devices

Production of

material-

epidemiology

data

Intellectual

property

Patent

monitoring

Patent-in-

principle

Specific

patents

Freedom of

operation

Process

patents

Intellectual

property of

clinical data

/ / Competitive

intelligence

3.2 Programmatic Maturity

An example of the milestones used to assign maturity

levels to the Programmatic axis is presented in Table

The second axis of the CML Santé France model

is the first real adaptation of the CML model as

developed by NASA. It conceptually takes up the

project management axis as well as part of its

regulatory axis, specifying it for the European and

French particularities which are organized for the

devices themselves (everything for access to market,

under the CE mark certification), but also for the

researches that focus on. The idea is also to follow the

“Good Clinical Practices” (GCP) and all ethical

principles protecting the individual that participates

in a research, from all kinds of risks when he or she is

involved in research aimed at acquiring new

biological or medical knowledge. It should be

remembered that researches are organized and carried

out on healthy or sick volunteers with a view to

developing new knowledge in the biological or

medical fields, and that the regulatory framework in

France is based on the european regulations, and with

recent updates for innovation that claims a medical

devices status (“clinical investigations” categories

proposed by ANSM french authority). It is also a

matter of ensuring that the methods for collecting and

processing health data comply with the General Data

Protection Regulation (GDPR) as well as the French

reference research methodologies (MR-00X), which

range from level 1 to 3.

Thus, the Programmatic axis of the CML Santé

France model enables the programmatic maturity of

the project to be assessed in three areas: project

management, regulatory aspects, and financial

aspects. The first axis, project management, is used to

assess the consortium formed for the project, from the

identification of the pilot to the renewal of the

development partnership, including the creation of

Test Beds, by assessing the nature of the partnerships

created. The second axis, called regulation, evaluates

the programmatic maturity from the first legal

investigation surrounding the project to the renewal

of the CE mark, including product risk analysis and

compliance with European (e.g. MDR, RGPD…) and

French (i.e. ethics; clinical investigations for medical

devices…) regulatory constraints. Finally, the last

axis, called financing, allows for a gradual evaluation

of the financial aspects, ranging from the

identification of potential sources of financing to the

updating of business economic assumptions in line

with the real-life use data of the device.

3.3 Needs Maturity

An example of the milestones selected for the

assignment of maturity levels on the Need axis are

presented in Table 4.

Finally, the last axis of the CML Santé France model

is the real innovation of the consortium in the

specification of the CML model as described by

NASA. It integrates the elements of the theory of

consumer behavior and in particular the jargon of its

barriers within the CML grid and thus makes it

possible to evaluate maturity in terms of needs on

three axes: uses, market, and clinical evidence.

Therefore, the first axis, Uses, provides an insight into

the value of use of the device as well as a metric of

the development process to ensure that the user has

been put at the center of the product development in

terms of uses. This axis allows us to gradually assess

its development from the identification of the social

context and in terms of public health, to the methods

of evaluation of the quality of care perceived by the

patient (PREM), through the performance of

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

234

acceptability studies. This axis thus makes it possible

to verify the removal of certain functional barriers, in

particular the conflict with usage patterns as

described by (Ram &Sheth, 1989). The second axis,

called the market axis, allows us to obtain information

on the competitive study that was carried out in terms

of uses, in particular from the establishment of a

review of the market literature to the evaluation of the

multicentricity of the market segments, including the

respective market access strategies. This axis thus

allows us to verify the removal of functional value

barriers as described by (Molesworth &Suortti,

2002), in particular by verifying the uniqueness of the

value proposition conveyed by the device. Finally, the

last axis, called Clinical Proof, makes it possible to

assess the quality of the clinical investigation that the

device has undergone, up to the establishment of

fundamental proofs through an exhaustive analysis of

the literature, up to the formalization of the processes

for evaluating the quality of the results of the device

as perceived by the patient (PROM). It is this last axis

that will manage the functional barrier of uncertainty

as described by (Stone &Grønhaug, 1993) and which

occurs when end-users have only limited access to

devices under development.

Table 4 : Example of the different maturity levels for each sub-area of the "Technological Maturity" domain.

CML1 CML2 CML3 CML4 CML5 CML6 CML7 CML8 CML9

Project

mangement

Driver

identification

Initial

analysis of

the project

risk

Tests beds Identification

complementa

ry skills

required

Detailed

development

plan of the

solution

Update of

project

elements and

risks

Identification

of marketing

and sales

skills

Closing of

the project

Review of

industrial

development

partnerships

Regulation Regulatory

framework

RGPD

Compliance

/ Ethical

analysis of the

product

Collection

of regulatory

data for

clinical

investigation

Consolidation

of the

technical file

for deposit

CE mark file Regulatory

framework

for data use

Renewal of

the CE

marking

Funding Identification

of funding

sources

Preparation

of the

business

plan

Demonstrator

financing plan

Formalization

of the business

plan

Financial

modeling

Minimum

Viable

Business

Model

Series A

Capital

Raising

Updating

economic

assumptions

with real-life

data

Table 5: Example of the different maturity levels for each sub-area of the "Need Maturity" domain (PREM : Patient Reported

Experience Measure ; PROM : Patient Reported Outcomes Measure).

CML1 CML2 CML3 CML4 CML5 CML6 CML7 CML8 CML9

Usage Social

and

public

health

context

Qualification

of a practice

situation

justifying the

need

Co-construction

of adapted

usage scenario

UX/UI lab

evaluations

Definition of the

usage

industrialization

scheme

Usability and

acceptability

assessment

Ecological

evaluation

of a pre-

series

Real-life

organizational

impact study

Quality

control

PREM

Market Review

of the

market

literature

Identification

of the value

proposition

Product

positioning and

expected impact

Quantificatio

of the

expected

impact

Market access

strategy

Characterization

of the device on

the basis of

usage surveys

Marketing

elements

(deployment,

export)

Refinement of

go-to-market

strategies by

customer type

Marketing

different

markets

Clinical

proofs

Review

of the

clinical

literature

Identification

of the

medical need

Clinical strategy Preliminary

clinical trials

Analysis of

clinical trials

Drafting of the

study report

(publication)

Multi-center

clinical trials

Medico-

economic

studies

Quality

control

PROM

Shaping User-Centered Health Innovation Through Assessment

235

3.4 An Example of the Implementation

of the Grid in a Health Project

In order to illustrate the use of the grid, we will now

present, as an example, some of the results obtained

during a project appraisal that was recently conducted

during the validation studies of the CML Health

questionnaire. They concern a particularly mature

project, but with some rooms for improvement

localized majoritarily at median and very advanced

CML levels.

3.4.1 General Maturity

Results for the self-reported General Maturity

measures for the sample project (anonymized) are

available in Figure 1. Data suggest that the project is

at the CML9 level, with an important homogeneity on

the needs and programmatic domains.

Regarding the general maturity and taking into

account all the sub-domains (Figure 1; Top-Left), the

self-reported data from the project leader suggest that

the entire sample project is at CML9 (M

=9), as the

project has exceeded the critical milestones on all the

maturity domains and sub-factors.

Furthermore, regarding maturity by CML domain

and considering the nine CML levels (Figure 1, Top-

Right), the self-reported data show that the most

mature and homogeneous domain is the Needs

domain [µ=5.67±.63], followed by the Programmatic

domain which seems slightly less mature and

homogeneous in its overall development

[µ=5.49±.81]. Finally, the data suggest that the most

heterogeneous domain in its development is the one

covering Technology, despite a higher average

maturity [µ=5.57±.89].

3.4.2 Focus on the Heterogeneity: Looking

at the Technological Domain

The results for the self-reported maturity measures of

the Technology domain are available in Figure 2. The

data suggest that the project is at the CML8 level in

the technology domain, with notable heterogeneity in

its constituent factors.

Regarding the maturity of the Technology domain

and considering all the factors that constitute it

(Figure 2; Top-Left), the self-reported data by the

project owner suggest that the sample project is at

CML8 level (M

=8), given that the project could not

exceed the critical milestones of CML9 at the level of

data management and intellectual property.

Regarding the maturity of the Technology domain

by CML factor and considering the nine CML levels

(Figure 2; Top-Right), the self-reported data show

that the most mature and homogeneous factor is

Technology Development [µ=5.81±.39], and whose

factorial CML level is 9. This factor thus contrasts

with the factors measuring data management

[µ=5.2±1.39] and intellectual property [µ=5.25±1.16]

whose data suggest a slightly lower level of maturity

(factorial CMLs at 8) with a more heterogeneous

development.

Regarding the maturity of the Technology

domain, for each CML level and each constituent

factor (Figure 2; Bottom), the self-reported data

suggest several possible areas of improvement.

Regarding intellectual property, these are mainly at

the CML1 level [µ

=4.5±.5]. This contrasts with the

data management factor, where the areas of

improvement are more likely to be found in CML5

[µ

=4±0] and CML6 [µ

=4±0] as well as in CML9

[µ=5.09±1.04] regarding data management

[µ

=3.75±2] and intellectual property specifically

[µ

=3±0].

Once the data was reported on the grid,

recommendations could be made to the project leader.

It was suggested for this specific domain to focus on

the production of solution exploitation data and to

update its competitive intelligence on services and

patents.

4 SHAPING USER-CENTERED

INNOVATION THROUGH

ASSESSMENT: THE LIVING

LAB MODEL

Now that we have described a model capable of a

priori catching the different parameters for estimating

the maturity of a health project, the framework in

which it can be used will be detailed and to formalize

the approach for its effective implementation.

In the current paper, we propose that the Living

Lab model, which emphasizes the collaboration

between the user and the designer throughout the

design process, is an effective method for developing

new devices. This model is based on iterative

evaluations, which are repeated over time, and

incorporates methodological techniques from both

the exact and social sciences. This approach allows us

to identify and understand the facilitators and

constraints associated with the use of the device, as

well as to ensure that these factors are taken into

account in future versions of the device under

examination. Additionally, this model allows for a

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

236

Figure 1: Dashboard of global maturity by level in self-assessment by the project leader (Association Innov’Autonomie -

DynSanté Concept Maturity Levels Questionnaire 180-items ; data taken from psychometric validation project runned by

Association Innov’Autonomie for illustration purpose). [Circle : outliers data].

Figure 2: Dashboard of Technological maturity by level in self-assessment by the project leader (Association

Innov’Autonomie - DynSanté Concept Maturity Levels Questionnaire 180-items ; data taken from psychometric validation

project runned by Association Innov’Autonomie for illustration purpose). [Circle : outliers data].

more efficient and effective development process, as

it allows for constant feedback and improvements

based on user needs and preferences (Zipfel et al.,

2022).

Generally speaking, a Living Lab is defined as a

place that practices user-centered research metho-

dologies to develop by co-design, use test, and

implement MedTech innovations in real-life contexts;

Shaping User-Centered Health Innovation Through Assessment

237

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

238

with a focus on placing the user

at the center of the

creation process and all of its stakeholders, such as

caregivers, academics, and entrepreneurs (Ballon et

al., 2005; Leminen et al., 2012; Pallot et al., 2010;

Veeckman et al., 2013).

A recent meta-analysis (Zipfel et al., 2022)

concluded that Living Labs methodologies had a

positive impact on the acceptability of the system and

the subsequent feasibility of the procedure and made

it possible to predict the perenniality of the

implementation which will be carried out later, as

observed previously by (Mulder et al., 2008). In

general, this work and its precursors have shown that

the implementation of a Living Lab evaluation

methodology allows us to hope for a better

subsequent adoption of the device (Kim & Chung,

2017).

Figure 3 describes the process of developing an

innovative solution in the Living Lab approach. The

first preparatory condition is to bring together all the

actors of the project ecosystem, from the end-users

with their caregivers, the academics, and the main

actors of the sector (e.g. industrials, funders...)

directly in the end-user's living place. The co-creation

process (1) then begins. It can take the form of

interviews or focus groups to highlight the needs of

the population in all their ecosystemic complexity as

well as their constraints of use, the main grievances

as well as to have a first estimation of the resultant

benefit/risk balance.

Once the device and its usage network have been

modeled for the first time, a validation phase (2) takes

place. It can take the form of technical tests (i.e. tests

on the basis of procedures to be followed) ensuring

the usability of the essential functions and especially

of use tests. A major importance must be given to the

evaluation methods in technical tests and usability

tests in order to ensure their interpretability and their

reproducibility on several subjects or several samples

and any hypothesis must be tested on a statistical

level.

Once the validation phase is completed, the

adaptation phase (3) begins. Its objective is to modify

the device in order to adapt its use to the constraints

of use that were identified in (2). Indeed, the first

circle of co-design, which was reduced, probably

contains biases that constrained the generalisability of

the device to a larger population (i.e. such as a

population of a market segment). These biases must

be corrected by the parameters measured during the

tests.

Thus, we understand that phases (2) and (3) are

cyclic, and that they allow to correct the device

iteratively until arriving at a satisfactory version on

the technical level (i.e. as evidenced by the technical

test) and on the usage level (i.e. as evidenced by the

usage test).

Figure 3: Co-creation process implemented in the Living Lab approach.

As a subject of experimentation rather than as a rights

holder in terms of intellectual property.

Shaping User-Centered Health Innovation Through Assessment

239

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

240

5 DISCUSSION AND

PERSPECTIVES

The objective of this work was to establish a

fundamental bibliographic link between successful

implementation and acceptability of a MedTech

innovation, as well as to provide a methodological

framework for applying this fundamental knowledge

in practice.

We first described the canonical models used by

our contemporaries, and in particular the TRL and

CML models developed by NASA (Mankins, 1995;

Wessen et al., 2013), which assess the maturity of a

project on technology development and proof-of-

concept metrics in the aerospace field. However,, we

have also highlighted through elements of the

literature that the acceptability and use of an

innovative MedTech device was not primarily based

on its intrinsic characteristics (Claudy et al., 2015),

contrary to spatial projects, but is indeed based on

criteria related to the consumer himself as well as his

usage traditions (Antioco&Kleijnen, 2010), where

the innovative parameter may even become a strong

barrier to the use of the device due to its perceived

uncertainty of use (Agarwal & Teas, 2001; Stone

&Grønhaug, 1993).

Thus, the TRL and CML models as canonically

described do not seem relevant for the evaluation of

consumer-related parameters rather than intrinsic

device parameters. The TRL and CML models are

particularly interesting for institutional evaluation

and the development of solutions for industries. The

TRL and CML models are particularly interesting for

institutional evaluation and lead to the development

of solutions for industries. These grids allow us to

arrive at solutions but they do not take enough into

account the acceptability and the use of these

solutions once developed..

We have therefore proposed a variant of the CML

model, the CML Santé France model, which

specializes the classical CML scales centered on

Technology by also opening its reading grid to the

elements of French and European regulations

(standards for scientific research methodology,

ethics, RGPD), and above all by granting a capital

place to the evaluation of the place of the end-user

and his ecosystem. Indeed, the CML Santé France

levels constitute a reference framework that does not

replace the interaction between the coach and the

entrepreneur, but provides to the coach a way to save

time, a knowledge base, a shared language, and to the

entrepreneur a rich, structured result that can be

shared and understood by other coaches or experts

who have adopted the method. The French field of

study is particularly interesting because regulation in

terms of health innovation is particularly heavy and

complex. This legislation is constantly evolving and

this is why there is a significant need for support in

this field. Even if there is a European will to define

common schemes for all countries, the regulation is

different from one country to another and must

therefore be adapted by country.

The use of this approach allows us to measure

maturity, to help companies optimize their evolution.

The goal is to carry out this type of assessment within

a controlled time frame, and to renew it at regular

intervals (for example, every year) to continue to

support the choices and decisions of start-ups.

In this sense, the CML Santé France

evaluation methodology seems particularly adapted

to the co-design methodological approach resulting

from Living Labs, where the user is placed at the

center of a creation process in partnership with all his

potential ecosystem (caregivers, funders...) and

making it possible to adapt the device to the reality of

the uses in an iterative fashion. Now that the

framework for the development of the system and a

relevant evaluation model have been described in the

literature, further research will focus on the

construction of CML Santé France tools for project

leaders and the experts who evaluate them. For the

time being, two standardized repeatable evaluation

questionnaires are being constructed, one for project

leaders, allowing them to quickly assess their level of

CML Santé France on all dimensions; the other for

expert evaluators of projects, enabling them to assess

the level of CML Santé France maturity on the basis

of a short oral presentation (i.e. pitch) by a project

actor. The data from these events are now being

analyzed as part of the psychometric validation of the

scale, in order to ensure the content and divergent

validity of the CML Santé France model as evaluated

in the construction of psychometric questionnaires for

clinical purposes (Gonzalez et al., 2021; Messick,

1989; Schmeiser et al., 2006). Moreover, its ease of

use and its effectiveness in comparison to existing

models is yet to be demonstrated in a large sample of

non-expert end-users.

ACKNOWLEDGEMENT

We would like to thank all those who contributed to

the preparation of the DYNSANTÉ project: Pierre-

Yves Traynard, Pôle ETP, Willy Allegre,

Cowork’Hit, Anne-Claude Lefevre, Cowork’HIt. The

project received a grant from the National Research

Agency to support the networking of partners

Shaping User-Centered Health Innovation Through Assessment

241

REFERENCES

Agarwal, S., & Teas, R. K. (2001). Perceived value :

Mediating role of perceived risk. Journal of Marketing

theory and Practice, 9(4), 1‑14.

Antioco, M., & Kleijnen, M. (2010). Consumer adoption of

technological innovations : Effects of psychological

and functional barriers in a lack of content versus a

presence of content situation. European Journal of

Marketing, 44(11/12), 1700‑1724. https://doi.org/

10.1108/03090561011079846

Asian, S., Pool, J. K., Nazarpour, A., &Tabaeeian, R. A.

(2019). On the importance of service performance and

customer satisfaction in third-party logistics selection:

An application of Kano model. Benchmarking: An

International Journal.

Ballon, P., Pierson, J., &Delaere, S. (2005). Test and

experimentation platforms for broadband innovation :

Examining European practice. Available at SSRN

1331557.

Broilo, P. L., Espartel, L. B., & Basso, K. (2016). Pre-

purchase information search: too many sources to

choose. Journal of Research in Interactive marketing.

Claudy, M. C., Garcia, R., & O’Driscoll, A. (2015).

Consumer resistance to innovation—A behavioral

reasoning perspective. Journal of the Academy of

Marketing Science, 43(4), 528‑544.

https://doi.org/10.1007/s11747-014-0399-0

Dawson, G. (2013). Soldier heroes: British adventure,

empire and the imagining of masculinities. Routledge.

Dunphy, S., &Herbig, P. A. (1995). Acceptance of

innovations: the customer is the key!. The Journal of

High Technology Management Research, 6(2), 193-

209.

Eagly, A. H., &Chaiken, S. (1998). Attitude structure and

function. In D. T. Gilbert, S. T. Fiske, & G. Lindzey

(Eds.), Handbook of social psychology (pp. 269–322).

New York: McGraw-Hill.

Gonzalez, O., MacKinnon, D. P., & Muniz, F. B. (2021).

Extrinsic convergent validity evidence to prevent jingle

and jangle fallacies. Multivariate Behavioral Research,

56(1), 3‑19.

Joachim, V., Spieth, P., & Heidenreich, S. (2018). Active

innovation resistance: An empirical study on functional

and psychological barriers to innovation adoption in

different contexts. Industrial Marketing Management,

71, 95-107.

Kano, N., &Seraku, K. (1984). F. and Tsuji, S..(1984).

Attractive Quality and Must-be Quality. The Journal of

the Japanese Society for Quality Control, 14(2).

Kim, J. S., & Chung, G. H. (2017). Implementing

innovations within organizations : A systematic review

and research agenda. Innovation, 19(3), 372‑399.

Laukkanen, T. (2007). Internet vs mobile banking:

comparing customer value perceptions. Business

process management journal, 13(6), 788-797.

Leminen, S., Westerlund, M., &Nyström, A.-G. (2012).

Living labs as open-innovation networks.

Mankins, J. C. (1995). Technology readiness levels. White

Paper, April, 6(1995), 1995.

Messick, S. (1989). Validity. Em r. Linn (org.), educational

measurement.(13-103). New York, NY: American

Council on Education and Macmillan Publishing

Company

Molesworth, M., &Suortti, J.-P. (2002). Buying cars

online : The adoption of the web for high‐involvement,

high‐cost purchases. Journal of Consumer Behaviour:

An International Research Review, 2(2), 155‑168.

Mulder, I., Velthausz, D., & Kriens, M. (2008). The living

labs harmonization cube : Communicating living lab’s

essentials. The Electronic Journal for Virtual

Organizations and Networks, 10, 1‑14.

Pallot, M., Trousse, B., Senach, B., &Scapin, D. (2010).

Living lab research landscape : From user centred

design and user experience towards user cocreation.

First European Summer School" Living Labs".

Ram, S., &Sheth, J. N. (1989). Consumer resistance to

innovations : The marketing problem and its solutions.

Journal of consumer marketing, 6(2), 5‑14.

Salazar, G., &Russi-Vigoya, M. N. (2021). Technology

readiness level as the foundation of human readiness

level. Ergonomics in Design, 29(4), 25-29.

Schmeiser, C. B., Welch, C. J., & Brennan, R. L. (2006).

Educational measurement. American Council on

Education and Praeger Publishers, Westport, CT.

See, A., Roller, S., Kiela, D., & Weston, J. (2019). What

makes a good conversation? how controllable attributes

affect human judgments. arXiv preprint

arXiv:1902.08654.

Sirgy, M. J. (1982). Self-concept in consumer behavior: A

critical review. Journal of consumer research, 9(3), 287-

300.

Stone, R. N., &Grønhaug, K. (1993). Perceived risk :

Further considerations for the marketing discipline.

European Journal of marketing, 27(3), 39‑50.

Tontini, G. (2007). Integrating the Kano model and QFD

for designing new products. Total Quality

Management, 18(6), 599-612.

Veeckman, C., Schuurman, D., Leminen, S., &Westerlund,

M. (2013). Linking living lab characteristics and their

outcomes : Towards a conceptual framework.

Technology Innovation Management Review, 3(12

december), 6‑15.

Wessen, R., Borden, C. S., Ziemer, J. K., Moeller, R. C.,

Ervin, J., & Lang, J. (2013). Space mission concept

development using concept maturity levels. AIAA

Space 2013 Conference and Exposition, 5454.

Xu, H., & Gupta, S. (2009). The effects of privacy concerns

and personal innovativeness on potential and

experienced customers’ adoption of location-based

services. Electronic Markets, 19, 137-149.

Zipfel, N., Horreh, B., Hulshof, C. T. J., de Boer, A. G. E.

M., & van der Burg-Vermeulen, S. J. (2022). The

relationship between the living lab approach and

successful implementation of healthcare innovations:

An integrative review. BMJ Open, 12(6), e058630.

https://doi.org/10.1136/bmjopen-2021-058630

ClinMed 2023 - Special Session on European Regulations for Medical Devices: What Are the Lessons Learned after 1 Year of

Implementation?

242