Comparison of Parametrization Approaches for Scenario-Based Testing

Christoph Glasmacher

, Marcel Sonntag

and Lutz Eckstein

Institute for Automotive Engineering, RWTH Aachen University, Steinbachstraße 7, Aachen, Germany

Keywords:

Scenario, Testing, Parametrization, Simulation, Automated Driving.

Abstract:

Scenario-based testing is a promising approach to assess and assure the safety of automated and connected

driving functions. In this approach, test scenarios are often described in an abstract way. Norms sometimes

even provide certain parameter values for, e.g., approaching maneuvers in lane-keeping situations. However,

the type of parametrization is often not fully speciﬁed - neither in databases nor in regulations. This paper

assesses differences in possible types of parametrizations for test scenarios and gives guidance about the im-

portance to choose a suitable parametrization for individual use cases. For this, different parametrization types

are categorized. The effects on the outcome of tests are investigated in a comprehensive study simulating

435,456 test cases in the CARLA simulator. Thereby, 8 different systems under test are investigated to ob-

serve the outcome on different parametrizations on intersections. The results show a high inﬂuence of the

parametrizations for different systems under test on the test outcomes leading to the need for carefully select-

ing a suitable parametrization approach.

1 INTRODUCTION

Due to technological advancements in machine learn-

ing and sensor systems, the development of auto-

mated driving systems (ADSs) sped up and ﬁrst appli-

cations are introduced to the market (Mercedes-Benz

AG, 2024; Waymo LLC, 2024). For the introduction

and extension of them, it is required by regulatory

bodies to assess the safety of ADSs before market in-

troduction (UNECE, 2021). To cope with the com-

plexity of today’s trafﬁc, scenario-based approaches

such as (PEGASUS Project Consortium, 2019) or

(Galbas et al., 2022) have emerged in recent years to

facilitate those safety analyzes. Scenario-based ap-

proaches also made it into standardization (Interna-

tional Organization for Standardization, 2022a) and

legislation (UNECE, 2021). Within scenario-based

testing, one goal is to derive the suitable scenarios

to represent the related operational design domain

(ODD) for a given use case or application (e.g., mo-

torway chauffeur) as realistically and completely as

possible. Based on those scenarios, ADSs are to be

tested in simulation as well as in real-world condi-

tions.

Substantial research has been performed on iden-

tifying and systematizing these scenarios (or scenario

https://orcid.org/0000-0003-4826-9706

https://orcid.org/0009-0003-1247-1782

categories) to represent the ODD (Weber et al., 2023a;

de Gelder et al., 2022). Using these scenarios for

testing purposes requires deﬁning models represent-

ing them. Usually, these models consist of parame-

ters deﬁning characteristics, such as the starting con-

ditions of actors involved. To generate concrete or

logical scenarios, parameter values or ranges are as-

signed to these models. Research has also been per-

formed on parameter value assignment (Glasmacher

et al., 2023b). However, there is no common ground

on how to deﬁne the models representing the scenar-

ios, e.g., in simulations to achieve meaningful out-

comes. In other words, there is no common ground

in which way and with which parameters (not the val-

ues of them) should deﬁne the scenarios for testing

to allow deriving valid conclusions for the speciﬁed

test case. This task is called scenario parametriza-

tion. Standards like OpenSCENRIO XML (ASAM

e.V., 2024) deﬁne a common language for scenario

modelling for simulations, but no guidance is given

on how to do the parametrization, i.e., how to deﬁne

the scenario elements in detail to, e.g., deﬁne dynamic

objects in the surrounding of the system under test

(SuT). One recent study (de Gelder and Camp, 2024)

provides ﬁrst analyzes on this topic, considering dif-

ferent levels of detail of scenario parametrization of

those objects. This is analyzed for scenarios in longi-

tudinal trafﬁc. However, different degrees of reactiv-

Glasmacher, C., Sonntag, M. and Eckstein, L.

Comparison of Parametrization Approaches for Scenario-Based Testing.

DOI: 10.5220/0013282200003941

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 11th International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS 2025), pages 439-447

ISBN: 978-989-758-745-0; ISSN: 2184-495X

439

ity of those objects or the general availability to react

are not analyzed. The dynamic objects could be de-

ﬁned, e.g., using ﬁxed trajectories or arbitrarily com-

plex driver models. The inﬂuence of these types of

parametrizations remains unclear in simulation out-

comes.

In this work, we systematically deﬁne and cate-

gorize relevant terms and representations for scenario

parametrization (see Section 3). Based on this cat-

egorization, three different scenario parametrizations

of the same scenario are set up for testing in simula-

tion, considering different degrees of reactivity of the

dynamic objects in the surrounding of the SuT (see

Section 4). Those parametrizations are analyzed by

conducting an extensive experiment. The three dif-

ferent parametrizations combined with four different

implementations of an SuT behavior as well as param-

eter values sampled from deﬁned parameter ranges

result in more than 400,000 simulated cases. This

allows an extensive analysis of the inﬂuence of the

parametrizations on simulation outcomes under vary-

ing conditions. The results (see Section 5) show that

the choice of the scenario parametrization has signif-

icant inﬂuence on the test results with respect to the

number and characteristics of the resulting collisions.

Based on this, guidance is given on how to select a

suitable scenario parametrization for the intended use

(see Section 6).

2 STATE OF THE ART

In recent years, several publications on scenario-

based testing have been made, applying the developed

concepts to different use cases for scenarios. In order

to structure this ﬁrst, an overview of some of these

various use cases is provided. Based on this, a de-

tailed look is taken at their parametrization and the

differences are elaborated.

2.1 Scenario-Based Testing and

Applications

Due to the high complexity of today’s trafﬁc, regu-

lar testing approaches solely based on on-road test-

ing are not applicable for testing ADSs, as the re-

quired mileage to be able to derive valid conclusions

is within an unreasonable order of magnitude (Winner

et al., 2018). Thus, scenario based approaches were

introduced to enable a more systematic way for test-

ing relevant situations (PEGASUS Project Consor-

tium, 2019). A scenario, as deﬁned in ISO 34501 (In-

ternational Organization for Standardization, 2022b),

is a sequence of scenes usually including the au-

tomated driving system(s) / subject vehicle(s), and

its/their interactions in the process of performing the

dynamic driving task (DDT).

While the term scenario is rather well-deﬁned, the

process of parametrization of scenarios is not deﬁned

precisely, leading to inconsistent use. Some use the

term for the process of selecting speciﬁc parameter

values from given ranges to come to concrete scenar-

ios (Bach et al., 2016). Others are using the term for

the process of identifying parameters that represent

the desired scenarios in a meaningful way (de Gelder

and Camp, 2024).

2.2 Different Parametrizations

When deﬁning scenarios for simulation, models for

the actors involved are to be deﬁned to model their

individual behavior. These models can have varying

complexity, leading to different parameters that are to

be deﬁned. This is an important part of the scenario

parametrization.

The most common standard for deﬁning scenar-

ios for simulations is the ASAM OpenSCENARIO

XML standard (ASAM e.V., 2024). It allows differ-

ent ways for the representation of the actors’ behavior.

First, ﬁxed trajectories can be used deﬁning dynamic

states of the actors at consecutive positions over time.

Second, behavior can be deﬁned using predeﬁned ac-

tions and triggers, allowing actors to react to deﬁned

circumstances and interactions. Third, custom con-

trollers can be speciﬁed to model further interactions.

Those different possibilities lead to the need for se-

lecting an appropriate approach for the desired use,

which might require different models for deﬁning the

actors’ behaviors.

Within the V4SAFETY project, for the purpose of

prospective safety impact assessment, different cate-

gories of models based on the actors’ reactivity have

been deﬁned (Fahrenkrog et al., 2024). Within the de-

ﬁned baseline generation approaches, one can either

use pre-simulation or in-simulation approaches. For

the former, the characteristics of the object’s trajec-

tory are deﬁned before the simulation is started. For

the latter, the trajectories are deﬁned during the sim-

ulation based on the actual interactions. Triggers and

actions might deﬁne this behavior. This implies that

different approaches require different parametriza-

tions of scenarios, leading to the need of deﬁning dif-

ferent parameters.

In addition, different levels of detail of modelling

objects’ behavior can be considered. Within the study

of (de Gelder and Camp, 2024), the inﬂuence of this

on simulation results is analyzed for the scenarios de-

ﬁned in the regulation UN ECE R157 (UNECE, 2021)

VEHITS 2025 - 11th International Conference on Vehicle Technology and Intelligent Transport Systems

440

focussing on longitudinal trafﬁc. This includes, e.g.,

different ways of parametrizing lane changes within

cut-in scenarios. A substantial inﬂuence of the differ-

ent parametrization approaches on the test outcomes

is shown. Though, the general inﬂuence of different

parametrizations, e.g., considering objects’ reactivity,

is not discussed in current research.

This stresses the need of further investigating

the inﬂuence of different scenario parametrizations

to give guidance on how to select appropriate ap-

proaches.

3 DEFINITION AND

CATEGORIZATION OF

PARAMETRIZATIONS

Section 2.2 shows that scenarios are often described

in different ways. In order to systematize different

types, deﬁnitions are ﬁrst discussed. Based on these,

different categories of scenario parametrization are

used to describe them.

3.1 Deﬁnitions

Although terms such as “parameter” or abstraction

levels for scenarios are frequently used in ISO doc-

uments (International Organization for Standardiza-

tion, 2022b), the level of detail leaves space for mis-

interpretation in the following methodology. To avoid

this, deﬁnitions are set up for the task of parametriza-

tion in the same manner as for the distinction of types

of logical scenarios as in (Glasmacher et al., 2023a):

• Scenario Parameter: Scenario parameters are el-

ements of a logical or concrete scenario which

quantify its characteristics (incl. involved actors).

• Scenario Parametrization: Process of modeling

a logical scenario with a set of parameters without

assigning concrete values, ranges or distributions.

• Scenario Parameter Value Assignment: Param-

eter value assignment describes the assignment of

values, ranges or distributions to predeﬁned sce-

nario parameters.

Using an analogy with mathematics, parametriza-

tion opens up the vector space that may be designed to

represent the real world. The assignment of the values

or ranges then only sets a point or a body in the (high-

dimensional) spanned space, which then represents a

concrete or logical scenario.

3.2 Categorization of Parametrization

Approaches

Scenarios are described differently in the literature

depending on the use case. In the context of this

paper, categories for the parametrization of scenar-

ios that contain parameters are of particular interest.

We distinguish among three categories for scenario

parametrization: level of abstraction, degree of reac-

tivity, and type of description language.

The category of abstraction levels is traditionally

used frequently and encompasses the level of detail.

Common levels are functional, abstract, logical and

concrete. As the level of detail increases with lower

abstraction level, information must be added to make

the description complete.

We deﬁne reactivity as the interactivity of the sce-

nario elements to actions of the system under test. If a

scenario consists exclusively of predeﬁned elements,

there is no reactivity of the scenario elements and the

control loop of action and reaction of the scenario el-

ements is not closed in the simulation. Thus, it is e.g.

a direct replay of a recorded scenario. On the other

side of the spectrum, all scenario elements would re-

act to actions of the ego over the entire time span of

the scenario. This is the case for road users, for exam-

ple, if they continuously adjust to the ego’s behavior

and adapt their own behavior. Further gradations are

possible between these two extremes.

The last category is the description language.

Based on the selected parameters, the reactivity and

the resulting level of abstraction, this can be selected

to allow an (machine readable) interpretation of the

scenario. Scenario description languages such as

ASAM OpenScenario XML/ DSL or proprietary de-

scriptions deﬁne those languages.

4 EXPERIMENT

The inﬂuence of different parametrization approaches

on a simulation result is assessed within an experi-

ment to investigate whether there is a relevant inﬂu-

ence on the outcome of a potential assessment of a

SuT focussing on the aspect of reactivity. In the ex-

periment, a total of 435,456 scenarios are generated,

simulated and evaluated to evaluate the inﬂuence of a

wide range of combinations within and between dif-

ferent parametrization approaches. Within those sce-

narios, aspects with potentially high inﬂuence on the

simulation result as the parametrization approach, un-

derlying parameters and the system under test are var-

ied for a comprehensive assessment. After setting up

the scenarios (Section 4.1) and SuTs (Section 4.2),

Comparison of Parametrization Approaches for Scenario-Based Testing

441

Figure 1: Constellation of ego road user (blue) and conﬂict-

ing road user (orange).

each system under test is confronted with the scenar-

ios using the simulation platform CARLOS (Geller

et al., 2024) which is based on the CARLA simulator.

They are assessed based on crashes and impact speeds

since those are some of the most common metrics for

assessing safety.

4.1 Scenario Parametrization and

Parameter Value Assignment

Assessing a scenario with multiple characteristics can

become a high dimensional problem. This increases

even more if multiple parametrization approaches and

different SuTs should be assessed. To limit this com-

plexity, one speciﬁc abstract scenario is chosen. We

decided to assess the scenario ego passing straight

with intersecting object from right passing straight

according to (Weber et al., 2023b) due to the high

relevance in recent assessments. It is designed with

vehicles for both road users, but since the models

don’t account for this difference, conclusions should

be equally valid for other road user types such as

VRUs or other situations (e.g. including occlusions).

The scenario is applied to the four-arm intersection

Frankenburg of the inD dataset (Bock et al., 2020).

The constellation is shown in Figure 1.

According to Section 3.2 numerous possible

parametrizations exist. The focus of this experiment

is to investigate the inﬂuence of different reactivi-

ties within the deﬁned scenario. Thus, three differ-

ent parametrizations for the conﬂicting road user (ob-

ject) in the same abstract scenario are set up accord-

ing to the reactivity. The focus of this experiment is

on the object’s velocity (-proﬁle) while keeping the

same path:

• Constant Velocity: As a representative of the

ﬁxed trajectory approach, the object is modeled

with a constant velocity not reacting to the behav-

ior of the ego vehicle.

• Synchronization: For this approach, the behav-

ior can be subdivided into two phases: before and

after a locally ﬁxed synchronization point for the

object. Before this point, the object tries to pro-

voke a collision (PL = 0) with the ego by acceler-

ating or decelerating. After synchronization, the

velocity does not change anymore. Thus, reactiv-

ity of the object is given to a certain point.

• Adaptive Function: This approach is compara-

ble to the synchronization approach, but always

tries to create a collision under given acceleration

constraints without stopping this synchronization

at a certain point. This reﬂects the highest degree

of reactivity.

These approaches are detailed by deﬁning param-

eters and assigning parameter values. For the parame-

ter value assignment, initial conditions are set equally

for the different approaches in a realistic range for

intersections. Besides these, speciﬁc parameters are

set for the individual approaches if necessary (Ta-

ble 1). To enable a simple sampling approach while

still generating crashes easily and allowing compar-

isons, whenever possible, parameters are set in a way

that the actors reach the conﬂict after a deﬁned time.

This means, for example, that the initial position is

calculated based on the sampled initial velocity in the

way, that the actor reaches the conﬂict area after a de-

ﬁned timespan, taking the sampled priority level into

account.

The desired conﬂict is described with a predicted

priority level (PrPL). According to (Hu and Li, 2017)

this is deﬁned in the interval [−1,1]. However, to de-

scribe also near misses, this is extended to [−1.5,1.5].

Using a uniform sampling, 6,804 concrete sce-

narios are created for the constant velocity approach,

20,412 for the synchronization approach and 27,216

for the adaptive approach. This sampling approach

ensures a good comparability between approaches

and does not include bias due to the parameter value

assignment.

4.2 Systems Under Test

The experiment is conducted with different SuTs, as

different characteristics of a SuT might lead to differ-

ent effects of parametrization approaches:

• No reaction (NR): As a baseline, this system un-

der test does not react on the object.

• Autonomous Emergency Braking System

(AEB): An AEB system is deﬁned with a

time-to-collision (TTC) threshold (T TC < 1s)

and a distance headway (DHW) threshold

(DHW < 3m). Once one condition is reached, an

emergency breaking maneuver is performed.

VEHITS 2025 - 11th International Conference on Vehicle Technology and Intelligent Transport Systems

442

Table 1: Parameter ranges for experimental setup.

Parameter Unit Constant velocity Synchronization Adaptive Function

Predicted pre-crash time [s] [0.3, 5.0]

ego,init

[m/s] [3.0, 20.0]

ob ject,init

[m/s] [3.0, 20.0]

Predicted Priority Level [-] [-1.5, 1.5]

Max. acceleration [m/s

] - [9.81] [1.0, 9.81]

Synchronization time to conﬂict [s] - [0, 2.0] -

Table 2: Logic system under tests.

SuT Logic Acc.

AEB T TC<1.0s ∨ DHW <1.5m -9.81 m/s

Assist

PrPET <1.0 ∧ PrPL>1.1

PrPET <1.0 ∧ PrPL<1.1

2 m/s

-2 m/s

• Intersection Assist (As): An intersection as-

sist (Assist) does not only break, but acceler-

ates and decelerates based on the predicted post-

encroachment-time (PrPET) and a predicted pri-

ority level (PrPL, (Hu and Li, 2017)) with a

smaller acceleration/ deceleration to avoid a con-

ﬂict (see Table 2).

• AEB and Assist (A-As): Both AEB and Assist

are used in combination. Once the AEB is acti-

vated, it overrides the Assist function.

For each SuT, two variants are set up to acknowl-

edge different approaching maneuvers for intersec-

tions. This is done by deﬁning a target speed to

which the system under test decelerates when ap-

proaching the intersection. Such a behavior can re-

ﬂect approaching an intersection on a minor or prior-

ity road. The approaching target speed is either set

to const meaning that it does not intent to reduce the

speed or to 3m/s for a more cautious approaching ma-

neuver when the SuT might need to yield. Taking the

number of different scenarios and the different SuTs

into account, 435,456 test cases are set up and simu-

lated in the CARLOS simulation environment.

5 RESULTS

According to the experiment description in Section 4,

the test cases created are evaluated a posteriori us-

ing various metrics. Whereas mainly results based on

accidents are shown in this section, analysis of vari-

ous TTX metrics such as TTC, THW, PET, but also

metrics with predictions such as PrPET and PrPL are

incorporated in the conclusion. However, due to the

limited space and similar statements, the focus within

the results is on collisions. For this purpose, the crash

ratio and collision speeds are primarily taken into ac-

count.

The inﬂuence of individual parameter values are

not analyzed below, but rather the inﬂuence of the

parametrization on the outcome of the scenario. Ac-

cordingly, the respective scenarios are not considered

individually, but the results are aggregated. The re-

sults are clustered in two sections: the inﬂuence of the

parametrization approach itself, and the inﬂuence of

certain parameters in combination with the approach.

5.1 Inﬂuence of Parametrization

The comparison of the different parametrizations

shows differences in the output of a simulation de-

pending on the parametrization approach and the SuT

used. These differences are not linear in each case,

they partly counteract each other and depend on indi-

vidual parameters (see Figure 2).

In order to assess the effects of different

parametrizations more deeply, a baseline is used as

the cases with the object road user with constant ve-

locity parametrization and with a SuT with no reac-

tion. Figure 2 shows for this baseline (two left blue

bars) that all generated cases with −1 < PL < 1 re-

produce a crash and according to the deﬁnition for

PL = −1.5 and PL = 1.5 no crashes are detected. For

PL = −1 and PL = 1, crashes are mostly detected but

not always due to numerical inaccuracies, since this

mathematical limiting case is extremely unstable. Al-

ready in the baseline, it can be seen that different SuTs

show signiﬁcantly different crash rates depending on

the deﬁned target speed (ts) for entering the intersec-

tion.

If we look at the changes caused by the inﬂuence

of the other parametrization approach on this base-

line, we can see signiﬁcant differences depending on

the behavior of the SuT. As expected, the synchro-

nization approach produces more crashes compared

to the baseline if no AEB is included since it reacts to

the behavior of the SuT til a certain point to provoke

the desired constellation - in this case a conﬂict sit-

uation. Less crashes are produced for the SuTs with

AEB especially at PL > 0 since the object want to pro-

voke a PL = 0 and so makes it easier for the AEB to

Comparison of Parametrization Approaches for Scenario-Based Testing

443

Figure 2: Inﬂuence of different parametrizations and SuTs on the case results.

prevent these cases. The adaptive approach produces

generally even more collisions than the synchroniza-

tion parametrization approaches. This is due to the

fact that in this case a collision is provoked up to the

conﬂict point and not only a prior point in time. In this

case the SuT can only prevent a collision by stopping

before entering the conﬂict area or if the initial con-

ditions don’t allow a conﬂict. This can be the case if

the pre-crash time is relatively short and PL /∈ [−1,1]

is selected. So, for a predeﬁned SuT we can already

see a signiﬁcant inﬂuence of the parametrization ap-

proach on the simulation outcome.

When investigating the inﬂuence of the

parametrization depending on different SuTs, a

similar conclusion can be drawn. However, there

is no clear correlation between the SuTs and

parametrization approaches affecting the crash rates

of the test cases. Although the synchronization

produces fewer crashes for SuTs compared to the

adaptive function, it is the opposite for the baseline

SuT since the synchronization includes on average

a higher acceleration intensity than the adaptive

approach within the study. Furthermore, the syn-

chronization approach produces even fewer crashes

compared to the baseline in those cases, in which

the synchronization is rather early and the change of

the object propagate until the conﬂict area no matter

how the SuT behaves. So, taking the inﬂuence of

the SuT in comparison with the parametrization into

account, the impact of the parametrization approach

on the simulation outcome becomes less predictable

between these SuTs.

Figure 3: Impact speed distributions of object in compari-

son to crash ratios.

These differences can be seen not only in the num-

ber of collisions, but also in the object road user’s

impact speed (see Figure 3). Thereby, it is not dis-

tinguished between different target velocities. This

shows that collisions not only occur at different fre-

quencies, but that the severity of a collision does not

change proportionally. Rather, a more differentiated

picture must be drawn. Although the highest colli-

sion speeds of the object road user are recorded for

the constant velocity, values for synchronization and

adaptive function are similar for the assist function

but signiﬁcantly different for the AEB. This can be

explained partly by the different intensities of the ac-

celeration of the object within different parametriza-

VEHITS 2025 - 11th International Conference on Vehicle Technology and Intelligent Transport Systems

444

Figure 4: Pre-crash time inﬂuence for crash rates (CR) for

SuTs with different target speeds const (blue) and 3 m/s

(yellow) as well as constant velocity (solid), synchroniza-

tion (dashed) and adaptive (dotted dashed) averaged over

all investigated PLs.

tions. Furthermore, a shift can be explained by the

later reaction of an AEB which may be more often

after the synchronization point. This shows that the

quantitative inﬂuence of different parametrizations on

the evaluation of a scenario can be signiﬁcant and dif-

ﬁcult to estimate a priori, as various factors usually

have to be taken into account.

5.2 Implications on Parameter Value

Assignment

In addition to the aggregated inﬂuence of the

parametrization approach on a simulation outcome it-

self, we investigated how that effect varies depending

on the actual parameter values assignment. This in-

ﬂuence is investigated by varying the pre-crash time

and PL. According to Figure 4, the number of crashes

with regard to the inﬂuence of the pre-crash time and

its dependency with parametrization approach and the

SuT can be signiﬁcant. The ﬁgure shows that the dif-

ferent characteristics of a parametrization and an SuT

change based on the pre-crash time. While a similar

outcome can be observed for a pre-crash time close

to zero seconds, this changes with increasing time

and converges in some cases. With regard to the ef-

fects of the SuT, it can be observed that shorter re-

action times of SuTs such implemented in the AEB

lead to faster convergence, while the assist function

takes longer. We show that different effects work

against each other, especially with the more adaptive

parametrizations. For scenarios without a designed

collision, the state of conﬂict must ﬁrst be reached,

Figure 5: Pre-crash time inﬂuence for PL = 0 distinguishing

target speed const (blue) and 3 m/s (yellow) as well as con-

stant velocity (solid), synchronization (dashed) and adap-

tive (dotted dashed).

resulting in more crashes (orange dotted dashed line

curve). Especially with long pre-crash times, how-

ever, the early inﬂuence of the delay towards the in-

tersection takes over the larger part, so that the curves

decrease again with longer lead times of scenarios.

Taking the PL as another parameter in combina-

tion into account, the picture can be drawn slightly

differently (see Figure 5). In this case, the effect of

the rise is eliminated, as collisions already occur at the

beginning and only the SuT can prevent them. How-

ever, the ability naturally goes hand in hand with the

time available and the intensity of the SuT, as well

as a counter movement of the parameterized object.

Accordingly, the inﬂuence of the parametrization is

shown. For example, the adaptive approach can en-

sure crashes for the longest pre-crash time, but falls

off relatively faster. This shows that the inﬂuence of

time also has a signiﬁcant inﬂuence, which in turn de-

pends on the parametrization and the SuT. However,

it also shows that saturation effects occur with a suf-

ﬁciently long pre-crash time, but these make the sce-

nario less robust in relation to the originally desired

constellation and therefore depending on the proce-

dure harder to incorporate in a safety argumentation

or assessment. So, it can be shown that even individ-

ual parameter value assignments itself may have rel-

evant impacts varying depending on the parametriza-

tion.

Comparison of Parametrization Approaches for Scenario-Based Testing

445

6 DISCUSSION

The results show that not only the parameters itself,

but also the approach of parametrization can have a

signiﬁcant inﬂuence on the simulation outcome. This

implies that the comparability of two analyses cannot

be described purely in terms of naming the outcome

of abstract scenarios. Similarly, a relative comparison

between a baseline and a SuT is not transferable as

the inﬂuence is not independent of the parametriza-

tion. Rather, for a direct comparison of both abso-

lute ﬁgures and relative evaluation a comparison of

the parametrization type of a scenario is also neces-

sary and should be presented transparently.

Furthermore, the analysis of the pre-crash time in

a scenario shows that its inﬂuence can be signiﬁcant.

If it is too short, functions cannot react adequately. If

it is too long, instabilities can play a role depending

on the simulation model. Accordingly, this should be

considered in advance for a valid result.

The results are based on the comprehensive anal-

ysis of one abstract scenario. However, a generaliza-

tion of the discussed results is possible for several

reasons: On the one hand, the scenario used repre-

sents a frequently occurring case. On the other hand,

the models used to control road users are so general

that no assumptions were made about the type of road

user. Although there will be deviations for other road

users due to the dimensions, there is no reason to

assume that observed effects (possibly in a different

form) will not also be found in other scenarios and

may be even more signiﬁcant depending on the com-

plexity.

Based on the results, multiple ﬁndings can be

summarized and recommendations can be derived:

• Different parametrizations produce fundamen-

tally different results. Accordingly, the design

should be closely coordinated with the purpose of

the test.

• It is not advisable to compare outputs from sce-

narios directly with each other without detailed

knowledge of their design, as these can differ.

• The longer the simulation, the less likely it is

that even small changes will affect an outcome.

Accordingly, the scenario should only be long

enough to see the desired effects.

• The pre-crash time before the effect to be inves-

tigated should be long enough to allow a reaction

within the scenario.

• For a good traceability and transparency, it is ad-

visable to document key design decisions of the

scenario parametrization.

As proposed in the ﬁrst recommendation, the

choice of scenario parametrization should depend on

the use case, the purpose of the test, and the models

used. Therefore, based on our ﬁndings, we propose

questions which may guide in the selection of a proper

approach.

• Can the SuT (or any other reactive in-simulation

model) show any reaction because of the sce-

nario component? If no reaction is to be expected

within the scenario with regards to the scenario el-

ement, there is no need to generate a reactive sce-

nario and actions can be predeﬁned. In this case,

further questions are not needed.

• Is the purpose of the scenario to challenge the SuT

as much as possible to test conﬂict avoidance? If

it should be used for a falsiﬁcation of a SuT, the

other road users should try to provoke an accident

until the end. This leads to a high reacivity within

the scenario.

• Should the behavior of the scenario components

reﬂect usual/ realistic behavior? If that is the case,

an interactivity leading to a falsiﬁcation may be

not suitable - neither may a purely trajectory fol-

lowing description be but a speciﬁc model may be

needed.

• Is an adequate reaction model for scenario com-

ponents available? If that is the case, it may make

sense to go for a higher reactivity since the used

model.

When answering these questions, limitations may

arise due to missing information, trade-offs that have

to be made, or missing models. These limitations

should be made transparent to allow contextualization

of test results.

7 CONCLUSION

Within the paper the authors assessed the inﬂuence of

different parametrization approaches on the result of

an assessment. Therefore, the difference between a

parametrization approach and the actual parametriza-

tion is deﬁned. Guidance is given which effects from

parametrization approaches should be considered for

different use cases. The inﬂuences of different ap-

proaches are shown in a large study assessing differ-

ent SuTs and parameter values. As one aspect, the

pre-crash time is shown as a relevant (meta) param-

eter affecting the outcome of a scenario. We show

that the parametrization has a relevant effect on the

outcome and should be chosen carefully. For this,

recommendations are ﬁnally given to account for the

VEHITS 2025 - 11th International Conference on Vehicle Technology and Intelligent Transport Systems

446

capabilities of a SuT and the purpose of a test. It is es-

sential to carefully choose and adequately document

the parametrization approach to allow for a reliable

statement within a safety assessment or safety assur-

ance argumentation. These recommendations could

be detailed for speciﬁc use cases.

ACKNOWLEDGEMENTS

The work of this paper has been done in the context

of the V4SAFETY project which is funded by the Eu-

ropean Commission’s Horizon Europe Research and

Innovation Programme under grant agreement num-

ber 101075068. Views and opinions expressed, are

those of the author(s) only and do not necessarily re-

ﬂect those of the European Union or the European

Climate, Infrastructure and Environment Executive

Agency (CINEA). Neither the European Union nor

the granting authority can be held responsible for

them.

REFERENCES

ASAM e.V. (2024). ASAM OpenSCENARIO® XML Ver-

sion 1.3.0.

Bach, J., Otten, S., and Sax, E. (2016). Model based sce-

nario speciﬁcation for development and test of auto-

mated driving functions. In 2016 IEEE Intelligent Ve-

hicles Symposium (IV), pages 1149–1155.

Bock, J., Krajewski, R., Moers, T., Runde, S., Vater, L.,

and Eckstein, L. (2020). The ind dataset: A drone

dataset of naturalistic road user trajectories at german

intersections. In 2020 IEEE Intelligent Vehicles Sym-

posium (IV), pages 1929–1934.

de Gelder, E. and Camp, O. O. d. (2024). Scenario-

based assessment of automated driving systems: How

(not) to parameterize scenarios? arXiv preprint

arXiv:2409.01117.

de Gelder, E., Paardekooper, J.-P., Saberi, A. K., Elrofai, H.,

Camp, O. O. d., Kraines, S., Ploeg, J., and De Schut-

ter, B. (2022). Towards an ontology for scenario def-

inition for the assessment of automated vehicles: An

object-oriented framework. IEEE Transactions on In-

telligent Vehicles, 7(2):300–314.

Fahrenkrog, F., Fries, A., Das, A., Kalisvaart, S., Cha-

roniti, E., Op den Camp, O., et al. (2024). Draft

prospective safety assessment framework - method.

Milestone report MS10 of the Horizon Europe project

V4SAFETY.

Galbas, R., Reich, J., Schittenhelm, H., and Wagener, N.

(2022). Safeguarding methods for complex trafﬁc sce-

narios for approval of automated driving functions.

ATZ worldwide, 124(9):56–61.

Geller, C., Haas, B., Kloeker, A., Hermens, J., Lampe, B.,

Beemelmanns, T., and Eckstein, L. (2024). Carlos: An

open, modular, and scalable simulation framework for

the development and testing of software for c-its. In

2024 IEEE Intelligent Vehicles Symposium (IV), pages

3100–3106. IEEE.

Glasmacher, C., Schuldes, M., Topalakatti, P., Hristov, P.,

Weber, H., and Eckstein, L. (2023a). Scenario-based

model of the odd through scenario databases.

Glasmacher, C., Weber, H., Schuldes, M., Wagener, N., and

Eckstein., L. (2023b). Generation of concrete param-

eters from logical urban driving scenarios based on

hybrid graphs. In Proceedings of the 9th Interna-

tional Conference on Vehicle Technology and Intel-

ligent Transport Systems - VEHITS, pages 215–222.

INSTICC, SciTePress.

Hu, M. and Li, Y. (2017). Drivers’ avoidance patterns in

near-collision intersection conﬂicts. In 2017 IEEE

20th International Conference on Intelligent Trans-

portation Systems (ITSC), pages 1–6.

International Organization for Standardization (2022a).

Road vehicles - safety of the intended functionality.

International Organization for Standardization (2022b).

Road vehicles - test scenarios for automated driving

systems - vocabulary.

Mercedes-Benz AG (2024). Drive pilot technology by

mercedes-benz. Accessed: 2024-11-08.

PEGASUS Project Consortium (2019). Pegasus method -

an overview. Technical report. Accessed: 2024-11-

04.

UNECE (2021). UN Regulation No. 157 on Automated

Lane-Keeping Systems (ALKS). Revision as of Jan-

uary 2023, adopted by UNECE’s World Forum for

Harmonization of Vehicle Regulations.

Waymo LLC (2024). Waymo driver. Accessed: 2024-11-

08.

Weber, H., Glasmacher, C., Schuldes, M., Wagener, N., and

Eckstein, L. (2023a). Holistic driving scenario con-

cept for urban trafﬁc. In 2023 IEEE Intelligent Vehi-

cles Symposium (IV), pages 1–8.

Weber, H., Glasmacher, C., Schuldes, M., Wagener, N., and

Eckstein, L. (2023b). Holistic driving scenario con-

cept for urban trafﬁc. pages 1–8.

Winner, H., Lemmer, K., Form, T., and Mazzega, J. (2018).

PEGASUS—ﬁrst steps for the safe introduction of au-

tomated driving. In Lecture Notes in Mobility, pages

185–195. Springer International Publishing.

Comparison of Parametrization Approaches for Scenario-Based Testing

447