Using the Projection-based Vehicle in the Loop for the Investigation
of in-Vehicle Information Systems: First Insights
Matthias Graichen
, Lisa Graichen
, Thomas Rottmann
and Verena Nitsch
Human Factors Institute, Bundeswehr University Munich, Werner-Heisenberg-Weg 39, Neubiberg, Germany
Department of Cognitive and Engineering Psychology, Chemnitz University of Technology, Chemnitz, Germany
Keywords: Driving Simulator, Vehicle in the Loop, Simulator Sickness, Gesture-based Interaction, Advanced Driver
Assistance Systems, in-Vehicle Information Systems, Human Machine Interaction.
Abstract: Most driving simulators cannot replicate real driving dynamics and thus fail to convey a realistic driving
experience. To overcome this issue, the Vehicle in the loop (VIL) had been developed, which combines a
virtual visual environment with the realistic kinaesthetic feedback of a vehicle while driving on a closed test
track. Previous VIL setups used a head-mounted display (HMD) for displaying the virtual environment. This
limits the driver’s visual input to the virtual environment and makes it difficult to investigate potential research
questions concerning driver interactions with in-vehicle information systems (IVIS). To address this issue, a
new version of the VIL has been developed, which uses a projector for displaying the driving simulation on
an inset in the windshield and two monitors mounted at the vehicle’s sides. This work presents the first
application of the Pro-VIL for investigating IVIS and their impact on driving performance in safety critical
situations. For this purpose, we built a setup for comparing the user experience when using either a gesture-
or touch-based interaction system, and the observation of driver attention. Results support the overall
practicability of the setup, but also revealed new challenges for experimental research design and execution.
Driving simulators are frequently used for analysing
driving behaviour that is related to diverse
endogenous and exogenous factors such as driving
experience, states of fatigue, stress, or inattention,
driving manoeuvres or complex traffic scenarios (e.g.
Fisher, Rizzo, Caird, & Lee, 2011). Moreover, they
are crucial to the investigation of driver adaptions to
advanced driver assistance systems (ADAS), (partial)
vehicle automation, or in-vehicle driver information
systems (IVIS; Stevens, Brusque, & Krems, 2014; cf.
recent research using driving simulators in Bengler,
Drüke, Hoffmann, Manstetten, & Neukum, 2018). In
comparison to observations in real traffic
environments, virtual driving simulations are more
cost-effective and allow for higher levels of
standardisation and safe study execution (e.g.
reaction on crossing pedestrians) with respect to both,
participants and involved experimenters. Typically,
static driving simulators (ranging from basic desktop
simulators to more realistic vehicle mock-ups) and
dynamic driving simulators (capable of simulating
actual vehicle motions) are distinguished and their
utilization would depend on research aims,
availability and operational costs. When research
aims focus on the usability of new HMI (human-
machine interface) concepts, a simulator that simply
provides the driver an appropriate driving
environment (and thereby evoke the need for
fulfilling the primary driving task) would suffice. In
contrast, designing, parametrising, and validating
technical aspects of new (safety-critical) driver
assisting functions (e.g. lateral offset in emergency
steering systems) often require testing processes that
involve accurate and realistic vehicle dynamics.
However, even dynamic driving simulators are not
able to simulate vehicle motions completely as a
driver would experience them under real
environmental conditions (e.g. longer transitional
forces cannot be displayed). As a result, a trade-off
must be defined concerning the levels of experimental
standardization and ecological validity, depending on
the results’ impact and consequences in terms of
controllability and safety issues, e. g. conducting
studies using a real test vehicle on a closed test track
(Purucker, Schneider, Rüger, & Frey, 2018).
Graichen, M., Graichen, L., Rottmann, T. and Nitsch, V.
Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights.
DOI: 10.5220/0006649102310237
In Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS 2018), pages 231-237
ISBN: 978-989-758-293-6
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
To address these requirements of realistic vehicle
dynamics, keeping high levels of experimental
standardisation (accurately timed and replicable
scenarios), and safe study execution, the Vehicle in
the loop (VIL) was developed by Thomas Bock in
cooperation with the AUDI AG (Bock, 2008), which
combines the immersion of the driver in a virtual
environment using a head-mounted display (HMD)
with the experience of realistic dynamic forces of a
real test vehicle on a test track (Berg, Nitsch, &
Färber, 2016; Bock, 2012). However, using a HMD
lowers an overall holistic driving experience as the
driver is bound to the mere virtual environment,
which limits potential research questions concerning
driver interactions with any HMI within the vehicle.
To overcome these issues, a new projection-based
VIL (Pro-VIL) has been proposed (Riedl & Färber,
2015), but has not been evaluated in terms of
applicability for the investigation of IVIS. In this
work, we present the first-time application of the Pro-
VIL aimed at evaluating both a gesture and a touch-
based interaction (GBI, TBI) system regarding user
experience, driver attention, and driving
performance. We will first describe the general
functioning of the (Pro-)VIL and the experimental
setup, and will then present results concerning the
driving experience and practicability of the setup for
investigating IVIS. In addition, new directions for the
further development of the VIL are discussed.
1.1 Vehicle in the Loop
In a previous version of the VIL, the driver sees a
complete virtual reality using a HMD but actually
receives realistic kinaesthetic, vestibular and auditory
feedback from the interaction with the real car while
driving on a test track (Berg et al., 2016). The
functional principle is shown in Figure 1. A virtual
environment (1) is first constructed based on the
available routes of the test track. While driving, the
exact position and orientation of the vehicle (2) is
located using differential GPS and an inertial
measuring unit. An optical tracker (3) keeps track of
the head movements of the driver (only in HMD-
VIL). Then, the image generation (4) of the driving
simulation is based on a sensor fusion of these signals
and displays the exact section of the virtual
environment the driver is currently looking at.
Various studies confirmed the validity of the
HMD-VIL for the investigation of longitudinal
driving behaviours (Karl, Berg, Rüger, & Färber,
2013) and steering responses in critical situations
(Rüger & Färber, 2018; Rüger, Nitsch, & Färber,
2015; Sieber et al., 2013; Weber, Blum, Ernstberger,
& Färber, 2015).
However, there are a number of drawbacks when
using a HMD for displaying the virtual environment
(Riedl & Färber, 2015). 1) The currently used HMD
(NVIS nVisor ST50; for a comparison of evaluated
VIL-HMDs see Berg, 2014) has a relatively narrow
field of view (40° with a resolution of 1280x1024),
which limits its applicability to scenarios which
require a widely spread gaze behaviour as in turning
manoeuvres or crossing situations when interacting
with other traffic participants. 2) As there is neither a
suitable display of the car body nor a view of the
actual interior (including the driver’s body), the
Figure 1: Functional principle of the VIL.
VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems
driver has the impression of flying over the street
(Karl et al., 2013). 3) The currently used headset has
no way of recording the gaze behaviour. 4) The HMD
has a relatively high weight (1050 g), which makes it
uncomfortable when wearing for a longer time.
With the emerging trend of virtual reality
applications, new techniques are considered for
replacing the previous headset, e.g. with the HTC
Vive, which provides a lower weight (470 g), broader
viewing angle and higher resolution (110°;
2160×1200), and can be upgraded to record gaze
behaviour with the Pupil Labs eye-tracking add-on
( Nevertheless, the
problem of a lacking interior view remains.
1.2 Projection-based VIL
To overcome the impression of flying on the street, a
new setup has been proposed by Riedl and Färber
(2015) using a short-distance projector, which is
directly mounted under the vehicle’s roof and projects
the ego’s front view on an inset fitted in front of the
windshield (1280x800). For side views, two monitors
(29 inches, 1920x1080) have been mounted at the
outside of the left and right front doors (see Figure 1).
The projector setup has been evaluated in terms of
simulator sickness and ecological validity concerning
the perception and production of longitudinal and
lateral distances (Riedl & Färber, 2015), which has
shown comparable results to those of the HMD-VIL.
Without the HMD, the Pro-VIL provides an
unimpaired driving experience with full visibility of
the interior of the vehicles and hands, enabling driver
interactions with passengers or interaction systems.
The development process of new IVIS is not only
guided by integrating new emerging technologies and
design trends, but also by an overall user experience
from a safety perspective. Thus, user efforts of using
the system while driving should be minimized in
order to reduce its risk of visually or cognitively
distracting the driver (cf. Bayly, Young, & Regan,
2009; Liang & Lee, 2010). A possibility of using IVIS
for even complex tasks (navigating through more than
two menu levels) and without requiring the driver to
visually search the controls is provided, e.g., by using
gestural control. In Graichen, Graichen, and Krems
(2018), the overall potential of GBI compared to TBI
for reducing driver distraction has been shown, by
significantly reducing the number and duration of
gazes while interacting with the IVIS using a static
driving simulator. To evaluate the potential of GBI to
be also less cognitively distracting than TBI and
thereby increasing the readiness of the driver to react
timely and adequately in safety critical situations, this
setup has been transferred to the Pro-VIL. Here, we
describe first insights into participants’ impression
regarding this setup, and report results on simulator
sickness before and after the study execution.
2.1 Vehicle Setup and Scenario
The experiment used the Wizard-of-Oz-Technique,
thus all user inputs were actually carried out by the
examiner on the rear seats. For displaying the IVIS
screens, an additional monitor (Tontec, 7 inches,
1024x500) was mounted on the centre console, which
also supposedly recognized touch inputs (see Figure
2). To demonstrate an apparent functioning gesture
recognizing device, the Leap Motion was used.
To record the driver and interaction behaviour,
two cameras were mounted on the rear mirror and the
front passenger seat respectively. For eye-tracking the
SMI ETG 2W device was used. Tracking markers
were placed around the wind shield and monitor to
capture gazes within these regions of interest.
Figure 2: Vehicle setup with wind shield projection, IVIS
monitor, gesture recognition device (below the hand),
gesture online-visualization, and eye-tracking markers.
2.2 Participants
An opportunity sample of 65 participants (16 female;
mostly students), with a mean age of 26.14 (Min = 18,
Max = 57, SD = 8.06). No restrictions were made
regarding driving experience or visual aids. Only
participants with a valid driving licence were
Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights
2.3 Measurements
For the purpose of this work, only the questionnaire
and results on simulator sickness (SSQ; Kennedy,
Lane, Berbaum, & Lilienthal, 1993) and subjective
driving experience will be described. A pre-post-test
measurement was chosen to observe the development
of sickness symptoms when sitting in the Pro-VIL for
about 60 minutes. The SSQ measures 16 sickness
symptoms, which can be assigned to three subscales:
Nausea (SSQ-N), oculomotor disturbances (SSQ-O),
and disorientation (SSQ-D), as well as to an overall
score for total severity (SSQ-TS). During each trial,
participants’ comments were noted that gave
indication to their personal driving experience.
Moreover, participants were encouraged to talk
about the driving experience and overall impression
on the Pro-VIL after each trial, and were again
explicitly asked at the end of the session.
2.4 Research Design and Procedure
A two-way (2x2) within-subjects design was chosen,
with the interaction type (GBI vs. TBI) being the first
factor, and the level of interaction (simple vs.
complex tasks) the second factor. All five trials were
based on the same urban scenario requiring one
turning manoeuvre and one U-turn, but differed
regarding the surrounding traffic. Each trial involved
three interaction tasks with the IVIS, with two of
them being directly followed by various safety critical
situations (e.g. unpredictable crossings or merging or
vehicles). Overall, participants drove more than 11.5
km with the Pro-VIL. The experiment took each
participant about 1.5 h.
Upon arrival, participants completed
questionnaires pertaining to demographics,
experience on GBI systems, and a pre-test of the SSQ.
Then, they were introduced to the vehicle and the
functional principle of the VIL. Afterwards, the
participants were trained to control the IVIS with both
possible input modes of GBI and TBI, including a
demonstration of the gesture recognition performance
using an online-visualization on the display (Figure 2).
After that, participants drove one training scenario to
familiarize themselves with the vehicle and the VIL.
Then, they drove five trials in total, including the first
trial without any safety critical situation to capture a
baseline of individual driving performance regarding
lane keeping and preferences on time headway in car-
following. Before each trial, the eye-tracking system
was calibrated. Within the four test trials, the trained
interaction tasks with the IVIS (e.g. zoom into
navigation map, or call traffic information) were
instructed by the examiner at fixed scenario positions.
After each trial, participants were debriefed,
completed questionnaires pertaining to different
aspects of user experience with both interaction
systems. At the end of the session, participants
completed the post-test SSQ and were interviewed on
their impressions of the Pro-VIL. Overall, the
experiment took about 90 minutes.
In the following sections, the results on simulator
sickness, qualitative data pertaining to driving
experience and immersion effects as well as
experiences from the perspective of the study
execution are described.
3.1 Driving Experiences
When explaining the functional principle of the VIL
to the participants, several expressed worry due to the
lack of visual feedback on their real position on the
test track, and asked about the reliability and
precision of the system. Noticeably, most participants
showed a cautious driving style (low speed and
acceleration) in the practicing trial, but drove
increasingly more confidently within the baseline
trial. However, the real driving speed was often
underestimated, which led to a higher radius in
turning manoeuvres. Some participants commented
on the different resolutions between the projection
and monitor displays. Overall, most participants
enjoyed the driving experience and were often
completely taken by the test environment (e.g.
reducing speed at a radar trap, emergency brakings in
safety critical situations, and addressing aggressive
comments, hand gestures and even using the car horn
to the virtual causer of the accident). However, after
failing in safety critical situations, participants often
became silent and later expressed negative feelings.
3.2 Simulator Sickness
During the trials, one participant requested to pause
due to light sickness symptoms, and one participant
reported severe symptoms about one hour after
completing the study (the participant already knew
about his sensibility from previous studies with the
HMD-VIL). Some participants reported light
headaches after the test phase. Two participants
reported sickness symptoms prior to the test phase
VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems
Figure 3: Average SSQ symptom profiles (left) and distributions of scores (right).
and were therefore excluded in the following
analysis. Two missing item values were identified in
the SSQ pre-test and imputed by predictive mean
matching using the R package ‘mice’ (Van Buuren &
Groothuis-Oudshoorn, 2011). Figure 3 (left) provides
an overview of the average values for each symptom,
compared to SSQ results for driving in complex
worlds taken from Karl et al. (2013).
The SSQ scores were computed according to
Kennedy et al. (1993). Reliability analysis for
subscales and the overall score showed low
Cronbach’s α values in T1 (ranging from .41 to .52)
and acceptable values ranging from .68 to .76 at T2.
As depicted in Figure 3 (right), symptoms for SSQ-D
were more severe than SSQ-N and SSQ-O in both,
pre- and post-surveys. To compare each of the scores
at T1 and T2 a paired Wilcoxon signed-rank was
conducted. Scores for SSQ-N differed not
significantly between T2 (M = 9.86; SD = 14.69) than
on T1 (M = 7.6; SD = 10.14), r = -0.15. Scores for
SSQ-O were significantly higher on T2 (M = 14.28;
SD = 16.58) and T1 (M = 9.38; SD = 10.38), p =
0.024, r = -0.29. Scores for SSQ-D were significantly
higher on T2 (M = 15.31; SD = 22.45) than on T1 (M
= 6.13; SD = 11.34), p = .003, r = -.39. The score for
SSQ-TS was significantly higher on T2 (M = 147.53;
SD = 182.03) than on T1 (M = 86.44; SD = 99.24), p
= .01, r = -.33.
3.3 Technical Issues and Procedure
Two study assistants are necessary to ensure a safe
and reliable study execution, as the handling of the
Pro-VIL and the used vehicle setup required sustained
attention to detail and the accurate order of working
steps. One technical issue of the presented study
concerns the power management of the vehicle,
which provided enough power to drive just about two
sessions. It was recommended to recharge the vehicle
in short intervals without driving.
Another issue was the complex study procedure,
including operating on the rear seat with two
computers simultaneously: One for the virtual
environment and monitoring the webcams (mounted
on the vehicle to ensure safe driving on the test track),
and another computer for timely controlling the IVIS
corresponding to the driver inputs, monitoring the
performance of the eye-tracking system, and
recording the vehicle data.
At first, participants expressed doubt regarding the
reliability of the VIL system, but then showed signs
of high immersion effects (e.g. strong behavioural
reactions against accident causers) and remarked that
they enjoyed the driving experience. Though some
scores for SSQ symptoms were significantly higher
after the study, results are lower compared to the
HMD-VIL shown in Karl et al. (2013), and even
lower than those shown in Riedl and Färber (2015)
using the Pro-VIL without side monitors. Some
symptoms might be attributed to different resolutions
between the projector and the monitor, which could
be overcome by lowering the resolution of the side
monitors. By now, there are new short distance
projectors available, which are also capable of a
higher resolution. But still, there would remain some
blurring in the projector image, as the pixels are more
stretched in the lower areas due to the sloping inset in
the windshield. A new projector (1920x1080) was
mounted after the study, which increases the pixel
density from 1.024 to 1.536 horizontally, and from
Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights
2.286 to 3.086 vertically. Overall, a new front view
to side view ratio of 0.4 in horizontal density and 0.8
in vertical density could be achieved. The potential
effects of the new projector on driver experience and
simulator sickness will be examined in the future.
As the vehicle did not allow for longer session
times without frequent recharging at the time of the
study, the power management was completely
redesigned afterwards to increase the operating time,
which is now at approximately 7 hours (without any
other instrumentation or study equipment).
Overall, the Pro-VIL was successfully applied in
the domain of investigating IVIS and their effects on
the driving performance in safety critical situations.
The additional setup of the Pro-VIL for investigating
the effect of GBI and TBI was well accepted and all
participants were able to handle the driving task and
secondary tasks with the IVIS, which allows for
analyses of realistic driving behavior.
Future plans involve a multi driver simulation,
combing both, the HMD and the Pro-VIL.
Bayly, M., Young, K. L., & Regan, M. A. (2009).
mitigation_pp210. In M. A. Regan, J. D. Lee, & K. L.
Young (Eds.), Driver distraction: Theory, effects, and
mitigation. Boca Raton, Fla.: CRC Press.
Bengler, K., Drüke, J., Hoffmann, S., Manstetten, D., &
Neukum, A. (Eds.). (2018). UR:BAN Human Factors in
Traffic: Approaches for Safe, Efficient and Stressfree
Urban Traffic. Wiesbaden: Springer Fachmedien
Berg, G. (2014). Das Vehicle in the Loop - Ein Werkzeug
für die Entwicklung und Evaluation von
sicherheitskritischen Fahrerassistenzsystemen
(Dissertation). Universität der Bundeswehr München,
Berg, G., Nitsch, V., & Färber, B. (2016). Vehicle in the
Loop. In H. Winner, S. Hakuli, F. Lotz, & C. Singer
(Eds.), Handbook of driver assistance systems: Basic
Information, Components and Systems for Active Safety
and Comfort (pp. 199210). Cham: Springer
International Publishing.
Bock, T. (2008). Vehicle in the loop: Test- und
Simulationsumgebung für Fahrerassistenzsysteme.
Audi Dissertationsreihe: Bd. 10. Göttingen: Cuvillier
Bock, T. (2012). Bewertung von Fahrerassistenzsystemen
mittels der Vehicle in the Loop-Simulation. In H.
Winner, S. Hakuli, & G. Wolf (Eds.), ATZ/MTZ-
Fachbuch. Handbuch Fahrerassistenzsysteme:
Grundlagen, Komponenten und Systeme für aktive
Sicherheit und Komfort (2nd ed., pp. 7683).
Wiesbaden: Vieweg + Teubner.
Fisher, D. L., Rizzo, M., Caird, J. K., & Lee, J. D. (Eds.).
(2011). Handbook of driving simulation for
engineering, medicine, and psychology. Boca Raton:
CRC Press.
Graichen, L., Graichen, M., & Krems, J. F. (2018).
Evaluation of gesture-based in-vehicle interaction: User
experience and the potential to reduce driver
distraction. Manuscript in preparation.
Karl, I., Berg, G., Rüger, F., & Färber, B. (2013). Driving
Behavior and Simulator Sickness While Driving the
Vehicle in the Loop: Validation of Longitudinal
Driving Behavior. IEEE Intelligent Transportation
Systems Magazine, 5(1), 4257.
Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal,
M. G. (1993). Simulator Sickness Questionnaire: An
Enhanced Method for Quantifying Simulator Sickness.
The International Journal of Aciation Psychology, 3(3),
Liang, Y., & Lee, J. D. (2010). Combining cognitive and
visual distraction: less than the sum of its parts.
Accident Analysis & Prevention, 42(3), 881890.
Purucker, C., Schneider, N., Rüger, F., & Frey, A. (2018).
Validity of research environments: Comparing
criticality perceptions across research environments. In
K. Bengler, J. Drüke, S. Hoffmann, D. Manstetten, &
A. Neukum (Eds.), UR:BAN Human Factors in Traffic:
Approaches for Safe, Efficient and Stressfree Urban
Traffic (pp. 423446). Wiesbaden: Springer
Fachmedien Wiesbaden.
Riedl, B., & Färber, B. (2015). Evaluation of a new
projection concept for the Vehicle in the Loop
(VIL) driving simulator. In H. Bülthoff, A. Kemeny, &
P. Pretto (Eds.), Proceedings of DSC 2015 Europe
Driving Simulation Conference & Exhibition:
Tübingen, Germany, September 16 - 18, 2015 (pp. 225
226). Tübingen.
Rüger, F., & Färber, B. (2018). Emergency steering systems
- Controllability investigations with the Vehicle in the
loop. In K. Bengler, J. Drüke, S. Hoffmann, D.
Manstetten, & A. Neukum (Eds.), UR:BAN Human
Factors in Traffic: Approaches for Safe, Efficient and
Stressfree Urban Traffic (pp. 447460). Wiesbaden:
Springer Fachmedien Wiesbaden.
Rüger, F., Nitsch, V., & Färber, B. (2015). Automatic
evasion seen from the opposing traffic - An
investigation with the Vehicle in the Loop. In 2015
IEEE 18th International Conference on Intelligent
Transportation Systems (ITSC 2015): Gran Canaria,
Spain, 15 - 18 September 2015 (pp. 10411048).
Piscataway, NJ: IEEE.
Sieber, M., Berg, G., Karl, I., Siedersberger, K.-H., Siegel,
A., & Färber, B. (2013). Validation of driving behavior
in the Vehicle in the Loop: Steering responses in critical
situations. In 2013 16th International IEEE Conference
VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems
on Intelligent Transportation Systems (ITSC 2013): The
Hague, The Netherlands, 6 - 9 October 2013 (pp. 1101
1106). Piscataway, NJ: IEEE.
Stevens, A., Brusque, C., & Krems, J. (Eds.). (2014). Driver
adaptation to information and assistance systems.
London: The Institution of Engineering and
Technology (IET).
Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice:
Multivariate Imputation by Chained Equations in R.
Journal of Statistical Software, 45(3), 167. Retrieved
Weber, S., Blum, K., Ernstberger, A., & Färber, B. (2015).
Standard Reactions Driver Reactions in Critical
Driving Situations. Procedia Manufacturing, 3, 2489
Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights