Using the Projection-based Vehicle in the Loop for the Investigation

of in-Vehicle Information Systems: First Insights

Matthias Graichen

, Lisa Graichen

, Thomas Rottmann

and Verena Nitsch

Human Factors Institute, Bundeswehr University Munich, Werner-Heisenberg-Weg 39, Neubiberg, Germany

Department of Cognitive and Engineering Psychology, Chemnitz University of Technology, Chemnitz, Germany

Keywords: Driving Simulator, Vehicle in the Loop, Simulator Sickness, Gesture-based Interaction, Advanced Driver

Assistance Systems, in-Vehicle Information Systems, Human Machine Interaction.

Abstract: Most driving simulators cannot replicate real driving dynamics and thus fail to convey a realistic driving

experience. To overcome this issue, the Vehicle in the loop (VIL) had been developed, which combines a

virtual visual environment with the realistic kinaesthetic feedback of a vehicle while driving on a closed test

track. Previous VIL setups used a head-mounted display (HMD) for displaying the virtual environment. This

limits the driver’s visual input to the virtual environment and makes it difficult to investigate potential research

questions concerning driver interactions with in-vehicle information systems (IVIS). To address this issue, a

new version of the VIL has been developed, which uses a projector for displaying the driving simulation on

an inset in the windshield and two monitors mounted at the vehicle’s sides. This work presents the first

application of the Pro-VIL for investigating IVIS and their impact on driving performance in safety critical

situations. For this purpose, we built a setup for comparing the user experience when using either a gesture-

or touch-based interaction system, and the observation of driver attention. Results support the overall

practicability of the setup, but also revealed new challenges for experimental research design and execution.

1 INTRODUCTION

Driving simulators are frequently used for analysing

driving behaviour that is related to diverse

endogenous and exogenous factors such as driving

experience, states of fatigue, stress, or inattention,

driving manoeuvres or complex traffic scenarios (e.g.

Fisher, Rizzo, Caird, & Lee, 2011). Moreover, they

are crucial to the investigation of driver adaptions to

advanced driver assistance systems (ADAS), (partial)

vehicle automation, or in-vehicle driver information

systems (IVIS; Stevens, Brusque, & Krems, 2014; cf.

recent research using driving simulators in Bengler,

Drüke, Hoffmann, Manstetten, & Neukum, 2018). In

comparison to observations in real traffic

environments, virtual driving simulations are more

cost-effective and allow for higher levels of

standardisation and safe study execution (e.g.

reaction on crossing pedestrians) with respect to both,

participants and involved experimenters. Typically,

static driving simulators (ranging from basic desktop

simulators to more realistic vehicle mock-ups) and

dynamic driving simulators (capable of simulating

actual vehicle motions) are distinguished and their

utilization would depend on research aims,

availability and operational costs. When research

aims focus on the usability of new HMI (human-

machine interface) concepts, a simulator that simply

provides the driver an appropriate driving

environment (and thereby evoke the need for

fulfilling the primary driving task) would suffice. In

contrast, designing, parametrising, and validating

technical aspects of new (safety-critical) driver

assisting functions (e.g. lateral offset in emergency

steering systems) often require testing processes that

involve accurate and realistic vehicle dynamics.

However, even dynamic driving simulators are not

able to simulate vehicle motions completely as a

driver would experience them under real

environmental conditions (e.g. longer transitional

forces cannot be displayed). As a result, a trade-off

must be defined concerning the levels of experimental

standardization and ecological validity, depending on

the results’ impact and consequences in terms of

controllability and safety issues, e. g. conducting

studies using a real test vehicle on a closed test track

(Purucker, Schneider, Rüger, & Frey, 2018).

Graichen, M., Graichen, L., Rottmann, T. and Nitsch, V.

Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights.

DOI: 10.5220/0006649102310237

In Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS 2018), pages 231-237

ISBN: 978-989-758-293-6

231

To address these requirements of realistic vehicle

dynamics, keeping high levels of experimental

standardisation (accurately timed and replicable

scenarios), and safe study execution, the Vehicle in

the loop (VIL) was developed by Thomas Bock in

cooperation with the AUDI AG (Bock, 2008), which

combines the immersion of the driver in a virtual

environment using a head-mounted display (HMD)

with the experience of realistic dynamic forces of a

real test vehicle on a test track (Berg, Nitsch, &

Färber, 2016; Bock, 2012). However, using a HMD

lowers an overall holistic driving experience as the

driver is bound to the mere virtual environment,

which limits potential research questions concerning

driver interactions with any HMI within the vehicle.

To overcome these issues, a new projection-based

VIL (Pro-VIL) has been proposed (Riedl & Färber,

2015), but has not been evaluated in terms of

applicability for the investigation of IVIS. In this

work, we present the first-time application of the Pro-

VIL aimed at evaluating both a gesture and a touch-

based interaction (GBI, TBI) system regarding user

experience, driver attention, and driving

performance. We will first describe the general

functioning of the (Pro-)VIL and the experimental

setup, and will then present results concerning the

driving experience and practicability of the setup for

investigating IVIS. In addition, new directions for the

further development of the VIL are discussed.

1.1 Vehicle in the Loop

In a previous version of the VIL, the driver sees a

complete virtual reality using a HMD but actually

receives realistic kinaesthetic, vestibular and auditory

feedback from the interaction with the real car while

driving on a test track (Berg et al., 2016). The

functional principle is shown in Figure 1. A virtual

environment (1) is first constructed based on the

available routes of the test track. While driving, the

exact position and orientation of the vehicle (2) is

located using differential GPS and an inertial

measuring unit. An optical tracker (3) keeps track of

the head movements of the driver (only in HMD-

VIL). Then, the image generation (4) of the driving

simulation is based on a sensor fusion of these signals

and displays the exact section of the virtual

environment the driver is currently looking at.

Various studies confirmed the validity of the

HMD-VIL for the investigation of longitudinal

driving behaviours (Karl, Berg, Rüger, & Färber,

2013) and steering responses in critical situations

(Rüger & Färber, 2018; Rüger, Nitsch, & Färber,

2015; Sieber et al., 2013; Weber, Blum, Ernstberger,

& Färber, 2015).

However, there are a number of drawbacks when

using a HMD for displaying the virtual environment

(Riedl & Färber, 2015). 1) The currently used HMD

(NVIS nVisor ST50; for a comparison of evaluated

VIL-HMDs see Berg, 2014) has a relatively narrow

field of view (40° with a resolution of 1280x1024),

which limits its applicability to scenarios which

require a widely spread gaze behaviour as in turning

manoeuvres or crossing situations when interacting

with other traffic participants. 2) As there is neither a

suitable display of the car body nor a view of the

actual interior (including the driver’s body), the

Figure 1: Functional principle of the VIL.

VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems

232

driver has the impression of flying over the street

(Karl et al., 2013). 3) The currently used headset has

no way of recording the gaze behaviour. 4) The HMD

has a relatively high weight (1050 g), which makes it

uncomfortable when wearing for a longer time.

With the emerging trend of virtual reality

applications, new techniques are considered for

replacing the previous headset, e.g. with the HTC

Vive, which provides a lower weight (470 g), broader

viewing angle and higher resolution (110°;

2160×1200), and can be upgraded to record gaze

behaviour with the Pupil Labs eye-tracking add-on

(https://pupil-labs.com/vr-ar). Nevertheless, the

problem of a lacking interior view remains.

1.2 Projection-based VIL

To overcome the impression of flying on the street, a

new setup has been proposed by Riedl and Färber

(2015) using a short-distance projector, which is

directly mounted under the vehicle’s roof and projects

the ego’s front view on an inset fitted in front of the

windshield (1280x800). For side views, two monitors

(29 inches, 1920x1080) have been mounted at the

outside of the left and right front doors (see Figure 1).

The projector setup has been evaluated in terms of

simulator sickness and ecological validity concerning

the perception and production of longitudinal and

lateral distances (Riedl & Färber, 2015), which has

shown comparable results to those of the HMD-VIL.

Without the HMD, the Pro-VIL provides an

unimpaired driving experience with full visibility of

the interior of the vehicles and hands, enabling driver

interactions with passengers or interaction systems.

2 INVESTIGATION OF IVIS

The development process of new IVIS is not only

guided by integrating new emerging technologies and

design trends, but also by an overall user experience

from a safety perspective. Thus, user efforts of using

the system while driving should be minimized in

order to reduce its risk of visually or cognitively

distracting the driver (cf. Bayly, Young, & Regan,

2009; Liang & Lee, 2010). A possibility of using IVIS

for even complex tasks (navigating through more than

two menu levels) and without requiring the driver to

visually search the controls is provided, e.g., by using

gestural control. In Graichen, Graichen, and Krems

(2018), the overall potential of GBI compared to TBI

for reducing driver distraction has been shown, by

significantly reducing the number and duration of

gazes while interacting with the IVIS using a static

driving simulator. To evaluate the potential of GBI to

be also less cognitively distracting than TBI and

thereby increasing the readiness of the driver to react

timely and adequately in safety critical situations, this

setup has been transferred to the Pro-VIL. Here, we

describe first insights into participants’ impression

regarding this setup, and report results on simulator

sickness before and after the study execution.

2.1 Vehicle Setup and Scenario

The experiment used the Wizard-of-Oz-Technique,

thus all user inputs were actually carried out by the

examiner on the rear seats. For displaying the IVIS

screens, an additional monitor (Tontec, 7 inches,

1024x500) was mounted on the centre console, which

also supposedly recognized touch inputs (see Figure

2). To demonstrate an apparent functioning gesture

recognizing device, the Leap Motion was used.

To record the driver and interaction behaviour,

two cameras were mounted on the rear mirror and the

front passenger seat respectively. For eye-tracking the

SMI ETG 2W device was used. Tracking markers

were placed around the wind shield and monitor to

capture gazes within these regions of interest.

Figure 2: Vehicle setup with wind shield projection, IVIS

monitor, gesture recognition device (below the hand),

gesture online-visualization, and eye-tracking markers.

2.2 Participants

An opportunity sample of 65 participants (16 female;

mostly students), with a mean age of 26.14 (Min = 18,

Max = 57, SD = 8.06). No restrictions were made

regarding driving experience or visual aids. Only

participants with a valid driving licence were

selected.

Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights

233

2.3 Measurements

For the purpose of this work, only the questionnaire

and results on simulator sickness (SSQ; Kennedy,

Lane, Berbaum, & Lilienthal, 1993) and subjective

driving experience will be described. A pre-post-test

measurement was chosen to observe the development

of sickness symptoms when sitting in the Pro-VIL for

about 60 minutes. The SSQ measures 16 sickness

symptoms, which can be assigned to three subscales:

Nausea (SSQ-N), oculomotor disturbances (SSQ-O),

and disorientation (SSQ-D), as well as to an overall

score for total severity (SSQ-TS). During each trial,

participants’ comments were noted that gave

indication to their personal driving experience.

Moreover, participants were encouraged to talk

about the driving experience and overall impression

on the Pro-VIL after each trial, and were again

explicitly asked at the end of the session.

2.4 Research Design and Procedure

A two-way (2x2) within-subjects design was chosen,

with the interaction type (GBI vs. TBI) being the first

factor, and the level of interaction (simple vs.

complex tasks) the second factor. All five trials were

based on the same urban scenario requiring one

turning manoeuvre and one U-turn, but differed

regarding the surrounding traffic. Each trial involved

three interaction tasks with the IVIS, with two of

them being directly followed by various safety critical

situations (e.g. unpredictable crossings or merging or

vehicles). Overall, participants drove more than 11.5

km with the Pro-VIL. The experiment took each

participant about 1.5 h.

Upon arrival, participants completed

questionnaires pertaining to demographics,

experience on GBI systems, and a pre-test of the SSQ.

Then, they were introduced to the vehicle and the

functional principle of the VIL. Afterwards, the

participants were trained to control the IVIS with both

possible input modes of GBI and TBI, including a

demonstration of the gesture recognition performance

using an online-visualization on the display (Figure 2).

After that, participants drove one training scenario to

familiarize themselves with the vehicle and the VIL.

Then, they drove five trials in total, including the first

trial without any safety critical situation to capture a

baseline of individual driving performance regarding

lane keeping and preferences on time headway in car-

following. Before each trial, the eye-tracking system

was calibrated. Within the four test trials, the trained

interaction tasks with the IVIS (e.g. zoom into

navigation map, or call traffic information) were

instructed by the examiner at fixed scenario positions.

After each trial, participants were debriefed,

completed questionnaires pertaining to different

aspects of user experience with both interaction

systems. At the end of the session, participants

completed the post-test SSQ and were interviewed on

their impressions of the Pro-VIL. Overall, the

experiment took about 90 minutes.

3 EVALUATION OF DRIVING

EXPERIENCE

In the following sections, the results on simulator

sickness, qualitative data pertaining to driving

experience and immersion effects as well as

experiences from the perspective of the study

execution are described.

3.1 Driving Experiences

When explaining the functional principle of the VIL

to the participants, several expressed worry due to the

lack of visual feedback on their real position on the

test track, and asked about the reliability and

precision of the system. Noticeably, most participants

showed a cautious driving style (low speed and

acceleration) in the practicing trial, but drove

increasingly more confidently within the baseline

trial. However, the real driving speed was often

underestimated, which led to a higher radius in

turning manoeuvres. Some participants commented

on the different resolutions between the projection

and monitor displays. Overall, most participants

enjoyed the driving experience and were often

completely taken by the test environment (e.g.

reducing speed at a radar trap, emergency brakings in

safety critical situations, and addressing aggressive

comments, hand gestures and even using the car horn

to the virtual causer of the accident). However, after

failing in safety critical situations, participants often

became silent and later expressed negative feelings.

3.2 Simulator Sickness

During the trials, one participant requested to pause

due to light sickness symptoms, and one participant

reported severe symptoms about one hour after

completing the study (the participant already knew

about his sensibility from previous studies with the

HMD-VIL). Some participants reported light

headaches after the test phase. Two participants

reported sickness symptoms prior to the test phase

VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems

234

Figure 3: Average SSQ symptom profiles (left) and distributions of scores (right).

and were therefore excluded in the following

analysis. Two missing item values were identified in

the SSQ pre-test and imputed by predictive mean

matching using the R package ‘mice’ (Van Buuren &

Groothuis-Oudshoorn, 2011). Figure 3 (left) provides

an overview of the average values for each symptom,

compared to SSQ results for driving in complex

worlds taken from Karl et al. (2013).

The SSQ scores were computed according to

Kennedy et al. (1993). Reliability analysis for

subscales and the overall score showed low

Cronbach’s α values in T1 (ranging from .41 to .52)

and acceptable values ranging from .68 to .76 at T2.

As depicted in Figure 3 (right), symptoms for SSQ-D

were more severe than SSQ-N and SSQ-O in both,

pre- and post-surveys. To compare each of the scores

at T1 and T2 a paired Wilcoxon signed-rank was

conducted. Scores for SSQ-N differed not

significantly between T2 (M = 9.86; SD = 14.69) than

on T1 (M = 7.6; SD = 10.14), r = -0.15. Scores for

SSQ-O were significantly higher on T2 (M = 14.28;

SD = 16.58) and T1 (M = 9.38; SD = 10.38), p =

0.024, r = -0.29. Scores for SSQ-D were significantly

higher on T2 (M = 15.31; SD = 22.45) than on T1 (M

= 6.13; SD = 11.34), p = .003, r = -.39. The score for

SSQ-TS was significantly higher on T2 (M = 147.53;

SD = 182.03) than on T1 (M = 86.44; SD = 99.24), p

= .01, r = -.33.

3.3 Technical Issues and Procedure

Two study assistants are necessary to ensure a safe

and reliable study execution, as the handling of the

Pro-VIL and the used vehicle setup required sustained

attention to detail and the accurate order of working

steps. One technical issue of the presented study

concerns the power management of the vehicle,

which provided enough power to drive just about two

sessions. It was recommended to recharge the vehicle

in short intervals without driving.

Another issue was the complex study procedure,

including operating on the rear seat with two

computers simultaneously: One for the virtual

environment and monitoring the webcams (mounted

on the vehicle to ensure safe driving on the test track),

and another computer for timely controlling the IVIS

corresponding to the driver inputs, monitoring the

performance of the eye-tracking system, and

recording the vehicle data.

4 DISCUSSION

At first, participants expressed doubt regarding the

reliability of the VIL system, but then showed signs

of high immersion effects (e.g. strong behavioural

reactions against accident causers) and remarked that

they enjoyed the driving experience. Though some

scores for SSQ symptoms were significantly higher

after the study, results are lower compared to the

HMD-VIL shown in Karl et al. (2013), and even

lower than those shown in Riedl and Färber (2015)

using the Pro-VIL without side monitors. Some

symptoms might be attributed to different resolutions

between the projector and the monitor, which could

be overcome by lowering the resolution of the side

monitors. By now, there are new short distance

projectors available, which are also capable of a

higher resolution. But still, there would remain some

blurring in the projector image, as the pixels are more

stretched in the lower areas due to the sloping inset in

the windshield. A new projector (1920x1080) was

mounted after the study, which increases the pixel

density from 1.024 to 1.536 horizontally, and from

Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights

235

2.286 to 3.086 vertically. Overall, a new ‘front view

to side view ratio’ of 0.4 in horizontal density and 0.8

in vertical density could be achieved. The potential

effects of the new projector on driver experience and

simulator sickness will be examined in the future.

As the vehicle did not allow for longer session

times without frequent recharging at the time of the

study, the power management was completely

redesigned afterwards to increase the operating time,

which is now at approximately 7 hours (without any

other instrumentation or study equipment).

Overall, the Pro-VIL was successfully applied in

the domain of investigating IVIS and their effects on

the driving performance in safety critical situations.

The additional setup of the Pro-VIL for investigating

the effect of GBI and TBI was well accepted and all

participants were able to handle the driving task and

secondary tasks with the IVIS, which allows for

analyses of realistic driving behavior.

Future plans involve a multi driver simulation,

combing both, the HMD and the Pro-VIL.

REFERENCES

Bayly, M., Young, K. L., & Regan, M. A. (2009).

Regan_2009_Driver-distraction_Theory-effects-and-

mitigation_pp210. In M. A. Regan, J. D. Lee, & K. L.

Young (Eds.), Driver distraction: Theory, effects, and

mitigation. Boca Raton, Fla.: CRC Press.

Bengler, K., Drüke, J., Hoffmann, S., Manstetten, D., &

Neukum, A. (Eds.). (2018). UR:BAN Human Factors in

Traffic: Approaches for Safe, Efficient and Stressfree

Urban Traffic. Wiesbaden: Springer Fachmedien

Wiesbaden.

Berg, G. (2014). Das Vehicle in the Loop - Ein Werkzeug

für die Entwicklung und Evaluation von

sicherheitskritischen Fahrerassistenzsystemen

(Dissertation). Universität der Bundeswehr München,

Neubiberg.

Berg, G., Nitsch, V., & Färber, B. (2016). Vehicle in the

Loop. In H. Winner, S. Hakuli, F. Lotz, & C. Singer

(Eds.), Handbook of driver assistance systems: Basic

Information, Components and Systems for Active Safety

and Comfort (pp. 199–210). Cham: Springer

International Publishing. https://doi.org/10.1007/978-

3-319-12352-3_10

Bock, T. (2008). Vehicle in the loop: Test- und

Simulationsumgebung für Fahrerassistenzsysteme.

Audi Dissertationsreihe: Bd. 10. Göttingen: Cuvillier

Verlag.

Bock, T. (2012). Bewertung von Fahrerassistenzsystemen

mittels der Vehicle in the Loop-Simulation. In H.

Winner, S. Hakuli, & G. Wolf (Eds.), ATZ/MTZ-

Fachbuch. Handbuch Fahrerassistenzsysteme:

Grundlagen, Komponenten und Systeme für aktive

Sicherheit und Komfort (2nd ed., pp. 76–83).

Wiesbaden: Vieweg + Teubner.

https://doi.org/10.1007/978-3-8348-8619-4_9

Fisher, D. L., Rizzo, M., Caird, J. K., & Lee, J. D. (Eds.).

(2011). Handbook of driving simulation for

engineering, medicine, and psychology. Boca Raton:

CRC Press.

Graichen, L., Graichen, M., & Krems, J. F. (2018).

Evaluation of gesture-based in-vehicle interaction: User

experience and the potential to reduce driver

distraction. Manuscript in preparation.

Karl, I., Berg, G., Rüger, F., & Färber, B. (2013). Driving

Behavior and Simulator Sickness While Driving the

Vehicle in the Loop: Validation of Longitudinal

Driving Behavior. IEEE Intelligent Transportation

Systems Magazine, 5(1), 42–57.

https://doi.org/10.1109/MITS.2012.2217995

Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal,

M. G. (1993). Simulator Sickness Questionnaire: An

Enhanced Method for Quantifying Simulator Sickness.

The International Journal of Aciation Psychology, 3(3),

203–220. https://doi.org/10.1207/s15327108ijap0303

Liang, Y., & Lee, J. D. (2010). Combining cognitive and

visual distraction: less than the sum of its parts.

Accident Analysis & Prevention, 42(3), 881–890.

https://doi.org/10.1016/j.aap.2009.05.001

Purucker, C., Schneider, N., Rüger, F., & Frey, A. (2018).

Validity of research environments: Comparing

criticality perceptions across research environments. In

K. Bengler, J. Drüke, S. Hoffmann, D. Manstetten, &

A. Neukum (Eds.), UR:BAN Human Factors in Traffic:

Approaches for Safe, Efficient and Stressfree Urban

Traffic (pp. 423–446). Wiesbaden: Springer

Fachmedien Wiesbaden. https://doi.org/10.1007/978-

3-658-15418-9_25

Riedl, B., & Färber, B. (2015). Evaluation of a new

projection concept for the Vehicle in the Loop

(VIL) driving simulator. In H. Bülthoff, A. Kemeny, &

P. Pretto (Eds.), Proceedings of DSC 2015 Europe

Driving Simulation Conference & Exhibition:

Tübingen, Germany, September 16 - 18, 2015 (pp. 225–

226). Tübingen.

Rüger, F., & Färber, B. (2018). Emergency steering systems

- Controllability investigations with the Vehicle in the

loop. In K. Bengler, J. Drüke, S. Hoffmann, D.

Manstetten, & A. Neukum (Eds.), UR:BAN Human

Factors in Traffic: Approaches for Safe, Efficient and

Stressfree Urban Traffic (pp. 447–460). Wiesbaden:

Springer Fachmedien Wiesbaden.

https://doi.org/10.1007/978-3-658-15418-9_26

Rüger, F., Nitsch, V., & Färber, B. (2015). Automatic

evasion seen from the opposing traffic - An

investigation with the Vehicle in the Loop. In 2015

IEEE 18th International Conference on Intelligent

Transportation Systems (ITSC 2015): Gran Canaria,

Spain, 15 - 18 September 2015 (pp. 1041–1048).

Piscataway, NJ: IEEE.

Sieber, M., Berg, G., Karl, I., Siedersberger, K.-H., Siegel,

A., & Färber, B. (2013). Validation of driving behavior

in the Vehicle in the Loop: Steering responses in critical

situations. In 2013 16th International IEEE Conference

VEHITS 2018 - 4th International Conference on Vehicle Technology and Intelligent Transport Systems

236

on Intelligent Transportation Systems (ITSC 2013): The

Hague, The Netherlands, 6 - 9 October 2013 (pp. 1101–

1106). Piscataway, NJ: IEEE.

https://doi.org/10.1109/ITSC.2013.6728379

Stevens, A., Brusque, C., & Krems, J. (Eds.). (2014). Driver

adaptation to information and assistance systems.

London: The Institution of Engineering and

Technology (IET).

Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice:

Multivariate Imputation by Chained Equations in R.

Journal of Statistical Software, 45(3), 1–67. Retrieved

from http://www.jstatsoft.org/v45/i03/

Weber, S., Blum, K., Ernstberger, A., & Färber, B. (2015).

Standard Reactions – Driver Reactions in Critical

Driving Situations. Procedia Manufacturing, 3, 2489–

2496. https://doi.org/10.1016/j.promfg.2015.07.510

Using the Projection-based Vehicle in the Loop for the Investigation of in-Vehicle Information Systems: First Insights

237