MIXING VIRTUAL REALITY AND 2D VISUALIZATION
Using Virtual Environments as Visual 3D Information Systems for Discussion of
Data from Geo- and Environmental Sciences
Bj¨orn Zehner
Helmholtz Center for Environmental Research - UFZ, Permoserstrasse 15, 04318 Leipzig, Germany
Keywords:
Virtual environments, GIS, 3D interaction, 3D visual information system, Visualization, Geosciences.
Abstract:
3D stereoscopic visualization and virtual reality techniques are increasingly used for quality control, analysis
and discussion of 3D geoscientific data in the oil and gas industry. They provide an excellent and easily com-
prehensible insight into complex 3D structures of the earth’s subsurface. However, in many research topics in
environmental and geosciences the analysis of data usually also involves data that might be better viewed in
2D. Examples are maps or histograms. The use of virtual environments as visual information systems for the
efficient communication and discussion of complex multi-attribute data sets also requires 2D data to be visu-
alized with a high quality. Further it is often not possible to show all the relevant information simultaneously
and so an interactive virtual environment is required that provides an overview and the necessary interaction
techniques to select additional information, e.g. from a database, to be visualized on request. This article
describes the hardware setup installed at the UFZ Centre for Environmental Research and a software solution
for how to use this setup efficiently to connect 2D data representations with 3D visualization and interaction.
1 INTRODUCTION
At the very end of the last millennium, oil & gas com-
panies started to become interested in using virtual
reality (VR) techniques. In order to investigate this
method and obtain demonstration software, they fi-
nanced research at public research institutions, such
as the GMD German National Research Center for
Information Technology (now part of the Fraunhofer
Society, (Fr¨ohlich et al., 1999)) or the Virtual Envi-
ronment Technology Laboratory at the University of
Houston (Harding and Loftin, 2000). Many compa-
nies, such as Norsk Hydro or ARCO, set up their own
virtual environments and developed their own soft-
ware ((Midttun et al., 2000), (Stark et al., 2000)). Due
to the very expensive hardware needed at that time,
little research has hitherto been done outside this in-
dustrial field, e.g. on the use of VR for visualizing
data from applied geophysical investigations, such as
ground penetrating radar or geoelectrical resistivity
measurements (Zehner et al., 2001), (Zehner, 2002).
Nowadays large companies like Shell or Schlum-
berger own a number of visualization centres around
the world and use commercial software, to quality
control and analyze their data and discuss their mod-
els. Further, more than 500 Geowalls, a low cost
stereoscopic visualization environment, have been
built at universities and elsewhere, allowing geoscien-
tists outside industry to make use of advanced visual-
ization methods (Johnson et al., 2006). Thus, current
examples of the use of VR in geosciences have started
to include more diverse subjects, such as the 3D in-
tegration of outcrop data with regional data (Johnes
et al., 2009) or numerical simulations in geosciences
(Billen et al., 2008). However, despite many topics in
geo- and environmental sciences often inherently in-
volving structurally complicated 3D data, the use of
these 3D technologies has still not reached maturity
outside of the oil & gas industry: the question is why.
Principally, VR-systems should be suitable for
discussing geoscience issues because they make it
easier for users with different backgrounds and for
novices to understand the structures and spatial coher-
ence: this article will therefore focus on their use in
presentations and discussions. It first discusses what
might be missing in current virtual environments to
make them more useful for the geo- and environmen-
tal sciences (section 2) and then gives an overview of
what is currently available to solve this problem (sec-
tion 3). In section 4 the design of our purpose-built
display system is explained and in section 5 the im-
plementation and interaction.
364
Zehner B. (2010).
MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information Systems for Discussion of Data from Geo-
and Environmental Sciences.
In Proceedings of the International Conference on Computer Graphics Theory and Applications, pages 364-369
DOI: 10.5220/0002841703640369
Copyright
c
SciTePress
2 WHAT IS MISSING?
The VR systems that are currently state of the art have
been primarily designed only with 3D tasks in mind,
such as design or assembly studies. When geoscience
data are presented in such a system, the 3D data are
shown in the original form and all other data need to
be converted into 3D representations. While planning
a visualization centre for the UFZ-Helmholtz Centre
for Environmental Research, it became obvious that
use cases in such research centres have different re-
quirements. Investigations in geo- and environmental
sciences often contain a large number of different data
sets that are spatially distributed and it is often im-
possible to show all this information simultaneously.
Therefore, scientists frequently use Geographic Infor-
mation Systems (GIS). GIS connect a spatial 2D visu-
alization (a map) with a relational data base that con-
tains additional information about the objects shown
on the map. The maps show the overall structure
(e.g. the topography) and additional objects, such as
streets, dwellings or borehole locations. If during a
discussion additional information is needed on one of
these objects, viewers can select this object and the
information is shown.
If the investigations involve complicated 3D struc-
tures that can not be presented on a map, this overall
paradigm has to be extended from two to three di-
mensions and virtual environments become the ideal
discussion platform due to the stereoscopic visualiza-
tion and the direct interaction in 3D space. However,
much of the additional information that belongs to the
objects that are shown in the virtual environment in-
volves data which can inherently be better visualized
in 2D, such as areal images, soil columns or log plots.
All these data need to be at the user’s disposal during
discussion and accessed by making selections.
Another important topic where additional 2D vi-
sualization helps is the orientation of the users which
becomes complicated when finding their way within a
structurally complicated virtual model of several kilo-
metres in size in each direction. 2D visualization
could help with this task by showing a map of the area
with an icon that indicates where and at what depth
the users are currently located and in which direction
they are looking.
3 PREVIOUS WORK
Little work and software is available that really com-
bines 3D visualization and 3D interaction with the op-
tion to augment the virtual scene with 2D informa-
tion. Broadly the work can be split into three cate-
gories: Showing virtual 2D widgets in 3D space, us-
ing additional gadgets such as PDAs and hand-held
computers, and using video wall controllers, such as
BARCO’s Transform-A/XDS-1000 or Vista’s Spider
that are explained in Figure 1.
One example which shows virtual 2D widgets in
3D space is the Inside Reality Software from Schlum-
berger. This has been designed for working with data
in virtual environments. For the system control, the
standard XMotif style window interface is shown on
planes in 3D space and interaction is done by pointing
at the different menus and icons with a ray.
The Virtual Environments group at the Fraunhofer
Society in Birlinghoven has developed several ex-
amples of the use of tracked PDAs ((Simon et al.,
2005), (Holtk¨amper et al., 2007)) and Tablet PCs
(Holtk¨amper et al., 2007) in virtual environments.
This approach has been mainly used for system con-
trol, but could also be used to show additional 2D
information which would, due to the small displays,
only be available to very few people.
Workstation
with 3D
application
Laptop with
presentation
further
sources
Video
wall
con-
troller
DVI
RGB
oor proj. not used
RGB signals
to the projectors
Figure 1: Our projection-based display system run by a
video wall controller (VWC). Different computers, e.g a
workstation with a stereoscopic visualization application
(here showing a volume visualization in the right window)
and a laptop with an explanatory powerpoint presentation
(here shown in the left window) are connected as external
sources to the VWC which does the image generation for
the projectors and solves problems, such as edge blending.
While allowing the use of standard software on the display,
this essentially means using the display as a big monitor in-
stead of as a virtual reality display.
4 UFZ’S DISPLAY SYSTEM
Our head-tracked stereoscopic display system is
mainly used for collaboration and presentation. So
one major driving factor for design has been high im-
age quality/high resolution and a large screen to sup-
port an audience of several people. Further the aim
MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information
Systems for Discussion of Data from Geo- and Environmental Sciences
365
Videowall
controller
Computer 1
Computer 2
Computer 3
Computer 12
Computer 13
Slaves
The computers no. 1, 5, 6 and 10 are for running
the side-projections in VR conguration and so are
not used in this mode (or run in the background).
Master
Figure 2: Picture of the UFZ’s visualization centre used in mixed mode at the top and the corresponding hardware setup at the
bottom. On the rear screen and on the floor a 3D model is shown with which the users can directly interact in virtual space.
This part of the system is run by a computer cluster. On the side wings additional 2D information is shown upon the user’s
request, such as detailed borehole information on rock types or graphs of physical values (windows at the top of the two side
wings). The users choose which information is shown by selecting the corresponding objects in the virtual model. A map
(bottom left wing) always indicates the orientation of the display in the virtual world and so helps when working with data
sets that show geographical information ranging across several kilometers.
was to employ the system for a wide range of use
cases, some of them using it as a CAVE while oth-
ers as a simple 2D presentation system. The system
has an approximately 6x3 meter large main screen
and corresponding projections on the floor and side-
wings. In order to achieve a high resolution of ap-
proximately 6400x1800 pixels (after subtracting the
pixels for edge blending) and a good image quality, 13
SXGA+ projectors are used to run this system. Fig-
ures 1 and 2 show pictures taken in our display system
and sketches of the overall technical setup. Rendering
is done by a cluster of up to 13 workstations, one for
each projector, and/or via a video wall controller. In
the UFZ’s visualization centre, rendering can be done
in three different modes.
1. The first is to run the system purely as a virtual
reality system using a cluster of 13 workstations.
2. The second uses a video wall controller for the
main screen and the side wings (see Figure 1).
3. The third mode uses a combination of the first and
the second one.
Figure 2 shows the configuration of the third mode
that serves as the basis for augmenting virtual real-
ity and 3D interaction with 2D views of data. From
the cluster, nine computers are used to generate the
images for the main screen and the floor. This part
of the display provides a virtual environment. On the
side screens the video wall controller is used to show
2D information that has been generated as graphical
output on a remote machine.
5 IMPLEMENTATION
In order to demonstrate a system that mixes virtual
reality and 2D visualization to visitors at our visual-
ization center a sample implementation has been pro-
grammed. Software development currently comprises
three or four basic elements: The open source scene-
graph OpenSG (Reiners et al., 2002), the commercial
software VRED from PI-VR GmbH which is based on
OpenSG, and Nokia’s (Trolltech’s) Qt toolkit for de-
veloping GUIs and Plugins for VRED. For processing
3D scientific data, we use the Visualization Toolkit
(VTK, (Schroeder et al., 1996)).
Figure 3 shows the main components necessary
for this software. At the heart of it is a 3D visualiza-
GRAPP 2010 - International Conference on Computer Graphics Theory and Applications
366
3D Vis. App (VRED from PI-VR GmbH)
Managing the scenegraph
In case of rendering via cluster acting
as master
Window for 2D
data display
Window for 2D na-
vigation display
Window for 2D
data display
Mediator
Tools - 3D/2D
interaction
Child windows
2D window-
manager
Data-
base,
Files ...
Info on which object has been picked
Info on required data window navigation
via video-
wall con-
troller
rendering
via cluster
rendering
via cluster
via video-
wall con-
troller
via video-
wall con-
troller
Figure 3: The different software components necessary to
run the system in the mixed mode as is shown in Figure 2.
tion application that manages the scenegraph, includ-
ing camera transformation, and provides functionality
for operating on the scenegraph and adjusting the ren-
dering attributes. Further it implements the function-
ality to act as a master for the computer cluster that
represents the slave computers.
In order to show supplementary 2D information,
additional child windows are added to VRED which
are shown on the display using the video wall con-
troller: Two for rendering additional data and one for
showing a map and indicating with a glyph how the
user is oriented within the whole scene (See Figure
2). The windows for showing data implement a com-
mon interface that provides functions for scrolling up
and down, zooming in and out or loading data. They
are steered from a 2D window manager which is con-
trolled by the 3D interaction device.
5.1 3D Interaction
An essential component is the set of tools that have
been implemented for system control and interaction
with the objects in 3D space. In the UFZ’s display
system interaction is done with a so-called Flystick
from Advanced Realtime Tracking GmbH, an interac-
tion device that has 4 buttons and a small bidirectional
joystick on the top, and another button at the bottom.
The tracking system evaluates the position and the
orientation of the Flystick and the status of the but-
tons and the joystick. Data transmission is done via
the open source VRPN toolkit (Taylor et al., 2001).
The four buttons on the top of the Flystick are used to
switch between different tools. Which tools are avail-
able can be configured in a so-called tools-manager
class. The tasks that can be done with the tools can be
placed in four different groups:
Navigation
Interaction with objects in 3D space, such as
translating objects or moving clipping planes
Selection of objects for which additional informa-
tion should be shown in the 2D windows
Steering the 2D windows.
All tools are derived from a few simple base classes
that are linked to the different buttons of the Flystick.
The navigation is linked to the small joystick on the
top. By pushing it forward, the users fly in the direc-
tion they point and by pulling it back, they fly back-
wards. Moving it to the left and right turns them.
For direct 3D interaction, tools are provided to trans-
late objects or to move objects temporarily which then
snap back to their original positions. This can be very
helpful during discussions when some of the objects
disturb the view.
When looking at the 3D distribution of scalar data,
multiple isosurfaces are often generated and the ones
that are near to the viewer often hide the ones behind
them which disturbs the view of the 3D course of the
scalar field. As an example of data-analysis support-
ing 3D interaction we have implemented a solution
where the user can use 3D tools to define a clipping
region by manipulating the 3D widget that is shown
in Figure 4. A rectangular parallelepiped is defined
by 8 cubes at its corners and represents the volume
of interest. Outside of this volume the isosurfaces are
clipped, and so are not visible. A small sphere can be
moved within the defined volume that represents the
point where 3 additional axis-parallel clipping planes
inside the volume intersect, so that chair-cuts can be
interactivelydefined within the volume. In Figure 4 in
both cases the left front upper octant, which lies near
to the viewer is clipped away.
For the manipulation of this widget, two tools are
provided. The first is for translating the cubes at the
corners and so for reshaping and transforming the vol-
ume of interest. The second tool is for defining the
chair-cut: translating the sphere moves the 3 clip-
ping planes within the volume while picking one of
the cubes at the corner selects which octant should be
clipped away and makes the necessary adjustments to
the clipping planes. In this way the user can analyze
the course of the different isosurfaces around a bore-
hole as shown in the lower picture of Figure 4.
The tool that connects 2D and 3D visualization is
derived from the pick-tool and makes use of two ad-
ditional classes. The first allows it to evaluate which
screen of the display the pointing ray intersects and
MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information
Systems for Discussion of Data from Geo- and Environmental Sciences
367
Figure 4: Example of the direct interaction with the 3D
model in virtual space: Definition of a clipping box and
a chair cut through a data set (isosurfaces) using a 3D wid-
get, at the top as a screen shot and at the bottom showing a
person interacting in our display system.
the second is the 2D window manager class. Depend-
ing on which screen has been intersected, different ac-
tions are performed: If the intersected screen is one of
the side screens, the user must keep the button pressed
and can then use the joystick, that is normally used for
3D navigation, to scroll and zoom the 2D window on
the corresponding screen. If the intersected screen is
the main screen showing the 3D model, the program
checks for the intersection of the ray with the scene
and reports the picked object to the 2D window man-
ager, so that the suitable supplementary information
can be shown on the side screens.
5.2 2D Information Display
For our 3D analogy of GIS functionality we can not
use a commercial system as these only support 2D
spatial representations. Instead we directly use the
scengraph that describes our 3D visualization and
within each node is associated with a unique name.
Upon making a selection in 3D space, this name is
used as an identifier and passed to the 2D window
manager which then retrieves the additional informa-
tion from the database. Depending on the nature of
the information a suitable widget is chosen and the in-
formation is shown in one of the 2D windows on the
side wings. Figure 5 shows an example of the applica-
tion of this functionality, using the above-mentioned
tool to connect 2D and 3D visualization.
Figure 5: Example of requesting additional information on
one of the objects shown in 3D space. The person picks one
of the boreholes in order to see the additional 2D informa-
tion, which is shown on the left screen.
5.3 2D Navigation Display
With regard to navigation, the information from the
3D interaction device goes through a mediator which
passes it to both the 3D window/3D display and the
2D navigation window. The current position and ori-
entation is always projected into the map view. The
2D navigation window then displays a map of the re-
gion and indicates the user’s position and orientation
using an arrow glyph.
6 CONCLUSIONS
This article first explains why current virtual environ-
ments need to be extended to be able to show non 3D
data, such as maps and graphs, if they are to be used as
visual information systems for complicated 3D geo-
science data sets with multiple attributes, just as Geo-
graphic Information Systems are used for discussing
2D data today. As one possible solution a system has
been described that combines hardware and software
and provides virtual reality capabilities, such as user-
tracking, stereoscopic visualization and direct interac-
tion in 3D space and that can augment the information
given in the 3D model interactively with supplemen-
tary data, using additional windows. For demonstra-
tion purposes real data are loaded which stem from
the initial site analysis of a project that deals with
direct gas injections for stimulation of the biological
degradation for remediating organically contaminated
groundwater (Beckmann et al., 2007).
GRAPP 2010 - International Conference on Computer Graphics Theory and Applications
368
We have not yet evaluated the system quantita-
tively. As visitors to our center often lost their orien-
tation when navigating within large scale virtual mod-
els (even the ones that actually modeled it), it is clear
that an additional 2D map view which shows the cur-
rent location and orientation is a great help. The same
goes for providing additional 2D information needed
to discuss the data set. We have yet to evaluate if
our approach, using the peripheral parts of our display
for showing this information is superior to other ap-
proaches, e.g. those involving hand-held PCs. How-
ever, we usually have several visitors simultaneously
and so it is preferable to show the additional 2D in-
formation on large screens. Problems with the system
as shown are the wide angle of view where it is diffi-
cult to maintain an overview of the information on all
the screens. Furthermore people sometimes complain
about the 3D visualization abruptly ending on the side
of the main screen.
ACKNOWLEDGEMENTS
People at various companies offered invaluable feed-
back and discussion on the different technical options
and the state of the art for this technology while the
author was preparing the public invitation to bid for
the display technology in 2007. He thanks Dirk de
Meulemeester and his team from BARCO BV for
working with us on the the final design of the dis-
play system. Matthias Stiller and Andreas Ziehringer
from PI-VR GmbH provided help when extending
their software VRED. Anika Beckmann und Mario
Schirmer provided the data that are necessary to
demonstrate the system and Thomas Schnicke and
Lars Bilke helped with computer administration and
technical problems. Finally thanks go to Alison E.
Martin for proofreading the manuscript.
REFERENCES
Beckmann, A., Gerhardt, M., Zittwitz, M., Martienssen,
M., Krieg, R., Geistlinger, H., and Schirmer, M.
(2007). Das OXYWALL-Projekt: Anwendung
eines Verfahrens zur Direktgasinjektion von Sauer-
stoff zur in situ Sanierung von organisch kon-
taminierten Grundw¨assern. Zeitschrift Altlasten Spek-
trum, (4):153–159.
Billen, M., Kreylos, O., Hamann, B., Jadamec, M., Kellogg,
L., Staadt, O., and Sumner, D. (2008). A geoscience
perspective on immersive 3d gridded data visualiza-
tion. Computer & Geosciences, 34:1056–1072.
Fr¨ohlich, B., Barrass, S., Zehner, B., Plate, J., and G¨obel,
M. (1999). Exploring geoscientific data in virtual en-
vironments. In David, E., Gross, M., and Hamann, B.,
editors, Proceedings of the 10th IEEE Conference on
Visualization (Vis99), pages 169–173.
Harding, C. and Loftin, B. (2000). Visualization and mod-
elling of geoscientific data on the interactive work-
bench. The Leading Edge, 19(5):506–511.
Holtk¨amper, T., Scholz, S., Dressler, A., and Bogen, M.
(2007). Co-located collaborative use of virtual envi-
ronments. In Proceedings AAPG Annual Convention
and Exhibition 2007.
Johnes, R., McCaffrey, K., Clegg, P., Wilson, R., Holliman,
N., Holdsworth, R., Imber, J., and Waggott, S. (2009).
Integration of regional to outcrop digital data: 3d visu-
alization of multi-scale geological models. Computers
& Geosciences, 35:4–18.
Johnson, J., Morin, P., and Keken, P. V. (2006). Ge-
owall: Stereoscopic visualization for geoscience re-
search and education. Computer Graphics and Appli-
cations, 26(6):10–14.
Midttun, M., Helland, R., and Finnstrom, E. (2000). Vir-
tual reality, adding value to exploration and produc-
tion. The Leading Edge, 19(5):538–544.
Reiners, D., Voss, G., and Behr, J. (2002). Opensg: Basic
concepts. In OpenSG Symposium 2002.
Schroeder, W., Martin, K., and Lorensen, B. (1996). The
Visualization Toolkit, An Object-Oriented Approach to
3D Graphics. Prentice-Hall Inc.
Simon, A., Dressler, A., Kr¨uger, H.-P., Scholz, S., and
Wind, J. (2005). Interaction and co-located collabo-
ration in large projection-based virtual environments.
In Proceedings of INTERACT 2005, pages 364–376.
Stark, T., Dorn, G., and Cole, M. (2000). Arco and immer-
sive environments, part 1: The rst two generations.
The Leading Edge, 19(5):526–532.
Taylor, R., Hudson, T., Seeger, A., Weber, H., Juliano, J.,
and Helser, A. (2001). VRPN: A device-independent,
network-transparent VR peripheral system. In Pro-
ceedings of the ACM Symposium on Virtual Reality
Software & Technology 2001 (VRST 2001).
Zehner, B. (2002). Anwendung immersiver Visual-
isierungssysteme zur Exploration geophysikalischer
und geologischer Daten, Ph.D Thesis. Rheinische
Friedrich-Wilhelms-Univesit¨at Bonn, Germany.
Zehner, B., Altek¨oster, C., and K¨umpel, H.-J. (2001). Visu-
alization of 3-D GPR surveys: Application of virtual
reality in geosciences. European Journal of Environ-
mental and Engineering Geophysics, 6:141–152.
MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information
Systems for Discussion of Data from Geo- and Environmental Sciences
369