MIXING VIRTUAL REALITY AND 2D VISUALIZATION

Using Virtual Environments as Visual 3D Information Systems for Discussion of

Data from Geo- and Environmental Sciences

Bj¨orn Zehner

Helmholtz Center for Environmental Research - UFZ, Permoserstrasse 15, 04318 Leipzig, Germany

Keywords:

Virtual environments, GIS, 3D interaction, 3D visual information system, Visualization, Geosciences.

Abstract:

3D stereoscopic visualization and virtual reality techniques are increasingly used for quality control, analysis

and discussion of 3D geoscientiﬁc data in the oil and gas industry. They provide an excellent and easily com-

prehensible insight into complex 3D structures of the earth’s subsurface. However, in many research topics in

environmental and geosciences the analysis of data usually also involves data that might be better viewed in

2D. Examples are maps or histograms. The use of virtual environments as visual information systems for the

efﬁcient communication and discussion of complex multi-attribute data sets also requires 2D data to be visu-

alized with a high quality. Further it is often not possible to show all the relevant information simultaneously

and so an interactive virtual environment is required that provides an overview and the necessary interaction

techniques to select additional information, e.g. from a database, to be visualized on request. This article

describes the hardware setup installed at the UFZ Centre for Environmental Research and a software solution

for how to use this setup efﬁciently to connect 2D data representations with 3D visualization and interaction.

1 INTRODUCTION

At the very end of the last millennium, oil & gas com-

panies started to become interested in using virtual

reality (VR) techniques. In order to investigate this

method and obtain demonstration software, they ﬁ-

nanced research at public research institutions, such

as the GMD German National Research Center for

Information Technology (now part of the Fraunhofer

Society, (Fr¨ohlich et al., 1999)) or the Virtual Envi-

ronment Technology Laboratory at the University of

Houston (Harding and Loftin, 2000). Many compa-

nies, such as Norsk Hydro or ARCO, set up their own

virtual environments and developed their own soft-

ware ((Midttun et al., 2000), (Stark et al., 2000)). Due

to the very expensive hardware needed at that time,

little research has hitherto been done outside this in-

dustrial ﬁeld, e.g. on the use of VR for visualizing

data from applied geophysical investigations, such as

ground penetrating radar or geoelectrical resistivity

measurements (Zehner et al., 2001), (Zehner, 2002).

Nowadays large companies like Shell or Schlum-

berger own a number of visualization centres around

the world and use commercial software, to quality

control and analyze their data and discuss their mod-

els. Further, more than 500 Geowalls, a low cost

stereoscopic visualization environment, have been

built at universities and elsewhere, allowing geoscien-

tists outside industry to make use of advanced visual-

ization methods (Johnson et al., 2006). Thus, current

examples of the use of VR in geosciences have started

to include more diverse subjects, such as the 3D in-

tegration of outcrop data with regional data (Johnes

et al., 2009) or numerical simulations in geosciences

(Billen et al., 2008). However, despite many topics in

geo- and environmental sciences often inherently in-

volving structurally complicated 3D data, the use of

these 3D technologies has still not reached maturity

outside of the oil & gas industry: the question is why.

Principally, VR-systems should be suitable for

discussing geoscience issues because they make it

easier for users with different backgrounds and for

novices to understand the structures and spatial coher-

ence: this article will therefore focus on their use in

presentations and discussions. It ﬁrst discusses what

might be missing in current virtual environments to

make them more useful for the geo- and environmen-

tal sciences (section 2) and then gives an overview of

what is currently available to solve this problem (sec-

tion 3). In section 4 the design of our purpose-built

display system is explained and in section 5 the im-

plementation and interaction.

364

Zehner B. (2010).

MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information Systems for Discussion of Data from Geo-

and Environmental Sciences.

In Proceedings of the International Conference on Computer Graphics Theory and Applications, pages 364-369

DOI: 10.5220/0002841703640369

 SciTePress

2 WHAT IS MISSING?

The VR systems that are currently state of the art have

been primarily designed only with 3D tasks in mind,

such as design or assembly studies. When geoscience

data are presented in such a system, the 3D data are

shown in the original form and all other data need to

be converted into 3D representations. While planning

a visualization centre for the UFZ-Helmholtz Centre

for Environmental Research, it became obvious that

use cases in such research centres have different re-

quirements. Investigations in geo- and environmental

sciences often contain a large number of different data

sets that are spatially distributed and it is often im-

possible to show all this information simultaneously.

Therefore, scientists frequently use Geographic Infor-

mation Systems (GIS). GIS connect a spatial 2D visu-

alization (a map) with a relational data base that con-

tains additional information about the objects shown

on the map. The maps show the overall structure

(e.g. the topography) and additional objects, such as

streets, dwellings or borehole locations. If during a

discussion additional information is needed on one of

these objects, viewers can select this object and the

information is shown.

If the investigations involve complicated 3D struc-

tures that can not be presented on a map, this overall

paradigm has to be extended from two to three di-

mensions and virtual environments become the ideal

discussion platform due to the stereoscopic visualiza-

tion and the direct interaction in 3D space. However,

much of the additional information that belongs to the

objects that are shown in the virtual environment in-

volves data which can inherently be better visualized

in 2D, such as areal images, soil columns or log plots.

All these data need to be at the user’s disposal during

discussion and accessed by making selections.

Another important topic where additional 2D vi-

sualization helps is the orientation of the users which

becomes complicated when ﬁnding their way within a

structurally complicated virtual model of several kilo-

metres in size in each direction. 2D visualization

could help with this task by showing a map of the area

with an icon that indicates where and at what depth

the users are currently located and in which direction

they are looking.

3 PREVIOUS WORK

Little work and software is available that really com-

bines 3D visualization and 3D interaction with the op-

tion to augment the virtual scene with 2D informa-

tion. Broadly the work can be split into three cate-

gories: Showing virtual 2D widgets in 3D space, us-

ing additional gadgets such as PDAs and hand-held

computers, and using video wall controllers, such as

BARCO’s Transform-A/XDS-1000 or Vista’s Spider

that are explained in Figure 1.

One example which shows virtual 2D widgets in

3D space is the Inside Reality Software from Schlum-

berger. This has been designed for working with data

in virtual environments. For the system control, the

standard XMotif style window interface is shown on

planes in 3D space and interaction is done by pointing

at the different menus and icons with a ray.

The Virtual Environments group at the Fraunhofer

Society in Birlinghoven has developed several ex-

amples of the use of tracked PDAs ((Simon et al.,

2005), (Holtk¨amper et al., 2007)) and Tablet PCs

(Holtk¨amper et al., 2007) in virtual environments.

This approach has been mainly used for system con-

trol, but could also be used to show additional 2D

information which would, due to the small displays,

only be available to very few people.

Workstation

with 3D

application

Laptop with

presentation

further

sources

Video

wall

con-

troller

DVI

RGB

oor proj. not used

RGB signals

to the projectors

Figure 1: Our projection-based display system run by a

video wall controller (VWC). Different computers, e.g a

workstation with a stereoscopic visualization application

(here showing a volume visualization in the right window)

and a laptop with an explanatory powerpoint presentation

(here shown in the left window) are connected as external

sources to the VWC which does the image generation for

the projectors and solves problems, such as edge blending.

While allowing the use of standard software on the display,

this essentially means using the display as a big monitor in-

stead of as a virtual reality display.

4 UFZ’S DISPLAY SYSTEM

Our head-tracked stereoscopic display system is

mainly used for collaboration and presentation. So

one major driving factor for design has been high im-

age quality/high resolution and a large screen to sup-

port an audience of several people. Further the aim

MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information

Systems for Discussion of Data from Geo- and Environmental Sciences

365

Videowall

controller

Computer 1

Computer 2

Computer 3

Computer 12

Computer 13

Slaves

The computers no. 1, 5, 6 and 10 are for running

the side-projections in VR conguration and so are

not used in this mode (or run in the background).

Master

Figure 2: Picture of the UFZ’s visualization centre used in mixed mode at the top and the corresponding hardware setup at the

bottom. On the rear screen and on the ﬂoor a 3D model is shown with which the users can directly interact in virtual space.

This part of the system is run by a computer cluster. On the side wings additional 2D information is shown upon the user’s

request, such as detailed borehole information on rock types or graphs of physical values (windows at the top of the two side

wings). The users choose which information is shown by selecting the corresponding objects in the virtual model. A map

(bottom left wing) always indicates the orientation of the display in the virtual world and so helps when working with data

sets that show geographical information ranging across several kilometers.

was to employ the system for a wide range of use

cases, some of them using it as a CAVE while oth-

ers as a simple 2D presentation system. The system

has an approximately 6x3 meter large main screen

and corresponding projections on the ﬂoor and side-

wings. In order to achieve a high resolution of ap-

proximately 6400x1800 pixels (after subtracting the

pixels for edge blending) and a good image quality, 13

SXGA+ projectors are used to run this system. Fig-

ures 1 and 2 show pictures taken in our display system

and sketches of the overall technical setup. Rendering

is done by a cluster of up to 13 workstations, one for

each projector, and/or via a video wall controller. In

the UFZ’s visualization centre, rendering can be done

in three different modes.

1. The ﬁrst is to run the system purely as a virtual

reality system using a cluster of 13 workstations.

2. The second uses a video wall controller for the

main screen and the side wings (see Figure 1).

3. The third mode uses a combination of the ﬁrst and

the second one.

Figure 2 shows the conﬁguration of the third mode

that serves as the basis for augmenting virtual real-

ity and 3D interaction with 2D views of data. From

the cluster, nine computers are used to generate the

images for the main screen and the ﬂoor. This part

of the display provides a virtual environment. On the

side screens the video wall controller is used to show

2D information that has been generated as graphical

output on a remote machine.

5 IMPLEMENTATION

In order to demonstrate a system that mixes virtual

reality and 2D visualization to visitors at our visual-

ization center a sample implementation has been pro-

grammed. Software development currently comprises

three or four basic elements: The open source scene-

graph OpenSG (Reiners et al., 2002), the commercial

software VRED from PI-VR GmbH which is based on

OpenSG, and Nokia’s (Trolltech’s) Qt toolkit for de-

veloping GUIs and Plugins for VRED. For processing

3D scientiﬁc data, we use the Visualization Toolkit

(VTK, (Schroeder et al., 1996)).

Figure 3 shows the main components necessary

for this software. At the heart of it is a 3D visualiza-

GRAPP 2010 - International Conference on Computer Graphics Theory and Applications

366

3D Vis. App (VRED from PI-VR GmbH)

Managing the scenegraph

In case of rendering via cluster acting

as master

Window for 2D

data display

Window for 2D na-

vigation display

Window for 2D

data display

Mediator

Tools - 3D/2D

interaction

Child windows

2D window-

manager

Data-

base,

Files ...

Info on which object has been picked

Info on required data window navigation

via video-

wall con-

troller

rendering

via cluster

rendering

via cluster

via video-

wall con-

troller

via video-

wall con-

troller

Figure 3: The different software components necessary to

run the system in the mixed mode as is shown in Figure 2.

tion application that manages the scenegraph, includ-

ing camera transformation, and provides functionality

for operating on the scenegraph and adjusting the ren-

dering attributes. Further it implements the function-

ality to act as a master for the computer cluster that

represents the slave computers.

In order to show supplementary 2D information,

additional child windows are added to VRED which

are shown on the display using the video wall con-

troller: Two for rendering additional data and one for

showing a map and indicating with a glyph how the

user is oriented within the whole scene (See Figure

2). The windows for showing data implement a com-

mon interface that provides functions for scrolling up

and down, zooming in and out or loading data. They

are steered from a 2D window manager which is con-

trolled by the 3D interaction device.

5.1 3D Interaction

An essential component is the set of tools that have

been implemented for system control and interaction

with the objects in 3D space. In the UFZ’s display

system interaction is done with a so-called Flystick

from Advanced Realtime Tracking GmbH, an interac-

tion device that has 4 buttons and a small bidirectional

joystick on the top, and another button at the bottom.

The tracking system evaluates the position and the

orientation of the Flystick and the status of the but-

tons and the joystick. Data transmission is done via

the open source VRPN toolkit (Taylor et al., 2001).

The four buttons on the top of the Flystick are used to

switch between different tools. Which tools are avail-

able can be conﬁgured in a so-called tools-manager

class. The tasks that can be done with the tools can be

placed in four different groups:

• Navigation

• Interaction with objects in 3D space, such as

translating objects or moving clipping planes

• Selection of objects for which additional informa-

tion should be shown in the 2D windows

• Steering the 2D windows.

All tools are derived from a few simple base classes

that are linked to the different buttons of the Flystick.

The navigation is linked to the small joystick on the

top. By pushing it forward, the users ﬂy in the direc-

tion they point and by pulling it back, they ﬂy back-

wards. Moving it to the left and right turns them.

For direct 3D interaction, tools are provided to trans-

late objects or to move objects temporarily which then

snap back to their original positions. This can be very

helpful during discussions when some of the objects

disturb the view.

When looking at the 3D distribution of scalar data,

multiple isosurfaces are often generated and the ones

that are near to the viewer often hide the ones behind

them which disturbs the view of the 3D course of the

scalar ﬁeld. As an example of data-analysis support-

ing 3D interaction we have implemented a solution

where the user can use 3D tools to deﬁne a clipping

region by manipulating the 3D widget that is shown

in Figure 4. A rectangular parallelepiped is deﬁned

by 8 cubes at its corners and represents the volume

of interest. Outside of this volume the isosurfaces are

clipped, and so are not visible. A small sphere can be

moved within the deﬁned volume that represents the

point where 3 additional axis-parallel clipping planes

inside the volume intersect, so that chair-cuts can be

interactivelydeﬁned within the volume. In Figure 4 in

both cases the left front upper octant, which lies near

to the viewer is clipped away.

For the manipulation of this widget, two tools are

provided. The ﬁrst is for translating the cubes at the

corners and so for reshaping and transforming the vol-

ume of interest. The second tool is for deﬁning the

chair-cut: translating the sphere moves the 3 clip-

ping planes within the volume while picking one of

the cubes at the corner selects which octant should be

clipped away and makes the necessary adjustments to

the clipping planes. In this way the user can analyze

the course of the different isosurfaces around a bore-

hole as shown in the lower picture of Figure 4.

The tool that connects 2D and 3D visualization is

derived from the pick-tool and makes use of two ad-

ditional classes. The ﬁrst allows it to evaluate which

screen of the display the pointing ray intersects and

MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information

Systems for Discussion of Data from Geo- and Environmental Sciences

367

Figure 4: Example of the direct interaction with the 3D

model in virtual space: Deﬁnition of a clipping box and

a chair cut through a data set (isosurfaces) using a 3D wid-

get, at the top as a screen shot and at the bottom showing a

person interacting in our display system.

the second is the 2D window manager class. Depend-

ing on which screen has been intersected, different ac-

tions are performed: If the intersected screen is one of

the side screens, the user must keep the button pressed

and can then use the joystick, that is normally used for

3D navigation, to scroll and zoom the 2D window on

the corresponding screen. If the intersected screen is

the main screen showing the 3D model, the program

checks for the intersection of the ray with the scene

and reports the picked object to the 2D window man-

ager, so that the suitable supplementary information

can be shown on the side screens.

5.2 2D Information Display

For our 3D analogy of GIS functionality we can not

use a commercial system as these only support 2D

spatial representations. Instead we directly use the

scengraph that describes our 3D visualization and

within each node is associated with a unique name.

Upon making a selection in 3D space, this name is

used as an identiﬁer and passed to the 2D window

manager which then retrieves the additional informa-

tion from the database. Depending on the nature of

the information a suitable widget is chosen and the in-

formation is shown in one of the 2D windows on the

side wings. Figure 5 shows an example of the applica-

tion of this functionality, using the above-mentioned

tool to connect 2D and 3D visualization.

Figure 5: Example of requesting additional information on

one of the objects shown in 3D space. The person picks one

of the boreholes in order to see the additional 2D informa-

tion, which is shown on the left screen.

5.3 2D Navigation Display

With regard to navigation, the information from the

3D interaction device goes through a mediator which

passes it to both the 3D window/3D display and the

2D navigation window. The current position and ori-

entation is always projected into the map view. The

2D navigation window then displays a map of the re-

gion and indicates the user’s position and orientation

using an arrow glyph.

6 CONCLUSIONS

This article ﬁrst explains why current virtual environ-

ments need to be extended to be able to show non 3D

data, such as maps and graphs, if they are to be used as

visual information systems for complicated 3D geo-

science data sets with multiple attributes, just as Geo-

graphic Information Systems are used for discussing

2D data today. As one possible solution a system has

been described that combines hardware and software

and provides virtual reality capabilities, such as user-

tracking, stereoscopic visualization and direct interac-

tion in 3D space and that can augment the information

given in the 3D model interactively with supplemen-

tary data, using additional windows. For demonstra-

tion purposes real data are loaded which stem from

the initial site analysis of a project that deals with

direct gas injections for stimulation of the biological

degradation for remediating organically contaminated

groundwater (Beckmann et al., 2007).

GRAPP 2010 - International Conference on Computer Graphics Theory and Applications

368

We have not yet evaluated the system quantita-

tively. As visitors to our center often lost their orien-

tation when navigating within large scale virtual mod-

els (even the ones that actually modeled it), it is clear

that an additional 2D map view which shows the cur-

rent location and orientation is a great help. The same

goes for providing additional 2D information needed

to discuss the data set. We have yet to evaluate if

our approach, using the peripheral parts of our display

for showing this information is superior to other ap-

proaches, e.g. those involving hand-held PCs. How-

ever, we usually have several visitors simultaneously

and so it is preferable to show the additional 2D in-

formation on large screens. Problems with the system

as shown are the wide angle of view where it is difﬁ-

cult to maintain an overview of the information on all

the screens. Furthermore people sometimes complain

about the 3D visualization abruptly ending on the side

of the main screen.

ACKNOWLEDGEMENTS

People at various companies offered invaluable feed-

back and discussion on the different technical options

and the state of the art for this technology while the

author was preparing the public invitation to bid for

the display technology in 2007. He thanks Dirk de

Meulemeester and his team from BARCO BV for

working with us on the the ﬁnal design of the dis-

play system. Matthias Stiller and Andreas Ziehringer

from PI-VR GmbH provided help when extending

their software VRED. Anika Beckmann und Mario

Schirmer provided the data that are necessary to

demonstrate the system and Thomas Schnicke and

Lars Bilke helped with computer administration and

technical problems. Finally thanks go to Alison E.

Martin for proofreading the manuscript.

REFERENCES

Beckmann, A., Gerhardt, M., Zittwitz, M., Martienssen,

M., Krieg, R., Geistlinger, H., and Schirmer, M.

(2007). Das OXYWALL-Projekt: Anwendung

eines Verfahrens zur Direktgasinjektion von Sauer-

stoff zur in situ Sanierung von organisch kon-

taminierten Grundw¨assern. Zeitschrift Altlasten Spek-

trum, (4):153–159.

Billen, M., Kreylos, O., Hamann, B., Jadamec, M., Kellogg,

L., Staadt, O., and Sumner, D. (2008). A geoscience

perspective on immersive 3d gridded data visualiza-

tion. Computer & Geosciences, 34:1056–1072.

Fr¨ohlich, B., Barrass, S., Zehner, B., Plate, J., and G¨obel,

M. (1999). Exploring geoscientiﬁc data in virtual en-

vironments. In David, E., Gross, M., and Hamann, B.,

editors, Proceedings of the 10th IEEE Conference on

Visualization (Vis99), pages 169–173.

Harding, C. and Loftin, B. (2000). Visualization and mod-

elling of geoscientiﬁc data on the interactive work-

bench. The Leading Edge, 19(5):506–511.

Holtk¨amper, T., Scholz, S., Dressler, A., and Bogen, M.

(2007). Co-located collaborative use of virtual envi-

ronments. In Proceedings AAPG Annual Convention

and Exhibition 2007.

Johnes, R., McCaffrey, K., Clegg, P., Wilson, R., Holliman,

N., Holdsworth, R., Imber, J., and Waggott, S. (2009).

Integration of regional to outcrop digital data: 3d visu-

alization of multi-scale geological models. Computers

& Geosciences, 35:4–18.

Johnson, J., Morin, P., and Keken, P. V. (2006). Ge-

owall: Stereoscopic visualization for geoscience re-

search and education. Computer Graphics and Appli-

cations, 26(6):10–14.

Midttun, M., Helland, R., and Finnstrom, E. (2000). Vir-

tual reality, adding value to exploration and produc-

tion. The Leading Edge, 19(5):538–544.

Reiners, D., Voss, G., and Behr, J. (2002). Opensg: Basic

concepts. In OpenSG Symposium 2002.

Schroeder, W., Martin, K., and Lorensen, B. (1996). The

Visualization Toolkit, An Object-Oriented Approach to

3D Graphics. Prentice-Hall Inc.

Simon, A., Dressler, A., Kr¨uger, H.-P., Scholz, S., and

Wind, J. (2005). Interaction and co-located collabo-

ration in large projection-based virtual environments.

In Proceedings of INTERACT 2005, pages 364–376.

Stark, T., Dorn, G., and Cole, M. (2000). Arco and immer-

sive environments, part 1: The ﬁrst two generations.

The Leading Edge, 19(5):526–532.

Taylor, R., Hudson, T., Seeger, A., Weber, H., Juliano, J.,

and Helser, A. (2001). VRPN: A device-independent,

network-transparent VR peripheral system. In Pro-

ceedings of the ACM Symposium on Virtual Reality

Software & Technology 2001 (VRST 2001).

Zehner, B. (2002). Anwendung immersiver Visual-

isierungssysteme zur Exploration geophysikalischer

und geologischer Daten, Ph.D Thesis. Rheinische

Friedrich-Wilhelms-Univesit¨at Bonn, Germany.

Zehner, B., Altek¨oster, C., and K¨umpel, H.-J. (2001). Visu-

alization of 3-D GPR surveys: Application of virtual

reality in geosciences. European Journal of Environ-

mental and Engineering Geophysics, 6:141–152.

MIXING VIRTUAL REALITY AND 2D VISUALIZATION - Using Virtual Environments as Visual 3D Information

Systems for Discussion of Data from Geo- and Environmental Sciences

369