SOFTWARE DEVELOPMENT ASPECTS OF OUT-OF-CORE DATA

MANAGEMENT FOR PLANETARY TERRAIN

Cody J. White, Sergiu M. Dascalu and Frederick C. Harris, Jr.

University of Nevada, Reno, U.S.A.

Keywords:

Terrain-rendering, Data management, Out-of-core, GPU, Software requirements, Software design.

Abstract:

Rendering terrain on a planetary scale can quickly become a large problem. Many challenges arise when

attempting to render terrain over a spherical body as well as deal with the large amount of data that needs to

be used to accurately display the terrain of a planet. Most research in the area of terrain rendering is speciﬁc

to a given region of a planet, therefore needing few datasets for a proper rendering. However, since planets are

made up of larger areas, a different approach needs to be taken in order to display high-detail terrain around a

viewer while sorting through the large amount of planetary data available. Additionally, since modern desktops

have a relatively small amount of memory, a system to swap data from the hard drive into graphics processing

unit (GPU) memory must be created. Therefore, we present the software design for a data caching mechanism

which can efﬁciently swap only the data around a viewer into and out of the GPU memory in real-time. We

also present a prototype of the software which achieves efﬁcient framerates for high-quality views of a planet’s

surface while minimizing the time it takes to ﬁnd data centered around a viewer and display it to the screen.

1 INTRODUCTION

Terrain rendering has been at the forefront of re-

search for many years, especially in the ﬁelds of video

games, scientiﬁc visualization, and training simula-

tions. While many problems have been solved, lit-

tle has been done to address the issues surrounding

rendering terrain on a planetary-scale. These com-

plexities arise from both the shape of a planet as

well as the amount of data that must be used in or-

der to accurately render the planet in detail. While

there have been many solutions for rendering plane-

tary bodies, (Mahsman, 2010), (Cignoni et al., 2003),

the problem of dealing with large amounts of data still

needs to be fully addressed.

A planet’s surface is typically made up of many

datasets which describe the various geographical ar-

eas, ranging form very small in size to terabytes of in-

formation. Fortunately, there has been a large amount

of research in dealing with extremely large datasets

for terrain rendering (Cignoni et al., 2003), (Dick

et al., 2009), (Kooima et al., 2009). As this is the

ﬁrst step to generating realistic terrain, it has received

much attention. Typically, the data is organized into a

spatial hierarchy and then chosen for rendering based

on a user-speciﬁed search criteria. Using this ap-

proach, only the data surrounding a user needs to be

in memory while the rest can safely remain on the

hard drive until it is needed. While this process is ex-

tremely helpful in handling large datasets efﬁciently,

it does not solve the problem of multiple datasets

needing to be considered. Therefore, an extension to

this approach must be devised.

As planets are of a fairly predictable shape, a

bounding box can be constructed which tightly con-

tains them. Using this information, it can be possi-

ble to construct a spatial subdivision hierarchy which

covers the region of the entire planet. This hierarchy

can then contain information for the system to deter-

mine which datasets that exist on the planet are cur-

rently in view. Using this approach, the use of a large

amount of datasets is simple to deal with. Addition-

ally, inserting new data into the hierarchy becomes

a trivial operation. Once the hierarchy has been cre-

ated, it can exist in the main memory without any data

loaded to simply determine the visibility of datasets.

As a core requirement of terrain renderers, the al-

gorithm should work in real time. To achieve this re-

sult, the data cacher should take advantage of both

CPU and graphics processing unit (GPU) parallel pro-

cessing. Much of the searching and data swapping

can occur in parallel, freeing up system resources

for the terrain renderer being used. Since the ter-

rain renderer can use data already loaded into mem-

185

J. White C., M. Dascalu S., C. Harris and Jr. F..

SOFTWARE DEVELOPMENT ASPECTS OF OUT-OF-CORE DATA MANAGEMENT FOR PLANETARY TERRAIN.

DOI: 10.5220/0003509201850191

In Proceedings of the 6th International Conference on Software and Database Technologies (ICSOFT-2011), pages 185-191

ISBN: 978-989-8425-77-5

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

ory, the data cacher can work independently of the

renderer for its searching operations. The GPU can

also be used to perform composition of terrain data

into a ﬁnal image for use in the generation of a three-

dimensional mesh, similar to (Kooima et al., 2009).

We present the software development aspects for

the creation of a data caching system as described in

the thesis Out-of-Core Data Management for Plan-

etary Terrain (White, 2011). This system processes

the many datasets that compose a full planet and ef-

ﬁciently swaps them in and out of GPU memory in

real time. As the data caching system should not in-

terfere with the rendering, the system utilizes parallel

processing to resolve this issue. Additionally, the sys-

tem supports the addition of new datasets at runtime

to allow the user to render data which is not part of

the pre-built data cache. Figure 1 shows a resulting

frame from our proposed algorithm and system.

Figure 1: Global view of the planet Mars.

In this paper,we start with a brief survey of related

work (Section 2). Our software has been designed

with several considerations in mind which strive to

hide the complexities of data caching from the user

(Section 3). Instead of creating a new terrain ren-

derer to test the software, we integrated our algorithm

with the Hesperian terrain rendering library (Mahs-

man, 2010). Upon testing our software with the ter-

rain renderer,we’ve seen a noticeable improvement in

both the visual quality of the rendered terrain as well

as the overallefﬁciencyof the system (Section 4). Ad-

ditionally, we present some ideas for a future expan-

sion of the software system (Section 5).

2 RELATED WORK

A common approach to breaking datasets into

smaller, more memory cohesive chunks is to subdi-

vide them into a quadtree hierarchy (Lindstrom et al.,

1996), (de Boer, 2000), (Lindstrom and Pascucci,

2001). This type of structure is used because it splits

data into four equal-sized chunks per node until a pre-

deﬁned threshold has been met whereupon mipmap-

ping can occur. Using this approach, datasets can be

broken down and stored as pieces of the whole im-

age so that only the parts that are needed can be used

for rendering. Typically, the high-resolution imagery

is stored in the leaf nodes of the tree and the parent

nodes contain successively lower resolution versions

of their four children. Therefore, different levels of

the tree relate to a different level-of-detail (LOD) for

the given dataset.

A simple algorithm for mapping many differ-

ent datasets together is known as deferred textur-

ing (Kooima et al., 2009). Using this, a texture atlas

can be created which stores all of the loaded datasets

for access by the GPU. This method easily relates to

planetary data as each chunk of the atlas can repre-

sent a geographical region of the visible area. While

this helps to solve the issue of compositing multiple

datasets together on a planetary scale, the system im-

plemented in (Kooima et al., 2009) is unable of ren-

dering without a search pass through their hierarchies.

Therefore, the overall speed of the system is depen-

dant on the efﬁciency of their data swapping mech-

anism. This limitation forces the terrain renderer to

work at the speed of the hard drive and memory bus,

which restricts how often a frame can be rendered.

However,a data-caching system should not inhibit the

renderer’s performance. Therefore, it is an important

aspect for our system to decouple the data caching

and rendering through the use of CPU parallel pro-

cessing. Additionally, a generic data cacher should

be independent of the type of renderer being used, be

it ray or triangle-based. Our system then, has been

designed with this constraint in mind.

3 SOFTWARE DEVELOPMENT

In order to accurately deﬁne our system, we must ﬁrst

determine the functional and non-functional require-

ments (Sommerville, 2010) (Pressman, 2010) (Sec-

tion 3.1). Additionally, it is imperative to understand

the user’s needs for the system (Section 3.2).

3.1 Requirements

Our proposed system is designed to work across dif-

ferent display types as well as different operating sys-

tems. In order to do this, we must make our imple-

mentation both thread-safe (Lin and Snyder, 2009)

and rendering context-safe. Additionally, we make

use of cross-platform APIs (Daughtry et al., 2009)

such as OpenGL (Angel, 2008) and GDAL (GDAL,

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

186

2011). This way, our library can be used on any oper-

ating system/display environment without any change

to the algorithm, making it more applicable to any ter-

rain renderer.

In this NASA funded project, we chose to ren-

der the planet Mars. However, because we use

GDAL to determine the projection information for

all of our datasets, any spherical body can be ren-

dered if the data is available. Most of this data,

which is represented as either height, color, or normal

data, can be obtained directly from NASA (NASA,

1976), (NASA, JPL, and University of Arizona,

2011), (NASA Goddard Space Flight Center, 1996).

Additionally, as NASA has many missions to map

similar regions of planets, there is a large possibil-

ity of there being overlapping data for a given region.

For this reason, a composition algorithm is used to

accurately deal with overlapping datasets.

At any time, the user is able to add new data to

the data-cache for viewing. Once processed, this data

will be shown to the screen if the viewer is in the area

where the new terrain is located. Since we do not want

the data cacher to inhibit the realistic terrain render-

ing, we utilize the multi-core power of modern CPUs

and push all searching, uploading of patch data, and

addition of new data to separate threads.

Using common projection equations (Eliason,

2007) we are able to place any dataset in the proper

geographic location. Currently, our implementation

supports both equirectangular and polar stereographic

projections. However, more projection types could

be trivially added. As different data can be stored in

these projections, it is imperative to use the proper

projection when transforming the projection coordi-

nates into texture coordinates or the image will have

continuity problems related to an improper projection.

These functional and non-functional requirements

are listed in Table 1 and Table 2 respectively.

Table 1: Functional requirements.

F01 The library will read a standard data format.

F02 The library will allow variable patch sizes.

F03 The library will allow for new data.

F04 The library will composite overlapped data.

F05 The library will allow variable LOD error.

F06 The library will allow scalable memory use.

F07 The library will preprocess terrain data.

F08 The library will select the proper LOD.

F09 The library will display datasets correctly

F10 The library will use threads.

F11 The library will make use of the GPU.

F12 The library will provide a simple interface.

Table 2: Non-functional requirements.

N01 The system will be implemented as a library

N02 The library will be implemented using C++.

N03 The library will be thread safe.

N04 The library will be rendering context safe.

N05 The library will use OpenGL.

N06 The library will use GLSL.

N07 The library will use GDAL

3.2 Use Cases

The user of this application is considered to be a ter-

rain renderer. Therefore, we provide a simple inter-

face into our data caching mechanism library for easy

use of its complex features.

Prior to any terrain renderer being able to use the

data cacher, a preprocessing step must be performed

which places the data into a state which is suitable

for rendering. This includes building mipmap hier-

archies (Lindstrom et al., 1996), the dataset bound-

ing volume hierarchy (BVH), and pre-determining

dataset errors. Once the data is processed, it is writ-

ten to the hard drive and stored for later use. These

written ﬁles then make up the data cache. This func-

tionality is simple to use in our library as the interface

can accept a list of datasets to preprocess. Once this

step has been completed, the renderer is able to use

the data cache for rendering purposes.

Figure 2: Use cases for the data-caching library.

At runtime, the terrain renderer can add new data

to the data cache which can either be saved to the data

cache or discarded upon program exit. Additionally,

the application can search for new data to render and

have that data composited into one ﬁnal image for use

in mesh generation. At program initialization, the ren-

derer can specify the maximum amount of memory to

SOFTWARE DEVELOPMENT ASPECTS OF OUT-OF-CORE DATA MANAGEMENT FOR PLANETARY TERRAIN

187

be in use by the data-caching system as well as the

maximum screen-space error allowable for LOD se-

lection.

These use cases (Som´e, 2006) are shown in Fig-

ure 2.

3.3 Classes

As we have strived to maintain usability and simplic-

ity in the use of our implementation, we have written

the code in as few classes as possible. For easy inte-

gration, the user only interacts with one class which is

used to hide the complexities of the system beneath it.

An overview of these classes can be seen in the class

diagram (Taylor et al., 2010) presented in Figure 3.

Figure 3: Class diagram for the data-caching library.

The

DataCacher

class acts as an interface into the

rest of the system. With this class, the user can pre-

process data, search for renderable terrain patches, in-

sert new data, and set the required parameters for li-

brary use. At runtime, we initialize our data by read-

ing the input ﬁles and building skeleton versions of

the data structures without any loaded image data.

Both the structure of the BVH as well as the datasets

are stored in a ﬁle for reading. As each node is

built into the structure, the accompanying datasets are

read and created in memory. Each dataset contains a

TextureQuadtree

which contains information about

the various levels-of-detail that each dataset supports

as well as projection data speciﬁc to the dataset. We

have developed the

ProjectionParser

class which

obtains all of the necessary projection information for

a particular dataset.

To simplify the code, all GDAL commands are

contained within the

DataExtractor

class. Using

this class, the system can determine projection infor-

mation, obtain regions of pixel data, and calculate the

projection coordinates of a dataset. In order to sup-

port rendering-context safety, we allow for multiple

terrain patch queues to exist (one per graphics con-

text). Therefore, different displays that are looking in

different directions in a virtual world are able to load

in their own relative data. To make this process work,

we also need to implement the search algorithm using

different search threads per context; therefore, each

context can be searched simultaneously without any

holdup by one display in the system. We support this

type of runtime environment by utilizing per context

data which segregates the terrain data for each con-

text into separate instances of any class that needs to

be context-safe.

Outside of providing just an interface, the

DataCacher

class also takes care of maintaining the

queue of visible terrain patches by determining if too

much memory has been consumed. Old data patches

are discarded to the hard drive until the system is

no longer using more than the maximum amount

of memory. This class also uploads all patches to

the GPU by the creation of a texture atlas, similar

to (Kooima et al., 2009). While the atlas is being up-

loaded, a legend is also being created which details

where each dataset resides in the atlas, as well as per

dataset information required by the GPU for compo-

sition. As a ﬁnal step, the center of each dataset in

world coordinates is saved in a vertex buffer object

(VBO) for rendering. Once the atlas has been up-

loaded, the GPU takes over for the ﬁnal steps of the

implementation.

3.4 GPU

Using programmable shaders (Rost, 2008), we are

able to speed up the composition step by use of the

GPU. OpenGL gives us easy access to the geometry

and fragment shaders which we can use for texture

overlay options in order to create a resulting texture

that the terrain renderer can use for mesh generation.

As mentioned in the previous section, a texture atlas

along with a legend for the atlas and a VBO of points

is created for use by the GPU. Both of the shaders

work together to produce the ﬁnal image.

Geometry Shader. The geometry shader in the

graphics pipeline can be used to turn incoming points

into solid geometry. For our purposes, we trans-

form the points deﬁning the center of each dataset

into screen-aligned quads which are received by the

fragment shader. An activity diagram (Arlow and

Neustadt, 2005) depicting the ﬂow of events in this

algorithm is shown in Figure 4.

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

188

Figure 4: Generation of screen-aligned quads.

Once the quad has been generated, it is automati-

cally sent to the fragment shader by the GPU.

Fragment Shader. Using the fragment shader, we

can directly affect the outcome of a pixel color on the

currently-bound frame buffer. Each fragment of the

output quad gets processed as shown in Figure 5.

Figure 5: Data composition into ﬁnal the texture.

3.5 Deployment

Since the data cacher is designed to handle three sepa-

rate types of data (color, height, and normal), a thread

is launched for each type to search their respective

subtrees individually. Therefore, each data type is

Figure 6: Deployment diagram for the system.

processed independently of each other so as to not

place a large search overhead on the system. The

main thread then takes care of uploading data to the

GPU and initiating the composition steps. This lay-

out can be seen in the deployment diagram (Arlow

and Neustadt, 2005) presented in Figure 6.

4 RESULTS

To test the functionality of the system, we imple-

mented the class structure from Section 3.3 along with

the Hesperian terrain renderer (Mahsman, 2010). To

accurately test our solution, we implemented it in a

virtual reality (VR) system via Hydra (Hydra, 2010)

as well as on a desktop platform using the Qt devel-

opment environment (Nokia, 2011). A resulting im-

age from the desktop version can be seen in Figure 7.

From the desktop view, a user is able to move about

the planet, adjust the datasets in use by the system,

add new data, and edit lighting and scaling factors.

Further ﬁgures are from the VR version.

4.1 Experimental Method

To test the execution time of the algorithm, we used

a machine with an Intel Core i7 processor running at

2.8GHz and 8GB of RAM. In addition, we used an

Nvidia GeForce GTX 480 graphics card.

For all tests performed, the system has loaded

5.5GB of terrain data for use in the visualization.

SOFTWARE DEVELOPMENT ASPECTS OF OUT-OF-CORE DATA MANAGEMENT FOR PLANETARY TERRAIN

189

Figure 7: Desktop version of the implemented system.

Since Hesperian is designed to render the planet Mars,

all resulting images shown are of the Martian surface.

4.2 Results and Analysis

To determine if the implemented system was success-

ful, we performed numerous tests against a base ver-

sion of the Hesperian terrain rendering library. This

base version supports no out-of-core data streaming

mechanism and therefore can only use data that ﬁts

within system memory. Visually, our algorithm per-

forms much better as it is capable of handling high-

quality data, as can be seen in the comparison be-

tween Figure 9 and Figure 8.

Figure 8: Olympus Mons from Hesperian.

Outside of visual results, it was determined that

our system ran nearly 10% faster than the base Hes-

perian implementation (White, 2011). This is largely

due the fact that our algorithm has knowledge of the

datasets and can create tightly-ﬁtting bounding ge-

ometry around them in the geometry shader whereas

Hesperian must render a full-screen quadric for every

dataset during the composition pass of the rendering

algorithm. Therefore, less screen pixels need to be

Figure 9: Olympus Mons from our algorithm.

executed by the GPU from our implementation, de-

creasing the runtime of the composition step.

In order to prove that the parallelization of the

data caching algorithm was effective, we needed to

determine that the implemented system was decou-

pled from the terrain renderer. From our analysis, we

calculated that approximately 4% of the overall exe-

cution time for each frame of the visualization is spent

dealing with our data caching system. Therefore, we

can show that our system is in fact decoupled from the

terrain rendereras the remaining 96% of the execution

time is spent in the Hesperian system rendering to the

screen. From this, we can determine that the system

is successful in not forcing a constraint on the over-

all rendering system performance from searching the

dataset hierarchies and uploading data to the GPU for

insertion into the texture atlas.

5 CONCLUSIONS

We have presented the software design for an out-of-

core data caching mechanism designed to work for

a planetary terrain renderer. By breaking the prob-

lem down into manageable classes, we have created a

simple to maintain and use system which can easily

be integrated into a terrain rendering system which

uses either a triangle or a ray-based terrain generation

approach. In utilizing the GPU, the system is able to

approach efﬁcient run time speeds. Additionally, the

design of a GPU algorithm means that this system will

need no modiﬁcation for future hardware as GPU ar-

chitecture improves due to the fact that the GPU code

is written in GLSL, which automatically scales with

the number of cores present (Rost, 2008).

From the analysis of our results, we have proven

that the system is both efﬁcient and decoupled from

the terrain renderer. Therefore, if the system were

to be integrated with other terrain renderers, no no-

ticeable drops in framerates should occur. We have

achieved this through the parallelization of our search

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

190

algorithms as well as a joint CPU-GPU design which

ofﬂoads heavy work onto the GPU.

To further improve the system, a data compres-

sion system could be implemented to allow for com-

pressed terrain data to exist on the hard drive. Using

this approach, data is compressed by the terrain pre-

processing step and stored for future searches. Once

the data is uploaded to the GPU, it is uncompressed

into its original form for use. The performance beneﬁt

from this method is directly related to the amount of

data being read from the hard drive and to the GPU.

ACKNOWLEDGEMENTS

This work was funded by NASA EPSCoR, grant #

NSHE 08-51, and Nevada NASA EPSCoR, grants #

NSHE 08-52, NSHE 09-41, and NSHE 10-69.

REFERENCES

Angel, E. (2008). Interactive Computer Graphics: A

Top-Down Approach Using OpenGL, pages 289–304,

492–495. Addison Wesley, 5th edition.

Arlow, J. and Neustadt, I. (2005). UML 2 and the Uniﬁed

Process. Addison-Wesley, 2nd edition.

Cignoni, P., Ganovelli, F., Gobbetti, E., Marton, F., Pon-

chio, F., and Scopigno, R. (2003). Planet-sized

batched dynamic adaptive meshes (P-BDAM). In Pro-

ceedings of the 14th IEEE Visualization 2003, pages

147–154. IEEE Computer Society.

Daughtry, J. M., Farooq, U., Stylos, J., and Myers, B. A.

(2009). API usability: CHI’2009 special interest

group meeting. In Proceedings of the 27th Interna-

tional Conference on Human Factors in Computing

Systems, CHI ’09, pages 2771–2774, New York, NY,

USA. ACM.

de Boer, W. (2000). Fast terrain rendering using geometical

mipmapping. http://www.connectii.net/emersion.

Dick, C., Kr¨uger, J., and Westermann, R. (2009). GPU

ray-casting for scalable terrain rendering. In Proceed-

ings of Eurographics 2009–Areas Papers, pages 43–

50. Eurographics Association.

Eliason, E. (2007). Hirise catalog. http://hirise.lpl.arizona.

edu/PDS/CATALOG/DSMAP.CAT (Accessed July

21, 2010).

GDAL (2011). GDAL - Geospatial Data Abstraction Li-

brary. http://www.gdal.org (Accessed July 21, 2010).

Hydra (2010). Hydra. http://www.cse.unr.edu/hpcvis/

hydra/ (Accessed August 26, 2010).

Kooima, R., Leigh, J., Johnson, A., Roberts, D., SubbaRao,

M., and DeFanti, T. (2009). Planetary-scale terrain

composition. IEEE Transactions on Visualization and

Computer Graphics, 15(5):719–733.

Lin, C. and Snyder, L. (2009). Principles of Parallel Pro-

gramming. Addison-Wesley, 1st edition.

Lindstrom, P., Koller, D., Ribarsky, W., Hodges, L. F.,

Faust, N., and Turner, G. A. (1996). Real-time, contin-

uous level of detail rendering of height ﬁelds. In SIG-

GRAPH ’96: Proceedings of the 23rd Annual Con-

ference on Computer Graphics and Interactive Tech-

niques, pages 109–118. ACM.

Lindstrom, P. and Pascucci, V. (2001). Visualization of

large terrains made easy. In IEEE Visualization 2001.

Mahsman, J. D. (2010). Projective grid mapping for plane-

tary terrain. Master’s thesis, Department of Computer

Science and Engineering, University of Nevada, Reno.

NASA (1976). Mars - images of mars.

http://www.nasa.gov/mission pages/mars/images/

index.html (Accessed December 10, 2010).

NASA Goddard Space Flight Center (1996). The Mars Or-

biter Laster Altimiter. http://mola.gsfc.nasa.gov (Ac-

cessed April 21, 2010).

NASA, JPL, and University of Arizona (2011). HiRISE:

High Resolution Imaging Science Experiment.

http://hirise.lpl.arizona.edu (Accessed July 21, 2010).

Nokia (2011). Qt - a cross-platform application and UI

framework. http://qt.nokia.com/products/.

Pressman, R. S. (2010). Software Engineering: A Practi-

tioner’s Approach. McGraw-Hill, 7th edition.

Rost, R. J. (2008). OpenGL Shading Language. Addison-

Wesley, 2nd edition.

Som´e, S. S. (2006). Supporting use case based requirements

engineering. Inf. Softw. Technol., 48:43–58.

Sommerville, I. (2010). Software Engineering. Addison-

Wesley, 9th edition.

Taylor, R. N., Medvidovic, N., and Dashofy, E. (2010).

Software Architecture: Foundations, Theory, and

Practice. Wiley, 1st edition.

White, C. J. (2011). Out-of-core data management for plan-

etary terrain. Master’s thesis, Department of Com-

puter Science and Engineering, University of Nevada,

Reno.

SOFTWARE DEVELOPMENT ASPECTS OF OUT-OF-CORE DATA MANAGEMENT FOR PLANETARY TERRAIN

191