REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL

WORLD GLOSSY SURFACES

Claus B. Madsen, Bjarne K. Mortensen and Jens R. Andersen

Computer Vision and Media Technology Lab, Aalborg University, Aalborg, Denmark

Keywords:

HDR, radiance, Image-Based Rendering, reﬂectance, real-time, non-isotropic textures.

Abstract:

A technique for real-time visualization of glossy surfaces is presented. The technique is aimed at recreating

the view-dependent appearance of glossy surfaces under some ﬁxed illumination conditions. The visualized

surfaces can be actual real world surfaces or they can be surfaces for which the appearance is precomputed with

a global illumination renderer. The approach taken is to image to surface from a large number of viewpoints

distributed over the viewsphere. From these images the reﬂected radiance in different directions is sampled

and a parameterized model is ﬁtted to these radiance samples. Two different models are explored: a very

low parameter model inspired by the Phong reﬂection model, and a general Spherical Harmonics model. It is

concluded that the Phong-based model is best suited for this type of application.

1 INTRODUCTION

The use of images as textures in computer graphics

is extremely common. These images, acquired with

a camera under controlled illumination, or syntheti-

cally generated (procedurally or manually), are typi-

cally/traditionally used to modulate the diffuse reﬂec-

tion of light from surfaces. Therefore albedo maps

could be a more proper term than the term texture

map normally used, (Yu et al., 1999). Increasingly

more visual realism is achieved by combining such

diffuse textures (appearance is independent on view-

ing direction) with a global glossiness term (for the

entire surface), normal maps, displacement maps, etc.

A special case application area for textures is

for image-based illumination of augmented objects

in Augmented Reality applications, (Debevec, 2005;

Madsen et al., 2003; Havran et al., 2005; Barsi et al.,

2005; Jensen et al., 2006; Madsen and Laursen, 2007;

Debevec, 1998; Debevec, 2002). Here, fully omni-

directional images of the environment are used for

determining the illumination conditions at the loca-

tion where the augmented object will be inserted into

the scene. These environment maps are acquired in

High Dynamic Range (HDR, where pixel values are

ﬂoating point numbers) in order to fully capture the

range of illumination in a real scene. Using omni-

directional environment maps have the clear disad-

vantage that the technique assumes that the surround-

ing scene is distant, i.e., that distances from the aug-

mented object to the surfaces of the rest of the scene

are large compared to the size of the augmented ob-

ject. Pasting the omni-directional environment map

onto a coarse model of the scene, (Gibson et al., 2003)

alleviates the distant scene assumption, but introduces

a new assumption (or problem): since the environ-

ment map was a acquired with a camera from a cer-

tain position in the scene it is assumed that the re-

ﬂected light from the surfaces of the scene is view-

independent (diffuse reﬂectors). This poses a severe

problem in scenes containing windows and/or non-

isotropic light ﬁxtures. For a survey of illumination

in mixed reality see (Jacobs and Loscos, 2004).

This paper addresses the general issue of explor-

ing techniques for view-dependent, glossy textures.

Our long term goal is to use such textures for improv-

ing illumination of augmented objects (as described

above). Our short term goal, as reported in the present

paper, is simply to develop techniques addressing the

following problem: How can we, by taking many im-

ages of a surface from a large number of viewpoints,

reconstruct, in real-time, the view-dependent appear-

ance of a glossy surface, faithfully recreating the illu-

mination conditions present during image acquisition.

We are thus not interested in being able to change

the illumination conditions, and therefore not inter-

ested in explicitly modelling reﬂectance functions

(BRDFs). Rather, we are interested in being able to

231

B. Madsen C., K. Mortensen B. and R. Andersen J. (2008).

REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL WORLD GLOSSY SURFACES.

In Proceedings of the Third International Conference on Computer Graphics Theory and Applications, pages 231-240

DOI: 10.5220/0001095602310240

 SciTePress

synthesize novel views of real world glossy surfaces

in real-time. Figure 1 illustrates the type of results of

this work.

2 RELATED WORK

Clearly, this work falls within the ﬁeld of Image-

Based Rendering, IBR. Overviews of the ﬁeld can be

found in (Kang, 1997; Oliveira, 2002). Related work

can be grouped in three categories: 1) BRDFs esti-

mated from multiple images, 2) Light Fields, and 3)

View-Dependent Texture Maps.

Estimation of reﬂectance functions from images

has been formulated as inverse rendering, (Yu et al.,

1999; Boivin and Gagalowicz, 2001; Boivin and

Gagalowicz, 2002), where one or a small number

images of a surface are used to estimate the pa-

rameters of low-parameter BRDFs. The advantage

of such an approach is that the surface can subse-

quently be subjected to novel illumination conditions,

but the clear disadvantage is that the estimation pro-

cess requires knowledge of the light sources in the

scene. Conversely, the Bidirectional Texture Func-

tion, (Dana et al., 1997), represents a technique for

storing surface appearance indexed by viewing and

illumination directions. Similarly, there is a large

body of research into capturing reﬂectance maps, e.g.,

(McAllister et al., 2002; Lensch et al., 2003; Zick-

ler et al., 2005). By capturing BRDFs from system-

atic and controlled imaging under varying view direc-

tion and illumination direction enormous ﬂexibility

is achieved and the appearance can subsequently be

reconstructed under arbitrary illumination conditions.

The disadvantage of this approach is that acquisition

requires sampling the illumination direction as well

as the viewing direction, which is impossible given

our long term goal of being able to capture the view-

dependent appearance of for instance a room under

totally normal illumination conditions.

Light Fields and Surface Light Fields (Levoy and

Hanrahan, 1996; Gortler et al., 1996; Miller et al.,

1998; Wood et al., 2000) represent techniques for

appearance reconstruction using more or less knowl-

edge about scene geometry. Appearance from novel

views is based on interpolation of acquired appear-

ances from sample viewing directions. These tech-

niques either impose limitations on the allowable set

of viewing directions for novel views, or require very

high quality geometric models of the scene. Fur-

thermore, the Surface Light Field approach is vertex

based and appearance reconstruction is a complicated

process ill suited for rendering using acceleration.

View-Dependent Texture maps (Debevec et al.,

1998) use geometric information to re-project input

images to any novel view, blending among the input

images based on considerations of view direction and

sampling rate to avoid aliasing. A disadvantage of

this technique is that is based simply on interpolation

on input images and does not represent any kind of

condensation of appearance samples into a compact

representation.

In the present paper we explore techniques for

compacting view-dependent appearance into a single

parametric representation per surface point, suitable

for accelerated (GPU-based) visualization in real-

time.

3 OVERVIEW OF APPROACH

The chosen approach is conceptually very simple. We

mount a particular surface, e.g., a poster or piece of

card board with various material samples glued onto

it, in a setup which allows for photographing the sur-

face from a large number of viewpoints distributed

across the upper hemisphere. Figure 1 demonstrates

an OpenGL application with special purpose pixel

shaders developed to reconstruct the view-dependent

appearance based on models ﬁtted to the measure-

ments from the photographs of the surface.

In the example in ﬁgure 1 the surface was pho-

tographed from approximately 100 different view-

points keeping illumination conditions ﬁxed. For ev-

ery point on the surface we therefore have 100 RGB

measurements of that point’s appearance from differ-

ent view directions. For every point we then ﬁt a low-

parameter model to those 100 measurements. In the

demonstrated case we have used 10 parameters per

surface point. These parameters are stored in a texture

which is mapped to quadrilateral in an OpenGL based

application. A purpose-built pixel shader then recon-

structs the view-dependent appearance of the surface.

As seen, highlights move across the surface as the

view changes.

4 CAPTURING APPEARANCE

In order to model and reconstruct the appearance of

a surface as a function of viewing direction we need

to sample (measure) the appearance from a number

of viewing directions. Furthermore, since the ap-

pearance changes across the surface, e.g., due to tex-

ture, and changes in glossiness, we need to sample

the appearance at a large number of distinct locations

across the surface. This section describes how images

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

232

Figure 1: Three different views of a surface with different materials (in the foreground some duct tape, alu-foil and various

plastics). The view-dependent appearance of the surface is reconstructed in a pixel shader in real-time (60 fps) based on a

per-pixel appearance model ﬁtted to data acquired from a large number of photos of the real surface.

are used in order to collect information about view-

dependent appearance changes for a given surface un-

der given illumination conditions.

The proper radiometricquantity to use for describ-

ing appearance is the outgoing radiance. Imaging

sensors, cameras and human eyes alike, produce re-

sponses that are proportional to the incident radiance

and the proportionality constant depends on the ge-

ometry of the sensor, (Dutr´e et al., 2003). Radiance

is a radiometric quantity measured inW/(m

· sr) and

describes the power leaving (or arriving at) a surface

point, per solid angle, and per projected area. Radi-

ance does not attenuate with distance and hence a cer-

tain surface looks (to a camera or to a human) equally

bright independent on the distance between the ob-

server and the surface, (provided that interaction with

participating media is disregarded).

4.1 Measuring Real Surface Radiances

Figure 2: The setup used to acquire photographs of the sur-

face from a all directions. The surface sample and the light

source are mounted together on a setup which can pan and

tilt relative to a static camera on a tripod. For these con-

trolled experiments the light source is the only illumination

in the room.

A high quality digital camera is used to take im-

ages of a surface. Figure 2 shows images of a simple

setup used to acquire measurements of surface radi-

ance. To be able to properly represent the actual dy-

namic range in reﬂected radiance from real specular

surfaces we capture multiple images from each view-

ing position by varying the exposure. These multi-

ple exposures are fused into a single High Dynamic

Range (HDR) ﬂoating point image using the free soft-

ware tool HDRShop, (Debevec et al., 2007), which is

based on the techniques presented in (Debevec and

Malik, 1997). In this work we have not taken steps to

radiometrically calibrate the camera (Canon EOS 1Ds

MARK II 16 Mega pixel digital SLR ﬁtted with a 28

- 135 mm zoom lens) we use for acquiring images of

real surfaces. Therefore the constant of proportion-

ality between HDR pixel value and reﬂected surface

radiance is unknown. We simply operate on the pixel

values directly in further processing, acknowledging

that there is an unknown scale factor. Nevertheless,

the pixel values will be referred to as measured radi-

ances, a radiance for each of the three color channels.

4.2 Surface Sample Points

A rectilinear sampling grid is imposed on the surface

patch. Figure 3 illustrates how the surface is sampled

at high density (typically 512x512 or 1024x1024). To

ensure that the sample points have the same location

on the surface for all viewpoints a camera calibration

process is run in order to compute the position and

orientation of the camera relative to the surface for

all views. When combined with an a priori internal

camera calibration of the lens and camera parameters

this allows for projecting world coordinate points (the

sample grid points) to the image plane. The black

squares at the corners of the surface patch in ﬁgure

3 are used to estimate camera position and orienta-

tion. A camera calibration toolbox for MATLAB has

been used, (Bouguet, 2007), which also exist as an

OpenCV toolbox for implementation in a C program,

(SourceForge.net, 2007).

Since the camera by no means is inﬁnitely far

awayfrom the surface being imaged the direction vec-

REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL WORLD GLOSSY SURFACES

233

(a) Resolution = 16x16.

(b) Resolution = 32x32.

Figure 3: Two different low sampling resolutions for illus-

trative purposes. In reality higher resolutions are required

to reconstruct text and detailed ﬁgures on the surface. The

black squares at the corners are used for calibrating the cam-

era position and orientation to the surface coordinate sys-

tem.

tor to the focal point of the camera is not the same for

all sample grid locations. That is, for a given image

the viewing direction for all the surface points is not

the same. Therefore, alongside with storing the RGB

radiances for a given surface point for a given image it

is also necessary to store the viewing direction which

the radiance measurement corresponds to.

4.3 Constructing an Observation Map

An Observation Map (OM) is a large multi-

dimensional texture. It contains all the information

accumulated from the sequence of images taken of the

surface. For a given surface point each image (each

viewpoint) produces three radiance values (RGB) and

two parameters describing the viewing direction in

spherical coordinates (the distance to the camera is ir-

relevant as radiance is independent on distance). The

OM thus contains S· T · N times ﬁve ﬂoating point val-

ues, where S and T give the resolution of the sampling

grid, and N is the number of viewpoints from which

the surface has been imaged.

The viewing directions (camera positions relative

to surface coordinate system) used are found by sub-

sampling an icosahedron in order to get uniformly

distributed positions across the upper hemisphere,

(Ballard and Brown, 1982). Between 100 to 300 di-

rections are used for the real surfaces shown in this

paper, which in reality is not quite enough for highly

glossy surfaces. Our rig for acquiring images (ﬁgure

2) is manual. An automated (motorized and control-

lable) rig would make acquisition much easier. Figure

4 shows the measured radiances for a speciﬁc sample

point on a surface.

Figure 4: Measured radiance from 278 views of a point on

a glossy poster. Notice the clear highlight spike caused by

the single light source illuminating the surface during ac-

quisition. The colors in the ﬁgure correspond to the balance

between the measured RGB radiances, but since the high-

light is so strong most of the radiances appear black. This

ﬁgure illustrates the content of the Observation Map at each

surface sample point.

A brute force Image-Based Rendering approach

using the OM directly for subsequent real-time visu-

alization would involve creating a pixel shader which

used the view vector to interpolate between the ob-

served radiances for a given texel. Apart from being

cumbersome this has three obvious disadvantages: 1)

the number of observations may vary from surface to

surface complicating the shader code, 2) the observa-

tions per texel would have to be sorted some how in

order to make it possible to efﬁciently search for near-

est neighbors among the observation directions for a

given view vector, and 3) the shear size of the OM

is prohibitive (1.5 GByte for a 512x512 map, with

300 observation directions). Obviously, some kind of

compression is needed and in this work we have opted

for ﬁtting a parameterized model to the measured ra-

diances in order to be able to reconstruct the radiation

diagram, an example of which was illustrated in ﬁg-

ure 4. Section 5 addresses the modelling issue.

4.4 Synthesized Surface Radiances

Experimentally we have for this paper worked pri-

marily with trying to re-visualize real world surfaces

but all techniques are equally applicable to basing

the modelling process on synthesized surface radi-

ances computed with a Global Illumination solver,

for example a ray tracer/path tracer. It would be

quite straight forward to alter a path tracer to di-

rectly produce the OM data needed per surface point.

This would make it possible to visualize precom-

puted view-dependent global illumination effects in

real time.

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

234

5 PARAMETRIC MODELLING

OF RADIANCES

We are trying to model the outgoing radiance distribu-

tion, e.g., ﬁgure 4, for a surface point as a function of

viewing direction. Depending on the surface material

properties and the illumination conditions of the en-

vironment this radiance distribution can be arbitrarily

complex. The worst case situation is a mirror surface

which reﬂects the environment and the outgoing radi-

ance from a point on the surface can thus change with

very high frequency as a function of view direction

change. Below we describe two models that can be

used to approximate the outgoing radiance distribu-

tion for surfaces with some degree of specularity. We

shall return brieﬂy to the issue of frequency content

of the outgoing radiance distribution in section 5.3.

5.1 The Multiple Highlight Model

This model is inspired by the Phong reﬂection model,

although it should be kept in mind that this work does

not attempt to model reﬂection properties. We are try-

ing to model reﬂected radiance, which obviously is a

combination of reﬂection properties and illumination

conditions as typically formulated by the Rendering

Equation, (Jensen, 2001; Dutr´e et al., 2003).

The Multiple Highlight model (henceforththe MH

model) is inspired by the Phong reﬂection model in

the sense that it models the reﬂected radiance from

a material which can be described by the Phong

reﬂection model, and which is illuminated by N

point/directional light sources. The MH model can

be formulated as a combination of an ambient term

and a sum of highlight cosine lobes:

L(~v) = K

∑

i=1



·~v



(1)

Here L(~v) is the outgoing radiance in the view-

ing direction,~v. N is a number of highlights that the

model must encompass. K

is a view-independent,

“ambient”, term and can be represented with 3 pa-

rameters (one for each color channel). K

is a specu-

lar highlight “amplitude” for each of the N highlights.

Again, since there are three color channels, 3 parame-

ters are needed to represent each K

. The dot product

between

and~v (both vectors are assumed to be unit

length) modulates the specular highlight with the co-

sine of the angle between the viewing direction and a

main “highlight direction”,

. The width of this co-

sine lobe is controlled by raising the cosine to some

power, controlled by the shininess parameter, m. The

shininess is a characteristic of the surface and is there-

fore the same for all the N highlights, i.e., does not

depend on i. The highlight direction for the ith high-

light,

, can be parameterized by two spherical di-

rections, φ

and θ

. When ﬁtting the MH model to

acquisition data we have opted to parameterize it by

its three cartesian coordinate components, though, in

order to have simpler expressions and less non-linear

behaviour. In total the MH model requires 3 (ambient

term), plus N times 6 (three for specular amplitude

plus three for highlight direction) plus 1 (shininess)

parameters, equalling 4+ N · 6 parameters.

A collection of cosine lobes somewhat similar to

eq. 1 were used in (Gibson et al., 2001) for mod-

elling non-isotropic radiation from virtual point light

sources, but in that work the exponent was a ﬁxed

number not estimated during ﬁtting.

The Phong reﬂection model is based on point or

directional light sources which in reality do not ex-

ist. Even the sun subtends a non-inﬁnitesimal solid

angle as seen from the Earth (a 0.53 degree diameter

disc). The reﬂected radiance from a specular surface

modeled by a Phong reﬂection model and illuminated

by an area light source will exhibit a thicker highlight

lobe for a given shininess m than the same surface il-

luminated by a point light source. Therefore, by using

the Phong-like specular highlight term we are not de-

limiting ourselves from modelling highlights caused

by area light sources. The estimated shininess, m,

would just be smaller than the actual shininess of the

surface, in order to encompass the thicker highlight

lobe.

As stated the MH model requires 4+ N · 6 param-

eters, e.g., 10 to model one highlight, 16 to model

2 highlights. Each image (viewing direction) of a

surface point provides three radiance sample values

(one for each of the three color channels). There-

fore, at a minimum four viewing directions for each

point is needed to provide enough equations to solve

for a model with one highlight, and six directions are

needed to solve for two highlights. In practice many

more observations are needed to ensure a good model

ﬁt the critical part being to ensure getting direction

samples that clearly describe the highlight(s).

Model ﬁtting has been implemented in MATLAB

using the built-in lsqcurveﬁt function. This function

allows speciﬁcation of validity ranges of the param-

eters to be estimated, in order to bound the search

space. K

and K

are constrained to the interval [0,1]

for all color channels, since the measured radiance

values are normalized to this range prior to perform-

ing the model ﬁtting. The highlight direction vector,

, is constrained to the upper hemisphere, and the

shininess, m, must be larger than 1.

To ensure fast convergence it is important with

reasonable initial estimates for the parameters to be

REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL WORLD GLOSSY SURFACES

235

ﬁtted. Initial estimates for K

are set to the average of

all direction radiance samples. The initial estimates

for K

are set to zero, whereas the highlight direction

is initialized to the direction of the strongest radiance

sample. Shininess, m, is initialized to 10. Conver-

gence is obtained within 5 to 10 iterations for syn-

thetic data and within 10 to 20 iterations for real data.

Figure 5 shows results of ﬁtting a 10 parameter

(one highlight) MH model to synthetically generated

radiance samples of a surface. Synthetic radiance

samples are generated in the following manner. A

square surface is modelled in Autodesk 3DS Max 8

and illuminated by a single, white point light source.

289 uniformly distributed viewpoints are generated

from the upper hemisphere of a subsampled icosahe-

dron and the surface is rendered from each of these

views. Using these images an observation map is gen-

erated as described in section 4.3, and the MH model

is ﬁtted to each of the surface sample points in the ob-

servation map. Figure 5 concerns an arbitrarily cho-

sen sample point on the surface.

(a) Input radiances.

(b) Fitted MH model.

Figure 5: Top: 289 synthetically generated reﬂected radi-

ance samples uniformly distributed across the upper hemi-

sphere. Bottom: One-highlight MH model ﬁtted to the sam-

ples. The mean percentage residuals are [0.94,2.34,2.34]

for R, G, and B, respectively.

5.2 The Spherical Harmonics Model

As an alternative to the MH model described above

we have also experimented with the much more gen-

eral Spherical Harmonics framework. Spherical Har-

monics (SH) comprise a linear set of basis function

which can be used to represent spherical functions,

similarly to how Fourier basis functions are used

describe normal N-dimensional functions. (Green,

2003) provides an excellent introduction to practical

use of SH based approaches. It is beyond the scope of

the present paper to give an exhaustive description of

the fundamentals of the SH approach. We shall focus

on the two important issues of 1) how to compute the

SH coefﬁcients for a given data input, and 2) how to

handle the issue that the SH approach in principle as-

sumes signals that are periodical over the sphere and

the data we are ﬁtting to are only valid for the upper

hemisphere.

In the continuous case the ith SH coefﬁcient, c

, is

found by projecting the signal (the reﬂected radiance

distribution) onto the ith SH basis function, y

(θ,φ).

This in turn is done through convolution over the en-

tire sphere. In the present case we have a set of radi-

ance samples, L

, k = 1...K, where K is the number

of views stored in the OM for the given surface point.

These samples of the radiance distribution function

can be projected onto the SH basis vectors using nu-

merical integration:

≈

k=1

∑

· y

(θ

,φ

(2)

where (θ

,φ

) are the spherical angles of the view-

ing direction of the kth sample point, and A

is the

solid angle subtended by the viewing direction (each

sample represents a certain potion of the sphere of di-

rections). These solid angles are computed by pro-

jecting all the viewing directions (on a unit sphere)

onto the plane, performing a Voronoi tessellation, and

computing the area of each Voronoi polygon. These

areas are in turned weighted by 1/cos(θ

) to compen-

sate for the projection onto the plane.

The other non-trivial issue concerns handling the

missing samples for the lower hemisphere. Several

different approaches have been tried: 1) only inte-

grating over the upper hemisphere (the Zero Hemi-

sphere, ZH, approach), (Sloan et al., 2002), 2) mir-

roring the upper hemisphere, (Westin et al., 1992),

and 3) the Least Squares Optimal Projection (LSOP)

method, (Sloan et al., 2003). Experimentally the

LSOP method performs better, conﬁrming results in

(Sloan et al., 2003).

5.3 Discussion On Frequency Content

Both models can be made arbitrarily complex, and

can thus in principle be used to model the outgoing ra-

diance from a surface point even for mirror surfaces.

The cost is obviously an increase in the number of pa-

rameters. For there to be any “space savings”, or com-

pression factor, in modelling the radiance distribution

with a function, it must be reasonable to assume that

the radiance distribution is sufﬁciently smooth that it

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

236

can be faithfully represented by the chosen parameter-

ization. In the case of the Spherical Harmonics model

there is also an aliasing issue to consider if the num-

ber of SH coefﬁcients is too small compared to the

frequency content of the radiance distribution.

In the experiments section (section 7) we demon-

strate that the SH model cannot faithfully model the

reﬂected radiance from a glossy poster even with 64

parameters, and this is for a case where only one

light source is illuminating the material. Obviously,

the more light sources the more highlights, which in

turn leads to a higher frequency content in the radi-

ance distribution. A 64 parameter (per color channel)

SH model for a 512x512 surface texture requires 192

MByte memory with no compression.

6 REAL-TIME VISUALIZATION

A demonstrator system has been implemented in C++

using OpenGL. A multi-dimensional texture contain-

ing the parameters estimated for the chosen model

(beit the MH model or the SH model) is mapped to

a surface through uv coordinates. Using the mouse

the user can rotate the surface using a world-in-hand

scheme combined with moving toward or away from

the surface. In a vertex program the viewing direction

is computed and transformed to tangent space. This

viewing direction is interpolated over the surface and

passed to a fragment shader.

The fragment shader depends on the chosen

model. Assuming that the number of parameters used

for the given model is passed to the shader there is

no signiﬁcant overhead in implementing the shader

generally enough that it can encompass an arbitrary

number of parameters. It is quite straight forward to

reconstruct reﬂected radiance given viewing direction

for both models, especially for the MH model, where

the reconstruction expression is given by equation 1.

For the SH model reconstruction is found by comput-

ing the values of all the basis functions in the given

viewing direction and ﬁnding the dot product with the

coefﬁcient vector:

L(~v) ≈

−1

∑

i=0

(~v) (3)

where n is the number of bands used. An SH

model with 64 parameters encompasses 8 bands, 0

through 7. In our implementation the y

(~v) values

are computed directly in the shader instead of storing

them in high resolution tables and interpolating.

The fragment shader performs all computations in

ﬂoating point, and as a result the reconstructed ra-

diance for a given fragment is ﬂoating point. Prior

to display the radiance values are tone-mapped to

LDR 8 bit RGB brightness values as brightness =

1 − exp(−γ · L(~v)), where γ is an interactively ad-

justable exposure value.

7 EXPERIMENTAL RESULTS

The described techniques have been implemented and

tested on a single processor 2.2 GHz AMD athlon 64

machine with 3 GByte RAM. The computer is run-

ning 32 bit Windows XP (emulating 32 bit). The

graphics card is an NVIDIA GeForce 6800 series

card. On this machine the visualization application

is running at 60 fps (limited to refresh rate) with a

10 parameter MH model, and 20 and 10 fps with a

36 and a 64 parameter (per color) SH model, respec-

tively. There is much room for improvement on efﬁ-

ciency of the shader code.

(a) 10 parameter MH model

(b) 3x64 parameter SH model

Figure 6: A glossy poster reconstructed using the MH and

the SH models, respectively. The MH models highlights

much better, but has problems with the white areas on the

poster. See text for explanation.

Figure 6 compares the visual performance of the

MH and SH models. At ﬁrst glance the SH model

performs much more pleasing, since the MH model

appears to give a very spotty result. In reality the SH

model basically fails to represent the highlight areas.

REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL WORLD GLOSSY SURFACES

237

The 64 parameter SH projection simply cannot handle

the narrow highlight lobe correctly. On the other hand

the MH model (the simple Phong inspired model) ac-

tually recreates the highlights very accurately, only

there are some areas where it does not catch them

(large regions of the white part of the poster). The

problem does not lie in the MH model, though. When

inspecting the data in the Observation Map (OM), it is

seen that in the white areas the highlight lobe is sim-

ply so narrow that it is not captured from any view-

point. Figure 7 shows the OM data for a sample point

in a white poster area.

Figure 7: For most of the white area on the poster the high-

light is not captured in the radiance samples and conse-

quently does not appear in the subsequent visual reconstruc-

tion. Compare to ﬁgure 4 which illustrates a sample point

in one of the few white poster areas where the highlight ac-

tually is properly captured.

Two lessons are learned immediately from this: 1)

roughly 300 viewpoint samples distributed over the

viewsphere is not enough to provide good samples

for a material such as a glossy poster, and 2) the SH

model requires more than 64 parameters to represent

highlights from such a material.

Another experimental result is illustrated in ﬁg-

ure 8. When looking at the yellow cloth in the fore-

ground it is seen that the visual reconstruction method

is capable of handlingeffects that are usually achieved

through displacement mapping. More speciﬁcally the

depth discontinuity along the edges of the yellow

cloth correctly exhibits occlusion/dis-occlusion and

shadows appear and vanish consistently with a visual

interpretation of a non-planar surface.

More generally the demonstrated approach

with modelling and reconstructing appearance

embeds illumination, reﬂectance, normal, and depth-

discontinuity information into a single compact

representation. The advantage being that it can

represent and reconstruct all these phenomena, and

the disadvantage being that it is just an appearance

model, which does allow for a change in any of the

embedded information.

(a) View 1

(b) View 2

Figure 8: Two different reconstructed views of a surface

with multiple materials. Notice the correct handling of the

displacement and shadow at the edges of the thick yellow

cloth material in the foreground.

8 DISCUSSIONS AND FUTURE

WORK

This work has explored the possibilities for creating

non-diffuse textures which can be reconstructed (vi-

sualized) in real-time based on direction to the viewer.

With the current size of GPU memory it is clear

that the parameter maps used to represent these non-

diffuse textures must be as economical as possible in

their memory requirements. As described a 10 pa-

rameter (one highlight) MH model for a 512x512 tex-

ture requires 10 MBytes. A three highlight MH map

requires 22 MBytes. In comparison a 64 parameter

(per color channel) SH map of the same resolution re-

quires a staggering 192 MBytes.

We therefore believethe MH model to be the more

viable approach. In addition the MH model captures

the nature of highlights from a real glossy surface bet-

ter than the SH model (given that it is unrealistic to in-

crease the number of SH parameters). It could be said

that the MH model in this context is a more “model-

based model” than the SH model, which is completely

general. The drawback of the MH model is that it is

necessary to a priori determine how many highlight

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

238

to include when ﬁtting the model to data. We have

successfully ﬁtted two-highlight MH models to data

(not reported in this paper), and we have ﬁtted one-

highlight models to data with two highlights. In the

latter case, if highlights are close (on the hemisphere)

the resulting ﬁt is a broad, soft highlight. If the high-

lights are further apart the smaller of the two high-

lights is basically ignored in the ﬁt. An important area

for future research is to explore robust techniques for

ﬁtting the MH model to general data with no a priori

knowledge, e.g., by gradually increasing the number

of highlights until the optimal ﬁt is achieved.

Another approach to combating the memory re-

quirements of the parameter maps is to switch from

a texel-based to a vertex-based approach. For some

applications (when surfaces are not highly textured) it

may be sufﬁcient to store radiance distribution models

per vertex in a high resolution mesh, and then perhaps

use SH models with higher number of parameters.

As hinted in section 1 we also wish to continue

this work in a direction where the view-dependent

textures are used for illuminating augmented objects

in Augmented Reality applications. We are exploring

the use of an Irradiance Volume, (Greger et al., 1998),

approach, where the Irradiance Volume is computed

in real-time based on the view-dependent textures.

This would alleviate the assumption that the envi-

ronment is inﬁnitely distant, an assumption typically

made when using image-based approaches to illumi-

nation in Augmented Reality, (Debevec, 2005; Mad-

sen et al., 2003; Havran et al., 2005; Cohen and De-

bevec, 2001; Barsi et al., 2005).

9 CONCLUSIONS

It has been demonstrated that it is possible to model

real-world glossy surface appearance with a low pa-

rameter model ﬁtted to measured reﬂected radiance

sampled over the viewsphere. The parameters of the

ﬁtted models are stored in texture maps which are

used by pixel shaders for hardware accelerated real-

time visualization.

Two different modelling schemes have been com-

pared, one being a speciﬁc highlight model (the Mul-

tiple Highlight model) inspired by the Phong reﬂec-

tion model, and the other being the completely gen-

eral Spherical Harmonics framework. Experiments

demonstratedthat the MH model is the only viable ap-

proach due to the fact that the SH models require too

many coefﬁcients to faithfully represent highlights

from glossy materials, resulting in parameter textures

that cannot ﬁt into contemporary texture memory.

It was also seen that the number of view sam-

ples needs to be very high (several hundreds) in order

to properly capture highlights if uniform sampling is

used.

The advantage of the approach is that photo-

realistic appearance of a real-world glossy surface

can be achieved, which compresses the combined ef-

fects of illumination, reﬂectance, surface normal and

moderate height differences into a single appearance

model which can be reconstructed in real-time.

ACKNOWLEDGEMENTS

This research is funded by the CoSPE project (26-04-

0171) under the Danish Research Agency.

REFERENCES

Ballard, D. H. and Brown, C. M. (1982). Computer Vision.

Prentice-Hall.

Barsi, A., Szirmay-Kalos, L., and Sz´ecsi, L. (2005). Image-

based illumination on the gpu. Machine Graphics and

Vision, 14(2):159 – 169.

Boivin, S. and Gagalowicz, A. (2001). Image-based render-

ing of diffuse, specular and glossy surfaces from a sin-

gle image. In Proceedings: ACM SIGGRAPH 2001,

Computer Graphics Proceedings, Annual Conference

Series, pages 107–116.

Boivin, S. and Gagalowicz, A. (2002). Inverse rendering

from a single image. In Proceedings: First European

Conference on Color in Graphics, Images and Vision,

Poitiers, France, pages 268–277.

Bouguet, J.-Y. (2007). Camera Calibration Toolbox for

Matlab, www.vision.caltech.edu/bouguetj/calib doc/.

Cohen, J. M. and Debevec, P. (2001). The Light-

Gen HDRShop plugin. www.hdrshop.com/main-

pages/plugins.html.

Dana, K. J., van Ginneken, B., Nayar, S. K., and Koen-

derink, J. J. (1997). Reﬂectance and texture of real-

world surfaces. IEEE Conference on Computer Vision

and Pattern Recognition.

Debevec, P. (1998). Rendering synthetic objects into real

scenes: Bridging traditional and image-based graph-

ics with global illumination and high dynamic range

photography. In Proceedings: SIGGRAPH 1998, Or-

lando, Florida, USA.

Debevec, P. (2002). Tutorial: Image-based lighting. IEEE

Computer Graphics and Applications, pages 26 – 34.

Debevec, P. (2005). A median cut algorithm for light probe

sampling. In Proceedings: SIGGRAPH 2005, Los An-

geles, California, USA. Poster abstract.

Debevec, P. and Malik, J. (1997). Recovering high dynamic

range radiance maps from photographs. In Proceed-

ings: SIGGRAPH 1997, Los Angeles, CA, USA.

REAL-TIME VIEW-DEPENDENT VISUALIZATION OF REAL WORLD GLOSSY SURFACES

239

Debevec, P. E., Borshukov, G., and Yu, Y. (1998). Efﬁcient

view-dependent image-based rendering with projec-

tive texture-mapping. In 9th Eurographics Rendering

Workshop,.

Debevec et al., P. (2007). www.hdrshop.com.

Dutr´e, P., Bekaert, P., and Bala, K. (2003). Advanced

Global Illumination. A. K. Peters.

Gibson, S., Cook, J., Howard, T., and Hubbold, R. (2003).

Rapic shadow generation in real-world lighting envi-

ronments. In Proceedings: EuroGraphics Symposium

on Rendering, Leuwen, Belgium.

Gibson, S., Howard, T., and Hubbold, R. (2001). Flexi-

ble image/based photometric reonstruction using vir-

tual light sources. In Chalmers, A. and Rhyne, T.-M.,

editors, Proceedings: Annual Conference of the Eu-

ropean Association for Computer Graphics, EURO-

GRAPHICS 2001, Manchester, United Kingdom.

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen,

M. F. (1996). The lumigraph. In SIGGRAPH ’96:

Proceedings of the 23rd annual conference on Com-

puter graphics and interactive techniques, pages 43–

54, New York, NY, USA. ACM Press.

Green, R. (2003). Spherical harmonic lighting: The gritty

details. Technical report, Sony Computer Entertain-

ment America.

Greger, G., Shirley, P., Hubbard, P. M., and Greenberg,

D. P. (1998). The irradiance volume. IEEE Computer

Graphics and Applications, 18(2):32–43.

Havran, V., Smyk, M., Krawczyk, G., Myszkowski, K., and

Seidel, H.-P. (2005). Importance Sampling for Video

Environment Maps. In Bala, K. and Dutr´e, P., editors,

Eurographics Symposium on Rendering 2005, pages

31–42,311, Konstanz, Germany. ACM SIGGRAPH.

Jacobs, K. and Loscos, C. (2004). State of the art report on

classiﬁcation of illumination methods for mixed real-

ity. In EUROGRAPHICS, Grenoble, France.

Jensen, H. W. (2001). Realistic Image Synthesis Using Pho-

ton Mapping. A. K. Peters.

Jensen, T., Andersen, M., and Madsen, C. B. (2006). Real-

time image-based lighting for outdoor augmented re-

ality under dynamically changing illumination condi-

tions. In Proceedings: International Conference on

Graphics Theory and Applications, Set´ubal, Portugal,

pages 364–371.

Kang, S. B. (1997). A survey of image-based rendering

techniques. Tehcnical report series CRL 97/4, Cam-

bridge Research Laboratory, Digital Equipment Cor-

poration, One Kendall Square, Building 700, Cam-

bridge, Massachusetts 02139.

Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and

Seidel, H.-P. (2003). Image-based reconstruction of

spatial appearance and geometric detail. ACM Trans-

actions on Graphics, 22(2):234–257.

Levoy, M. and Hanrahan, P. (1996). Light ﬁeld render-

ing. In SIGGRAPH ’96: Proceedings of the 23rd an-

nual conference on Computer graphics and interac-

tive techniques, pages 31–42, New York, NY, USA.

ACM Press.

Madsen, C. B. and Laursen, R. (2007). A scalable gpu-

based approach to shading and shadowing for photo-

realistic real-time augmented reality. In Proceedings:

International Conference on Graphics Theory and Ap-

plications, Barcelona, Spain, pages 252 – 261.

Madsen, C. B., Sørensen, M. K. D., and Vittrup, M. (2003).

Estimating positions and radiances of a small num-

ber of light sources for real-time image-based lighting.

In Proceedings: EUROGRAPHICS 2003, Granada,

Spain, pages 37 – 44.

McAllister, D. K., Lastra, A. A., and Heidrich, W. (2002).

Efﬁcient rendering of spatial bi-directional reﬂectance

distribution functions. In Graphics Hardware 2002,

pages 79–88.

Miller, G. S. P., Rubin, S. M., and Ponceleon, D. B. (1998).

Lazy decompression of surface light ﬁelds for pre-

computed global illumination. In Rendering Tech-

niques, pages 281–292.

Oliveira, M. M. (2002). Image-based modelling and render-

ing: A survey. RITA - Revista de Informatica Teorica

a Aplicada, 9(2):37 – 66. Brasillian journal, but paper

is in English.

Sloan, P.-P., Hall, J., Hart, J., and Snyder, J. (2003). Clus-

tered principal components for pre-computed radiance

transfer. In Proceedings: SIGGRAPH 2003, New

York, New York, USA, pages 382 – 391.

Sloan, P.-P., Kautz, J., and Snyder, J. (2002). Precomputed

radiance transfer for real-time rendering in dynamic,

low-frequency lighting environments. In Proceedings:

SIGGRAPH 2002, San Antonio, Texas, USA, pages

527 – 536.

SourceForge.net (2007). OpenCV Computer Vision Library,

www.sourceforge.net/projects/opencv/.

Westin, S., Arvo, J., and Torrance, K. (1992). Predict-

ing reﬂectance functions from complex surfaces. In

Proceedings: SIGGRAPH 1992, New York, New York,

USA, pages 255 – 264.

Wood, D. N., Azuma, D. I., Aldinger, K., Curless, B.,

Duchamp, T., Salesin, D. H., and Stuetzle, W. (2000).

Surface light ﬁelds for 3D photography. In Proceed-

ings: SIGGRAPH 2000, pages 287–296.

Yu, Y., Debevec, P., Malik, J., and Hawkins, T. (1999).

Inverse global illumination: Recovering reﬂectance

models of real scenes from photographs. In Pro-

ceedings: SIGGRAPH 1999, Los Angeles, California,

USA, pages 215 – 224.

Zickler, T., Enrique, S., Ramamoorthi, R., and Belhumeur,

P. (2005). Reﬂectance sharing: Image-based render-

ing from a sparse set of images. In Rendering Tech-

niques 2005: 16th Eurographics Workshop on Ren-

dering, pages 253–264.

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

240