POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

Mohammad Amin Sadeghi, Seyyed Mohammad Mohsen Hejrati and Niloofar Gheissari

Computer Vision Group, School of Mathematics

Institute for studies in theoretical Physics and Mathematics, Tehran, Iran

Keywords:

Image stitching, Poisson equation, Panorama, Color correction, Registration, Image mosaicing.

Abstract:

A new method for seamless image stitching is presented. The proposed algorithm is a hybrid method which

uses optimal seam methods and smoothes the intensity transition between two images by color correction. A

dynamic programming algorithm that ﬁnds an optimal seam along which gradient disparities are minimized

is used. A modiﬁcation of Poisson image editing is utilized to correct color differences between two images.

Different boundary conditions for the Poisson equation were investigated and tested, and mixed boundary

conditions generated the most accurate results. To evaluate and compare the proposed method with competing

ones, a large image database consisting of more than two hundred image pairs was created. The test image

pairs are taken at different lighting conditions, scene geometries and camera positions. On this database the

proposed approach tested favorably as compared to standard methods and has shown to be very effective in

producing visually acceptable images.

1 INTRODUCTION

Since two registered images usually have local and

global intensity differences, in order to create visu-

ally acceptable images it is essential to complement

the registration process with seamless image stitch-

ing. As noted in (Hasler and Susstrunk, 2000), sev-

eral factors reﬂect the intensities recorded by cameras.

They include: exposure variances, white balancing,

gamma correction, vignetting and digitizer parame-

ters. Seamless stitching has important applications in

many high level tasks such as building panoramic im-

ages, virtual reality, super resolution and texture syn-

thesis. Recently there have been many breakthrough

attempts in seamless image stitching such as (Levin

et al., 2003), (Uyttendaele et al., 2001), (Jia and Tang,

2005a), (Jia and Tang, 2005b).

There are two main approaches for the elimination

of seams and artifacts in the overlapping images. The

ﬁrst approach is to ﬁnd an optimal seam along which

some energy function is minimized (Jia and Tang,

2005a), (Agarwala et al., 2004). Jia and Tang (Jia and

Tang, 2005a) have described different possible energy

functions (weighting functions) that might be use to

ﬁnd the optimal seam. These energy functions may

be based on intensity or gradient differences of the

two images. Dynamic programming or graph cut is

used to minimize the energy function. One advantage

of optimal seam methods is that they are applicable

to dynamic scenes. This is because the optimal seam

does not pass through areas of high gradient disparity.

However, optimal seam algorithms are less suitable

when the illuminations of the input images are not the

same or their overlapping area consists of thin strips.

The second approach is based on enhancing the

intensity values of one or both images so that the re-

sulting stitched image looks natural. This task can be

performed either globally or locally.

Szeliski and Shum (Shum and Szeliski, 2001) pro-

pose a method for building panoramic images capa-

ble of correcting local and global intensity misalign-

ments. Their feathering algorithm results in more nat-

ural blending effects. Still in many cases it produces

unnatural seams under different illumination condi-

tions. Belt and Adelson (Burt and Adelson, 1983)

use multiresolution splines for color correction. Uyt-

tendaele et al. (Uyttendaele et al., 2001) proposed a

method capable of stitching multiple images taken at

various exposures and illuminance conditions. They

allow for large scene changes or misregistrations be-

tween two images. Levin et al. (Levin et al., 2003)

introduced Gradient Domain Image Stitching (GIST)

method. GIST utilizes on an optimization algorithm

to minimize a cost function over image derivatives.

275

Amin Sadeghi M., Mohammad Mohsen Hejrati S. and Gheissari N. (2008).

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING.

In Proceedings of the Third International Conference on Computer Vision Theory and Applications, pages 275-282

DOI: 10.5220/0001077102750282

 SciTePress

This cost function is a dissimilarity measure between

the derivatives of the resulting stitched image and the

derivatives of the input images. Jia and Tang (Jia and

Tang, 2005b) proposed a tensor voting method for

global and local intensity alignment. They use a sym-

metric tensor representation to model the joint image

space (source image and target image intensities) as a

token. They extract curves in the image and let tokens

on a curve communicate mutually to reinforce each

other. A dense tensor ﬁeld postulates smooth connec-

tions among tokens and ensures natural color transi-

tions. This method can successfully handle occlusion

and motion blur. Their local intensity alignment ap-

proaches is devised to correct vignetting effects.

We have observed that a hybrid method that can

take advantage of the strengths of both main ap-

proaches, generates more reliable results. Therefore

the proposed hybrid method ﬁrst ﬁnds an optimal

seam to stitch two images. Then it uses the Pois-

son equation with mixed boundary conditions in the

color domain. As a result, the local color differences

between two images are corrected. Poisson equation

has been already used for image editing (P´erez et al.,

2003) or correcting structure misalignments produced

by stitching (Jia and Tang, 2005a). An advantage of

the proposed method is its low computational time

that makes it particularly desirable for applications to

high resolution images. The proposed method relies

on a model based registration algorithm to align two

images (Gheissari et al., 2008). In fact, the registra-

tion algorithm computes the transformation parame-

ters after recovering the transformation model.

This paper is organized as follows: Section 2,

gives a brief summery of the registration method

used. Methods for ﬁnding optimal cutting curve are

described in section 3. In section 4, the proposed

method, Poisson Local Color Correction (PLCC) is

explained. Different boundary conditions for the so-

lution of the Poisson equation are described in the

following section (4). Some challenging images for

local color correction and the results of applying the

proposed algorithm on them are shown in the section

2 MODEL-BASED IMAGE

REGISTRATION

In this work, a registration method introduced in

(Gheissari et al., 2008) is utilized. In summary this

registration method ﬁrst ﬁnds a set of SIFT inter-

est points (Lowe, 1999) and the correspondences be-

tween them. The outliers are eliminated to gener-

ate an outlier-free set of correspondences. This is

achieved by robustly ﬁtting a translation model to the

correspondences. Next all models in the model library

are ﬁtted to the above set of inliers.

(a) (b)

(e)

Figure 1: Different transformation models for registration:

(a) Translation, (b) Euclidian (c) Similarity, (d) Afﬁne and

(e) Projective transformation.

CAIC is applied to select the correct transforma-

tion model from a library of candidate models (Boz-

dogan, 1987). Figure 1 shows the set of candidate

models. The ﬁnal set of inliers is determined by ap-

plying the chosen model to the data set. The algorithm

estimates the parameters of the chosen model (based

on the set of inliers) to generate the transformation

matrix.

Before ﬁnding the optimal seam and using the

Poisson equation to correct color difference, we ap-

ply the transformation matrix to the source image.

This results in two registered images with an over-

lapping area which can be a triangle, a rectangle, or

a higher order polygon depending on the transforma-

tion model.

3 OPTIMAL SEAM METHODS

Optimal seam algorithms are used widely in appli-

cations such as pattern synthesis and image stitching

(Kwatra et al., 2003), (Jia and Tang, 2005a), (Uyt-

tendaele et al., 2001). These methods ﬁnd a cutting

curve in the overlapping area and stitch the images

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

276

along the cutting curve as their common boundary.

The main approach in optimal seam algorithms is to

specify a weight for each pixel belonging to the over-

lapping area. Then a curve that minimizes this weight

function is selected as the optimal cutting seam.

(a)

(b)

(e)

Figure 2: (a) Two registered images and their overlapping

area, (b) optimal cutting curve, (c) ﬁnal stitched images.

Assume we have registered images I

and I

. The

overlapping area is I

in image I

and I

in image I

(as shown in ﬁgure 2). We would like to cut and stitch

and I

along an optimal curve C.

Let C be a cutting curve, then the cost of havingC

as cutting curve is deﬁned as:

W(C) =

∑

p∈C

W(p), (1)

where W(p) is the value of the weight function asso-

ciated with pixel p. The optimal curve(seam) is the

curve which minimizes the cost W(C).

Different candidates for weight function are

(p) = ||∇I

(p) − ∇I

(p)||

, (2)

and

(p) = ||I

(p) − I

(p)||

, (3)

where ||.||

is the ℓ

–norm. A linear combination of

(p) and W

(p) could also be considered as weight

function (Jia and Tang, 2005a). We have tested differ-

ent energy functions and observed that the function

W(p) = ||∇I

(p) − ∇I

(p)||

(4)

(a) (b)

( f) (g) (h)

Figure 3: (a) And (b) are the input images. The images are

borrowed from ﬁgure 6 of (Jia and Tang, 2005a). The im-

ages, used in our experiments, are of our size 337 × 334.

(p) = ||I

(p) − I

(p)||

. (d) Stitching using intensity

based weight function without color correction (without

PLCC) (e) Stitching using intensity based weight function

with color correction (PLCC) (f) cutting curve using gra-

dient based weight function W

(p) = ||∇I

(p) −∇I

(p)||

(g) Stitching using gradient based weight function without

color correction (without PLCC) (h) Final result after using

gradient based weight function and applying PLCC method.

is more reliable than the other candidates specially

when the illuminations of I

and I

are different. Af-

ter ﬁnding the optimal seam PLCC is applied to cor-

rect color and illumination differences between two

input images. Setting n = 1 (using ℓ

–norm) leads

to better results because small intensity differences

are ignored in higher norms. We use a 3-dimensional

RGB color vector for each pixel as I(p), and observed

that using HSV color space has the same performance

as the RGB color space.

The result of using intensity based weight func-

tion and gradient based weight function is shown in

Figures 3 and 5.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

277

There are various algorithms with different com-

plexities for ﬁnding the optimal cutting curve. Min-

Cut(Max-ﬂow) algorithm (Kwatra et al., 2003) works

accurately. However due to its high complexity, it im-

poses signiﬁcant computational burden on images of

high resolution. We used a combination of dynamic

programming and BFS (Breadth First Search) algo-

rithm in our experiments to ﬁnd optimal cutting curve.

Its complexity is O(n) where n is the number of pixels

in the overlapping area.

Figure 4: In some cases the optimal seam is not a simple

curve.

In some cases the optimal seam gets more com-

plex as shown in Figure 4. Therefore an optimal seam

is not necessarily one simple curve.

4 COLOR CORRECTION BASED

ON POISSON EQUATION

Poisson equation has been used for image editing

and image alignment applications, for example (P´erez

et al., 2003), (Jia and Tang, 2005a) and (Agarwala

et al., 2004). Here we use Poisson equation to cor-

rect color differences between two already registered

images.

Lets Ω be the region (in I

) that lies on one side of

the optimal seam (the side belonging to the source im-

age). The boundary of Ω is denoted by ∂Ω. We sep-

arate ∂Ω into two parts: the common boundary with

(referred to as ∂Ω

), and the other boundaries (de-

ﬁned as ∂Ω

). Our notations are illustrated in Figure

To eliminate artifacts and color differences over

the seam ∂Ω

, we deﬁne a correction function Ψ on

Ω as

= I

+ Ψ (5)

where

is the corrected image. Then the Poisson

equation is solved on Ω and the resulting correction

function Ψ, is added to I

to compensate for color dif-

ferences between two images. For colored images,

we apply the same method on each color channel of

the input image. According to our experiments the

performance of the method is the same in RGB and

(a) (b) (c)

(d) (e) ( f)

(g) (h) (i)

Figure 5: Illustration of applying PLCC on two images.

(a) and (b) input images of size 800 × 600. (c) the result

of putting two registered images together without applying

any corrections. (d) a cutting curve found by using inten-

sity based weight function. (e) the solution of the poisson

equation solved on the corresponding boundary. (f) the re-

sult of stitching the images using an intensity based seam

line. (g) shows a cutting curve found using Gradient based

weight function. (h) the solution of the poisson equation

associated with the cutting curve in (g) and (i) the ﬁnal re-

sult of stitching using intensity based seam line. As you

can see, the color is not corrected perfectly in (f) because

it’s corresponding cutting curve has passed dark areas and

the boundary conditions of the poisson equation in (e) is not

valid.

Figure 6: Notation of the region Ω: Ω is the region of the

visible part of I

. The common boundary with I

is deﬁned

as ∂Ω

, and the other boundaries are deﬁned as ∂Ω

HSV color spaces. For solving Poisson equation on Ω

several boundary conditions could be assumed. The

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

278

(a) (b)

(e) (f )

Figure 7: Comparison between the results of using Drich-

let boundary conditions or Neumann boundary conditions

on ∂Ω

. (a) and (b) source and target images, both of size

800× 600. (c) the solution of Poisson equation with Drich-

let boundary conditions on ∂Ω

as color correction factor.

(d) the respective result of color correction. It is equivalent

to applying Poisson image editing (P´erez et al., 2003) (e)

the solution of Poisson equation with Neumann boundary

conditions on ∂Ω

and (f) the the result of color correction

using the correction factor (e). The correction factor (c) and

(e) are solved on a rectangular domain before applying the

transformation and added to source images after transfor-

mation.

Dirichlet boundary condition equal to the difference

between intensity of two input images along ∂Ω

was

imposed on ∂Ω

. For ∂Ω

the Neumann boundary

condition was used.

Imposing the proper boundary condition is critical in

our application. One might use Dirichlet boundary

condition on ∂Ω

(equation 6). However, the prob-

lem here is the choice of the boundary function on

∂Ω

. For the Dirichlet boundary condition, an ap-

proximation to the value on ∂Ω

is necessary. The

trivial approximation zero for the boundary value re-

sults in imperfect color correction as shown in Fig.

∆Ψ = 0 over Ω with







Ψ = I

− I

, On ∂Ω

Ψ = Constant, On ∂Ω

(6)

where ∆ denotes the Laplace operator.

The Neumann boundary condition on ∂Ω

PLCC (equation 7) produced signiﬁcantly better re-

sults. The superior performance of the Neumann

boundary condition on ∂Ω

is explicable in view of

the fact that it does not ﬁx intensity values along ∂Ω

but constraints the gradient values along ∂Ω

to be

zero. Fixing the intensity on ∂Ω

results in Ψ to

be constant over ∂Ω

, while the ideal values of Ψ

over the ∂Ω are not necessarily constant. Further-

more, there is no straightforward theoretical way to

set the values of Ψ over ∂Ω

and an approximation is

inevitable. This problem is aggravated where the il-

lumination condition between the input images differ

signiﬁcantly.

∆Ψ = 0 over Ω with







Ψ = I

− I

, On ∂Ω

∂Ψ

∂n

= 0, On ∂Ω

(7)

where ˆn is normal vector at ∂Ω. This Poisson equa-

tion is the result of the following minimization prob-

lem (equation 8).

(a)I

(b)I

(a) (b)

Figure 8: Applying PLCC and optimal seam to dynamic

scene. (a) and (b) input images of size 800× 600. (c) Opti-

mal seam based on gradient disparities. (d) Stitching with-

out using optimal seam. The shadow of pedestrians walk-

ing behind is visible, while the pedestrians body are elimi-

nated from the stitching result. The color difference is very

prominent. (e) Stitching using optimal seam. The shadow

of pedestrians are no longer visible. The color differences is

still prominent (f) Final result of applying color correction

(PLCC) and optimal seam.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

279

(a) (b)

(e)

( f)

Figure 9: Creating super resolution images using images

of poor quality. (a), (b), (c) and (d) are input images (of a

puzzle hanged to a wall) taken at four different viewpoints

and illumination conditions. input image (a) is taken un-

der yellow light source, (b) taken under yellow and white

light sources, (c) taken at low exposure time under white

light source and (d) taken under white light source. (e) the

stitching result without PLCC. (f) the stitching result after

PLCC.

min

Ω

|∇Ψ|

with







Ψ = I

− I

, On ∂Ω

∂Ψ

∂n

= 0, On ∂Ω

(8)

where the gradient operator [

∂

∂x

∂

∂y

] is represented by

∇. This minimization problem results in visually nat-

ural output images because it minimizes the ℓ

–norm

of the gradient of intensities near the boundary.

We used the method of ﬁnite differences and the

central difference formula to solve the Poisson equa-

tion and the resulting system of linear equations.

5 EXPERIMENTAL RESULTS

A database for testing and evaluating image stitching

algorithms was created. This database contains over

two hundred pairs of images from over forty scenes.

All pairs are available in three differentstandard sizes.

The scenes vary in color, camera properties, camera

location and lighting. The database will be publicly

available at ”math.ipm.ac.ir/vision”. The transforma-

tion matrix for each pair of images was determined

using (Gheissari et al., 2008). This database contains

challenging images for registration, color correction,

and building panoramic images.

As mentioned before the optimal seam method is

applicable to dynamic scenes. This is because the op-

timal seam does not pass through areas of high gra-

dient disparity. This capability is shown in ﬁgure 8.

As can be seen from this ﬁgure, two different groups

of pedestrians are present in the input images. The

optimal seam passes through the narrow gap between

these two groups and therefore both groups are pre-

sented in the resulting stitched image. If the opti-

mal seam has not been used, then the shadow of the

pedestrian group behind would be visible without the

pedestrians themselves.

Performance of the proposed method is compared

with some standard competing methods. As shown in

ﬁgure 10, a combination of PLCC and optimal seam

outperforms feathering (Uyttendaele et al., 2001),

pyramid blending (Adelson et al., 1984), GIST (Levin

et al., 2003) and Poisson image editing (P´erez et al.,

2003).

The proposed algorithm is desirable for making

super resolution images due to its high accuracy and

low time complexity. To show this capability we

chose four images of a puzzle hanged to a wall. The

images were taken at four different view points and

signiﬁcantly various illumination conditions as shown

in ﬁgure 9. The results of applying the proposed al-

gorithm shows its success .

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

280

(a)I

(b)I

(c)

(d)

(e) ( f) (g) (h) (i) ( j)

Figure 10: Comparing PLCC with some competing meth-

ods: (a) and (b) are input images borrowed from ﬁgure 4

of (Levin et al., 2003). The images are of size 348 × 255

and 290× 257 respectively. (c) Stitched image after apply-

ing optimal seam , (d) The color correction result generated

by PLCC after ﬁnding optimal seam, (e) Feathering (Uyt-

tendaele et al., 2001), (f) Pyramid blending (Adelson et al.,

1984), (g) Feathering on the gradients (Uyttendaele et al.,

2001), (h) Poisson editing (P´erez et al., 2003), (i) GIST

(Levin et al., 2003) (j) The result of PLCC method.

6 CONCLUSIONS

A new method for seamless image stitching was pre-

sented. The proposed algorithm is a hybrid method,

that combines optimal seam approaches and methods

which smooth the intensity transition between two

images (by color correction). Using a dynamic pro-

gramming algorithm, an optimal seam along which

gradient disparities are minimized is determined. A

local color correction method based on Poisson equa-

tion is used to enhance color variations between im-

ages. Different boundary conditions to solve the Pois-

son equation were investigated and tested. We con-

cluded that using mixed boundary conditions gen-

erates the most accurate results. To evaluate and

compare the proposed method (with some competing

methods), a large image database consisting of over

two hundred image pairs was utilized. The test im-

age pairs are taken at different lightening conditions,

scene geometry and camera positions. Compared to

standard methods, proposed approach performed fa-

vorably on this database and has shown to be very

effective in producing visually acceptable images.

ACKNOWLEDGEMENTS

We would like to acknowledge Dr. Mohammad Reza

Mokhtarzade for his valuable advices on solving

PDEs. In addition, we would like to acknowledge

Dr. Mehrdad Shahshahani for his kind helps in this

project and in writing this paper. Mr. Mostafa Ka-

mali Tabrizi helped us in modifying the source code

for registration. We appreciate the valuable discus-

sions he had with us. Finally we thank Zohre Sharaﬁ,

Parisa Mirshams, Paria Mehrani, Sheyda Beigpoor for

providing us with beautiful test images. This project

was supported by Institute for studies in theoretical

Physics and Mathematics (IPM).

REFERENCES

Adelson, E. H., Anderson, C. H., Bergen, J. R., Burt, P. J.,

and Ogden, J. M. (1984). Pyramid methods in image

processing. RCA Engineer, 29(6).

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S.,

Colburn, A., Curless, B., Salesin, D., and Cohen, M.

(2004). Interactive digital photomontage. ACM Trans.

Graph., 23(3):294–302.

Bozdogan, H. (1987). Model selection and akaike’s infor-

mation criterion (aic): The general theory and its ana-

lytical extensions.

Burt, P. J. and Adelson, E. H. (1983). A multiresolution

spline with application to image mosaics. ACM Trans.

Graph., 2(4):217–236.

Gheissari, N., Tabrizi, M. K., Sharaﬁ, Z., and Mirshams, P.

(2008). Model-based global image registration. In

Proc. of the 3rd International Conference on Com-

puter Vision Theory and Applications, page to appear.

POISSON LOCAL COLOR CORRECTION FOR IMAGE STITCHING

281

Hasler, D. and Susstrunk, S. (2000). Colour handling in

panoramic photography.

Jia, J. and Tang, C.-K. (2005a). Eliminating structure and

intensity misalignment in image stitching. In ICCV

’05: Proceedings of the Tenth IEEE International

Conference on Computer Vision, pages 1651–1658,

Washington, DC, USA. IEEE Computer Society.

Jia, J. and Tang, M.-C.-K. (2005b). Tensor voting for im-

age correction by global and local intensity alignment.

IEEE Trans. Pattern Anal. Mach. Intell., 27(1):36–50.

Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A.

(2003). Graphcut textures: image and video synthe-

sis using graph cuts. In SIGGRAPH ’03: ACM SIG-

GRAPH 2003 Papers, pages 277–286, New York, NY,

USA. ACM Press.

Levin, A., Zomet, A., Peleg, S., and Weiss, Y. (2003).

Seamless image stitching in the gradient domain.

Lowe, D. G. (1999). Object recognition from local scale-

invariant features. In Proc. of the International

Conference on Computer Vision ICCV, Corfu, pages

1150–1157.

P´erez, P., Gangnet, M., and Blake, A. (2003). Poisson im-

age editing. In SIGGRAPH ’03: ACM SIGGRAPH

2003 Papers, pages 313–318, New York, NY, USA.

ACM Press.

Shum, H.-Y. and Szeliski, R. (2001). Construction of

panoramic image mosaics with global and local align-

ment. pages 227–268.

Uyttendaele, M., Eden, A., and Skeliski, R. (2001). Elim-

inating ghosting and exposure artifacts in image mo-

saics. pages II–509– II–516 vol.2.

VISAPP 2008 - International Conference on Computer Vision Theory and Applications

282