A NOVEL REGION BASED IMAGE FUSION METHOD USING

DWT AND REGION CONSISTENCY RULE

Tanish Zaveri

Electronics & Communication Engineering Dept., Institute of Technology, Nirma University, Ahmedabad, India

Mukesh Zaveri

Department of Computer Engineering, Sardar Vallabhbhai National Institute of Technology, Surat, India

Keywords: Normalized cut, Discrete wavelet transform, Region consistency.

Abstract: This paper proposes a novel region based image fusion scheme using discrete wavelet transform and region

consistency rule. In the recent literature, region based image fusion methods show better performance than

pixel based image fusion methods. The graph based normalized cutset algorithm is used for image

segmentation. The region consistency rule is used to select the regions from discrete wavelet transform

decomposed source images. The new MMS fusion rule is also proposed to fuse multimodality images.

Proposed method is applied on large number of registered images of various categories of multifocus and

multimodality images and results are compared using standard reference based and nonreference based

image fusion parameters. It has been observed that simulation results of our proposed algorithm are

consistent and more information is preserved compared to earlier reported pixel based and region based

methods.

1 INTRODUCTION

In the recent years image fusion has emerged as an

important research area because of its wide

application in many image analysis task such as

target recognition, remote sensing, wireless sensor

network and medical image processing. We use the

term image fusion to denote a process by which

multiple images or information from multiple

images is combined. These images may be obtained

from different types of sensors. There has been a

growing interest in the use of multiple sensors to

increase the capabilities of intelligent machines and

systems. Actually the computer systems have been

developed those are capable of extracting

meaningful information from the recorded data

coming from the different sources. In other words,

image fusion is a process of combining multiple

input images of the same scene into a single fused

image, which preserves full content information and

also retains the important features from each of the

original images. The integration of data, recorded

from a multisensor system, together with

knowledge, is known as data fusion (Devid, 2001).

Because of the limited depth of focus in digital

camera, one part of the object is well focused while

the other parts are being out of focus. Parts of the

image which are out of focus have less depth of field

so in order to get all the detailed information from

out of focus area, the image fusion method is used.

The image fusion method is used to combine

relevant information from two or more source

images into one single image such that the single

image contains most of the information from all the

source images. Most of advance sensors used in

recent years have capability to produce an image.

The sensors like optical cameras, millimetre wave

cameras, infrared cameras, x-ray cameras and radar

cameras are examples of it. In this paper, IR camera

and MMW camera images are used to apply our

proposed method.

Image fusion methods (Piella, 2003) are

classified mainly into two categories: (i) pixel based

and (ii) region based. Pixel based methods generally

deal with pixel level information directly. In pixel

based image fusion method, the average of the

source images is taken pixel by pixel. However this

339

Zaveri T. and Zaveri M. (2009).

A NOVEL REGION BASED IMAGE FUSION METHOD USING DWT AND REGION CONSISTENCY RULE.

In Proceedings of the International Joint Conference on Computational Intelligence, pages 339-346

DOI: 10.5220/0002283203390346

 SciTePress

leads to undesired side effects in the resultant image.

There are various techniques for image fusion at

pixel level available in literature (Zhong, 1999)

(Anna, 2006). The region based algorithm has many

advantages over pixel base algorithm. It is less

sensitive to noise, better contrast, less affected by

misregistration but at the cost of complexity (Piella,

2003). Researchers have recognized that it is more

meaningful to combine objects or regions rather than

pixels. Piella (Piella, 2002) has proposed a

multiresolution region based fusion scheme using

link pyramid approach. Recently, Li and Young

(Shutao, 2008) have proposed region based

multifocus image fusion method using spatial

frequency as a fusion rule.

Zhang and Blum (Zhong, 1999) had proposed a

categorization of multiscale decomposition based

image fusion schemes for multifocus images. As per

the literature, large part of research on

multiresolution (MR) image fusion has emphasized

on choosing an appropriate representation which

facilitates the selection and combination of salient

features. The issues to be addressed are how to

choose the specific type of MR decomposition

methods like pyramid, wavelet, morphological etc.

and the number of decomposition levels. More

decomposition levels do not necessarily produce

better results (Zhong, 1999) but by increasing the

analysis depth, neighbouring features of lower band

may overlap. This gives rise to discontinuities in the

composite representation and thus introduces

distortions, such as blocking effect or ringing

artifacts into the fused image. The first level discrete

wavelet transform (DWT) based decomposition is

used in proposed algorithm to mitigate the drawback

of Multiscale transform. Also a region is more

meaningful structure in multifocus image and it has

many advantages over pixel based algorithm.

Proposed algorithm uses region based approach. The

heart of our algorithm is the segmentation of an

image. The normalized cut based (Shi, 2000) image

segmentation method is used in proposed algorithm.

The new region consistency and mean max and

standard deviation (MMS) fusion rule are proposed

in this algorithm to improve the fusion image

quality.

The paper is organized as follows. Proposed

algorithm is described in the following section. The

reference based and nonreference based image

fusion evaluation parameters are introduced in

section 3. The simulation results and assessment are

described in Section 4. It is followed by the

conclusion.

2 PROPOSED ALGORITHM

In this section first framework of proposed region

based image fusion method using DWT and region

consistency rule is introduced. The discrete wavelet

transform divides the source image into sub images

details are explained in (Mallat, 1989). The sub

images arise from separable applications of vertical

and horizontal filter. The resultant first level

decomposed four images include LL

sub band

image corresponding to coarse level approximation

image. Also the other three decomposed image

include (LH

, HL

, HH

) sub band images

corresponding to finest scale wavelet coefficient

detail images. Most image fusion method based on

DWT apply max or average fusion rule on DWT

decomposed approximate and detailed images to

generate final fused image. This fusion rule based on

DWT produces significant distortion in final fused

image as described in Fig. 1.

Figure 1: DWT computation process for fusion.

In Fig. 1, two image ia & ib of 2 x 2 size are

considered as the input source images. After

applying DWT, image is decomposed into four

components which are represented app, hor_D,

ver_D and dia_D. Among these four coefficients;

one app is approximate coefficient and hor_D,

ver_D and dia_D are three detail coefficients as

shown in the Fig. 1. Now if we apply Max rule to

select fused coefficient to decomposed images;

hor_D, ver_D, dia_D coefficient will be selected

from image ia while approximate coefficient will be

selected from image ib. So when inverse DWT is

applied on these images, it adds undesired

information distortion in final fused image. After

applying inverse DWT, the pixel values of resultant

fused matrix are shown in Fig. 1 which is not related

to any of the input images and the difference

between two pixels also changed. Therefore, it is

IJCCI 2009 - International Joint Conference on Computational Intelligence

340

Figure 2: Block diagram of the proposed method.

necessary to design a technique which can solve this

problem and generate consistent pixel value related

to either of the input images. The effect of this

problem is more severe in region based image fusion

scheme. To solve this problem region consistency

rule is introduced in the proposed method which is

explained in detail later in this section. The block

diagram of proposed algorithm is shown in Figure 2.

The fused image can be generated by following

steps as describe below.

Step 1 The DWT explained in (Gonzalez, 2006) is

applied on image IA which gives first level

decomposed image of one approximate image

(LL

) and three detail images (LH

, HL

, HH

Step 2 The Normalized cut segmentation algorithm

is applied on image IA. Segmentation is then down

sampled to match the size of DWT decomposed

image.

Step 3 The n numbers of segmented regions are

extracted from approximate component of image IA

and IB using segmented image. How the value of n

is decided is explained later in this step when fusion

rules are explained.

We have used two different fusion rules to

compare extracted regions from different kind of

source images. The SF is widely used in many

literatures (Shutao, 2008) to measure the overall

clarity of an image or region. The n

region of an IA

image is defined by RA

. The spatial frequency of

that region is calculated using Row frequency (RF)

and Column frequency (CF) as described in (1) and

finally SF is calculated using (2). SF parameter

presents the quality of details in an image. Higher

the value of SF, more details of image are available

in that region extracted. First fusion rule is region

based image fusion rule with spatial frequency (SF)

as described in (3), which is used to identify good

quality region extracted from multifocus source

images and image I

rla1

is generated. SF of nth region

of Image IA and IB is defined as SF

and SF

respectively.

[F(i,j)-F(i,j-1)]

RF =

[F(i,j)-F(i-1,j)]

CF =

∑∑

(1)

SF = RF +SF

(2)

nAnBn

rla

nAnBn

RA SF SF

≥

⎧

⎨

⎩

(3)

Here An and Bn are number of regions in image

IA and IB respectively. The value of n varies from 1

to i, where n = {1, 2, 3….i}. The value of i equals to

9 produces best results and it is determined after

analyzing many simulation results of various

categories of source image dataset. Regions

extracted after applying normalized cut set

segmentation algorithm on approximate image LL

are represented as RA

and RA

respectively. I

rla1

A NOVEL REGION BASED IMAGE FUSION METHOD USING DWT AND REGION CONSISTENCY RULE

341

is resultant fused image after applying fusion rule 1

as described in (3). This rule or any other existing

fusion parameter is not enough to capture desired

region from all the type of source images especially

multisensor images so new mean max and standard

deviation (MMS) rule is proposed in our algorithm.

MMS is an effective fusion rule to capture

desired information from multimodality images.

This proposed fusion rule exploits standard

deviation, max and mean value of images or regions.

The MMS is described as

max/*An An An AnMMS ME SD R=

(4)

Where

An, An ME SD are mean, standard deviation

of nth region of image LL

and

AnmaxR

maximum intensity value of same region of image

. The advantage of using MMS is that it

provides a good parameter to extract a region which

has more critical details.

(a) (b)

Figure 3: Input source multimodality IR image (a) visual

image (b) IR image ((c) Region method (d) Proposed

method.

This evident from simulation results described

later in this paper. MMS based fusion rule is very

important in the case of multimodality multisensor

images shown in Figure 3. This is evident from the

following example. In this example, two source

images (i) using visual camera & (ii) using IR

camera for surveillance application as shown in Fig.

3 (a) & (b) respectively. In visual image,

background is visible but a person is not visible

which an object of interest is. In IR image this man

is visible.

From our study, it is analyzed that for visual

images, SD is high and ME is low where in

multisensor images captured using sensors like

MMW & IR have ME value high & SD is low so in

our algorithm we have used both SD & ME with

maximum intensity value

AnmaxR to derive new

parameter MMS. From the experiments, it is

observed that the low value of MMS is desired to

capture critical regions especially man in this

multisensor images. The fusion rule 2 is described as

below

An An Bn

rla

Bn An Bn

RA MMS MMS

≤

⎧

⎨

⎩

(5)

Intermediate fused image

1rla

is generated by

fusion rule 2 which is applied for multimodality

images and first fusion rule is applied for multifocus

images. In Fig. 3(c), only region based image fusion

algorithm is applied as described in (Shutao, 2008)

with SF fusion rule. The fusion result generated after

applying MMS fusion rule is shown in the Fig. 3 (d).

It is clearly seen from the results that the MMS rule

is very effective to generate good quality fused

image for multimodality source images. After taking

approximate component by above method, region

consistency rule is applied to select detail

component from both decomposed images, which is

described in (6).

Region consistency rule states that chose detail

components from the results of fusion rule 1 of

approximate component.

rla1 An

rla1 Bn

if I =RA

rlv

⎧

⎨

⎩

(6)

Where

1rlv

is first region of vertical component of

image HL

. Similarly

1rlh

and

1rld

is computed

from image LH

and HH

respectively. If after

applying fusion rule first region is selected from

image IA then all the detail component regions are

also selected from the same image IA. Fused Images

generated with and without applying region

consistency rule are shown in Fig. 4 (c) & (f)

respectively.

Step 4 IDWT is performed to generate Irl1.

Step 5 Repeat the step 1 to 4 for image IB and

generate intermediate fused image Irl2

Step 6 Both Irl1 and Irl2 are averaged to improve

the resultant fused image IFUSE.

This new frame work is an efficient way to

improve the consistency in final fused image and it

avoids distortion due to unwanted information added

without using region consistency rule. The activity

level measured in each region is decided by the

spatial frequency and novel MMS statistical

parameter which is used to generate good quality

IJCCI 2009 - International Joint Conference on Computational Intelligence

342

fused image for all categories of multimodality and

multifocus images. The next section describes image

fusion evaluation criteria in brief.

3 EVALUATION PARAMETERS

We evaluated our algorithm using two categories of

performance evaluation parameters; subjective and

objective for the set of images of various categories.

The objective image fusion parameters are further

divided into reference and non reference based

quality assessment parameters. Fusion performance

can be measured correctly by estimating how much

information is preserved in the fused image

compared to source images.

3.1 Reference based Image Fusion

Parameters

Most widely used reference based image fusion

performance parameters are Entropy, Structural

Similarity Matrix (SSIM), Quality Index (QI),

Mutual Information (MI), Root mean square error

(RMSE). The RMSE and Entropy are well known

parameters to evaluate the amount of information

present in an image (Gonzalez, 2006). Mutual

information (MI) indices are also used to evaluate

the correlative performances of the fused image and

the reference image as explained in (Zheng, 2006)

which is used in this paper as MIr. A higher value of

mutual information (MIr) represents more similar

fused image compared to reference image.

The structural similarity index measure (SSIM)

proposed by Wang and Bovik (Zhou, 2002) is based

on the evidence that human visual system is highly

adapted to structural information and a loss of

structure in fused image indicates amount of

distortion present in fused image. It is designed by

modelling any image distortion as a combination of

three factors; loss of correlation, radiometric

distortion, and contrast distortion as mentioned in

(Zhou, 2002). The dynamic range of SSIM is [-1, 1].

The higher value of SSIM indicates more similar

structures between fused and reference image. If two

images are identical, the similarity is maximal and

equals 1.

3.2 Non Reference Based Image Fusion

Parameter

The Mutual information (MI), the objective image

fusion performance metric (Q

AB/F

), spatial frequency

(SF) and entropy are important image fusion

parameters to evaluate quality of fused image when

reference image is not available. MI described

(Xydeas, 2000) can also be used without the

reference image by computing the MI between

reference image IA, IB and fused image IFUSE. MI

between image IA and IFUSE called as

AFI and

similarly compute

BFI between image IB and IFUSE.

Total MI is computed as described by (7)

FBF

II I

(7)

The objective image fusion performance metric

AB/F

which is proposed by Xydeas and Petrovic

(Xydeas, 2000) reflects the quality of visual

information obtained from the fusion of input

images and can be used to compare the performance

of different image fusion algorithms. The range of

AB/F

is between 0 and 1. The value 0 means all

information is lost and 1 means all information is

preserved.

4 SIMULATION RESULTS

The novel region based image fusion algorithm as

described in section 2 has been implemented using

Matlab 7. The proposed algorithm is applied and

evaluated using large number of dataset images

which contain broad range of multifocus and

multimodality images of various categories like

multifocus with only object, object plus text, only

text images and multi modality IR (Infrared) and

MMW (Millimetre Wave) images. The large image

dataset is required to verify the robustness of an

algorithm. The simulation results are shown in Fig. 4

to 8. Proposed algorithm is applied on various

categories of images for different segmentation

levels and after analyzing the results, we have

considered nine segmentation levels for all our

experiments which improve visual quality of final

fused image. The performance of proposed

algorithm is evaluated using standard reference

based and nonreference based image fusion

evaluation parameters explained in previous section.

4.1 Fusion Results of Multi-focus

Images

The multifocus images available in our dataset are of

three kinds (1) object images (2) only text images

and (3) object plus text images which are shown in

Fig. 4 (a) & (b) clock image, Fig. 5 (a) & (b) pepsi

image, Fig. 6 (a) & (b) book image and Fig. 7 (a) &

A NOVEL REGION BASED IMAGE FUSION METHOD USING DWT AND REGION CONSISTENCY RULE

343

(a) (b)

(f)

Figure 4: Fusion results of multi-focus clock image (a), (b) Multi-focus source images (c) Proposed method (d) DWT & (e)

Li’s method (f) Without region consistancy rule.

(a) (b)

Figure 5:Fusion results of multi-focus pepsi image (a), (b) Multi-focus source images (c)Proposed method (d)DWT & (e)

Li’s method.

(a) (b)

Figure 6:Fusion results of multi-focus book image (a), (b) Multi-focus source images (c) Proposed method (d) DWT & (e)

Li’s method.

(a) (b)

Figure 7:Fusion results of multi-focus text image (a), (b) Multi-focus source images (c) Proposed method (d) DWT & (e)

Li’s method.

(b) text images respectively. In Fig. 4 to 7 column

(a) multifocus images, left portion is blurred and in

column (b) of same figure, right portions of images

is blurred and column (c) shows the corresponding

fused image obtained by applying proposed method

and column (d) and (e) are resultant fused image

obtained by applying pixel based DWT method

proposed by Wang (Anna, 2006) and region based

fusion method proposed by Li and Yang (Shutao,

2008).

The visual quality of the resultant fused image of

proposed algorithm is better than the fused image

obtained by other reported methods. The reference

based and non reference based image fusion

IJCCI 2009 - International Joint Conference on Computational Intelligence

344

(a) (b)

Figure 8 Fusion results for multimodality IR image (a) visible image (b)IR source image (c)Proposed method (d) DWT &

(e) Li’s method.

Table 1: Ref. based image fusion parameters.

Image

Fusion

Methods

Fusion Parameters

SF MIr RMSE SSIM

Text

image

DWT

Based

8.1956 1.5819 6.3682 0.9259

Li’s

Method

10.405 1.4713 5.2671 0.9749

Proposed

Method

10.736 1.9997 2.8313 0.9904

Book

Image

DWT

Based

12.865 3.8095 7.5994 0.9274

Li’s

Method

16.416 3.6379 5.0847 0.9782

Proposed

Method

17.415 6.5173 1.6120 0.9974

parameter comparisons are depicted in Table 1 and

Table 2. All reference based image fusion

parameters SF, MIr, RMSE and SSIM are

significantly better for proposed algorithm compared

to other methods as depicted in Table 1. Also non

reference based image fusion parameters as depicted

in Table 2 are better than compared methods. In

Table 2, SF and

AB/F

are remarkably better than

other compared fusion methods which is also

evident from the visual quality of resultant fused

image.

Table 2: Non Ref. Based Image Fusion Parameters.

Image

Fusion

Methods

Fusion Parameters

SF MI Q

AB/F

Entropy

Clock

image

DWT

Based

8.1501 5.848 0.5696 8.1506

Li’s

Method

10.113 7.356 0.7156 8.7803

Proposed

Method

10.8792 7.713 0.6850 8.7813

Pepsi

image

DWT

Based

11.6721 2.521 0.9287 8.7293

Li’s

Method

13.532 2.703 0.9683 7.1235

Proposed

Method

13.648 5.863 0.7857 7.2350

4.2 Fusion of Multimodality Images

The effectiveness of the proposed algorithm can be

proved by extending its application to the

multimodality concealed weapon detection (MMW

images) and IR images. MMW camera image with

the gun is shown in Fig. 9 (a) and visible images of a

group of persons are shown in Fig. 9 (b). Here the

aim is to detect gun location in the image by using

the visible image.

In visual camera image details of surrounding

area can be observed in shown Fig. 8 (a) and IR

camera detect the human in captured image as

shown in Fig. 8 (b). The aim of applying fusion

algorithm on IR image is to detect the human and its

location using both source images information. The

visual quality of resultant fused images generated by

applying proposed method is better than other

reported methods. New MMS fusion rule is used in

proposed algorithm. This rule preserves critical

regions in fused image which is also evident in

Table 3. The entropy is significantly better than

region based methods as depicted in Table 3.

Entropy is considered to evaluate the final fusion

results of both IR and MMW multimodality source

images because in both the cases IR and MMW

sensor source images are blurred however in that

case SF and Q

AB/F

do not give significant values for

comparison. The simulated results depicted in Table

1, 2 and 3 show that proposed method performs

better than other compared methods for broad

categories of multifocus and multimodality images.

Table 3: Fusion Parameter for Multisensor Images.

Image Fusion Method Entropy

IR Image

DWT Based

6.6842

Li’s Method

6.0472

Proposed Method

6.7814

MMW Image

DWT Based

4.9802

Li’s Method

3.7593

Proposed Method

6.9502

A NOVEL REGION BASED IMAGE FUSION METHOD USING DWT AND REGION CONSISTENCY RULE

345

(a) (b)

Figure 9: Fusion results for multimodality MMW image(a)visual image(b)MMW image(c)Proposed method (d) DWT & (e)

Li’s method.

5 CONCLUSIONS

In this paper, new region based image fusion method

using region consistency rule is described. This

novel idea is applied on large number of dataset of

various categories and simulation results are found

with superior visual quality compared to other

earlier reported pixel and recently proposed Li’s

region based image fusion methods. The novel

MMS fusion rule is introduced to select desired

regions from multimodality images. Proposed

algorithm is compared with standard reference based

and nonreference based image fusion parameters and

from simulation and results, it is evident that our

proposed algorithm preserves more details in fused

image. Proposed algorithm can be further improved

by designing more complex fusion rule.

REFERENCES

Anna, Wang, Jaijining, Sun, Yueyang, Guan, 2006. The

application to wavelet transform to multimodality

medical image fusion. IEEE International Conference

on Networking, Sensing and Control, pp. 270-274.

Devid, Hall, James, LLians, 2001. Hand book of

multisensor data fusion. CRC Press LLC,

Gonzales, Rofael, Richard, Woods, 2006. Digital Image

Processing, Pearson Education, 2nd ed.

Mallat, S., 1989. A theory for multiresoultuion signal

decomposition: the wavelet representation, IEEE

Trans. On Pattern Analysis and Machine Intelligence,

Vol. 2(7), pp. 674-693.

Miao, Qiguang, Wang, Baoshul, 2006. A novel image

fusion method using contourlet transform.

International Conference on Communications, Circuits

and Systems Proceedings, Vol. 1, pp 548-552.

Piella, G., 2003. A general framework for multiresolution

image fusion: from pixels to regions. Journal of

Information Fusion, Vol. 4 (4), pp 259-280.

Piella, Gemma, 2002. A region based multiresolution

image fusion algorithm. Proceedings of the Fifth

International Conference on Information Fusion, Vol.

2, pp 1557- 1564.

Shi, J., Malik, J., 2000. Normalized cuts and image

segmentation, IEEE Transactions on Pattern Analysis

and Machine Intelligence, Vol. 22 (8), 888–905.

Shutao, Li, Bin, Yang, 2008. Multifocus image fusion

using region segmentation and spatial frequency,

Image and Vision Computing, Elsevier, Vol. 26, pp.

971–979.

Timothee Cour, Florence Benezit, Jianbo Shi, Multiscale

Normalized Cuts Segmentation Toolbox for

MATLAB, available at http://www.seas.upenn.edu/

~timothee.

Xydeas, C., S., V., Petrovic, 2000. Objective image fusion

performance measure, Electronics Letters, Vol. 36 (4),

pp 308-309. February 2000.

Yin Chen, Rick S. Blum, 2008 A automated quality

assessment algorithm for image fusion,” Image and

vision computing, Elsevier.

Zheng, Liu, Robert, Laganiere, 2006. On the use of phase

congruency to evaluate image similarity”, IEEE

International Conference on Acoustics, Speech and

Signal Processing, ICASSP, Vol. 2, pp 937-940.

Zhong, Zhang, Rick, Blum, 1999. A categorization of

multiscale decomposition based image fusion schemes

with a performance study for digital camera

application. Proceedings of IEEE, Vol. 87 (8), pp

1315-1326.

Zhou Wang, Alan Bovik, 2002. A universal image quality

index, IEEE signal processing letters, Vol. 9, pp. 81-

81.

IJCCI 2009 - International Joint Conference on Computational Intelligence

346