AN IMPROVED SUPER RESOLUTION RECONSTRUCTION

ALGORITHM FOR VIDEO SEQUENCE

Hyo-Moon Cho

School of Electrical, University of Ulsan, 7-524 san29 Muger-dong Nam-gu, Ulsan, South Korea

Sang-Bok Cho

School of Electrical, University of Ulsan, 7-524 san29 Muger-dong Nam-gu, Ulsan, South Korea

Keywords: High resolution, Motion compensation error, Super resolution, Low resolution.

Abstract: In this paper, we introduce the input image selection-method to improve the reconstructed high-resolution

(HR) image quality. To obtain ideal super-resolution (SR) reconstruction image, all input images are well-

registered. However, the registration is not ideal in practice. By reason of this, the number of input images

with low registration error is more important than the number of input images in order to obtain good

quality of a HR image. The input image suitability could be evaluated by using statistical and restricted

registration properties. Therefore, we propose the input image evaluation-method in automatic manner as

pre-processing of SR reconstruction and its architecture. In video sequences, all input images in specified

region are allowed to use SR reconstruction as low-resolution (LR) input image and/or the reference image.

The evaluation basis is decided by the threshold value and this threshold is calculated by using the

maximum motion compensation error (MMCE) of the reference image. If the motion compensation error

(MCE) of LR input image is in the range of 0 < MCE < MMCE then this LR input image is selected for SR

reconstruction, else then LR input image are neglected. The optimal reference LR (ORLR) image is decided

by comparing the number of the selected LR input (SLRI) images for each reference LR input (RLRI)

image. Finally, we generate a HR image by using optimal reference LR image and selected LR images and

by using the Hardie’s interpolation method. This proposed algorithm is expected to improve the quality of

SR without any user intervention.

1 INTRODUCTION

The super-resolution (SR) algorithms, which

produce a high-resolution (HR) image from multiple

low-resolution (LR) images, are one of the most

promising approaches for image quality

enhancement.

The basic framework of SR is to define the

reference LR image from given multiple LR images

and to generate the HR reconstruction image by

fusing the several LR images based on reference LR

image.

The various SR algorithms have been proposed

by using frequency-domain approach and spatial-

domain approach. The SR algorithm was first

presented by Tsai and Huang. They used the

frequency-domain approach to obtain one improved

resolution image from several down-sampled noise-

free version of it, based on the spatial aliasing effect.

Usually, it assumed that there is some or small

relative motion between the camera and the scene. If

there is relative motion between the camera and the

scene, then the first step of SR is to register or align

the image, i.e., it computes the motion vector from

one image to others. The accurate registrations

between LR images are very important. To obtain

ideal SR reconstruction, the ideal registration is

executed among the LR input images and the point

spread function (PSF) of camera is accurate

estimated. However, in practice, these are not ideal.

To overcome such SR constraints, the recognition-

based SR algorithms have been proposed and the

methods considering the registration error in SR

reconstruction process have been tried.

Generally, the improved SR algorithms

considering the registration error have been applied

by using the restricted input images. However, all

input images of the specified region in video

192

Cho H. and Cho S. (2007).

AN IMPROVED SUPER RESOLUTION RECONSTRUCTION ALGORITHM FOR VIDEO SEQUENCE.

In Proceedings of the Second International Conference on Signal Processing and Multimedia Applications, pages 188-191

DOI: 10.5220/0002139601880191

 SciTePress

sequence can be used to SR reconstruction.

Therefore, to improve SR algorithm, we designed

the automatic input image evaluation and selection

block as pre-processing in order to solve a

registration error problem.

This automatic input image evaluation and

selection block designates the reference image

arbitrary because all input images are allowed to use

super-resolution reconstruction and it selects the

suitable LR input images from all LR input (LRI)

images for designated reference image. Finally, it

selects the optimized reference input LR (RILR)

image by using statistical property. All of these

processing are executed automatic manner and this

method achieves the HR image reconstruction

without any user interventions and fast and low

computational complexity. And also we design its

hardware architecture. This article is organized as

follows: in section 2, we describe the importance of

the ILR images and RILR image selection; in

section 3, we introduce the proposed algorithm and

its architecture; experimental results are shown in

section 4, and we conclude with section 5.

2 THE LR IMAGE AND RLRI

IMAGE SELECTION

The SR reconstruction is achieved when the LR

images with sub-pixel distance from reference

registered onto the HR image grid. Therefore, the

motion estimation and motion compensation are

essentially needed.

Figure 1: Influence of unsuitable ILR image selection.

The reconstructed HR image is distorted when the

sub-pixel motion is estimated inaccurately. In this

case, the distortion can be called the registration

error (RE) noise. In many conventional approaches,

it is assumed that the RE noise is neglected or

considered the same for all LR images: i.e., all LR

images are ideally confirmed.

In most and practice, ideal registration has

constraint by registration model or restricted

searching area. As these reasons, the quality of

reconstructed HR image is decreased.

Figure 1 shows contaminated HR reconstruction

image when the RE is high. Therefore the LR input

images with low RE is more important than the

number of LR input images.

3 PROPOSED ALGORITHM

We designate a reference LR image arbitrary in

specified region of input video since all LR input

images are allowed to reference LR input image and

we restrict the number of maximum ILR images up

to five frames. The reason of this, if many LR input

images are used then it is difficult to evaluate the

performance of algorithms comparing with others

and the SR reconstruction processing time is very

long. Therefore, we restricted the number of LR

images and make analysis of the influence of a

registration error.

To select the suitable LR input images, first, our

algorithm decides the threshold value by using

designated reference LR input image and its motion

compensated image. Secondly, our algorithm

evaluates the SAD between a LR input image and

reference image is less than the threshold value.

Figure 2 shows the flow chart to select the selected

LR input (SRLI) images. Therefore, if the LR input

image is selected, then it has a low RE noise.

Figure 2: The flowchart of SLRI selection method.

The SAD (Sum of Absolute Difference)

computation is used as calculation of the motion

compensation error (MCE).The maximum motion

AN IMPROVED SUPER RESOLUTION RECONSTRUCTION ALGORITHM FOR VIDEO SEQUENCE

193

compensation error (MMCE) for each reference

image is calculated by computing SAD between the

reference LR input image and its motion estimated

in sequence order.

Generally, the distribution of registration error

(RE) for the sub-pixel shifts in practical video, the

RE has maximum value when shifting displacement

is at (dx, dy)=(0.5, 0.5) as shown figure 3.

To decide optimized reference LR input

(ORLRI) image, we count the number of the

selected LR input (SLRI) images for each designated

reference LR input image and compare the number

of SLRI[f(n)] of each designated reference input

image in specified region as shown figure 4.

To decrease computation time and complexity,

we designed advanced architecture as shown figure

5. As shown figure 5, the duplicated SAD operators

on each frame are removed. And the number of

selected low resolution input (SLRI) images for one

reference image equals to 4 then the remaining the

selection of LR input image processes are stopped.

The partial distortion elimination (PDE) method

is used to decrease the SAD calculation. The basic

concept of PDE is that the motion estimation is more

efficient as the larger initial accumulated SAD value

is selected.

Figure 3: Distribution of RE depending on sub-pixel shift.

Figure 4: ORLRI block flowchart.

4 EXPERIMENTAL RESULTS

We used the video sequences to experiment by PC

camera which is made by China. The camera

features are as below:

- CMOS image sensor with 1.3M pixel

- Maximum resolution 1280x960

- 24bit true color video mode

- 640 x 480@30fps

- USB 2.0 interface

We used the Hardie algorithm as interpolation with

16 restricted searching-area and 20 iterations. Figure

7 is shown the sequenced images for experiment.

Figure 5: Advanced architecture of the proposed algorithm.

Figure 6: The sample imaged which used to experiment.

SIGMAP 2007 - International Conference on Signal Processing and Multimedia Applications

194

We take a picture of the sample object and

modified the image size into 128x128 from 512x512

original HR image. The modified images are used as

LR images, the f(3) frame is selected as reference

image by our algorithm procedure and it takes three

selected LR input (SLRI) images and other cases

have two or under SLRI images. Figure 8 is shown

the result image by using the proposed SR

reconstruction method Figure 8 is shown the result

image by using conventional SR reconstruction

method is shown the result image with bilinear

interpolation method. As shown figure 7, 8, figure 7

is better of them..

Figure 7: Experimental result image.

Figure 8: Result image with conventional SR

reconstruction method.

Figure 9: Result image with bilinear interpolation method.

5 CONCLUSIONS

We introduced the improved SR reconstruction

algorithm which selects the suitable input images

and the optimized reference image in automatic

manner. And also we presented its architecture.

However, we restrict the maximum input frame

number is five, so this is too small. If we increase

maximum input frame number then the image

quality higher than this. To do this, which restrict the

maximum frame number as 5 frames, is only for fast

and easy evaluation of our algorithm. This algorithm

is used to moving picture like as surveillance

systems.

ACKNOWLEDGEMENTS

This research was financially supported by the

Ministry of Commerce, Industry and Energy

(MOCIE) and Korea Industrial Technology

Foundation (KOTEF) through the Human Resource

Training Project for Regional Innovation and 2 level

BK21 (EVERDEC(e-Vehicle Research & human

Resource Development Center)) in MOE (Ministry

of Education & Human Resources Development).

REFERENCES

S.C. Park, M.K. Park and M.G. Kang, “Super-resolution

image reconstruction: a technical overview,” IEEE

Signal Processing magazine, pp. 21-36, May 2003.

E.S. Lee and M. G. Kang, “Regularized adaptive high-

resolution image reconstruction considering inaccurate

subpixel registration,” IEEE Trans. Image Processing,

vol. 12, pp. 826-837, July 2003.

Y. Altunbasak, A.J. Patti, and R.M. Mersereau, “Super-

resolution still and video reconstruction from MPEG-

coded video,” IEEE Trans. Circuits Syst. Video

Technol., Vol. 12, pp. 217-226, April 2002.

S. Baker and T. Kanade, “Limits on sper-resolution and

how to break them,” IEEE Trans. Pattern Analysis

Machine Intelligence, vol. 24, pp. 1167-1183, Sept.

2002.

R.C. Hardie, K.J. Barnard, and E.E. Armstrong, “Joint

MAP registration and high-resolution image

estimation using a sequence of undersampled image,”

IEEE Trans. Image Processing, vol. 6, pp. 1621-1633,

Dec. 1997.

R.R Schultz and R.L. Stevenson, “Extraction of high-

resolution frames from video sequences,” IEEE Trans.

Image Processing, vol. 5, pp. 996-1011, June 1996.

AN IMPROVED SUPER RESOLUTION RECONSTRUCTION ALGORITHM FOR VIDEO SEQUENCE

195