OVERLAPPED BLOCK MOTION COMPENSATION

FOR FRAME RATE UP CONVERSION

Suk Kim, Taeuk Jeong and Chulhee Lee

Dept. of Electrical and Electronic Engineering, Yonsei Univ., 134 Shinchon-Dong, Seodaemun-Gu, Seoul, Korea

Keywords: Frame Rate Conversion, Overlapped Block Motion Compensation, Multiple Motion Estimation.

Abstract: Recently, motion compensated interpolation methods are used to generate new frames in the frame rate up

conversion. However, they often yield undesirable blocking artifacts due to inaccurate motion estimation.

In order to mitigate these artifacts, we propose a new motion-compensated frame rate up conversion

algorithm with overlapped block motion estimation. The new interpolated frame is obtained by overlapped

block motion compensation of multiple motion estimation between the current and previous frames.

Experimental results show the proposed method produces better perceptual quality and outperforms

conventional motion-compensated methods in terms of PSNR.

1 INTRODUCTION

Currently, a number of video frame rates are used in

various applications. The typical frame rate ranges

from 15 to 60 frames per second. Recently, flat

panel displays are widely available and most

consumer TV monitors use either LCD or PDP

technologies, which display videos with a high

frame rate (50 to 120Hz). In order to achieve such

high frame rates, the frame rate up conversion

(FRUC) is required. FRUC generates additional

frames from adjacent frames, thereby improving

visual quality with flicker reduction. It can be also

applied to low bit rate video coding. If some of

frames are skipped at the encoder to archive low bit

rate compression, the missing frames can be

reconstructed at the decoder using FRUC (Haan,

1999; Dufaux and Moscheni, 1995).

For example, FRUC can be easily implemented

using frame repetition (FR) and temporal linear

interpolation (TLI), though FR may produce motion

jerkiness and TLI produces perceived blurring

artifacts. In order to reduce such artifacts, motion

compensated FRUC (MC-FRUC) algorithms have

been proposed.

Many motioncompensated interpolation (MCI)

algorithms with motion estimation (ME) have been

proposed for FRUC (Choi et al., 2000; Gao et al.,

2008; Haan, 1999; Ha et al., 2004; Ojo and Haan,

1997). These methods use the motion information of

two or more adjacent frames and motion vectors

(MVs) are used to reconstruct a new frame from the

corresponding frames using motion compensated

interpolation (MCI). In general, block matching

algorithms are employed for motion estimation and

compensation between adjacent frames due to the

simplicity and easy implementation. However, they

often yield undesirable artifacts such as blocking and

incompatible block degradation resulted from

inaccurate motion information.

To mitigate this problem, we propose a new MC-

FRUC with overlapped block motion compensation

(OBMC) (Orchard and Sullivan, 1994), which uses

multi-estimators and shifted grid. Then, we

produced new frames by averaging the overlapped

block motion compensation results from the mulitple

motion estimations.

2 PROPOSED METHOD

2.1 Motion Estimation and Motion

Compensation Interpolation

Methods

To obtain motion vectors, we used uni-directional

ME (UDME, Figure 1) and bi-directional ME

(BDME, Figure 2) methods (Choi et al., 2000).

Kim S., Jeong T. and Lee C. (2010).

OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION.

In Proceedings of the International Conference on Imaging Theory and Applications and International Conference on Information Visualization Theory

and Applications, pages 68-71

DOI: 10.5220/0002837400680071

 SciTePress

Current

Frame

Interpolated

Frame

…

Figure 1: The uni-directional motion estimation (UDME)

method.

Current

Frame

Interpolated

Frame

…

Figure 2: The bi-directional motion estimation (BDME)

method.

The proposed method first performs block-based

motion estimation for each macro block between the

previous and current frames. Let

B be the best

matching block in the previous frame for the j-th

block of the current frame (

B ). The block size is

wh

and the sum of absolute difference (SAD) is

used. The new block

in the interpolated frame is

obtained by averaging the two matching blocks of

the previous and current frames:

(, ) ( , ) ( , )

IC P

jjxyjxy

Bxy Bxvy v Bxvy v





(1)

where,

(, )

represents a motion vector.

2.2 OBMC using Multiple Estimators

and Shifted Grid

Figure 3 shows the parameters used in the motion

estimation and Figure 4 shows the block diagram of

the proposed MC-FRUC methods. In the proposed

method, two motion estimators (UDME and BDME)

are used and FRUC is performed by combining the

results obtained by applying several motion

compensations. In other words, we performed

several motion compensations for each pixel by

shifting the macro block by a small amount. Thus,

the proposed method produces a new frame by

combining several motion compensation results

while the conventional MC methods perform just

one motion compensation.

…

: Frame Size

: Macro Block Size

( , ) : Shift Size

Nx Ny



Figure 3: Parameters of the overlapped block estimation.

Image

Sequence

Motion

Estimators

Frame

Buffer1

Frame

Delay

Uni-Direction

Average

Interpolated

Frame

Cumulate Blocks

(, )mu nv

Shift

Frame

Buffer2

Current

Frame

MVs

Bi-Direction

Figure 4: Block diagram of the proposed method.

In the proposed method, the new pixel value of

the interpolated frame is obtained by averaging all

the overlapped blocks as follows:

(,)

(, ) (, )

(, )

jIxy

xy B xy

Ixy







(2)

OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION

where

(, )Ixy

denotes a set of block indices which

contains pixel

(, )

}),(|{),(

ByxkyxI 

(3)

 represents the number of elements of a set,

which counts the number of elements of the set. To

perform multiple motion compensations, we move

the block grid by u (horizontal increment) and v

(vertical increment). It is noted that u<w and v<h.

3 EXPERIMENTAL RESULTS

The proposed methods were evaluated using two

video sequences: “Foreman” with swaying

background and complex motions and “Table

Tennis” with a stable background and fast motions.

The two sequences are CIF and use the YUV420

format.

The n-th interpolated frame is generated from the

original (n-1)-th and (n+1)-th frames, and the PSNR

between the original and the interpolated n-th frames

are calculated to measure the objective quality of the

interpolated frames. We first performed motion

estimation in the Y channel and use the same motion

vectors for the UV channels.

The size of macro block was set to

16 16



and

the search ranges for two motion estimators are set

16

for the UDME method and

8

for the

BDME method. In spite of the different search

ranges, the computational complexities of the two

estimators are the same since the BDME method

searches motion vectors in half-pixel precision.

Table 1-2 show the performance comparison

with various shift sizes. It is noted that the

conventional method is the single ME method with

uw

and

vh

(u:16, v:16).

As can be seen in the tables, the proposed

UDME and BDME methods outperformed the

conventional method by about 0.5dB in terms of

PSNR and showed slight improvement as the shift

size was decreased.

The proposed multi-estimator (UD&BD) method

showed better results than the single estimators and

also provided better performance as the shift size

was decreased.

Figure 5 shows sub-images of the interpolated

frames of Table Tennis when different shift sizes

were used. As can be seen in Figure 5, the blocking

artifacts were reduced, though some blurring effects

appeared as the shift size is smaller.

Table 1: Average PSNR of interpolated frames using the

single ME and multiple MEs of Foreman.

u:16, v:16

u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1

UDME 31.27 31.88 32.04 32.04 31.99

BDME 31.09 31.76 31.90 31.93 31.91

Proposed

UD&BD

33.54 34.15 34.29 34.31 34.25

Table 2: Average PSNR of interpolated frames using

single ME and multiple MEs of Table Tennis.

u:16, v:16

u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1

UDME 32.45 33.80 34.12 34.18 34.19

BDME 32.55 33.57 33.81 33.86 33.87

Proposed

UD&BD

34.10 35.06 35.27 35.31 35.33

u:16, v:16 u:8, v:8

u:4, v:4 u:2, v:2

Figure 5: Sub-images of the interpolated frames with

various shift sizes of Table Tennis. (BDME).

The conventional BDME method produced

blocking artifacts and edge degradation in the fast

motion area due to unreliable motion estimation. On

the other hand, the proposed overlapped methods

reduced this problem and provided improved

perceptual quality.

Figure 6 shows sub-images of interpolated

frames using the single ME and the proposed

multiple MEs. As can be seen in Figure 6, the

proposed overlapped motion estimation methods

substantially removed those blocking artifacts. Also,

the proposed methods provided better PSNR.

IMAGAPP 2010 - International Conference on Imaging Theory and Applications

BDME (u:16, v:16) BDME (u:8, v:8)

UDME (u:16, v:16) UDME (u:8, v:8)

BD&UD (u:16, v:16) BD&UD (u:8, v:8)

Figure 6: Sub-images of the interpolated frames of

Foreman.

Table 3: Comparison of computational complexity of the

proposed methods.

u:16, v:16

u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1

UDME 1 4 16 64 256

BDME 1 4 16 64 256

Proposed

UD&BD

2 8 32 128 512

Table 3 shows the computational complexity

ratio of the proposed methods. If we assume the

amount of calculation of UDME is 1, the complexity

of the small shift size (u:1, v:1) is 256 times larger.

Although the overlapped UDME and BDME

methods with small shift sizes provided best PSNRs,

they suffer from the high implementation cost and

complexity.

Table 4 shows the performance comparison and

computational complexity of the proposed method

and the OBME method (Overlapped Block Motion

Estimation) (Ha et al., 2004). As can be seen in

Table 4, the proposed method produced better

PSNRs and lower complexity than the OBME

methods.

Table 4: PSNR and computational complexity of the

proposed and OBME (Ha et al., 2004) methods.

PSNR

(complexity)

Foreman Table Tennis

(u:16, v:16) (u:8, v:8) (u:16, v:16) (u:8, v:8)

Proposed

UD&BD

33.54

(2)

34.15

(8)

34.10

(2)

35.06

(8)

Proposed UD&BD

with sub-sampling

32.72

(0.5)

33.32

(2)

33.95

(0.5)

34.96

(2)

OBME

31.46

(4)

31.87

(16)

32.50

(4)

33.29

(16)

OBME with

sub-sampling

30.85

(1)

31.24

(4)

32.49

(1)

33.3

(4)

4 CONCLUSIONS

In this paper, we propose a new MC-FRUC

algorithm with overlapped motion estimation and

compensation. Experimental results show that the

proposed methods substantially reduced blocking

artifacts and were robust against inaccurate motion

estimation.

REFERENCES

Choi, B. T., Lee, S. H, and Ko, S. J. (2000). New frame

rate up-conversion using BI-Directional Motion

Estimation. IEEE Trans. on Consumer Electronics,

46(3), 603-609.

Dufaux, F. and Moscheni, F. (1995). Motion estimation

techniques for digital TV: a review ad a new

contribution. Proc. of the IEEE, 832, 858-876.

Gao, X., Yang, Y., and Xiao, B. (2008). Adaptive frame

rate up-conversion based on motion classification.

Signal Processing, 88, 2970-2988.

Haan, G. D. (1999). Video Scanning Format Conversion

and Motion Estimation. Digital Signal Processing

Handbook, Chap. 54, CRC Press.

Ha, T.-H., Lee, S. J., and Kim, J. S. (2004). Motion

Compensated Frame Interpolation by new Block-

based Motion Estimation Algorithm. IEEE Trans. on

Consumer Electronics, 50(2), 752-759.

Ojo, O. A. and Haan, G. D. (1997). Robust motion-

compensated video upconversion. IEEE Trans. on

Consumer Electronic, 43(4), 1045-1056.

Orchard, M. T. and Sullivan G. J. (1994). Overlapped

block motion compensation: An estimation-theoretic

approach. IEEE Trans. on Image Process, 3(5), 693-

699.

OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION