OVERLAPPED BLOCK MOTION COMPENSATION
FOR FRAME RATE UP CONVERSION
Suk Kim, Taeuk Jeong and Chulhee Lee
Dept. of Electrical and Electronic Engineering, Yonsei Univ., 134 Shinchon-Dong, Seodaemun-Gu, Seoul, Korea
Keywords: Frame Rate Conversion, Overlapped Block Motion Compensation, Multiple Motion Estimation.
Abstract: Recently, motion compensated interpolation methods are used to generate new frames in the frame rate up
conversion. However, they often yield undesirable blocking artifacts due to inaccurate motion estimation.
In order to mitigate these artifacts, we propose a new motion-compensated frame rate up conversion
algorithm with overlapped block motion estimation. The new interpolated frame is obtained by overlapped
block motion compensation of multiple motion estimation between the current and previous frames.
Experimental results show the proposed method produces better perceptual quality and outperforms
conventional motion-compensated methods in terms of PSNR.
1 INTRODUCTION
Currently, a number of video frame rates are used in
various applications. The typical frame rate ranges
from 15 to 60 frames per second. Recently, flat
panel displays are widely available and most
consumer TV monitors use either LCD or PDP
technologies, which display videos with a high
frame rate (50 to 120Hz). In order to achieve such
high frame rates, the frame rate up conversion
(FRUC) is required. FRUC generates additional
frames from adjacent frames, thereby improving
visual quality with flicker reduction. It can be also
applied to low bit rate video coding. If some of
frames are skipped at the encoder to archive low bit
rate compression, the missing frames can be
reconstructed at the decoder using FRUC (Haan,
1999; Dufaux and Moscheni, 1995).
For example, FRUC can be easily implemented
using frame repetition (FR) and temporal linear
interpolation (TLI), though FR may produce motion
jerkiness and TLI produces perceived blurring
artifacts. In order to reduce such artifacts, motion
compensated FRUC (MC-FRUC) algorithms have
been proposed.
Many motioncompensated interpolation (MCI)
algorithms with motion estimation (ME) have been
proposed for FRUC (Choi et al., 2000; Gao et al.,
2008; Haan, 1999; Ha et al., 2004; Ojo and Haan,
1997). These methods use the motion information of
two or more adjacent frames and motion vectors
(MVs) are used to reconstruct a new frame from the
corresponding frames using motion compensated
interpolation (MCI). In general, block matching
algorithms are employed for motion estimation and
compensation between adjacent frames due to the
simplicity and easy implementation. However, they
often yield undesirable artifacts such as blocking and
incompatible block degradation resulted from
inaccurate motion information.
To mitigate this problem, we propose a new MC-
FRUC with overlapped block motion compensation
(OBMC) (Orchard and Sullivan, 1994), which uses
multi-estimators and shifted grid. Then, we
produced new frames by averaging the overlapped
block motion compensation results from the mulitple
motion estimations.
2 PROPOSED METHOD
2.1 Motion Estimation and Motion
Compensation Interpolation
Methods
To obtain motion vectors, we used uni-directional
ME (UDME, Figure 1) and bi-directional ME
(BDME, Figure 2) methods (Choi et al., 2000).
68
Kim S., Jeong T. and Lee C. (2010).
OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION.
In Proceedings of the International Conference on Imaging Theory and Applications and International Conference on Information Visualization Theory
and Applications, pages 68-71
DOI: 10.5220/0002837400680071
Copyright
c
SciTePress
Current
Frame
Interpolated
Frame
Previous
Frame
Figure 1: The uni-directional motion estimation (UDME)
method.
Current
Frame
Interpolated
Frame
Previous
Frame
Figure 2: The bi-directional motion estimation (BDME)
method.
The proposed method first performs block-based
motion estimation for each macro block between the
previous and current frames. Let
P
j
B be the best
matching block in the previous frame for the j-th
block of the current frame (
C
j
B ). The block size is
wh
and the sum of absolute difference (SAD) is
used. The new block
I
j
B
in the interpolated frame is
obtained by averaging the two matching blocks of
the previous and current frames:
1
(, ) ( , ) ( , )
2
IC P
jjxyjxy
Bxy Bxvy v Bxvy v


(1)
where,
(, )
x
y
vv
represents a motion vector.
2.2 OBMC using Multiple Estimators
and Shifted Grid
Figure 3 shows the parameters used in the motion
estimation and Figure 4 shows the block diagram of
the proposed MC-FRUC methods. In the proposed
method, two motion estimators (UDME and BDME)
are used and FRUC is performed by combining the
results obtained by applying several motion
compensations. In other words, we performed
several motion compensations for each pixel by
shifting the macro block by a small amount. Thus,
the proposed method produces a new frame by
combining several motion compensation results
while the conventional MC methods perform just
one motion compensation.
Nx
Ny
w
h
u
v
: Frame Size
: Macro Block Size
( , ) : Shift Size
Nx Ny
wh
uv
Figure 3: Parameters of the overlapped block estimation.
Image
Sequence
Motion
Estimators
Frame
Buffer1
Frame
Delay
Uni-Direction
Average
Interpolated
Frame
Cumulate Blocks
(, )mu nv
Shift
Frame
Buffer2
Current
Frame
Previous
Frame
MVs
Bi-Direction
Figure 4: Block diagram of the proposed method.
In the proposed method, the new pixel value of
the interpolated frame is obtained by averaging all
the overlapped blocks as follows:
(,)
1
(, ) (, )
(, )
II
j
jIxy
f
xy B xy
Ixy
(2)
OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION
69
where
(, )Ixy
denotes a set of block indices which
contains pixel
(, )
x
y
:
}),(|{),(
I
k
ByxkyxI
(3)
represents the number of elements of a set,
which counts the number of elements of the set. To
perform multiple motion compensations, we move
the block grid by u (horizontal increment) and v
(vertical increment). It is noted that u<w and v<h.
3 EXPERIMENTAL RESULTS
The proposed methods were evaluated using two
video sequences: “Foreman” with swaying
background and complex motions and “Table
Tennis” with a stable background and fast motions.
The two sequences are CIF and use the YUV420
format.
The n-th interpolated frame is generated from the
original (n-1)-th and (n+1)-th frames, and the PSNR
between the original and the interpolated n-th frames
are calculated to measure the objective quality of the
interpolated frames. We first performed motion
estimation in the Y channel and use the same motion
vectors for the UV channels.
The size of macro block was set to
16 16
and
the search ranges for two motion estimators are set
to
16
for the UDME method and
8
for the
BDME method. In spite of the different search
ranges, the computational complexities of the two
estimators are the same since the BDME method
searches motion vectors in half-pixel precision.
Table 1-2 show the performance comparison
with various shift sizes. It is noted that the
conventional method is the single ME method with
uw
and
vh
(u:16, v:16).
As can be seen in the tables, the proposed
UDME and BDME methods outperformed the
conventional method by about 0.5dB in terms of
PSNR and showed slight improvement as the shift
size was decreased.
The proposed multi-estimator (UD&BD) method
showed better results than the single estimators and
also provided better performance as the shift size
was decreased.
Figure 5 shows sub-images of the interpolated
frames of Table Tennis when different shift sizes
were used. As can be seen in Figure 5, the blocking
artifacts were reduced, though some blurring effects
appeared as the shift size is smaller.
Table 1: Average PSNR of interpolated frames using the
single ME and multiple MEs of Foreman.
u:16, v:16
u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1
UDME 31.27 31.88 32.04 32.04 31.99
BDME 31.09 31.76 31.90 31.93 31.91
Proposed
UD&BD
33.54 34.15 34.29 34.31 34.25
Table 2: Average PSNR of interpolated frames using
single ME and multiple MEs of Table Tennis.
u:16, v:16
u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1
UDME 32.45 33.80 34.12 34.18 34.19
BDME 32.55 33.57 33.81 33.86 33.87
Proposed
UD&BD
34.10 35.06 35.27 35.31 35.33
u:16, v:16 u:8, v:8
u:4, v:4 u:2, v:2
Figure 5: Sub-images of the interpolated frames with
various shift sizes of Table Tennis. (BDME).
The conventional BDME method produced
blocking artifacts and edge degradation in the fast
motion area due to unreliable motion estimation. On
the other hand, the proposed overlapped methods
reduced this problem and provided improved
perceptual quality.
Figure 6 shows sub-images of interpolated
frames using the single ME and the proposed
multiple MEs. As can be seen in Figure 6, the
proposed overlapped motion estimation methods
substantially removed those blocking artifacts. Also,
the proposed methods provided better PSNR.
IMAGAPP 2010 - International Conference on Imaging Theory and Applications
70
BDME (u:16, v:16) BDME (u:8, v:8)
UDME (u:16, v:16) UDME (u:8, v:8)
BD&UD (u:16, v:16) BD&UD (u:8, v:8)
Figure 6: Sub-images of the interpolated frames of
Foreman.
Table 3: Comparison of computational complexity of the
proposed methods.
u:16, v:16
u:8, v:8 u:4, v:4 u:2, v:2 u:1, v:1
UDME 1 4 16 64 256
BDME 1 4 16 64 256
Proposed
UD&BD
2 8 32 128 512
Table 3 shows the computational complexity
ratio of the proposed methods. If we assume the
amount of calculation of UDME is 1, the complexity
of the small shift size (u:1, v:1) is 256 times larger.
Although the overlapped UDME and BDME
methods with small shift sizes provided best PSNRs,
they suffer from the high implementation cost and
complexity.
Table 4 shows the performance comparison and
computational complexity of the proposed method
and the OBME method (Overlapped Block Motion
Estimation) (Ha et al., 2004). As can be seen in
Table 4, the proposed method produced better
PSNRs and lower complexity than the OBME
methods.
Table 4: PSNR and computational complexity of the
proposed and OBME (Ha et al., 2004) methods.
PSNR
(complexity)
Foreman Table Tennis
(u:16, v:16) (u:8, v:8) (u:16, v:16) (u:8, v:8)
Proposed
UD&BD
33.54
(2)
34.15
(8)
34.10
(2)
35.06
(8)
Proposed UD&BD
with sub-sampling
32.72
(0.5)
33.32
(2)
33.95
(0.5)
34.96
(2)
OBME
31.46
(4)
31.87
(16)
32.50
(4)
33.29
(16)
OBME with
sub-sampling
30.85
(1)
31.24
(4)
32.49
(1)
33.3
(4)
4 CONCLUSIONS
In this paper, we propose a new MC-FRUC
algorithm with overlapped motion estimation and
compensation. Experimental results show that the
proposed methods substantially reduced blocking
artifacts and were robust against inaccurate motion
estimation.
REFERENCES
Choi, B. T., Lee, S. H, and Ko, S. J. (2000). New frame
rate up-conversion using BI-Directional Motion
Estimation. IEEE Trans. on Consumer Electronics,
46(3), 603-609.
Dufaux, F. and Moscheni, F. (1995). Motion estimation
techniques for digital TV: a review ad a new
contribution. Proc. of the IEEE, 832, 858-876.
Gao, X., Yang, Y., and Xiao, B. (2008). Adaptive frame
rate up-conversion based on motion classification.
Signal Processing, 88, 2970-2988.
Haan, G. D. (1999). Video Scanning Format Conversion
and Motion Estimation. Digital Signal Processing
Handbook, Chap. 54, CRC Press.
Ha, T.-H., Lee, S. J., and Kim, J. S. (2004). Motion
Compensated Frame Interpolation by new Block-
based Motion Estimation Algorithm. IEEE Trans. on
Consumer Electronics, 50(2), 752-759.
Ojo, O. A. and Haan, G. D. (1997). Robust motion-
compensated video upconversion. IEEE Trans. on
Consumer Electronic, 43(4), 1045-1056.
Orchard, M. T. and Sullivan G. J. (1994). Overlapped
block motion compensation: An estimation-theoretic
approach. IEEE Trans. on Image Process, 3(5), 693-
699.
OVERLAPPED BLOCK MOTION COMPENSATION FOR FRAME RATE UP CONVERSION
71