A Hybrid Algorithm using Metaheuristics Applied to H.264/AVC
Video Encoder
Arthur Tórgo Gómez and Iris Correa C. Linck
Universidade do Vale do Rio dos Sinos, Av. Unisinos - 950, São Leopoldo, Brazil
Keywords: Metaheuristics, Tabu Search, Genetic Algorithm, H.264 Video Encoder, CODEC.
Abstract: This paper focuses on the study and the analysis of the dynamic relationship among six parameters of the
H.264/AVC video encoder, that are: frame rate, bit rate, quantization parameter for I slice, B slice, and P
slice, and the number of B slices in the GOP (Group of Pictures). For this study, it was developed and
implemented a hybrid algorithm called Simulator of Metaheuristics applied to a CODEC (SMC). The SMC
algorithm consists of two metaheuristics that are Tabu Search and Genetic Algorithm. It tries to find the best
configuration of the studied parameters in order to obtain a good quality and compression of the encoded
video in the H.264/AVC standard. The SMC algorithm uses a maximization objective function as an
objective evaluation method to reach the proposed goals.
1 INTRODUCTION
Algorithms for compression and decompression
video, called video CODECs, have been continually
improved over the last decade to meet the demands
of the market (Golston and Rao, 2006). One of the
latest CODECs is the H.264/MPEG-4 AVC (ITU-T,
2007) which was defined as a standard for encoding
video for the Brazilian Digital Television System
(BDTS), (ABNT NBR 15601:2008; ABNT NBR
15602-1:2008). This standard allows that several
parameters can be configured allowing a high
flexibility to obtain a good video quality and
therefore influencing its performance.
This work focuses on the development of an
hybrid algorithm (Glover et al., 1999) called
Simulator of Metaheuristics applied to a CODEC
(SMC) to define the configuration parameters of the
H.264 video CODECs for the BDTS and thus
optimize its performance. The solution proposed
consists in two metaheuristics, Tabu Search (Glover,
1986) and Genetic Algorithm (Holland, 1975).
The H.264 CODEC configuration problem is
treated as a combinatorial optimization problem
known as the Selection Problem of Parties
(Gonçalves and Resende, 2004) and classified as
NP-Hard (Papadimitriou, 1994).
This paper is organized as follow: Section 2
presents the related works. Section 3 presents the
proposed solution for the Simulator of Metaheuristics
applied to a CODEC (SMC). The Section 4 presents
the experiments and results, and the section 5 presents
the conclusions about this work.
2 RELATED WORK
A number of authors have proposed algorithms to
optimize the performance of the H.264 CODEC
(Yasakethu et al., 2008; Cermak et al., 2011; Huang
et al., 2006; Nemethova et al., 2004). For evaluating
the performance of these algorithms, methods and
techniques have been developed to evaluate the
perceived quality level of the video content (Sikora,
2005; Moriyoshi et al., 2000; Malvar et al., 2003;
ITU-R, 2002). These methods can be mainly
categorized into two major classes: the subjective
and objective methods (Ries et al., 2007;
Seshadrinathan et al., 2009; (Pinson and Wolf, 2003;
Wolf and Pinson, 2007).
The objective methods are, in the most of cases,
based on an sensitivity framework of the error, being
one of the metrics most widely used, the Peak Signal
to Noise Ratio - PSNR (Winkler and Mohandas,
2008). In this context, this paper presents an
objective evaluation method, based on an objective
function proposed by a Simulator of Metaheuristics
applied to a CODEC (SMC).
212
Tórgo Gómez A. and Correa C. Linck I..
A Hybrid Algorithm using Metaheuristics Applied to H.264/AVC Video Encoder.
DOI: 10.5220/0004038602120217
In Proceedings of the 9th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2012), pages 212-217
ISBN: 978-989-8565-21-1
Copyright
c
 2012 SCITEPRESS (Science and Technology Publications, Lda.)

3 PROPOSED SOLUTION
The SMC computational model in the figure 1 is
basically composed of three modules: initial
solution, tabu search, genetic algorithm and
functional blocks of H.264/AVC.
Figure 1: SMC – Computational Model of the Simulator
of Metaheuristics applied to a CODEC ( SMC).
In the model shown in Figure 1 is provided a
viable initial solution to the TS, which consists in a
set of six parameters of the H.264. They were
obtained by the JIGA of tests from the DigConv
project (UNISINOS, 2008) and reached an
acceptable PSNR.
The TS explores the search space around this
initial solution using six types of neighborhood in
order to find a better solution and it uses the
equation (1) as the Objective Function. The
neighborhood structure consists in 6 movements and
the tabu list stores the last 7 tabu movements. The
stop criterion applied is the maximum number of
iteration without improvement of the Objective
Function value (nbmax) and its size was defined in
100. A list of the twenty best solutions found is
stored in an elite candidate list. The Genetic
Algorithm uses as its initial population the elite
candidate list generated from TS which consist in
twenty individuals or elite candidates. The
chromosome is represented by the six studied
parameters. The fitness function is represented by
Equation (1). The selection strategy used is
tournament selection; the reproduction strategy
consists in crossover and mutation with probability
equal of 0.8 and 0.2 respectively. The number of
generations is 100 and it is used as the stop criteria.
If the solution found by the Genetic Algorithm
module is better than the solution found by the Tabu
Search module then the SMC returns to the Tabu
Search module in order to optimize this solution.
Otherwise, the SMC is finished.
3.1 Objective Function
The SMC algorithm uses the Objective Function
(OF) which is estimated with six parameters
(decision variables) as shown by equation (1) and it
is a function of maximizing.
Max OF= α
1
BR + α
2
FR + α
3
1
QI
+α
4
1
QP
+
+α
5
1
QB
+α
6
1
PF
(1)
The coefficients α
1
=7,α
2
=1,α
3
=32,α
4
=
24, α
5
=21,α
6
= 122, were estimated by
calculating the not tendentious weights, the
parameters QI, QP and QB are the quantization
parameters for I slices, P slices and B slices
respectively. The BR value is the bitrate out of the
video, the PF value is the number of B coded frames
inserted between P slices that is used to determine
the initial picture ordering entrance for the video to
be encoded. FR value is the framerate for the
entrance of the video.
4 EXPERIMENTS
Experimental simulations were performed on the
first frames of well-known QCIF (176x144) video
sequences. The set of parameters adopted as initial
solution by the SMC algorithm is BR=64bps,
FR=30.3fps, QI=16, QP=16, QB=18, PF=1. The
ranges of values for each parameter are: 0.1 to 192.0
for BR; 0.1 to 100.0 for FR; 0.1 to 51.0 for QI, QP
and QB; 1 to 5 for PF.
Two experiments were performed using the SMC
algorithm, in order to analyze the behavior of the
studied parameters and to verify if they are in
agreement with the dynamic H.264 parameters
studied in the literature.
In the first experiment 20 units were added to the
AHybridAlgorithmusingMetaheuristicsAppliedtoH.264/AVCVideoEncoder
213

non-biased weight of BR parameter. The weights of
the other variables were maintained with their
original non-biased weights. The SMC algorithm
was performed 300 times, where in each thirty times
it was calculated the average value of each
parameter (Table 1).
The experiment continued repeatedly, by adding
20 units to the BR parameter weight until the 200
units. The SMC algorithm was performed 300 times
for each change of weight value. All this process
was done for all studied parameters. The goal of this
experiment is to evaluate the behavior of the mean
values of the parameters when one of them has a
gradual increase in your non-biased weight and the
others parameters remain with their original non-
biased weights. These values are observed through
the values assumed by the parameters of the
objective function.
In the second experiment, the non-biased weight
of BR parameter was set aside by assigning zero to it
and the other variables remained with their non-
biased original weights. The SMC algorithm was
performed 300 times, where in each 30 times was
calculated the average value of each parameter
(Table 3) and was observed how the other
parameters behaved without the influence of BR
parameter in the objective function. This same
process was performed for the all studied parameters
and the results are shown in the section 4.1.
4.1 Results of the First Experiment
In the graph 1 (Figure 2) is shown the behavior
curves among QI, QP, QB and PF parameters when
the non-biased weight of the BR parameter is
intensified in the objective function.
In the graph 2 (Figure 2) is shown the curves of
behavior between the Objective Function (OF) and
BR. In the graph 3 (Figure 2) is shown the curves of
behavior between FR and BR.
In the Table 1 is shown the mean values of the
parameters and the objective function found by the
SMC algorithm in the first experiment.
In the Table 1, each line corresponds to a set of
mean values found by the SMC algorithm after it
runs 30 times. This set of mean values was used to
construct the graphs in the Figure 2. The values
highlighted in the table 1 correspond to the
minimum and the maximum values found by the
SMC for each parameter and the Objective Function.
The graph 1 (Figure 2) is also shown that QP and
QB tend to be inversely related and PF has a direct
influence on QB. In sum, the more B frames (PF
parameter) we have, the more compression a video
will suffer (by increasing parameter QB).Thus
Figure 2: Graphs 1, 2 and 3 of Experiment 1 using the
intensification of BR parameter.
Table 1: Mean values of the experiment 1 using BR
parameter intensification.
decreasing the video quality can be observed that the
algorithm proposed in this paper tries to compensate
this loss of video quality by decreasing the
quantization parameter of P frames (QP). The P
frames serve as a reference to B frames, and thus the
algorithm tries to obtain a better image quality.
According to the literature (Yasakethu et al.,
2008), quantization parameters, which in this case
are represented by QP, QB and QI, influence the
amount of spatial detail of the video to be saved.
In Table 1 is observed that the higher values of
QP, QB and QI not reach high values in comparison
with the values in Table 3 of the experiment 2. This
occurs because in the Graph 2 (Figure 2) it is shown
a constantly growing of BR parameter. The BR
parameter tends to decrease the maximum values of
QP, QB and QI parameters, since quantization
parameters are inversely proportional to BR when it
comes to achieve an improvement image quality.
According to the literature (Yasakethu et al., 2008;
Kim et al., 2006), quantization parameters influence
the bitrate (BR parameter), which means that when
ICINCO2012-9thInternationalConferenceonInformaticsinControl,AutomationandRobotics
214

the bitrate is increased, consequently decreases the
compression of video (lower values of QP, QB and
QI) and vice versa, in order to achieve a better
quality image.
In the Graph 2 (Figure 2) shows that as the
bitrate increases the objective function also
increases, which means that the image quality is
better. However, according to the literature
(Koumaras et al., 2005) improvements of the video
quality is not significant for bitrates higher than a
specific threshold.
The graph 3 (Figure 2) shows that as the BR
grows, the FR fluctuates and it tends to stabilize.
According to the literature (Ries et al., 2007), the
framerate (FR) may increase or decrease in relation
to the increasing of bitrate (BR) due to the type of
video content. Ries et al (2005) states that panoramic
videos receive a better rating in video quality when
the framerate drops. However, dynamic videos
receive a better rating when the framerate and bitrate
grow together.
It was found that PF and FR parameters obtained
a low standard deviation in the SMC. This fact
means that the SMC algorithm explored a restricted
search space, while the others parameters reached a
high deviation standard that means a larger search
space was explored. The objective function reached
a high deviation standard due to the high deviation
standard from the most of parameters.
Figure 3: Graphs 1, 2 and 3 of Experiment 1 using QI
parameter.
In Figure 3 is shown the graphs of experiment 1
where the non-biased weight of QI is increased in
the objective function.
The Graph 1 (Figure 3) shows the curves of
behavior of the QB, QP, PF and FR parameters in
relation to QI parameter.
In Graph 2 (Figure 3) the QI parameter
contributed to increase the objective function (OF).
The objective function (OF) reached lower values in
comparison with the OF values of the Table 1. It
means that the increasing compression of I frames
(higher QI) causes a lower bitrate (BR) and it
contributes for a lower image quality and
consequently a lower objective function value (OF).
Table 2: Mean values of the experiment 1 using the
intensification of QI parameter.
In Graph 3 (Figure 3) is shown the relationship
between BR and QI parameters. Note that in the first
five iterations BR parameter tends to oscillate in a
range of its higher values while the QI parameter is
growing in the range of its lower values, but as QI
tends to grow, BR tends to oscillate in a range of its
lower values. In sum, as the compression of I frames
increases (QI), the SMC algorithm tries to decrease
the bitrate (BR) in order to improve the quality of
the image.
The improvement of the image quality can be
seen in the Graph 3 (Figure 3) where is shown the
relationship between QI and OF. As higher the
objective function (OF) is, the higher the image
quality is, according to the SMC algorithm.
In Table 2 is shown the results of the experiment
1 where the non-biased weight of QI was intensified
in the objective function. The highlighted values in
the Table 2 are the higher and the lower values of
each parameter that were found by the SMC
algorithm.
It was observed that when the SMC increases the
QB non-biased weight, BR remains in a higher range
of values since QB keeps in constant growth. At this
time, QB is still within a smaller range of values.
When the QB switches to a higher range of values,
BR tends to oscillate in a smaller range of values. In
sum, the SMC algorithm tries to compensate the
high compression of B slice (QB) by a lower bitrate
(BR) and vice versa. This contributes for a better
image quality.
The experiment 1 done with the FR, QP and PF
parameters confirmed the conclusions obtained in
the experiments with BR, QI and QB parameters that
AHybridAlgorithmusingMetaheuristicsAppliedtoH.264/AVCVideoEncoder
215

were described in this section.
4.2 Results of the Second Experiment
In graph 1 (Figure 4), that shows the results obtained
in the experiment 2, it is observed the behavior of
the QI, QP, QB, and PF parameters when the BR is
not considered in the objective function.
The parameters kept the same behavior of the
experiment 1. However, the higher and the lower
values of the QI, QP and QB parameters that are
shown in Table 3, were increased if compared with
the values of the Table 1 (experiment 1).
The range of the values of QI, QP, and QP
increased because the BR parameter was not
considered (its weight have been set to zero) in the
objective function of the experiment 2. In sum, the
BR parameter influences the limits of these
parameters (Kim et al., 2005).
In graph 2 (Figure 4), it is shown the curves of
OF and FR parameters in the experiment 2. The OF
parameter tends to fluctuate until the iteration 7.
After this interaction it stabilizes in lower values
than the values obtained in the experiment 1, which
means a lower quality image.
Figure 4: Graphs 1 and 2 of the Experiment 2 using BR
parameter.
The point is that BR is an important parameter in
the objective function. When the BR parameter is
considered in this function (experiment 1), it tends to
increase the OF value and contributes for a better
image quality. When the BR parameter is not
considered in the OF (Experiment 2), the
quantization parameters (QI, QP, QB) assume higher
values that contribute for a worse image quality.
In Graph 2 (Figure 4) was observed that FR
tends to fluctuate because BR was not considered in
the objective function. The BR directly influences
the FR parameter and when the BR is not considered
the FR fluctuates.
Table 3: Mean values of the experiment 2 not considering
the BR parameter.
In this experiment the standard deviations of the
FR and PF mean values reached a low standard
deviation while the QI, QP and QB reached a higher
value. These standard deviations reflected in the
objective function standard deviation.
The experiment 2 done with the FR, QP, QB, QI
and PF parameters confirmed the conclusions
obtained in the experiments with BR that was
described in this section.
5 CONCLUSIONS
It was presented a new hybrid algorithm called
Simulator of Metaheuristics applied to a CODEC
(SMC). The SMC uses two metaheuristics: Tabu
Search and Genetic Algorithm. It is used to identify
the best configuration to a CODEC H.264 for the
Brazilian Digital Television System (BDTS) by
using six parameters: BR, FR, QI, QP, QB and PF.
The model proposed in this paper was accurate. The
parameters behavior was according to the literature.
The experiments proved that the SMC algorithm
tries to improve the H.264 configuration through the
best combination of the six studied parameters. The
SMC algorithm proved to be robust and reliable. The
SMC algorithm takes just four minutes to have 300
executions. It was developed in ANSI C language
and it was run in an Intel Core 2 Duo processor.
REFERENCES
ABNT - Associação Brasileira de Normas Técnicas - NBR
15602-1, 2008. Digital Terrestrial Television – video
coding, audio coding and multiplexing. Part 1: video
coding. Rio de Janeiro.
ICINCO2012-9thInternationalConferenceonInformaticsinControl,AutomationandRobotics
216

ABNT - Associação Brasileira de Normas Técnicas. NBR
15601, 2008. Televisão digital terrestre – Sistema de
transmissão. Rio de Janeiro.
Cermak, G., Pinson M. and Wolf, S., 2011. The
relationship among video quality, screen resolution,
and bit rate. IEEE transactions on broadcasting, v.
30(2).
Glover, F., Kelly, P. J., Laguna, M., 1995. Genetic
algorithm and Tabu Search: Hybrids for optimization.
Computers Ops Res., vol. 22, no. 1, pp. 111−134.
Golston, J., Rao, A., 2006. Video codecs tutorial: Trade-
offs with H.264, VC-1 and other advanced codecs.
Embedded Systems Conference Silicon Valley.
Gonçalves, J. F., Resende, M. G. C., 2004. An
evolutionary algorithm for manufacturing cell
formation. Computers & Industrial Engineering, v. 47,
pp. 247-243.
Huang, Y.–W et al., 2006. Analysis and complexity
reduction of multiple reference frames motion
estimation in H.264/AVC. IEEE Transactions on
Circuits and Systems for Video Technology, v. 16
ITU-R, 2002. Recommendation ITU-R BT.500-11:
Methodology for the subjective assessment of the
quality of television pictures. Swiss.
ITU-T, 2007. ITU-T Recommendation H.264. Advanced
video coding for generic audiovisual services.
Kim, C., Shih, H.-H.,Kuo, C.-C. J., 2006. Fast H.264 intra
prediction mode selection using joint spatial and
transform domain features. Journal of Visual
Communication & Image Representation, v. 17, p.
291-310.
Koumaras, H., Kourtis, A., Martakos, D., 2005.
Evaluation of video quality based on objectively
estimated metric. Journal of Communications and
Networks, Vol.7, No.3
Malvar, H.S., Hallapuro, A., Karczewicz, M., Kerofsky,
L., 2003. Low-complexity transform and quantization
in H.264/AVC. IEEE transaction on Circuits and
Systems for video technology, vol. 13, pp. 598−603.
Moriyoshi, T., Shinohara, H., Miyakazi, T., Kuroda, I.,
2000. Real-time software video codec with a fast
adaptive motion vector search. Journal of VLSI Signal
Processing, vol. 29, pp. 239−245.
Nemethova, O., Ries, M., Siffel, E., Rupp, M.,2004.
Quality Assessment for H.264 Coded Low-Rate and
low-Resolution Video Sequences. Proc. of Conf. on
Internet and Inf. Technologies (CIIT), St. Thomas, US
Virgin Islands, pp. 136-140
Papadimitriou, C. H. Computational Complexity, 1994.
Reading: Addison-Wesley.
Pinson, H. M., Wolf, S., 2003. A new standardized
method for objectively measuring video quality.
Institute for Telecommunication Science, Boulder CO
80305 USA.
Ries, M., Nemethova, O. and Rupp, M., 2007.
Performance evaluation of video quality estimators. In
Proc. European Signal Processing Conference
EUSIPCO, pp. 159-163.
Ries, M., Nemethova, O., Rupp, M., 2005. Reference-Free
Video Quality Metric for Mobile Streaming
Applications. In Proc. of the DSPCS 05 & WITSP 05,
pp. 98-103, Sunshine Coast, Australia.
Seshadrinathan, K., Soundararajan, R., Bovik, A. C.,
Cormack, L. K., 2009. Study of subjective and
objective quality assessment of video. IEEE
Transaction of Image Processing.
Sikora, T., 2005. Trend and Perspectives in Image and
Video Coding. Proceedings of IEEE, vol. 93.
Unisinos – University of the Sinos Valley, 2008.
Especificação de Software CODEC – Codificação e
Decodificação de Sinais Fonte.
Winkler, S.; Mohandas, P., 2008. The evolution of video
quality measurement: from PSNR to hybrid metrics.
IEEE Transactions on Broadcasting, pp. 660 – 668.
Yasakethu , S. L. P., Fernando, W. A. C., Adedoyin , S.,
and Kondoz , A., 2008. A rate control technique for
off line H.264/AVC video coding using subjective
quality of video. IEEE Trans. Consumer Electronics,
vol. 54, no. 3, pp. 1465−1473
AHybridAlgorithmusingMetaheuristicsAppliedtoH.264/AVCVideoEncoder
217