A New Face Beauty Prediction Model based on Blocked LBP
Guangming Lu
1
, Xihua Xiao
1
and Fangmei Chen
2
1
Shenzhen Graduate School, Harbin Institute of Technology, Shenzhen, China
2
Department of Electronic Engineering, Tsinghua University, Beijing, China
Keywords: Face Beauty, ASMs, Texture Feature, Blocked-LBP.
Abstract: In recent years, many scholars use machine learning methods to analyze facial beauty and achieve some
good results, but there are still some problems needed to be considered, for instance, the face beauty
degrees are not widely distributed, and previous works emphasized more on face geometry features, rather
than texture features. This paper proposes a novel face beauty prediction model based on Blocked Local
Binary Patterns (BLBP). First, we obtain the face area by ASMs model, then, the BLBP algorithm is
proposed in accordance with texture features. Finally, we use Pearson correlation coefficient between the
output of the facial beauty by our algorithm and subjective judgments by the raters for evaluation.
Experimental results show that the method can predict the beauty of face images automatically and
effectively.
1 INTRODUCTION
Pursuing beauty is the nature of human beings,
especially in terms of facial beauty. In ancient
China, “three court five eyes” was considered as an
evaluation criterion about facial beauty. The
Western society also has a “golden ratio” evaluation
criterion. Currently, more and more people pursue
beauty, but they are confused on what kind of
photos can attract more people’s attention when
they post photos on the social network. How to
define the beautiful faces and how to beautify their
face images? Is there a method which can give a
reliable beauty index about their photograph? How
to evaluate the plastic surgery results? In the beauty
pageant, participants evaluate face beauty according
to their own tastes, which is often not convincing to
the public. Can we use computer technology for the
pageant?
In recent years, with the development of
computer technology, some scholars began to use
computer-related technology to analyze facial beauty
(Eisenthal et al., 2006; Kagian et al., 2008). They try
to find the common properties of facial beauty and
provide a quantitative evaluation. Aarabi (Aarabi et
al., 2001) established an automatic scoring system
for face beauty. They defined the face beauty in three
levels, and chose 40 face images for training and
other 40 face images for testing. They got the final
classification accuracy of 91% by using
nearest-neighbors (KNN). Irem (Irem et al., 2007)
proposed a two-levels (beauty or not) model based
on a training dataset with 150 female faces, in which
the principal component analysis (PCA) and support
vector machine (SVM) methods were used for
feature extraction and classification, respectively.
Finally, the highest accuracy of 89% was achieved
by using 170 female face images as the testing data.
Gunes (Gunes and Piccardi, 2006) proposed a
method based on supervised learning, in which 11
features were involved to describe the face beauty
degree. The 17 “golden ratios” rules for face beauty
was given by Schmid (Schmid et al., 2008), but it
only used some geometric features to describe the
face beauty. Douglas (Douglas et al., 2010)
contributed a method of quantifying and predicting
female facial attractiveness using an automatically
learned appearance model which did not require
landmark features. Zhang (Zhang et al., 2011)
mapped faces on to a human face shape space, and
then quantitatively analyzed the effect of facial
geometric features on human facial beauty. The
experiments showed that human face shapes lay in a
very compact region of the geometric feature space
and that female and male average face shapes were
very similar. Mao (Mao et al., 2011) proposed a
computational method for estimating facial
activeness based on Gabor features and SVM,
experimental results showed that the FeaturePoint
Gabor features performed best and obtained the
Lu, G., Xiao, X. and Chen, F.
A New Face Beauty Prediction Model based on Blocked LBP.
DOI: 10.5220/0005670500870092
In Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016) - Volume 4: VISAPP, pages 87-92
ISBN: 978-989-758-175-5
Copyright
c
2016 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
87
correlation of 0.93 with average human ratings, but
there were only 100 Chinese female faces in their
database, the database was not big enough. Gan (Gan
et al., 2014) utilized deep self-taught learning to
learn the concept of facial beauty and produce
human-like predictors, but its processing was
relatively complex.
In summary, previous works have used image
processing and machine learning techniques to
classify face beauty, and they have achieved initial
success on some different data sets. However, there
are still some problems: 1) the number of face
images for experiment is inadequate. 2) Face beauty
degrees are not widely distributed. 3) Previous works
emphasized more on face geometry features, rather
than texture features.
This paper aims to predict the beauty for female
faces by using the texture features. First, the active
shape models (ASMs) is used for facial landmark
extraction, and 77 landmarks are extracted to
represent the face shape. Then, a new method,
named Blocked-LBP (BLBP), is proposed based on
the rotation invariance of Local Binary Patterns, and
Histogram Matching method is employed to analyze
face beauty. The BLBP has some advantages: 1) it
can generate more samples based on our current
samples. 2) In aesthetic research of human face, we
mainly use geometric features or global texture
features to analyze the beauty of human face. But by
experiments, we find that it is more reliable to
predict the face beauty by using the local features of
face skin. 3) It defines a new scoring model which is
different from the usual scoring model. Experiments
show that BLBP is effective for face beauty analysis,
which gets a Pearson correlation of 0.874 in our
database and 0.852 in the database of (Zhang et al.,
2011).
The paper is organized as follows: Section 2
describes the preprocessing procedure of obtaining
the face area. In Section 3, face texture feature
extraction method based on the BLBP algorithm is
introduced. Experimental results are shown in
Section 4, and the conclusion is given in Section 5.
2 FACE AREA EXTRACTION
Landmark extraction is important in face area
extraction. Landmarks refer to the locations of key
points of nose, mouth, eyebrows, eyes, and face
contours in a face image. In this paper, the active
shape models (ASMs) algorithm (Cootes et al.,
1995, Sukno et al., 2007) is used for landmark
extraction. ASMs are statistical models of the shape
of objects which deform iteratively to fit to an
example of the object in a new image (Cui et al.,
2012). Using this model, 77 landmarks are extracted
to represent the face shape. Figure 1(a) is an input
image, and Figure 1(b) shows the extracted
landmarks.
There is usually some background information
which would impact the texture feature in a face
image. We find a way to segment the face region
out so as to eliminate this impact. The main steps
are as follows:
(1) Use the ASMs method to get the landmarks, as
shown in Figure 1(b).
(2) Find the location of face contour points
,
including the points from No. 0 to No. 15 in
Figure 1(b).
(3) Use the face contour points to get a mask image,
whose size should be the same with the face
image.
(4) Segment the face image based on the mask
image, so the hair and the background can be
removed. Figure 2 shows the result.
(a)Original image (b) Extracted landmarks
Figure 1: Landmark extraction by using ASMs.
Figure 2: The process to locate the face region.
3 BLOCKED-LBP
3.1 Texture Features of LBP
The basic idea of LBP (Local Binary Patterns)
VISAPP 2016 - International Conference on Computer Vision Theory and Applications
88
which can learn by Guo (Guo Z et al., 2010) is to
summarize the local structure in an image by
comparing each pixel with its neighborhoods. For
each pixel
,
, the LBP value is calculated as
follows:
(1) Compare the intensity of 3*3 neighbor pixels
of
,
with the intensity of
,
. If
the intensity of any neighbor pixel is smaller
than the intensity of
,
, then, denote
this neighbor pixel as 0, otherwise 1. This
procedure is shown as Figure 3. We can end
up with a binary number for each pixel, like
11001111.
Figure 3: The process for calculating a LBP.
(2) Calculate the LBP value of
,
with the
following equation :
,
2
(1)
Where
is the grey value of
,
,
is the
gray value of the
neighbor of
,
.In the
original LBP algorithm, P is set as 8. The function
of is a symbolic function:
1 0
0
(2)
Many researchers have proposed some extensions
about the original LBP, such as the rotation
invariance LBP which is based on a circular domain.
Figure 4 is a circular LBP operator.
Figure 4: A model for circular LBP operator.
For a point
,
, and its neighboring point
,
can be calculated by:
2
2
(3)
where is the radius of the circular. If
,
is
not on the image coordinates, its interpolation point
can be calculated:
,
1
0,0
0,1
1,0
1,1
1
(4)
3.2 BLBP
In this paper, a Blocked-LBP (BLBP) is proposed
based on the rotation invariance LBP (LBPROT) to
extract the texture features. It is different from the
block LBP for face recognition which is shown in
Figure 5, and the detail explaining is given in (Jia et
al., 2014).
Figure 5: The flowchart of the block LBP for face
recognition.
A face image will get low beauty score if there
are some bad features on the face, such as scars.
Accordingly, we use local features instead of global
features and propose the BLBP descriptor, which
shown in Figure 6. The detailed description about
the BLBP method is as following.
(1) Divide each LBP image which comes from the
training set into smaller blocks. (Here the face
image is divided into 12 blocks)
(2) During the testing procedure, the first step is to
obtain the face region of the test images by the
method introduced in Section 3.1. Then, the
rotation invariance LBP of the image is
calculated. Finally, the images are divided into
A New Face Beauty Prediction Model based on Blocked LBP
89
blocks. For each block, find the most similar
block by Histogram Matching. The beauty score
of the training sub-block is set to the human
rated score of the corresponding face image.
When testing the
sub-block’s score, we find
the most similar block according to each training
sample’s
sub-block and set its score to the
testing sub-block.
(3) The sum of each sub-block’s score is treated as
the testing image’s final score.
Figure 6: The flowchart of BLBP extraction.
Let
denote the
training image where
i=1,2,3…M. M is the number of training images,
,
∈
denotes a vector which is a LBP image
histogram for the
block of
, where j =
1,2,3,…N, and N is the number of block. Let
∈
∗
denote a histogram matrix and
,
,
,
,
,
,…
,
(5)
Let
denote the
testing image,
,
∈
denotes a histogram vector of the
block of
,
and
denotes the
mean human rater of
training image, the testing score of
,
can be
calculated by the following steps:
1) Calculating the distances between
,
and
each row of
:
1
,
∗
,
∑
,
∗
∑
,
(6)
2) Finding the min distance of
:
(7)
3) The testing score for the
block of
is:
(8)
4) The total score of
is:
(9)
Finally,
is used for evaluation.
Comparing the block LBP for face recognition
model with BLBP model, there were some
differences: BLBP model can generate more
samples than block LBP, as shown in Figure 5.
For
facial beauty prediction,
the score of the test
sample can be determined by the number of training
samples in BLBP model, but the score of the test
sample is determined by one sample in block LBP
for face recognition model.
4 EXPERIMENTS
A beauty face data base has been established for
experiments, which contains 400 high-quality female
face images, including some well-known beautiful
faces collected from some webs (e.g. Miss World,
movie stars, and super models), several existing
databases (the Shanghai database of (Zhang et al.,
2011)) and some profile pictures. The size of the face
images is 480∗600. The face images are confined
to be frontal and have neutral or gentle smile
expressions.
To obtain the human-rated beauty scores, an
annotation interface was developed, which displayed
the face images in random order and asked the raters
to give a score for each face image. The scores are
integer from 1 to 10, where “10” means the most
beautiful face, and “1” means the ugliest face. Nine
volunteers attended the annotation task for all the
images. Therefore, we obtained 9 annotations for
each image. The average beauty ratings are
considered as the human-rated beauty scores. Then,
the 10 folds cross-validation technique was chosen
as the training and testing method. Some face images
are shown in Figure 7. Finally, the Pearson
correlation is used for model evaluation (Pearson,
1920; Rodgers and Nicewander, 1988).
VISAPP 2016 - International Conference on Computer Vision Theory and Applications
90
Figure 7: Some examples in the image gallery.
Firstly, the optimal block number of BLBP
should be determined. We change this parameter
from 1*1, 2*2… to 10*10, and the corresponding
results are shown in Figure 8. It shows that the best
block number is 8*8, and the highest Pearson
correlation is 0.874 under this parameter. The face
regions need not to normalize to the same size, for
example, let ∗ be the size of the face region,
where , may be different for different images.
When the block number is ∗, then each block
size should be
/
∗
/
.
Figure 8: Comparing with different block numbers in
BLBP.
Table 1: The results by using the texture feature to
analysis face beauty in different database.
Database
Feature
Our database
(Zhang, 2011)
database
LBP 0.656 0.588
(Jia et al., 2014) 0.744 0.727
Gabor 0.693 0.678
BGabor 0.739 0.704
(Mao et al., 2011) 0.853 0.837
BLBP 0.874 0.852
Comparing with LBP, Gabor, Blocked-Gabor
(BGabor), and LBP (Jia et al., 2014) which was the
block LBP for face recognition model, the proposed
BLBP method has better performance in face beauty
analysis. The results are listed in Table 1. BLBP can
get a correlation coefficient of 0.874 in our database.
Testing these methods on the database used in
(Zhang et al., 2011), the proposed BLBP also gets
the highest correlation coefficient of 0.852.
5 CONCLUSIONS
This paper proposes a novel Face Beauty Prediction
Model Based on BLBP. First, we extract the face
area by ASMs model and obtain 77 landmark points
to segment the face area. Then, a novel Blocked
Local Binary Patterns (BLBP) algorithm is
proposed in accordance with texture features.
Finally, we use Pearson correlation coefficient
between the output of the facial beauty by our
algorithm and subjective judgments by the rater for
evaluation. Experimental results show that the
method can predict the face images automatically
and effectively, and obtain a correlation coefficient
of 0.874 in our database.
ACKNOWLEDGEMENTS
The work is supported by the NSFC funds under
Contract No. 61271344, Shenzhen Fundamental
Research Fund No. JCYJ20140508160910917, and
JCYJ20150403161923528, China.
REFERENCES
Eisenthal Y., Dror G., Ruppin E., 2006. Face
Attractiveness: beauty and the machine, Neural
Computation, pp. 110-150.
Kagian A., Dror G., LeyVand T., Cohen-Or D., Ruppin E.,
2008. A humanlike predictor of face attractiveness,
Advance Neural Information Processing Systems, pp.
676-683.
Aarabi P., Hughes D., Mohajer K., et al., 2001. The
automatic measurement of face beauty, Proc of the
IEEE International Conference on System Man and
Cybernetics, pp. 2644-2647.
Irem H., Turkmen Z., Kurt M., et al., 2007. Global feature
based female face beauty decision system,
Proceedings of the 15th European Signal Processing
Conference, Lausanne, Switzerland, pp. 1945-1949.
Gunes H., Piccardi M., 2006. Assessing face beauty
through proportion analysis by image processing and
supervised learning, International Journal of
A New Face Beauty Prediction Model based on Blocked LBP
91
Human-Computer Studies, pp. 1184-1199.
Schmid K., Marx D., and Samal A., 2008. Computation of
a face attractive- ness index based on neoclassical
canons, symmetry, and golden ratios, Pattern
Recognition, pp. 2710-2717.
Douglas G., Kai Y., Wei X., Gong Y., 2010. Predicting
face beauty without landmarks, Computer
Vision-ECCV 2010 Lecture Notes in Computer
Science, pp. 434-447.
Zhang D., Zhao Q., Chen F., 2011. Quantitative analysis
of human face beauty using geometric features,
Pattern Recognition, pp. 940-950.
Mao H., Chen Y., Jin L. and Du M., 2011. Evaluating face
attractiveness: a gabor feature approach, Journal of
Communication and Computer, pp. 674-679.
Gan J., Li L., Zhai Y., et al., 2014. Deep self-taught
learning for facial beauty prediction. Neurocomputing,
2014, pp. 295-303.
Cootes T., Taylor C., et al., 1995. Active shape
models---their training and application, Computer
Vision and Image Understanding, pp. 38-59.
Sukno F., Ordas S., Butakoff C., Cruz S., Frangi A., 2007.
Active shape models with invariant optimal features:
application to facial analysis, IEEE Transactions on
Pattern Analysis and Machine Intelligence, pp.
1105-1117.
Cui Y., Jin Z., Yang W., 2012. A two steps face alignment
approach using statistical models, International
Journal of Advanced Robotic System, pp. 1-6.
Guo Z., Zhang L., Zhang D., 2010. Rotation invariant
texture classification using LBP variance (LBPV)
with global matching, Pattern Recognition, pp.
706-719.
Jia M. Zhang Z., Song P., Du J., 2014. Research of
Improved Algorithm Based on LBP for Face
Recognition. Biometric Recognition Lecture Notes in
Computer Science, pp. 111-119.
Pearson K., 1920. Notes on the history of correlation
royal society. Proceedings Biometrika, pp. 25-45.
Rodgers J., Nicewander W., 1988. Thirteen ways to look
at the correlation coefficient, The American
Statistician, pp. 59-66.
VISAPP 2016 - International Conference on Computer Vision Theory and Applications
92
