DETECTION AND RECOVERY OF OCCLUDED FACE IMAGES
BASED ON CORRELATION BETWEEN PIXELS
Ji-eun Lee and Nojun Kwak
Department of Electrical & Computer Engineering, Ajou University, Suwon, 443-749, South Korea
Keywords: Correlation coefficient, Occluded face, Occlusion detection, Face image recovery.
Abstract: In this paper, we propose a method to detect and recover the occluded parts of face images using the
correlation between pairs of pixels. In the training stage, correlation coefficients between every pairs of
pixels are calculated using the occlusion-free training face images. Once a new face image is shown, the
occluded area is detected and recovered using correlation coefficients obtained in the training stage. We
compare the performance of the proposed method with the conventional method based on PCA. The results
show that the proposed method detects and recovers occluded area with much smaller noises than the
conventional PCA based method.
1 INTRODUCTION
The problem of detecting and recovering the
occluded parts of a face image is a very important
task to solve. There has been works on detecting and
recovering the occlusion of a face and the
representative approach is to apply PCA (principal
component analysis). These methods includes
automatic eyeglasses removal from face images (Wu,
2004), reconstruction of the occluded parts by fast
recursive PCA (Wang, 2007), application of
probabilistic PCA (XiaoFeng, 2010), image
completion (Efros, 1999; Sun, 2005; Komodakis,
2006) and so on (Lin, 2007).
The conventional PCA based methods use a
weight matrix composed of the eigenvectors of the
training data consisting of non-occluded face images
to detect and recover the occluded parts.
In this paper, we propose to use correlation
between pixels. The proposed algorithm can be
outlined as follows. Firstly, the proposed method
calculates the correlation coefficients between every
pairs of pixels using the training images. Secondly,
we estimate the pixel values of occluded parts of
each test images using the correlation coefficients.
Then we detect the occluded parts and recover them
until the difference between the reconstructed and
the occluded images are small.
The paper is organized as follows. In Section 2,
the method based on correlation between pairs of
pixels is proposed. Then we will compare the result
of the proposed method with that of the conventional
PCA based method through experiments in Section 3.
Finally, conclusions follow in Section 4.
2 THE PROPOSED METHOD
2.1 Correlation Coefficient
The basic idea of the proposed method is that the
pixel values of the occluded part can be predicted
from the pixel values of the non-occluded part which
are highly related to the ones in question.
Suppose each pixel value be a random variable.
Then the relationship between a pair of pixels can be
measured by the correlation coefficient.
The correlation coefficient ρ
between two
pixels x
and x
is defined as
ρ
=
=
(
)
[(
)
]
.
(1)
Here, C
is the covariance between two random
variables and σ
,σ
are the standard deviations of
each random variable. The μ
and μ
are the mean
values of x
and x
respectively and E[∙] is the
expectation operation. If the correlation coefficient
is close to 1 or -1, those two random variables are
much related to each other and if it is close to 0,
those two are hardly related at all.
568
Lee J. and Kwak N. (2012).
DETECTION AND RECOVERY OF OCCLUDED FACE IMAGES BASED ON CORRELATION BETWEEN PIXELS.
In Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods, pages 568-571
DOI: 10.5220/0003772805680571
Copyright
c
SciTePress
Normally, the expectation of random variables
cannot be calculated without knowing the
underlying joint distribution. However, it can be
estimated by the samples and the estimated value of
the correlation coefficient (1) using N samples can
be calculated as follows.
ρ
=
∑(
x
−x
)
x
−x
∑(
x
−x
)
∑
x
−x
(2)
Here, x
is the i
pixel value of the n
training
image and x
and x
are i
and j
sample mean of
pixel values respectively.
Figure 1 shows correlation coefficient maps of a
face. The pixel marked X is the reference pixel
(cheek and eye). Looking at the correlation
coefficient map of an eye, we can see that the pixels
in the right eye are highly related to the one in the
left eye. This shows that in addition to the local
pixels, far away pixels from the reference point can
also provide important information in estimating the
pixel value of the reference point.
2.2 Jointly Gaussian Distribution
In this part, we present a way to estimate an
unknown pixel value from the other known ones.
Assume that the pixel value of x
is not known,
while that of x
is known. We also assumed that
pixel i and j follows jointly Gaussian distribution.
Then, the pixel value of x
can be obtained using a
set of pixels S
= {j| indexs of m number of x
which are highly related to x
} that are highly related
to x
.
When
|S
|=1 and the j
pixel value is ν, the
conditional probability of x
given x
is
Px
|x
=ν=
Px
,x
=ν
Px
=ν
.
(3)
Figure 1: Correlation coefficient map (cheek and eye).
Because Px
=ν is a constant, Px
|x
=ν is
proportional to
Px
,x
=ν. With the assumption that
the pixel i and j follows jointly Gaussian distribution,
Px
,x
=ν becomes
Px
,x
=ν∝exp−
1
2
x
ν
Σ
x
ν
=exp
−
1
2
1
1−ρ
σ
x
−
σ
σ
ρ
ν
(4)
Here, Σ
is an inverse of the covariance matrix
Σ of x
and x
.
Σ
=
1
1−ρ
σ
σ
σ
−ρ
σ
σ
−ρ
σ
σ
σ
(5)
Using (4), the conditional mean and the variance
of x
are as follows:
Ex
|x
=ν=
σ
σ
ρ
ν
(6)
Varx
|x
=ν=1−ρ
σ
.
(7)
In case of
|
S
|
=2 and assuming that the value
of x
is ν and x
is ω, we can estimate the value of
x
using a weighted sum of Ex
|x
=ν and
E
[
x
|x
=ω
]
because Ex
|x
=ν,x
=ω is hard
to represent.
Ex
|x
=ν,x
=ω
≃αEx
|x
=ν+βE
[
x
|x
=ω
]
(8)
In the above equation α and β should meet the
condition α+β=1. The weight α can be regarded
as the contribution of x
in estimating the value of x
.
If the conditional variance of x
given x
decreases,
the confidence of the estimation grows and the
weight α should be increased. Therefore, the weight
is set to be proportional to 1 divided by the standard
deviation which is the square root of (7) in this paper.
For example, when
|
S
|
=2, α∝1
1−ρ
⁄
,
β∝1
1−ρ
⁄
and it is made that α+β=1.
In case of
|
S
|
>2, the weights are set in the
similar way. If x
and x
are highly correlated to each
other, we can assume high confidence resulting in a
high weight. On the contrary, if the correlation
between x
and x
are small, the corresponding
weight is set to be small. In this paper, to alleviate
computational complexity, we only make use of the
pixels that have the highest correlation coefficient in
estimating the unknown pixel value.
2.3 Detection and Recovery of
Occluded Face Images
Obtaining Correlation Coefficient: The first step
is to get the correlation coefficient for all the pixels
using the non-occluded training face images X
.
This is done by using (2). After getting the
correlation coefficient of all the pairs of pixels, for
each pixel, a list of highest correlation coefficient
are stored.
Reconstruction: Assume that we do not know the
first pixel value of the test face image which are
centerized by the mean image of training data. The
DETECTION AND RECOVERY OF OCCLUDED FACE IMAGES BASED ON CORRELATION BETWEEN PIXELS
569
pixel value can be predicted using a set of pixels
which are highly correlated with the one in question.
The method to calculate each pixel value x
with
the condition
|
S
|
=2 is shown from (6) to (8). This
is generalized to a case of
|
S
|
=m as follows:
w
=
1
1−ρ
(9)
z
=w
σ
σ
ρ
x
+⋯+w
σ
σ
ρ
x
=w
σ
σ
ρ
x
(10)
x
=x+
z
∑
w
(11)
Here, w
means weight (or confidence) of x
derived from the correlation coefficient between i
and j
pixels. The pixels x
(j=1,⋯,m) are the
pixels that have the highest correlation coefficient
with the i
pixel.
Occlusion Detection: The reconstructed face image
x
and the original test face image x are compared to
decide whether a pixel is occluded or not. This is
achieved by checking the difference between pixel
values of x and x
at the same pixel is more than a
threshold ϵ or not. This procedure is described in the
following equation:
|
x
−x
|
<ϵ ⟹ i∈O
≥ϵ ⟹ i∈O.
(12)
Here, i denotes the specific location of the pixel
and x
and x
are i-th pixel values of x and x
respectively. The set O is the set of occluded pixels
and O
is the complement of O which corresponds to
the set of non-occluded pixels.
Recovery: After detecting occluded parts from the
face image, occluded parts are filled with the
reconstructed image x
while non-occluded parts
remains the same as the original image x. The
resulting recovered image x
is as follows:
x
=
x
i
f
i∈O
x
i
f
i∈O
(13)
After the recovery stage, x is replaced with x
and the process is repeated until the difference
between x and x
becomes small enough.
3 EXPERIMENTS
3.1 BioID Data
BioID data contains 1521 gray-scale face images. It
consists of frontal face images of 23 people and the
size of each image is 100 by 100. Among 1521
images 1000 images were used for training and the
other 521 images were used for testing.
We covered each test image with a squared box
which has a random height and width on a random
position to make an occlusion like Figure 3.
3.1.1 Conventional PCA Based Method
The first row of figure 3 shows the difference of
pixel values between the original occluded image
and the recovered image. We represent the occluded
parts as value 1 and the non-occluded parts as value
0. Final recovered images can be seen in second row
of figure 3. As a result of the detection and recovery,
we can see a lot of noise in the figures.
3.1.2 The Proposed Method
The first row of figure 4 shows the detected
occluded area. In the figure, we can see some noise
around the occluded parts, but the noise far from and
within the occluded parts is not high compared to
Figure 3. The second row of figure 4 shows the
recovered image. As a result of recovery, we can see
the continuity between the occluded parts and the
non-occluded parts is enhanced resulting in
smoother images than those in Figure 3.
Figure 2: Occluded face images for BioID data.
Figure 3: Detected and recovered face images for BioID
data using the conventional PCA based method.
ICPRAM 2012 - International Conference on Pattern Recognition Applications and Methods
570
Figure 4: Detected and recovered face images for BioID
data using the proposed method.
Table 1: MAE and MSE between recovered BioID data
and non-occluded BioID data.
MAE MSE
PCA based 17052.45 788.06
Correlation based 10500.65 532.22
Table 2: Number of iterations and processing time for
BioID data.
# of iteration Processing time
Avg. Var. Avg. Var.
PCA based 3.117 1.226 0.092 0.051
Proposed 5.633 3.820 1.173 0.421
3.1.3 Numerical Comparison
The MAE and MSE are shown on the Table 1. As
we expected, the proposed method showed less error
than the conventional PCA-based method.
Both the number of iterations and the processing
time of the proposed method are more than those of
the PCA-based method in Table 2. The reason can
be attributed to the fact that the conventional PCA
based method reconstructs the image at once by
multiplying the weight matrix W, while in the
proposed method reconstruction is done pixel by
pixel.
4 CONCLUSIONS
In this paper, we proposed a new method to recover
the occluded face images using the correlation
coefficient between pairs of pixels. The simple idea
that a pixel value can be determined by the weighted
sum of other pixel values which are highly
correlated with the one in question.
The proposed method is compared with the
conventional PCA based method and it showed
better recovery performance in both qualitatively
and quantitatively. The blurring of the recovered
images is much less and the border lines between
occluded and non-occluded parts are connected well.
Moreover, the mean absolute error and the mean
squared error value of the proposed method are
smaller than the PCA-based method. However, the
proposed method was comparatively slower than the
PCA-based method and this should be enhanced in
the future work.
ACKNOWLEDGEMENTS
This work was supported by Korea Research
Foundation Grant funded by Korean Government
(KRF-2011-0005324).
REFERENCES
Wu, C., Liu, C., Shum, H., Y., Xy, Y., Q., Zhang, Z.,
2004. “Automatic eyegalsses removal from face
images”. Asian Conference on Computer Vision, IEEE
Trans. Pattern Analysis and Machine Intelligence.,
Vol. 26, no. 3, pp. 322-336.
Wang, Z., M., Tao, J., H., 2007. “Reconstruction of
partially occluded face by fast recursive PCA”.
International Conference on Computational
Intelligence and Security Workshops., pp . 304-307.
XiaoFeng, S., Min, T., Deren, C., RuoFeng. T., 2010. “A
two step method to recover occluded part of face”.
International Conference on Information Sciences and
Interaction Sciences., pp . 169-173.
Lin, D., Tang, X., 2007. “Quality-driven face occlusion
detection and recovery”. IEEE Conference of
Computer Vision and Pattern Recognition., pp. 1-7.
Efros, A., A., Leung, T., K., 1999. “Texture synthesis by
non-parametric sampling”. IEEE International
Conference on Computer Vision.
Sun, J., Yuan, L., Jia, J., Shum, H.,-Y., 2005. “Image
Completion with Structure Propagation”. SIGGRAPH.
Komodakis, N., Tziritas, G., 2006. “Image Completion
Using Global Optimization”. IEEE CVPR.
DETECTION AND RECOVERY OF OCCLUDED FACE IMAGES BASED ON CORRELATION BETWEEN PIXELS
571
