RELEVANCE FEEDBACK WITH MAX-MIN POSTERIOR
PSEUDO-PROBABILITY FOR IMAGE RETRIEVAL
Yuan Deng, Xiabi Liu
*
and Yunde Jia
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, P. R. China
Keywords: Content-based image retrieval, relevance feedback, discriminative training, Gaussian mixture models, max-
min posterior pseudo-probabilities.
Abstract: This paper proposes a new relevance feedback method for image retrieval based on max-min posterior
pseudo-probabilities (MMP) framework. We assume that the feature vectors extracted from the relevant
images be of the distribution of Gaussian mixture model (GMM). The corresponding posterior pseudo-
probability function is used to classify images into two categories: relevant to the user intention and
irrelevant. The images relevant to the user intention are returned as the retrieval results which are then
labelled as true of false by the user. We further apply MMP training criterion to update the parameter set of
the posterior pseudo-probability function from the labelled retrieval results. Subsequently, new retrieval
results are returned. Our method of relevance feedback was tested on Corel database and the experimental
results show the effectiveness of the proposed method.
1 INTRODUCTION
One of the main problems in content-based image
retrieval (CBIR) is how to bridge the ‘semantic gap’
(Zhou and Huang, 2003). Relevance feedback (RF)
technique was introduced to CBIR in mid 1990s,
with the intention to bring user in the retrieval loop
to reduce the gap and improve retrieval performance
(Liu et al., 2007).
Currently, some researchers have regarded
image retrieval as a supervised learning problem and
some machine learning methods have been
combined with RF to improve retrieval performance.
Since many RF methods treat each image as a whole
while the user only concerns a few parts of the
image, some researchers transformed CBIR into a
Multiple Instance Learning (MIL) problem to find
those regions in which the user was interested (Chen
et al., 2006). Support Vector Machine (SVM) itself
has some drawbacks such as unstable for small-sized
training set, biased optimal hyper plane for
imbalanced sample set, overfitting, etc. To overcome
those problems, some other methods have been
applied, such as integrating bagging and random
subspace (Tao et al., 2006), active learning (Cheng
and Wang, 2007), Boosting (Yu et al., 2007), Biased
Minimax Probability Machine (BMPM) (Peng and
King, 2006), etc. Besides, due to the user’s
subjectivity and strict binary classification problem,
fuzzy SVM (Rao et al., 2006) and Bayesian learning
(Zhang and Zhang, 2006) were proposed to reduce
misclassification and refine retrieval precision.
Furthermore, Bayesian classifier combining with
incremental learning was adapted to realize long
term feedback (Goldmann et al., 2006).
In this paper, a new relevance feedback method
for image retrieval based on max-min posterior
pseudo-probabilities (MMP) (Liu et al., 2006) is
proposed to learn user’s intention during feedback.
We assume that the feature vectors extracted from
the relevant images should be of the distribution of
Gaussian mixture model (GMM). The posterior
pseudo-probability function for the relevant images
is used as user intention model. According to the
posterior pseudo-probabilities, the images in the
database are classified into two categories: relevant
to the user intention and irrelevant. The optimum
parameter set of user intention model is learned from
relevant and irrelevant images that user labelled
during feedback using MMP criterion. Then the
model obtained after learning is utilized to classify
all images and return new results to user.
Experiments on 5,000 Corel images show the
effectiveness of our proposed method for improving
retrieval performance.
* Corresponding Author
286
Deng Y., Liu X. and Jia Y. (2008).
RELEVANCE FEEDBACK WITH MAX-MIN POSTERIOR PSEUDO-PROBABILITY FOR IMAGE RETRIEVAL.
In Proceedings of the Third International Conference on Computer Vision Theory and Applications, pages 286-289
DOI: 10.5220/0001079602860289
Copyright
c
 SciTePress

2 IMAGE CLASSIFICATION
ACCORDING TO USER’S
INTENTION
2.1 Statistical Modelling of User’s
Intention
Each image is represented as an 80-D feature vector
which consists of 9-D color moments and 71-D
Gabor based texture features.
Relevant images that user labelled during
feedback can reflect user’s intention; therefore we
can use them to describe user’s intention. We
assume that the feature vectors extracted from the
relevant images should be of the distribution of
Gaussian mixture model. Let
X be the feature
vector of the image, and
ω
be the relevant image.
Let
K
be the number of Gaussian components,
k
w ,
k
μ
and
k
Σ
be the weight, the mean, and the
covariance matrix of the
K
-th Gaussian component
respectively,
1
1
=
∑
=
K
k
k
w
,
(
)
ω xp
be the class-
conditional probability density function for
X
:
() ()
∑
=
Σ=
K
k
kkk
xNwxP
1
, μω
(1)
where
()
() ()()
⎟
⎠
⎞
⎜
⎝
⎛
−−−=
−
−
−
kk
T
kkkk
N μxΣμxΣΣμx
1
2
1
40
2
1
exp2,
π
(2)
k
Σ
is further assumed to be diagonal for simplicity:
[]
80
1=
=
j
kj
σ
k
Σ
.
2.2 Image Classification using
Posterior Pseudo-Probability
Posterior class probabilities are generally used to
realize classification in classical Bayesian
classifiers. Because it is not practicable to collect the
representative examples of irrelevant images,
posterior class probability is not adequate for the
classification problem discussed here. We use
posterior pseudo-probability to approximate
(
)
x ω P
by embedding
(
)
ω xp
in a smooth, monotonically
increasing function which takes value in
]1 ,0[
:
(
)
(
)
(
)
(
)
(
)
ωλωω xxx ppfP −−=≈ exp1
(3)
where
λ
is a positive number.
For more details of posterior pseudo-probability,
please refer to (Liu et al., 2006).
By substituting Eq. 1 into Eq. 3, we get user
intention model:
)),(exp(1);(
1
∑
=
−−=
K
k
kkk
Nwf ΣxΛX μλ
(4)
where
Λ
denotes the unknown parameter set:
Kkw
kkk
,,1},,,,{ "=
=
ΣμΛ
λ
(5)
We can use the posterior pseudo-probability
function to classify the images into two categories:
relevant and irrelevant. We compute the values of
the posterior pseudo-probabilities for all images in
the database using Eq. 4 and sort those images in
descending order according to their posterior
pseudo-probabilities. Then top
N images are
returned as the retrieval results.
3 MMP LEARNING OF USER’S
INTENTION
In order to classify images using Eq. 4, the unknown
parameter set Λ must be determined. We use MMP
criterion to learn those parameters from the training
data. We collect the relevant and irrelevant images
that user labelled as positive and negative examples
respectively. MMP method is introduced below
briefly.
The main idea of MMP method is to maximize
the class separability by producing the posterior
pseudo-probability function of each class to
maximize the posterior probabilities for its positive
examples, at the same time to minimize those for its
negative examples. Let
i
x
ˆ
and
i
x be the feature
vector of the
i
–th positive and negative example of
the user intention model respectively. Let
m
and
n
be the number of positive and negative examples of
user intention model respectively. Then the objective
function of the MMP learning for user intention
model is designed as:
()
() []
()
[]
∑∑
==
+
+−
+
=
n
i
i
m
i
i
f
nm
m
f
nm
n
F
1
2
1
2
;1;
ˆ
ΛxΛxΛ
(6)
It is obvious that
(
)
0=ΛF
means the hundred-
percent class separability: the less the value of
(
)
ΛF
is, the more class separability is. Consequently, we
can obtain the optimum parameter set
∗
Λ
of user
intention model by minimizing
()
ΛF
:
RELEVANCE FEEDBACK WITH MAX-MIN POSTERIOR PSEUDO-PROBABILITY FOR IMAGE RETRIEVAL
287

() ΛΛ
Λ
Fminarg=
∗
(7)
The optimum parameter set of user intention
model is updated iteratively through the gradient
descent method until convergence or a prefixed
maximum number of iteration is reached.
For more details of the MMP criterion, please
also refer to (Liu et al., 2006).
4 EXPERIMENTS AND
DISCUSSIONS
Relevance feedback experiments for querying by
concept and querying by example (QBE) on 5,000
Corel images were taken to evaluate our proposed
method. Those images are divided into 50
categories, such as African people, beach, buildings,
etc. Each category includes 100 images. We also
compared our method with other approaches.
4.1 Relevance Feedback Experiment
for Querying by Concept
In this experiment, concept refers to the “name” of
image category. Therefore relevant images are those
images that belong to the specified image category
that user query. We assumed that the feature vectors
extracted from images with the same image category
be of the distribution of Gaussian mixture model. 50
concept models were trained with 2,500 images (50
images each category). Concept retrieval experiment
was performed on the remaining 2,500 images. After
user input the concept, we computed the posterior
pseudo-probabilities of the corresponding concept
model for 2,500 images. Then those images were
sorted in descending order according to the value of
the posterior pseudo-probability functions and the
top 50 images were returned as the results. Please
refer to (Deng et al., 2007) for more details about
querying by concept. During feedback, top 50
images were labelled automatically as relevant to the
concept or irrelevant and then used as the training
data for user intention model to obtain the optimum
parameters set using MMP criterion. Then 2,500
images were classified according to user intention
model after learning.
P20 and P50 were used as the performance
measure. Table 1 shows the experiment data.
Table 1: Average precision for top 20 and top 50 images.
Iteration times P20 P50
#0 0.4220 0.3280
#1 0.5770 0.3752
#2 0.6120 0.4024
#3 0.6350 0.4052
#4 0.6550 0.4144
#5 0.6720 0.4288
Gosselin and Cord proposed a retrieval method
which combined transductive SVM with active
learning strategy (Gosselin and Cord, 2004). Their
retrieval experiment was performed on 11 Corel
image categories, nine iterations were carried out
and 20 images were labelled each time. We did
similar experiment on 50 Corel image categories.
Table 2 shows the experiment data between our
method and Gosselin and Cord’s method, which are
denoted as MMP and RETINAL respectively.
Table 2: Average precision for top 20 images after nine
iterations in two methods.
P20 #9
RETINAL 0.61
MMP 0.8790
4.2 Relevance Feedback Experiment
for Querying by Example
This experiment was designed to find images that
were similar to the query image. Two images with
the same image category are similar; therefore
relevant images are those images from the same
image category as the query image. We assumed that
the difference between feature vectors of two images
from the same category be of the distribution of
Gaussian mixture model. We randomly chose 20
images from each image category, or, 1,000 images
in all, to train similarity model. QBE experiment
was performed on the remaining 4,000 images. After
the user input the query image, the system computed
the posterior pseudo-probabilities for the query
image and the target image in the database. Then
those target images were sorted in descending order
according to the value of the posterior pseudo-
probability functions and the top 80 images were
returned as the results. During feedback, top 80
images were labelled automatically as relevant to the
query image or irrelevant and then used as the
training set for user intention model to obtain the
optimum parameters set using MMP criterion. Then
4,000 images were classified according to user
intention model after learning. Table 3 shows the
experiment data.
VISAPP 2008 - International Conference on Computer Vision Theory and Applications
288

Table 3: Average precision for top 20 and top 50 images.
Iteration times P20 P50
#0 0.4982 0.3878
#1 0.6026 0.4570
#2 0.6446 0.4853
#3 0.6694 0.5013
#4 0.6894 0.5107
#5 0.7028 0.5192
Rao et al. proposed a querying by example
method based on Fuzzy SVM and performed
experiment on 2,000 Corel images (Rao et al., 2006).
Top 20 images were labelled each time. We did
similar QBE experiment on 5,000 Corel images.
Table 4 shows the experiment results between two
methods at three iteration steps (#1, #5, and #10),
Rao et al.’s method is denoted as Fuzzy SVM.
Table 4: Average precision for top 20 images in two
methods at three iteration steps.
P20 #1 #5 #10
Fuzzy SVM About 0.53 About 0.74 About 0.77
MMP 0.5730 0.6681 0.7172
5 CONCLUSIONS
In this paper, we have proposed a new relevance
feedback method based on max-min posterior
pseudo-probabilities framework for learning pattern
classification. We assume that the feature vectors
extracted from the relevant images be of the
distribution of Gaussian mixture model. The
corresponding posterior pseudo-probability function
is used to determine whether the image is relevant to
the user intention. In each feedback process, those
images relevant to the user intention are returned as
the retrieval results and then labeled as true or false
by the user. According to labeled retrieval results,
MMP training criterion is used to update the
parameter set of posterior pseudo-probability
function and subsequent retrieval results. We
conducted concept retrieval and example retrieval
experiments of relevance feedback on Corel
database. After five iterations, P20 has been raised
from 42.20% to 67.20% and from 49.82% to 70.28%
respectively.
ACKNOWLEDGEMENTS
This research was partially supported by the 973
Program of China (2006CB303105) and BIT
Excellent Young Scholars Research Fund
(2006Y1202).
REFERENCES
Chen, S.C., Rubin, S.H., Shyu, M.L., Zhang, C., 2006. A
dynamic user concept pattern learning framework for
content-based image retrieval. IEEE Transactions on
Systems, Man and Cybernetics Part C: Applications
and Reviews, 36(6): 772-783.
Cheng, J., Wang, K., 2007. Active learning for image
retrieval with Co-SVM. Pattern Recognition, 40(1):
330-334.
Deng, Y., Liu, X., Jia, Y., 2007. Learning Semantic
Concepts for Image Retrieval Using the Max-min
Posterior Pseudo-Probabilities Method. In ICME 2007,
IEEE International Conference on Multimedia and
Expo.
Goldmann, L., Thiele, L., Sikora, T., 2006. Online image
retrieval system using long term relevance feedback.
In CIVR 2006, International Conference on Image and
Video Retrieval.
Gosselin, P.H., Cord, M., 2004. RETIN AL: an active
learning strategy for image category retrieval. In ICIP
2004, International Conference on Image Processing.
Liu, X., Jia, Y., Chen, X., Deng, Y., Fu, H., 2006. Max-
Min Posterior Pseudo-Probabilities Estimation of
Posterior Class Probabilities to Maximize Class
Separability. Technical Report BIT-CS-TP-
20060002, Beijing: Beijing Institute of Technology.
(http://www.mcislab.org.cn/technicalreports/MMP.pdf)
Liu, Y., Zhang, D., Lu, G., Ma, W.Y., 2007. A Survey of
Content-Based Image Retrieval with High-level
Semantics. Pattern Recognition, 40(1): 262-282.
Peng, X., King, I., 2006. Imbalanced learning in relevance
feedback with biased minimax probability machine for
image retrieval tasks. In ICONIP 2006, International
Conference on Neural Information Processing.
Rao, Y., Mundur, P., Yesha, Y., 2006. Fuzzy SVM
Ensembles for Relevance Feedback in Image
Retrieval. In CIVR 2006, International Conference on
Image and Video Retrieval.
Tao, D., Tang, X., Li, X., Wu, X., 2006. Asymmetric
bagging and random subspace for support vector
machines-based relevance feedback in image retrieval.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 28(7): 1088-1099.
Yu, J., Lu, Y., Xu, Y., Sebe, N., Tian, Q., 2007.
Integrating relevance feedback in boosting for content-
based image retrieval. In ICASSP 2007, IEEE
International Conference on Acoustics, Speech, and
Signal Processing.
Zhang, R., Zhang, Z., 2006. BALAS: Empirical bayesian
learning in the relevance feedback for image retrieval.
Image and Vision Computing, 24(3): 211-223.
Zhou, X.S., Huang, T.S., 2003. Relevance feedback in
image retrieval: A comprehensive review. Multimedia
Systems, 8(6): 536-544.
RELEVANCE FEEDBACK WITH MAX-MIN POSTERIOR PSEUDO-PROBABILITY FOR IMAGE RETRIEVAL
289