Sampled Multi-scale Color Local Binary Patterns

Yu Zhang

, Stephane Bres

and Liming Chen

Universite de Lyon, CNRS, Ecole Centrale de Lyon, LIRIS, UMR5205, F-69134, Lyon, France

LIRIS-INSA de Lyon, 20 Avenue Albert Einstein, 69621 Villeurbanne Cedex, France

Keywords:

Local Binary Pattern, Feature Extraction, Object Recognition, Patch Reorganization.

Abstract:

In this paper, we propose a novel representation, called sampled multi-scale color Local Binary Pattern (SMC-

LBP), and apply it to Visual Object Classes (VOC) Recognition. The Local Binary Pattern (LBP) has been

proven to be effective for image representation, but it is too local to be robust. Meanwhile such a design cannot

fully exploit the discriminative capacity of the features available and deal with various changes in lighting and

viewing conditions in real-world scenes. In order to address these problems, we propose SMC-LBP, which

randomly samples the neighboring pixels across different scale circles, instead of pixels from individual circu-

lar in the original LBP scheme. The proposed descriptor presents several advantages: (1) It encodes not only

single scale but also multiple scales of image patterns, and hence provides a more complete image information

than the original LBP descriptor; (2) It cooperates with color information, therefore its photometric invari-

ance property and discriminative power is enhanced. The experimental results on the PASCAL VOC 2007

image benchmark show signiﬁcant accuracy improvement by the proposed descriptor compared with both the

original LBP and other popular texture descriptors.

1 INTRODUCTION

Texture, color and local gradients features play a ma-

jor role in content-based image categorization task.

Identifying patches with texture features is at the heart

of many computer vision algorithms. It is widely ap-

plied in object category recognition and image re-

trieval application(Ozuysal et al., 2010). Identify-

ing patches is difﬁcult because of drastic surface

appearance which depends on how the image tex-

ture information is captured. To address this prob-

lem, many texture descriptors have been proposed

in the literature, such as Grey Co-occurrence Ma-

trix(GLCM)(Tuceryan and Jain, 1998), Texture Auto

Correlation(TAC)(Tuceryan and Jain, 1998), Gabor

ﬁlter(Zhang et al., 2000), Brief(Calonder et al., 2010)

and LBP(Ojala et al., 2002).

Among all these texture features, LBP is one of

the most popular texture descriptors. It was intro-

duced and used in texture classiﬁcation based on local

binary patterns and nonparametric discrimination of

sample and prototype distributions(Ojala et al., 2002).

It can be seen as a uniﬁed approach to statistical and

structural texture analysis. Fig. 1 gives an example.

The LBP descriptor encodes one pixel of an image by

thresholding the neighborhood of each pixels with the

center value. Then the threshold results are multiplied

Figure 1: Calculation of the original LBP descriptor.

with weights given by powers of two. Finally the LBP

code is obtained by summing up all the weighted re-

sults. This process is done for each pixel, and the

image representation is obtained by counting the his-

togram based on these codes. The LBP descriptor is

further extended to multi-scale using a circular neigh-

borhood with variant radius and variant number of

neighboring pixels, as shown in Fig 2.

Because of its descriptive power for analyzing

both micro and macro texture structures, and com-

putational simplicity, LBP has been widely applied

for texture classiﬁcation(Ojala et al., 2002) and object

recognization(Zhu et al., 2010), and is demonstrated

excellent results and robustness against global illumi-

nation changes. It has also been used successfully for

texture segmentation(Blas et al., 2008), recognition of

facial identity(Guo et al., 2010) and expression(Shan

et al., 2009).

303

Zhang Y., Bres S. and Chen L..

Sampled Multi-scale Color Local Binary Patterns.

DOI: 10.5220/0004282403030308

In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2013), pages 303-308

ISBN: 978-989-8565-47-1

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

(a) R=1,P=8 (b) R=2,P=16 (c) R=3,P=24

Figure 2: Single-scale LBP descriptor.

However, the original LBP descriptor also has

several drawbacks in its application. It covers a

small spatial support area, hence the bit-wise com-

parisons are made through single circular pixel val-

ues with the central of pixel. This means that the

LBP codes are easily affected by noise(Liao et al.,

2007). Moreover, features calculated in a single cir-

cular neighborhood cannot capture larger scale struc-

ture (macrostructure) that may be dominant features.

Meanwhile, the original LBP descriptor ignores all

color information (its calculation is based on gray im-

age), while color plays an important role for distinc-

tion between objects, especially in natural scenes(Zhu

et al., 2010). There can be various changes in light-

ing and viewing conditions in real-world scenes, lead-

ing to large variations of objects in surface illumina-

tion, scale, etc., which make the original LBP per-

formance is not very good in VOC recognition tasks.

In order to address these drawbacks, many improve

method of LBP descriptors have been proposed, such

as Multi-scale Block LBP(Liao et al., 2007), Hierar-

chical Multi-scale LBP(Guo et al., 2010), Multi-scale

Color LBPs(Zhu et al., 2010) and so on.

Traditionally, in order to capture larger scale struc-

ture (macrostructure), there are the histogram fusion

and extending radius approaches to be proposed. The

histogram concatenation approach. Firstly, LBP fea-

tures of different scale are extracted, and then the his-

tograms are concatenated into a long feature. Vector

joint distribution could contain more information, but

it suffers from huge feature dimension. Meanwhile

this approach can not represent the image with the his-

togram uniquely. Usually, considering bigger neigh-

borhood (more neighboring pixels with bigger radius)

could lead to better performance because more local

information is obtained. However, the drawback lies

in the high dimensional histogram produced by the

LBP codes. According to the deﬁnition, if the length

of binary bitstring is p, the resulting histogram will be

of 2

dimension. The dimensionality growth is expo-

nential when the number of neighboring pixels is in-

creasing, and it is impractical to feed the histograms

with such huge dimension into the classiﬁer for clas-

siﬁcation. Although many approaches reduced the

dimension(e.g. ri, u2(Ojala et al., 2002)) were pro-

posed, the drawback are still not solved completely.

In this work, we propose a novel representation,

called Sample Multi-scale Color Local Binary Pat-

tern (SMC-LBP), to overcome the mentioned limi-

tations of LBP and extend the LBP feature to patch.

To validate the proposed feature, we apply it to VOC

Recognition problem. In SMC-LBP, the computation

is done based on randomly sampling the neighboring

pixels from multi-scale circles. Furthermore, in or-

der to enhance photometric invariance property and

discriminative power, the proposed descriptor is com-

puted in different color spaces. To summarize, the

SMC-LBP descriptor presents several advantages:

• It encodes not only single scales but also multiple

scale of image patterns, extends the LBP to the

patch, and hence provides a more complete image

representation than the original LBP descriptor.

• It corporates with color information, therefore its

photometric invariance property and discrimina-

tive power are enhanced.

In section 2, we introduce Sample Multi-scale Bi-

nary Pattern in detail. Section 3 presents Sample

Multi-scale Color Local Binary Pattern. The Frame-

work of the experiment is introduced in section 4. The

experimental results are shown in section 5. Finally

some conclusions and future work are given in sec-

tion 6.

2 SAMPLE MULTI-SCALE

LOCAL BINARY PATTERN

2.1 SM-LBP Approach

Our approach is inspired by earlier work(Ozuysal

et al., 2010) that image patches could be effectively

classiﬁed on the basis of a relatively small num-

ber of pairwise intensity comparisons(Calonder et al.,

2010). Here we randomly sample across different

scale circles, as shown in Fig 3 and is further extended

to use the circular neighborhood with variant radius

and variant number of neighboring pixels.

More speciﬁcally, the SM-LBP descriptor at pixel

location g

) is deﬁned as follows:

SM −LBP

:= Σ

n=1

τ(g

(1)

where g

,··· ,g

) N = 2π/θ , g

is the

pixel gray value of the multi-scale circular neighbor-

hood. N is the number of neighbor pixels which we

randomly choose from different scale circles. How to

generate the g

is introduced in the next section.

VISAPP2013-InternationalConferenceonComputerVisionTheoryandApplications

304

−15 −10 −5 0 5 10 15

−10

−5

(a) Uniform(−10, 10)

−15 −10 −5 0 5 10

−10

−5

(b) Gaussian(0, 100)

−15 −10 −5 0 5 10

−10

−5

Figure 3: Different approaches to choosing the g

location. All the radius R are selected by randomly sampling from different

circulares.

τ(g

) =

1 if g

< g

0 otherwise

(2)

Compared to the original LBP, the MS-LBP re-

places comparisons between the central pixel and sin-

gle circular pixels with comparisons between the cen-

tral pixel and the pixels which are randomly chosen

from multi-scale circles. In this way, the neighboring

pixels randomly chosen could come from the different

scales, this means that our new descriptor can capture

more information from larger region. In this paper,

the follow experiments we consider N = 8, 16, 24;

∈ {R

2.2 Sample Arrangement of SM-LBP

There are many options for generating the radius

from different distributions. We experiment with

three sampling approaches. Assuming the origin of

the patch coordinate system to be located at the patch

center. The patch size S is max(R

). The center point

) to be located at the patch center, g

)

are given by (−R

sin(2πn/N),R

cos(2πn/N)), R

can be described as follows.

• R

∼ i.i.d. Uniform(−R,R): The g

locations are

evenly distributed over the patch.

• R

∼ i.i.d. Gaussian(0,S

) ,the radius R

is sam-

pled from a Gaussian distribution with mean pa-

rameter 0 and standard deviation parameter S

centered around the origin g

. This forces the g

to be more local. g

locations outside the patch

are clamped to the edge of the patch.

• R

∼ i.i.d. Gaussian(R

) , the radius R

is sam-

pled from a Gaussian distribution with mean pa-

rameter R

6= 0 and standard deviation parameter

centered around the origin g

. g

locations

outside the patch are clamped to the edge of the

patch.

3 SAMPLE MULTI-SCALE

COLOR LOCAL BINARY

PATTERN

3.1 Model Analysis for Illumination

Changes

The VOC task is important to access visual informa-

tion on the level of objects and scene types(van de

Sande et al., 2010). In order to enhance the de-

scriptor’s illumination invariance and discriminative

power, we farther proposed color MS-LBP, called

SMC-LBP. The diagonal model (3) and the diagonal-

off model (4) can be used to model changes in the

illumination(van de Sande et al., 2010).

a 0 0

0 b 0

0 0 c

(3)

a 0 0

0 b 0

0 0 c

(4)

where u is an light source, and c is the canonical

illumination. The eq. (4) presents that maps colors

that are taken under an unknown light source to their

corresponding colors under the canonical illumina-

tion(Ozuysal et al., 2010). In order to deal with a

wider range of imaging conditions, Finlayson et al

extend the diagonal model to the diagonal-off model

with an offset (O

)

(Finlayson et al., 2005).

Based on above two models, illumination change

can be deﬁned. If a constant factor in all channels

(a = b = c) In eq. (3), it presents the light intensity

change; If Image values change by an equal offset in

all channels (a = b = c = 1,O

= O

) in eq. (4),

SampledMulti-scaleColorLocalBinaryPatterns

305

Input

images

Joint

Probability Density

Functions (jPDFs)

Feature

extraction

LBP code

Kernel

compution

Classifier

(SVM)

Object

Class

prediction

K-length

Feature

vectors

Figure 4: Flow chart of our system for VOC.

it presents light intensity shift. If (a = b = c,O

= O

) in eq. (4), it means light intensity change

and shift. light color change depend on all channels

independently (a 6= b 6= c), as eq. (3) and light color

change depend on all channels independently with ar-

bitrary offsets (a 6= b 6= c,O

6= O

), as eq. (4).

3.2 SMC-LBP Descriptors

In order to enhance SM-LBP’s photometric invari-

ance property and discriminative power, three color

SMC-LBP descriptors are proposed. The main idea is

to compute the SMC-LBP descriptor independently

over all the channels of certain color space.

RGB-SM-LBP This descriptor is obtained by com-

puting LBP over all three channels of the RGB color

space independently, and then concatenating the re-

sults together. It is invariant to monotonic light inten-

sity change due to the property of the original LBP,

and has no additional invariance properties.

Opponent-SM-LBP This descriptor is obtained by

computing LBP over all three channels of the oppo-

nent color space:

(R −G)/

√

(R + G −2B)/

√

(R + G + B)/

√

(5)

Due to the subtraction, O

and O

channels are invari-

ant to light intensity shift. O

channel represents the

intensity information, and has no invariance proper-

ties.

Hue-SM-LBP This descriptor is obtained by comput-

ing LBP for the Hue channel of the HSV color space:

Hue = arctan(

) = arctan(

√

3(R −G)

R + G −2B

) (6)

Due to the subtraction and the division, Hue channel

is scale-invariant and shift-invariant, therefore this de-

scriptor is invariant to light intensity change and shift.

4 THE FRAMEWORK OF VOC

Our framework for VOC is depicted in Fig. 4

4.1 Feature Extraction

The SM-LBP descriptors extracted from input im-

ages at every pixel location as their features. With

the radius R

which is sampled from a Gaussian or

Uniform, the neighboring pixels g

) is gener-

ated. By this way, the LBP descriptor is extended to

use the multi-circular neighborhood with variant ra-

dius and variant number of neighboring pixels. It is

more suitable for VOC task. Moreover, in order to in-

crease photometric invariance property and discrimi-

native power of the SM-LBP descriptors, The SMC-

LBPs are used in this system.

4.2 Classiﬁcation

Once all the jPDFs representations of the input im-

ages are obtained, they are then feed into certain clas-

siﬁer for classiﬁcation. Here we apply the Support

Vector Machine (SVM) for the ﬁnal classiﬁcation.

The beneﬁts of SVM for histogram-based classiﬁca-

tion have been clearly demonstrated in (Caputo et al.,

2005).

In our experiments, the χ

distance is computed to

measure the similarity between each pair of the fea-

ture vectors F and F

(n is the size of the feature vec-

tor):

dist

(F, F

) =

i−1

−F

)

+ F

(7)

Then, the kernel function based on the χ

distance

is used for SVM to train the classiﬁer:

(F, F

) = e

−

dist

(F,F

)

(8)

where D is the parameter for normalizing the dis-

tances. Here D is set to the average distance of all the

training data. Finally, for each test image, the output

VISAPP2013-InternationalConferenceonComputerVisionTheoryandApplications

306

(a) Aeroplane (b) Dog (c) Horse

(d) Bicycle (e) Car (f) TV/monitor

Figure 5: Example images of the PASCAL VOC 2007

benchmark.

probabilities of SVM classiﬁer are used to predict the

object categories.

5 EXPERIMENT

We perform the VOC experiments on the standard

PASCAL VOC 2007 benchmark. The dataset has 20

different object classes, such as sheep, train, boat, bus,

sofa, table, etc. Some example images are shown in

Fig. 5. The dataset is pre-deﬁned into 50% for train-

ing/validation and 50% for testing. In total there are

9,963 images, where 2501 are for training, 2510 are

for validation and 4952 are for test. For evaluation we

use mean average precision (mAP)(Yue et al., 2007).

5.1 Experiment Results

In order to evaluate the performance of our descrip-

tors, we compare SM-LBP and SMC-LBP descriptors

with the other texture features. Meanwhile we also

compare these descriptors with the SIFT(Lowe, 2004)

which is one of the most powerful image descriptors.

5.1.1 Comparison with the Original LBP

The proposed SM-LBP descriptors are compared with

the original LBP. In our experiment, we set the N =

8,16,24, and g

is generated by the Gaussian distri-

bution and the Uniform distribution. The ﬁnal mAP

value is obtained by the mean of 20 experimental re-

sults. Table 1 shows the comparison of proposed SM-

LBP descriptors and the original LBP on PASCAL

2007. It can be seen that the SM-LBP gets the bet-

ter performance of mAP. Compared with the original

LBP, the SM-LBP obtains a better performance im-

provement (nearly 2%). Fig. 6 shows comparison of

the proposed SMC-LBP descriptors and original color

LBP. It shows that the SMC-LBP all further outper-

form the original color LBP, with the improvements

from 2% to 5.8%.

Table 1: Comparison of proposed SM-LBP descriptors and

the original LBP on PASCAL 2007(original LBP: N=8, the

circle of radius R=1; N=16, the circle of radius R=2; N=24,

the circle of radius R=3. U,G: U is the Uniform distribution;

G is the Gaussian distribution; ).

mAP(%) N=8 N=16 N=24

LBP(original) 28.40 31.64 29.78

SM-LBP(U(-5,5)) 30.40 33.83 33.02

SM-LBP(G(0, 25)) 30.61 33.42 33.20

SM-LBP(G(2, 25)) 29.98 33.34 33.32

0 5 10 15 20 25 30 35 40

LBP(original)

LBP(RGB)

LBP(Opponent)

LBP(hue)

SMC-LBP(RGB)

SMC-LBP(Opponent)

SMC-LBP(hue)

Figure 6: Comparison of the proposed SMC-LBP descrip-

tors and original color LBP(For original color LBPs, N=24,

R=3; For the SMC-LBPs, N=24, the distribution is Gaus-

sian(0, 25)).

5.1.2 Comparison with other Texture

Descriptors

As one kind of texture feature, SM-LBP and SMC-

LBP are compared with three widely-used popu-

lar texture descriptors, including Gabor ﬁlter, Tex-

ture Auto Correlation (TAC), and Grey Level Co-

occurrence Matrix (GLCM). We set 5 scales and 8 ori-

entations for Gabor ﬁlter. For TAC, the rang of x and

y directions is [0,8] with interval of 2. For GLCM,

4 directions (horizontal, vertical and diagonal) with 1

offset between two pixels are considered.

From the results shown in Fig 7, it can be seen

that the original LBP already outperforms other pop-

ular texture descriptors, proving that LBP is one of

the best texture features available today. Our new de-

scriptors further improve the performances to almost

double of the other texture descriptors, demonstrating

that the strong power of the proposed descriptors ben-

eﬁt from the properties of illumination-invariant and

scale-invariant.

5.1.3 Comparison with SIFT Descriptor

Nowadays SIFT, a kind of local gradient descriptors

SampledMulti-scaleColorLocalBinaryPatterns

307

0 10 20 30 40

GLCM

TAC

Gabor

LBP(orginal)

SMC-LBP(RGB)

SMC-LBP(Opponent)

SMC-LBP(hue)

Figure 7: Comparison of the proposed SMC-LBP descrip-

tors and other texture descriptors(SMC-LBPs, N=24, the

distribution chosen Gaussian(0, 25)).

Table 2: Comparison of the proposed SMC-LBP and the

SIFT(SMC-LBPs, N=24, the distribution chosen Gaus-

sian(0, 25)).

mAP(%)

LBP(original) 28.40

SMC-LBP(hue) 34.82

SMC-LBP(Opponent) 35.87

SMC-LBP(RGB) 35.59

SIFT 38.00

is one of the most powerful image descriptors in the

literature. Comparison of the proposed SMC-LBP

and the SIFT, the performance of our texture SMC-

LBP descriptor is close to SIFT.

6 CONCLUSIONS

In this paper, we propose a novel SM-LBP which can

obtain multi-scale patterns and provide a patch texture

representation. Moveover, in order to deal with the

deﬁciency of color information and sensitivity to non-

monotonic lighting condition changes, SMC-LBP de-

scriptor is proposed. The main contributions are that

SM-LBP and SMC-LBP not only have more discrimi-

native power by obtaining more local information, but

also possess invariance properties to different light-

ing condition changes. In addition, they keep the ad-

vantage of computational simplicity from the original

LBP descriptor. The proposed descriptors are vali-

dated by applying on on the PASCAL VOC 2007 im-

age benchmark. Compared with the original LBP, the

experimental results exhibit better recognition accu-

racy.

REFERENCES

Blas, M. R., Agrawal, M., Sundaresan, A., and Konolige,

K. (2008). Fast color/texture segmentation for outdoor

robots. In IROS, pages 4078–4085.

Calonder, M., Lepetit, V., Strecha, C., and Fua, P. (2010).

Brief: binary robust independent elementary features.

In Proceedings of the 11th European conference on

Computer vision: Part IV, ECCV’10, pages 778–792,

Berlin, Heidelberg. Springer-Verlag.

Caputo, B., Hayman, E., and Mallikarjuna, P. (2005). Class-

speciﬁc material categorisation. IEEE International

Conference on Computer Vision (ICCV), 2:1597–

1604.

Finlayson, G. D., Hordley, S. D., and Xu, R. (2005). Convex

programming colour constancy with a diagonal-offset

model. In ICIP (3)’05, pages 948–951.

Guo, Z., 0006, L. Z., Zhang, D., and Mou, X. (2010). Hier-

archical multiscale lbp for face and palmprint recog-

nition. In ICIP, pages 4521–4524. IEEE.

Liao, S., Zhu, X., Lei, Z., Zhang, L., and Li, S. Z. (2007).

Learning multi-scale block local binary patterns for

face recognition. In Lee, S.-W. and Li, S. Z., editors,

ICB, volume 4642 of Lecture Notes in Computer Sci-

ence, pages 828–837. Springer.

Lowe, D. G. (2004). Distinctive image features from scale-

invariant keypoints. Int. J. Comput. Vision, 60(2):91–

110.

Ojala, T., Pietik

ainen, M., and M

aenp

a, T. (2002). Mul-

tiresolution gray-scale and rotation invariant texture

classiﬁcation with local binary patterns. IEEE Trans.

Pattern Anal. Mach. Intell., 24(7):971–987.

Ozuysal, M., Calonder, M., Lepetit, V., and Fua, P. (2010).

Fast keypoint recognition using random ferns. IEEE

Trans. Pattern Anal. Mach. Intell., 32(3):448–461.

Shan, C., Gong, S., and McOwan, P. W. (2009). Fa-

cial expression recognition based on local binary pat-

terns: A comprehensive study. Image Vision Comput.,

27(6):803–816.

Tuceryan, M. and Jain, A. K. (1998). Texture analysis. In

Handbook of Pattern Recognition and Computer Vi-

sion, pages 235–276.

van de Sande, K. E. A., Gevers, T., and Snoek, C. G. M.

(2010). Evaluating color descriptors for object and

scene recognition. IEEE Transactions on Pattern

Analysis and Machine Intelligence, 32(9):1582–1596.

Yue, Y., Finley, T., Radlinski, F., and Joachims, T. (2007).

A support vector method for optimizing average preci-

sion. In Proceedings of the 30th annual international

ACM SIGIR conference on Research and development

in information retrieval, SIGIR ’07, pages 271–278,

New York, NY, USA. ACM.

Zhang, D., Wong, A., Indrawan, M., and Lu, G. (2000).

Content-based image retrieval using gabor texture fea-

tures. In IEEE Transactions PAMI, pages 13–15.

Zhu, C., Bichot, C.-E., and Chen, L. (2010). Multi-scale

Color Local Binary Patterns for Visual Object Classes

Recognition. In IEEE, editor, International Confer-

ence on Pattern Recognition (ICPR), pages 3065–

3068.

VISAPP2013-InternationalConferenceonComputerVisionTheoryandApplications

308