ROBUST MULTIMODAL BIOMETRIC SYSTEM USING

MARKOV CHAIN BASED RANK LEVEL FUSION

Maruf Monwar and Marina Gavrilova

Computer Sciene, University of Calgary, Canada

Keywords: Pattern recognition, Multimodal biometric system, Rank level fusion, Markov chain.

Abstract: Multimodal biometrics is an emerging area of pattern recognition research that aims at increasing the

reliability of biometric systems through utilizing more than one biometric in decision-making process. But

an effective fusion scheme is necessary for combining information from various sources. Such information

can be integrated at several distinct levels, such as sensor level, feature level, match score level, rank level

and decision level. In this research, we develop a multimodal biometric system utilizing face, iris and ear

features through rank level fusion method. We apply Fisherimage technique on face and ear image

databases for recognition and Hough transform and Hamming distance techniques for iris image

recognition. We introduce Markov chain approach for biometric rank aggregation. We investigate various

rank fusion techniques and observe that Markov chain approach gives us the best result. Also this approach

satisfies the Condorcet criterion which is essential in any fair rank aggregation system. The system can be

effectively used by of security and intelligence services for controlling access to prohibited areas and

protecting important national or public information.

1 INTRODUCTION

The biometrics based controlled access to the protected

resources has emerged shown to offer higher security

and convenience to the users.

The optimal biometric

recognition would be one having the properties of

distinctiveness, universality, permanence,

acceptability, collectability, and resistance to

circumvention (Ross et al., 2006). No existing

biometric system simultaneously meets all of these

requirements, however the use of more than one

biometric can help lead to a system which is closer

to these ideals.

The most immediate advantage of multimodal

authentication is increased recognition accuracy.

Multimodal systems fuse information for more than

one source, each of which offers additional evidence

about the authenticity of an identity claim.

Therefore, one can have more confidence in the

result (Dunstone and Yager, 2009).

Multibiometric systems can address the non-

universality problem and reduce the FTER (Failure-

to-Enroll Rate) and FTCR (Failure-to-Capture Rate).

For instance it is estimated that 2% of the population

may not be able to provide a fingerprint due to

medical/genetic conditions, accidental destruction,

or temporary damage (Maio et al., 2004). That group

of persons can still be recognized using other

biometric traits in a multimodal biometric system.

Multimodal biometric systems are more resistant

to spoof attacks because it is difficult for the attacker

to simultaneously spoof multiple biometric sources.

All multimodal biometric systems need a fusion

module that takes two or more data and combines

them in order to obtain the authentication result:

impostor or genuine user. Figure 1 shows a sample

multimodal biometric system.

The fusion strategies are divided into two main

categories: pre-mapping fusion and post-mapping

fusion (Revett, 2008). The first strategy deals with

the sensor data fusion level and feature vector fusion

level. These techniques are not used because they

give many implementation problems (Bubeck,

2003). The second strategy is realized through the

decision level fusion, based on some algorithms

which combine single decisions for each component

system, or through the matching score level fusion,

which combines the matching scores of each

component system, or through the rank level fusion,

which is used when the output of each component

system is a subset of possible matches (i.e.,

identities) sorted in decreasing order of confidence.

458

Monwar M. and Gavrilova M. (2010).

ROBUST MULTIMODAL BIOMETRIC SYSTEM USING MARKOV CHAIN BASED RANK LEVEL FUSION.

In Proceedings of the International Conference on Computer Vision Theory and Applications, pages 458-463

DOI: 10.5220/0002851404580463

 SciTePress

Figure 1: A sample multimodal biometric system.

In this research, we investigate rank level fusion

for face, ear and iris biometrics as the other fusion

methods have been extensively studied in the

literature from the last ten years. Fusion at the rank

level is a significantly understudied problem, which

has a high potential for efficient consolidation of

ranked information obtained from multiple unimodal

matchers (Bhatnagar et al., 2007). We introduce

Markov chain (Dwork et al., 2001) approach for

fusing rank information in this multimodal system.

2 RANK LEVEL FUSION

Rank-level fusion is used only in identification systems

and is applicable when the individual matcher’s output

is a ranking of the “candidates” in the template

database. The system is expected to assign a higher

rank t

o a template that is more similar to the query.

Very few methods can be found in the literature for

consolidation of biometric rank information as it is

still an understudied problem. Three methods

described by Ho, Hull, and Srihari in (Ho et al.,

1994) to find out the final decision in a general

multiple classifier system, can be used for rank level

fusion in multimodal biometric systems. These

methods are highest rank, Borda count and logistic

regression methods. Recently Nandakumar and

others (Nandakumar et al., 2009) introduced

Bayesian approach for rank level fusion. All of these

methods for rank level fusion is briefly discussed in

the next subsections.

2.1 Highest Rank Method

The highest rank method is good for combining a

small number of specialized matchers and hence can

be effectively used for a multimodal biometric

system where the individual matchers are the best. In

this method, the consensus ranking is obtained by

sorting the identities according to their highest rank.

The advantage of this method is the ability to utilize

the strength of each matcher. The disadvantage of

this method is that the final ranking may have many

ties (Monwar et al., 2009).

2.2 Borda Count Method

The Borda count (Borda, 1781) method is the most

widely used rank aggregation method and uses the

sum of the ranks assigned by individual matchers to

calculate the final rank. This method assumes that

the ranks assigned to the users by the individual

matchers are statistically independent and the

performances of all three matchers are equally well.

The advantage of this method is that it is easy to

implement and requires no training stage. These

properties made the Borda count method feasible to

incorporate in multimodal biometric systems. The

disadvantage of this method is that it does not take

into account the differences in the individual

matcher’s capabilities and assumes that all the

matchers perform equally.

2.3 Logistic Regression Method

The logistic regression method calculates the

weighted sum of the individual ranks. In this

method, the final consensus rank is obtained by

sorting the identities according to the summation of

their rankings obtained from individual matchers

multiplied by the assigned weight.

The weight to be assigned to the different

matchers is determined by a ‘logit’ function using

logistic regression (Agresti, 2007). This method is

very useful when the different matchers have

significant differences in their accuracies but

requires a training phase to determine the weights

which can be computationally expensive. Also one

of the key factors that have direct effect on the

performance of a biometric system is the quality of

the biometric samples. Hence the single matchers’

performance can vary with different sample sets

which make the weights allocating process more

challenging and inappropriate weight allocation can

eventually reduce the recognition performance of

this multimodal biometric system (using logistic

regression) compared to unimodal matchers.

2.4 Bayesian Approach

Bayesian approach for biometric rank fusion is

based on Bayes decision theory. This approach uses

the rank distribution (probability that an identity is

assigned a rank by an individual matcher is a true

identity) which can be estimated provided the

marginal genuine and impostor match score

densities are known (Nandakumar et al., 2009). The

consensus rank is obtained as the product of the

ROBUST MULTIMODAL BIOMETRIC SYSTEM USING MARKOV CHAIN BASED RANK LEVEL FUSION

459

posterior probabilities of the individual matchers.

The size of the multimodal biometric database is

usually huge and thus only the top few results are

usually considered for the final reordered ranking.

Hence, a very common scenario of a rank based

multimodal biometric system is that some results

may rank at top by a few classifiers and the rest of

the classifiers do not even output the result. In this

situation, the above approaches cannot produce a

good recognition performance.

To deal with these shortcomings, in this

research, we introduce Markov chain rank

aggregation method to find out the consensus rank

for person identification. Previously, this approach

has successfully been used in web search (Dwork et

al., 2001). Due to the ease in implementation and its

successful usage in the web ranking, we decide to

employ Markov chain approach for multimodal

biometric rank fusion.

3 MARKOV CHAIN APPROACH

We consider the biometric rank aggregation as an

evaluation of a voting method. In a voting method

evaluation, the most important thing is to ensure the

fairness of the voting system. Among the fairness

criteria, the two most important criteria are

Condorcet Winner Criterion and the Condorcet

Loser Criterion (Condorcet, 1785).

Condorcet Winner Criterion: If there exists an

alternative a, which would win in pairwise votes

against each other alternative, then a should be

declared the winner of the election. Note that there is

not necessarily such an alternative a. This alternative

is called the Condorcet winner.

Condorcet Loser Criterion: If there exists an

alternative a, which would loose in pairwise votes

against each other alternative, then a should not be

declared the winner of the election.

None of the approaches described in section 2

ensures the election of Condorcet Winner. This

motivates us to employ the Markov chain approach

for biometric rank fusion in this multimodal

biometric system.

In the Markov chain biometric rank aggregation

method, it is assumed that there exists a Markov

chain on the enrolled identities and the order

relations between those identities in the ranking lists

(obtained from different biometric matchers)

represent the transitions in the Markov chain. The

stationary distribution of the Markov chain is then

utilized to rank the entities (Dwork et al., 2001). The

construction of the consensus ranking list from the

Markov chain can be summarized as below:

1) Map the set of ranked lists to a single Markov

chain, with one node of the chain represents one

identity in the initial ranking lists.

2) Compute the stationary distribution on the

Markov chain.

3) Rank the identities based on the stationary

distribution. That is, the node with the highest score

in the stationary distribution is given the top rank,

and so on down to the node with the lowest score in

the stationary distribution which is given the last

rank.

The proposed Markov chain approach for

biometric rank aggregation has several advantages.

This method handles the partial ranking list very

well and provides a more holistic viewpoint of

comparing all candidates against each other. To do

so, this method use only the available comparisons

(in the partial lists) between the identities to

determine the transition probabilities and exploit the

connectivity of the chain to infer comparison

outcomes between pairs that were not explicitly

ranked by any of the matcher. The Markov chain

method also handles the uneven comparison, i.e.,

when the results of the initial ranking lists are very

much different. Heuristics for combining rankings

are motivated by some underlying principle and the

Markov chain model can be viewed as the natural

extensions of those heuristics. For example, Borda’s

method is based on the idea “more wins is better.” It

is natural to extend this and say “more wins against

good players is even better,” and so on, and

iteratively refine the ordering produced by a

heuristic. Some Markov chain models for biometric

rank aggregation can be viewed as the natural

extensions of Borda’s method, sorting by Geometric

mean or Copeland’s method (sort the candidates by

the number of pairwise majority wins minus

pairwise majority losses) (Copeland, 1951).

There are four specific Markov chains (Dwork et

al., 2001), which can be used for biometric rank

aggregation. Among those four methods, the last

method satisfies the Copeland method and according

the literature, the best performing one. This specific

Markov chain can be termed as MC

and can be

defined as follows:

: If the current state is a, then choose an

identity b uniformly from the union of all identities

ranked by the unimodal matchers. If the rank of b is

lower than the rank of a for majority of the matchers

that rank both a and b, go to b, else stay in a.

Figure 2 shows a Markov chain with its transition

matrix build on MC

. There are three matchers

which outputs three different ranking lists. Based on

the ranking list, a Markov chain is constructed

VISAPP 2010 - International Conference on Computer Vision Theory and Applications

460

Figure 2: Markov chain and the transition matrix

constructed from three ranking lists based on MC

according to MC

The final ranking list can be

obtained by applying the Copeland method, i.e., by

sorting the nodes in the majority graph (Markov

chain) by outdegree minus indegrree.

4 THE PROPOSED SYSTEM

The design of a multimodal biometric system is

strongly dependent on the application scenario. A

number of multimodal biometric systems have been

proposed for the last ten years but they differ from one

another in terms of their architecture, the number and

choice of biometric modalities, the level at which the

evidence is accumulated, and the methods used for the

integration or fusion of information (Chandran and

Rajesh, 2009). The proposed system adopts multiple

biometric traits of an individual, to establish the

identity.

The system employs three unimodal matchers for

face, ear an iris biometric traits. T

he main goal of this

research is to evaluate the performance of the

multimodal biometric system based on rank level

fusion over the unimodal biometric system. So, we

decide to use face, ear and iris biometric traits for

this system. Although ear is not a frequently used

biometric trait, but we choose this trait because w

want to use biometrics from the similar region of the

human body keeping in mind that, it will help us to

create the multimodal database in future.

All the biometric traits that will be used in this

project are images. For face and ear images, we use

Fisherface (Belhumeur et al., 1997), as this method

has significant advantages over the popular

eigenface method (Turk and Pentland, 1991) in case

of images of the same subject with certain

illumination change. Fisherimage is a combination

of principle component analysis (PCA) and linear

discriminant analysis (LDA). The prime difference

between LDA and PCA is that PCA does more of

feature classification and LDA does data

classification. Researchers have demonstrated that

the LDA based algorithms outperform the PCA

algorithm for many different tasks (Zhao et al.,

1998).

For iris recognition, hamming distance method is

used for recognition after the iris image pre-

processing and encoding. At first, the iris part of the

eye image (from inside the limbus (outer boundary)

and outside the pupil (inner boundary)) are

localized. For iris localization, Hough transform

(Wildes, 1997) is used. After localizing the region of

interest, the Rubber Sheet Model (Daugman, 2004)

is used for un-wrapping the iris image. Then a Gabor

filter encodes the iris data. After encoding, the

binary data is available which is compared by

Hamming distance method.

A detailed diagram of the proposed system is

shown in figure 3. In the enrolment phase, face, ear

and iris images will be acquired first and then will

be processed according to the training algorithms

and saved as face, ear and iris templates.

Figure 3: Proposed system architecture.

In the Identification phase, face and ear images

will be recognized measuring the Euclidian distance

between the test image and the images in the

fisherfaces and fisherears. For iris, the Hamming

distance will be calculated between the codes

generated from the test iris with the iris codes in the

database. In each of the three cases, five identities

will be obtained as output that will be ranked

ROBUST MULTIMODAL BIOMETRIC SYSTEM USING MARKOV CHAIN BASED RANK LEVEL FUSION

461

according to their distances. The identities of these

three ranking list then be integrated using the rank

level fusion approach to find out a consensus rank of

the identities and the identity at the top of the

consensus ranking list will be identified as the

desired identity. For rank level fusion, highest rank,

Borda count, logistic regression and Markov chain

approaches are used to find out the consensus

ranking from the three ranking lists.

5 EXPERIMENTS AND RESULTS

Due to the inherent cost and effort associated with

constructing a multimodal database, the database

used in this system is not the “original” multimodal

database (different biometric traits are collected

from the same person), but rather we use a “virtual”

database which contains records created by

consistently pairing a user from one unimodal

database (e.g., face) with a user from another

database (e.g., iris) (Ross et al., 2006). The creation

of virtual users is based on the assumption that

different biometric traits of the same person are

independent.

For iris, we use the CASIA Iris Image Database

(ver 1.0) from the Chinese Academy of Science

(CASIA, 2004). CASIA database (ver 1.0) includes

756 black and white iris images from 108 eyes. For

each eye, 7 images are captured in two sessions.

The ear images are from the USTB, China

database (USTB, 2002). The database contains ear

images with illumination and orientation variation.

The images are 300 x 400 pixels in size.

For face, Facial Recognition Technology

(FERET) database (Phillips et al., 1998) is used.

There are 14,051 images (256 x 384) of 1199

person. There are various face images with

expression, pose and illumination variation.

To build the virtual multimodal database for the

proposed system, we consider 600 iris images from

300 subjects of CASIA database. In addition, 600

ear images and 600 face images are also be

considered from USTB and FERET database

respectively. Then each sample of these 600 iris

images will randomly be combined with one sample

of 600 ear images and one sample of 600 face

images. Half of these 600 combined samples are

used for training purposes and the remaining half are

used for testing. Thus we have a virtual multimodal

database containing 300 training and 300 testing

multimodal samples. Fig 4 shows a portion of our

sample multimodal database.

Figure 4: A small portion of our virtual database.

After experiment, we observe the result by

plotting the recognition value on a Cumulative

Match Characteristic (CMC) curve. CMC curve is

used to summarize the identification rate at different

rank values. As rank level fusion method can only

be applied in identification (not in verification)

systems, so, we insist on the identification rate

which is the proportion of times the identity

determined by the system is the true identity of the

user providing the query biometric sample. If the

biometric system outputs the identities of the top x

matches, the rank-x identification rate, is defined as

the proportion of times the true identity of the user is

contained in the top m matching identities.

Figure 5 shows the CMC curves of the

individual face, ear and iris matchers and for

Markov chain and logistic regression rank fusion

approaches. We investigate highest rank, Borda

count, logistic regression and Markov chain

approaches on this virtual multimodal database and

obtained the best identification rates through

Markov chain approach (98.2%). Among the other

three, logistic regression approach is better (97%).

Figure 5: CMC curve of Markov chain and logistic

regression rank fusion approaches along with face, iris and

ear unimodal systems.

As the performances of our individual matchers are

not equal, hence we report only the identification

rates of Markov chain and logistic regression

VISAPP 2010 - International Conference on Computer Vision Theory and Applications

462

approaches. Also we report the identification rates of

the face, iris and ear matchers on the CMC curves to

show the differences.

6 CONCLUSIONS

The design of a multimodal biometric system is a

challenging task due to heterogeneity of the

biometric sources in terms of the type of

information, the magnitude of information content,

correlation among the different sources and

conflicting performance requirements of the

practical applications. Extensive research has been

done to identify better methods to combine the

information obtained from multiple sources. In this

research, we combine face, ear and iris biometric

information using rank level fusion method. We

introduce Markov chain approach for biometric rank

fusion and obtain better identification rate over other

rank fusion approaches. Thus, Markov chain method

can be a reliable solution of integrating biometric

ranking lists to obtain a consensus rank list and can

be effectively used in various security systems.

REFERENCES

Agresti, A., 2007. An introduction to categorical data

analysis. Wiley-Interscience, 2

edition.

Belhumeur, P. N., Hespanha, J. P., and Kriegman, D. J.,

1997. Eigenfaces vs. Fisherfaces: Recognition using

class specific linear projection. IEEE Transaction on

Pattern Analysis and Machine Intelligence, vol. 19,

no. 7, pages 711-720.

Bhatnagar, J., Kumar, A., and Saggar, N., 2007. A novel

approach to improve biometric recognition using rank

level fusion. In Proc. IEEE Conference on Computer

Vision and Pattern Recognition, pages 1-6,

Minneapolis, USA.

Borda, J. C., 1781. M´emoire sur les ´elections au scrutin.

Histoire de l’Acad´emie Royale des Sciences, France.

Bubeck, U. M., 2003. Multibiometric authentication – An

overview of recent developments. San Diego

University.

http://www.thuktun.org/cs574/papers/multibiometrics.

pdf

CASIA: CASIA iris image database, 2004. Retrieved on

May 23, 2008. www.sinobiometrics.com

Chandran, J. G. C., and Rajesh, R. S., 2009. Performance

analysis of multimodal biometric system

authentication. IJCSNS International Journal of

Computer Science and Network Security, vol. 9, no.3,

pages 290-296.

Condorcet, M.-J., 1785. E´ssai sur l’application de

l’analyse a` la probabilite´ des de´cisions rendues a`

la pluralite´ desvoix.

Copeland, H., 1951. A reasonable social welfare function.

Mimeo, University of Michigan, USA.

Daugman, J. G., 2004. How iris recognition works. IEEE

Transaction on Circuits and Systems for Video

Technology, vol. 14, no. 1, pages 21-30.

Dunstone, T., and Yager, N., 2009. Biometric system and

data analysis: Design, evaluation, and data mining.

Springer, New York.

Dwork, C., Kumar, R., Naor, M., and Sivakumar, D.,

2001. Rank aggregation methods for the web. In Proc.

of 10

International World Wide Web Conference,

pages 613–622, Hong Kong, China.

Ho, T. K., Hull. J. J., and Srihari, S. N., 1994. Decision

combination in multiple classifier systems. IEEE

Trans. on Pattern Analysis and Machine Intelligence,

vol. 16, no. 1, pages 66-75.

Kim, J., Cho, S., Kim, D., and Chung, S.-T., 2006. Iris

recognition using a low level of details. Lecture Notes

in Computer Science, vol. 4292, pages 196-204,

Springer.

Maio, D., Maltoni, D., Cappelli, R., Wayman, J. L., and

Jain, A. K., 2004. In Proc. International Conference

on Biometric Authentication, pages 1-7, Hong Kong,

China.

Monwar, M. M., and Gavrilova, M., 2009. A Multimodal

Biometric System using Rank Level Fusion Approach.

IEEE Transactions on Systems, Man, and Cybernetics

- Part B: Cybernetics (special issue on Cognitive

Informatics and Cybernetics), vol. 39, no. 4, pages

867-878.

Nandakumar, K., Jain, A.K., and Ross A., 2009. Fusion in

multibiometric identification systems: What about the

missing data? In M. Tistarelli and M.S. Nixon,

Editors, International Conference on Biometrics, vol.

LNCS 5558, pages 743–752, Springer.

Phillips, P. J., Moon, H., and Rauss, P., 1998. The FERET

database and evaluation procedure for face recognition

algorithms. Image and Vision Computing, vol. 16. no.

5, pages 295-306.

Revett, K., 2008. Behavioral biometrics: A remote access

approach. Wiley, West Sussex, UK.

Ross, A., Nandakumar, K., and Jain, A. K., 2006.

Handbook of multibiometrics. Springer, New York.

Turk, M., and Pentland, A., 1991. Eigenfaces for

recognition. Journal of Cognitive Science, pages 71-

86.

USTB ear database, China. Retrieved on May 11, 2008.

http://www.ustb.edu.cn/resb/

Wildes, R., 1997. Iris recognition: An emerging biometric

technology. In Proc. IEEE, vol. 85, no. 9, pages 1348-

1363.

Zhao, W., Chellappa, R., and Nandhakumar, N., 1998.

Empirical performance analysis of linear discriminant

classifiers. In Proc. 1998 Conference on Computer

Vision and Pattern Recognition, pages 164–169, Santa

Barbara, CA.

ROBUST MULTIMODAL BIOMETRIC SYSTEM USING MARKOV CHAIN BASED RANK LEVEL FUSION

463