MELON AUTHENTICATION BY AGRI-BIOMETRICS

Identifying Individual Fruits using a Single Image of Rind Pattern

Rui Ishiyama

, Yoichi Nakamura

, Akira Monden

, Lei Huang

and Seiji Yoshimoto

Media and Information Processing Research Laboratories, NEC Corporation,

1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa 211-8666, Japan

Space Systems Division, NEC Corporation, 1-10, Nisshincho, Fuchu, Tokyo 183-8501, Japan

Software Purchasing Division, NEC Corporation, 5-7-1 Shiba, Minato-ku, Tokyo 108-8001, Japan

Keywords: Agri-Biometrics, Authentication, Fruits, Melon, Identification, Verification, Pose, Feature Points, Minutiae,

Fingerprint Matching, Image.

Abstract: We propose a new method of biometric authentication, called agri-biometrics that identifies an individual

fruit using a single image of its rind patterns. Our proposed method normalizes the rotations in depth of the

fruit and extracts a set of image features, which are compatible to the 'minutiae', from the normalized image;

thus, it enables us to apply a state-of-the-art technique of fingerprint matching to identify the rind patterns of

fruit. We conducted large-scale experiments to identify/verify 1,776 individual melons in practical

situations where the images were taken under different pose and illumination conditions on different days.

Our method in the experiments achieved excellent recognition of EER=0.06%. The agri-biometric

authentication we propose accomplishes 'verifiable' agri-food traceability and brand protection; once the

producers register pictures of their products into the database, anyone can verify the products on hand with

the camera of a mobile phone.

1 INTRODUCTION

1.1 Product Identification for

Traceability and Anti-Counterfeit

There have been growing demands for agri-products

that have diverse added values including those

originating from branded varieties, well-known

growing districts, premium grades, and organic

cultivation. Advertising these added values is now

important for marketing, and gaining the trust of

consumers is mandatory to sell products at high

prices. The problem is that consumers, and even

retailers and traders, cannot find these added values

by just looking at the products themselves. Thus,

product information has more impact on the price

than the product itself, and this explains why

counterfeiting is so enticing.

Increasing problems with counterfeited and fake

products are being reported as the supply chain

expands globally and internationally. These not only

misappropriate revenues from the sales of genuine

producers, but they can also have significant

consequences on consumers. Consumers are

currently having to pay a great deal of attention to

traceability, which is the ability to trace the history

of a product through all its stages of production,

processing, and distribution.

Traceability is established based on methods of

identifying and verifying individual examples of the

product. We not only need to search databases, but

also to verify that individual products on hand are

genuine and not counterfeits. The way products are

authenticated, identified, and verified determines the

accessibility and anti-counterfeit capabilities of

traceability systems.

The traditional way of identifying products is by

attaching tags that directly display product

information or serial numbers. Barcodes and RFIDs

have recently been used to improve accessibility

(Regattieri, 2007). As barcodes can be read by

cameras installed on standard mobile phones, these

offer greater access to consumers to obtain detailed

information on products through the Internet.

Although these technologies may provide

increasingly more information, which may ease

consumer confidence, the risk of counterfeiting is

not reduced.

RFIDs and hologram tags have been proposed as

698

Ishiyama R., Nakamura Y., Monden A., Huang L. and Yoshimoto S..

MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind Pattern.

DOI: 10.5220/0003842706980704

In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 698-704

ISBN: 978-989-8565-03-7

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

technologies to combat counterfeiting (Bernardi,

2008). Chromatography and DNA analysis

techniques have been conducted to inspect agri-

products themselves to prevent counterfeiting (Lees,

2003). Numerous anti-counterfeit technologies have

been utilized, and surveillance has been conducted

by public institutions. However, counterfeiting of

various agri-products is increasingly being reported.

1.2 Problems in Traceability of

Agri-Product

Existing methods encounter two main difficulties in

being effective deterrents against counterfeiting:

 Cost of tagging

Anti-counterfeiting tags are inexorably

expensive and an enormous number of tags needs to

be attached to all agri-products. The risk of tags

being swapped cannot be avoided even after this

high cost is incurred.

 Usability of verification

As inspecting tags and products require special

devices or skilled staff, only limited numbers of

products on the global market can be checked.

Consequently, counterfeiting is rarely discovered.

Consumers are not only unaware of anti-

counterfeiting measures but they do not want to pay

for these.

A novel method is required to solve these

problems so that agri-products can be authenticated

by anyone, anywhere, and at any time without

having to rely on costly tags or inspection

procedures.

1.3 Agri-Biometric Authentication

We propose a new methodology in this paper to

identify individual agri-products by having single

photographs taken of rind patterns (e.g., net, stripe,

and dot patterns on the rinds, see Figure 1) and by

matching these to an image database of

authenticated products. Since methods of

authenticating people using facial and fingerprint

features are called 'biometrics', we have called our

proposed method 'agri-biometrics'. The new method

authenticates the fruit bodies themselves through the

use of rind patterns, without the need to attach tags.

Rind patterns of fruit are generated depending on the

environment in which they are grown, and these are

unique to individual fruit. Even if fake fruit are

grown from the same seed and with the same

method of cultivation, creating an identical rind

pattern is supposed to be impossible. Thus, fake fruit

cannot be cultivated, at least not within reasonable

costs that would offset the expense of counterfeiting.

The key feature of the proposed method is that

only a single photograph is required that is taken

with handy standard cameras such those in mobile

phones to authenticate the individual fruit on hand

from the enormous amount of fruit on the market.

Producers in practical traceability systems register

images of shipped fruit into a database. As many

producers adopt automated systems for grading and

inspecting the quality of fruit (Kondo, 2010),

capturing images of individual fruit in a database

can easily be automated. If a traceability service to

match images with those in the database is provided

over the Internet, anyone can authenticate fruit using

his/her smartphone from everywhere and at any

time. As the whole market is monitored by everyone

at all times, counterfeiting is expected to be

effectively suppressed. Furthermore, as consumers

are able to check the products themselves, they

actually feel it is worth paying for added values.

Figure 1: Rind patterns of netted honeydew melon, water

melon, and green apple.

1.4 Previous Study and Proposed

Architecture

In the literature, a similar approach has been

reported. It identifies individual apples using

appearances of multiple images (Niigaki, 2009).

Since it requires numerous images to be taken for

each authentication to compensate for different

poses of apples, it is far from being a practical

application. We propose a new method that

normalizes pose variations to achieve authentication

using only a single image, and that utilizes

fingerprint matching technology to achieve

extremely accurate authentication. Figure 2 outlines

our new approach.

In our proposed method, a 3D model (sphere for

melons) approximates a fruit’s average shape to an

image and cancels out rotations in depth. This

simulates the same process as that with

fingerprinting, which also involves patterns on

curved 3D surfaces that are flattened onto a scanner;

the scanned image of the fingerprints does not

contain deformation due to rotations in depth.

The rind patterns of fruit differ greatly from

MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind

Pattern

699

fingerprints, but their features have the common

nature that feature points are located randomly for

each individual, which is different from facial

features. Our method extracts a feature set, which is

compatible to the 'minutiae' used to match

fingerprints, from a pose-normalized image. This

make it possible to utilize state-of-the-art methods of

fingerprint matching using minutiae features, whose

accuracy has been demonstrated to be sufficient

even for law enforcement applications (Jain, 2007).

In this study, which appears as the 1st report of

agri-biometrics research, netted honeydew melons

were chosen to be the targeted agri-product in this

study. There are numerous premium brands,

growing districts for melons, and the prices of

melons differ from $5 up to $100 depending on such

added value information. Thus the melon is

considered to be a typical example of the agri-

products having serious risks of counterfeits. We

actually identified thousands of melons in the

experiments to demonstrate high degrees of

accuracy in practical situations.

(Registration ) (Query)

Take a single image

Pose normalization

(Proposed method)

Feature extraction

(Compatible to minutiae)

Matching

(Using fingerprint matching algorithm)

Figure 2: Architecture of our proposed method to

authenticate the fruit using rind pattern image.

2 PROPOSED METHOD

A new method of authenticating fruit with agri-

biometrics that uses a single image is proposed in

this section. Our architecture utilizes the technique

of fingerprint matching to match the rind patterns of

fruit. It consists of three steps to make it feasible.

The first step solves the problem of variations in

object poses, which cannot be solved by

conventional techniques of fingerprint matching.

The second step extracts the feature set that is

minutiae-compatible from the rind pattern image.

The third step involves an extension of the

conventional technique of fingerprint matching.

2.1 Pose Normalization

The pose normalization step in our proposed method

is described in this subsection. As outlined in Figure

3, this step is used to take an image with the location

of its stem point as input, and it cancels out the

rotations in depth of the image. The contour of the

object is extracted in this step, and a standard 3D

shape model is fitted to the image. The input image

is mapped to the pose-normalized image with the

texture-mapping technique, which simulates an

image being taken by setting up the camera and the

fruit accurately in a predetermined normal pose.

The camera is modelled with weak perspective,

and a sphere is used as the standard 3D shape model

for melons (which is good approximation of the 3D

shape of melons, especially for the premium graded

honeydew melons we used). Let us denote the image

coordinates and the camera’s optical axis to

correspond to the x, y, and z-axes. The normal pose

is predetermined as the 'top view', where the melon's

axis piercing from its stem point to the base is

aligned with the z-axis.

Suppose that point P is on the sphere’s surface

and this corresponds to pixel (x

, y

) in the pose-

normalized image, and that

denotes the angle

between the line passing through the sphere’s center

to point P and the yz-plane. Also suppose that

denotes the angle between the line and xz-plane.

Here, the following equations are obtained.

= sin

, y

= cos

sin

(1)

If the sphere is rotated by angle

around the y-

axis, P is moved to a pixel corresponding to a point

whose polar coordinates are (

) on the sphere;

its image coordinates (x

, y

) are then obtained as:

= sin(

), y

= cos(

) sin

(2)

If the sphere is additionally rotated by angle

around the z-axis, the image pixel corresponding

to P is moved to (x

, y

), which is obtained as:

= x

cos

- y

sin

, y

= x

sin

+ y

cos

(3)

Here, the contour of the melon is detected as a

VISAPP 2012 - International Conference on Computer Vision Theory and Applications

700

(a) Input image with

Stem point

(b) Circle detection

(

)

Estimate rotations in de

(

)

Pose-normalized Ima

, p

)

Figure 3: Pose normalization step in our proposed method

to cancel out rotations in depth.

circle. For simplicity, we assume that the

background of the image is dark monotone, thus the

melon’s body region is detected by applying the

following well-known image processing algorithms:

Otsu’s digitization (Otsu, 1979) for the whole image,

Canny edge detector, and Hough transform for circle

finding (Figure 3(b)). In this study, the stem point is

manually inputted. If its automation is necessary for

the applications, numerous pattern-finding methods

can be applied.

The original image is translated and scaled so

that the detected circle is located at the origin and

has a unit radius (see Figure 3(c)). If the stem point

is located at (p

, p

), the rotations

and

, are

estimated as:

,sin

pp +=

pp=

tan

(4)

The pose-normalized image (Figure 3(d)) is

generated by calculating (x

, y

) for each pixel (x

) and the pixel values are mapped from the input

image. Although the input image has arbitrary

rotations in depth (Figure 3(a)), the rotations are

cancelled out in the pose-normalized image (Figure

3(d)).

2.2 Minutiae-Compatible Feature

Extraction

This subsection describes the feature extraction step

in our proposed method. As outlined in Figure 4, this

step is used to extract a set of feature points and their

directions. The feature set is compatible with the

‘minutiae’ that are widely used in fingerprint

matching, and it can be matched with the state-of-

the-art techniques of fingerprint matching.

First, the input image is digitized to extract the

contours of the melon’s netted rind pattern. Since the

shading on the melon’s surface differs locally and

depends on the illumination environments in which

the image was taken, locally adaptive threshold

(Niblack, 1986) was adopted in digitization (a

survey is given in (Sezgin, 2004)). This method

determines the thresholds for each of all pixels based

on the average and the variance in pixel intensities

of each neighbouring pixels. The threshold was

determined to be the local average plus the local

standard deviation multiplied by a predetermined

coefficient. The size of the neighbouring area and

the coefficient were fixed for the all images. The

fixed size and coefficient were determined in a

preliminary experiment using separated image

database to be sufficient to work well for extracting

the mesh rind pattern from any image taken in

general indoor environments.

After digitization, the image was filtered by

successive morphological operations of dilation and

erosion, and a median filter was used to remove

noise and to smooth the contours. There is an

example of the resulting image in Figure 4(b).

(

a) Pose-normalized image

(b) Digitized image

(Points and directions)

(d) Minutiae-compatible

Feature se

Figure 4: Feature extraction step of our proposed method

to extract minutiae-compatible feature set from pose-

normalized rind pattern image.

Next, our proposed method was used to extract

image features that were compatible with the

‘minutiae’. A minutia consists of the location of a

feature point and the direction attached to it. Pixels

MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind

Pattern

701

on borders are traced to calculate the curvature to

extract such features. If the curvature takes a local

maximum that is larger than the predetermined

threshold, the pixel is extracted as a feature point.

The normal direction of the border at the feature

point is determined to be the ‘direction’ of the

feature point (see Figure 4(c)). Tracing all the

borders of the rind pattern yields a set of hundreds of

feature points and their directions, as shown in

Figure 4(d). The set is compatible with ‘minutiae’.

The feature points are extracted from a region

that looks like a doughnut, as seen in Figure 4(d).

The stem point region is omitted because it has few

feature points that looks similar all over individuals.

2.3 Matching Image Features with

Fingerprint Matching Technique

This subsection describes the feature matching step

in our proposed method, where the similarity in pairs

of images is evaluated by matching their feature sets

using a state-of-the-art technique of fingerprint

matching.

We propose utilizing the conventional algorithm

originally proposed for fingerprint identification

using the accidental coincidence probability

(Monden, 2002), which is referred to as ACP in the

following. The object pose in our applications is

always unstable and varying numbers of feature

points are missing in every match. ACP offers hope

in such situations, because it has been shown to

output a stable similarity score regardless of missing

features. We have to ensure that ACP is based on the

assumption that there is no correlation between the

patterns of different individuals. Fortunately like

fingerprints, the features of rind patterns of melons

(and many other fruit) conform to this assumption,

as fingerprints do.

How the ACP algorithm evaluates the similarity

score is summarized here. Suppose that two feature

sets, P

and P

consisting N

i and

feature points,

respectively, are matched by determining their

common corresponding feature points, whose

locations and directions are sufficiently similar after

applying global affine transformation between the

two sets. It is assumed that any pair of feature points

in either set is more distant than 2d. The threshold of

the distance to determine corresponding feature

points is d.

When P

is a random pattern, i.e. N

feature points

are randomly located in the area of S, the probability

that n feature points out of

will accidentally

correspond to any of

feature points in P

, i.e., the

distance is less than d, is estimated as:

⎟

⎠

⎞

⎜

⎝

⎛

⎟

⎠

⎞

⎜

⎝

⎛

−

⎟

⎠

⎞

⎜

⎝

⎛

pit

NNNnp ),,;(

(5)

where

is an integer that is defined as:

⎥

⎤

⎢

⎡

(6)

Therefore, when P

is matched to any random

pattern P

, the probability that more than N

corresponding points will be accidentally determined

is estimated as:

∑

),min(

),,;(),,;(

pitpitfar

NNNnpNNNNp

(7)

The

far

in Equation 7 indicates ACP. Since the

smaller value for

far

leads to a higher likelihood of

the coincidence of patterns

and P

, we use (1-

far

) as the similarity score.

3 EXPERIMENTS

A total of 1,776 honeydew melons of the same strain,

i.e., it was difficult to identify individuals, were

collected to construct a large-scale image database

for the experiments (there are examples in Figure 5).

The experimental results obtained from

authenticating the melons indicated that the

proposed method of pose normalization made it

feasible to apply the technique of fingerprint

matching to match images of melons, i.e., 3D

objects. The proposed approach significantly

improved the accuracy of authentication.

3.1 Experimental Setup

The experiments simulated a realistic situation in

which a melon producer registered images of melons

into a database when they were shipped, and a

consumer took an image with the camera of his

mobile phone to authenticate the melon at a retail

store. Four main changes inevitably occur in the two

images to be matched for authentication in such

practical use:

 Camera device

 Pose of melon

 Illumination environment

 Melon’s colour and shape (subtle changes due

to passage of time in supply chain)

The image database consists of the two sets: the

VISAPP 2012 - International Conference on Computer Vision Theory and Applications

702

registration set and query set which is taken two or

three days later in a different location, by a different

camera. Photographers were directed to take images

from the tops of stem points to construct the image

database, but no instruments were used to control the

pose of the melon or the camera. A melon was

directly placed on a desk, and an image was

captured with a hand-held camera. Consequently,

the camera axes of images were slanted by 10

degrees on average from the melon's vertical axis.

Figure 5 has example photographs in the registration

and query sets. The two images in the same column

are of the same individual honeydew melon.

The resolution of the images was reduced to 640

x 640 at the beginning of the feature extraction step

(see Section 2.2).

One query image of an individual melon was

matched with one registration image of the same

fruit in the experiments and 1,775 registration

images of other individual melons. Consequently,

1,776 genuine pairs and 1,775 x 1,776 = 3,154,200

imposter pairs were matched in the experiments to

evaluate the accuracy of authentication using the

false accept rate (FAR) and false reject rate (FRR).

To evaluate the rank recognition accuracy, an error

was recorded if any of the imposter pairs including

the query individual had a higher score than a

genuine pair, which should have been ranked first.

3.2 Experimental Results

We compared our proposed approach with a

conventional method that applies fingerprint

matching without the pose normalization step

described in Section 2. The recognition accuracies

for the two methods are compared in Table 1 and

Figure 7. Figure 7 plots the ROC curves whose

horizontal axis indicates FAR and vertical axis

indicates FRR. Table 1 also lists the first-rank

matching error percentages (percentages of genuine

pairs whose scores were less than any of the

imposter pairs including each individual query).

The method without pose normalization resulted

in a first-rank matching error of 3.1% and an EER

(Equal Error Rate: the error rate (expressed as a

percentage) when the authentication threshold was

set so that FAR and FRR were equal) of 1.5%.

These accuracies were much inferior to those

reported in the studies of fingerprint matching

techniques, which indicate that such direct

application of fingerprint matching is not

sufficiently rigorous.

In contrast, the first-rank matching error rate

and EER (expressed as a percentage) were

drastically reduced to 0.06% using our proposed

pose normalization. Even when the authentication

threshold was so rigorous that FAR=1.E-6 (one error

in a million), FRR still remained quite low (0.06%).

This implies that our proposed method offers anti-

counterfeit checking that is so accurate that it only

allows one in a million fake products, with only

0.06% error in the authentication of genuine

products.

Consequently, the experimental results

demonstrated that our proposed method reduces the

error in authentication down to less than 1/50 that

induced by the direct application of fingerprint

matching without the pose normalization, and it

offers a practical error rate that is much lower than

the results obtained from benchmark tests of

fingerprint matching (Jain, 2007).

Two or three days later

(a) Registration image set

(b) Query image set

Figure 5: Image database of netted melons for the

experiments.

Table 1: Recognition accuracy of proposed and

conventional methods.

EER

FRR

@FAR1.e-6

Top-rank

ID Erro

Proposed

0.06% 0.06% 0.06%

w/o pose-

normalization

1.5% 5.6%

3.10%

0.00

0.01

0.02

0.03

0.04

0.05

0.0

1.E-6 1.E-5 1.E-4 1.E-3 1.E-2 1.E-1 1.E+0

Proposed

w/o 3D

pose norm .

False Accept Rate (log-scale)

False Reject Rate

(FAR, FRR)=

(1.E-6, 0.06%)

(1.E-6, 5.6%)

EER=0.06%

EER=1.5%

0.00

0.01

0.02

0.03

0.04

0.05

0.0

1.E-6 1.E-5 1.E-4 1.E-3 1.E-2 1.E-1 1.E+0

Proposed

w/o 3D

pose norm .

False Accept Rate (log-scale)

False Reject Rate

(FAR, FRR)=

(1.E-6, 0.06%)

(1.E-6, 5.6%)

EER=0.06%

EER=1.5%

Figure 6: ROC curves to show the recognition

performances of our proposed method (solid line) and

conventional method (dotted line).

MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind

Pattern

703

4 CONCLUSIONS

A new method of agri-biometric authentication was

proposed to identify and verify individual pieces of

fruit, using a single image of their rind patterns. We

also proposed an architecture that normalized the

rotations in depth of the target in the image,

extracted minutiae-compatible features from a pose-

normalized image, and then utilized a fingerprint

matching technique to match the feature sets of

images. The proposed architecture achieved

excellent recognition accuracy in experiments using

images of 1,776 honeydew melons.

Our new method enabled a traceability system to

be attained that could protect fruit from being

counterfeited without having to use wasteful and

costly anti-counterfeiting tags. It would also enable

anyone in a global supply chain to authenticate

registered fruit with a standard camera from

anywhere and at any time.

We chose melons as the first target in our

research on methods of agri-biometrics

authentication, because their 3D shape is simple and

their rind patterns have an abundance of features.

Our proposed architecture can be applied to various

other fruit and agri-products. The two main

requirements to apply our method are to model and

fit a standard 3D shape model of the target to the

image and extract the feature points and their

directions from the rind pattern. Since the accuracy

of recognition depends on uniqueness and the

number of features in the rind pattern, we intend to

investigate cases of other agri-products in future

work to extend the applications of agri-biometrics.

REFERENCES

Regattieri, A., Gamberi, M., Manzini, R., 2007.

Traceability of food products: General framework and

experimental evidence. In Journal of Food

Engineering, Volume 81, Issue 2, July 2007, Pages

347-356.

Bernardi, P., 2008. An anti-counterfeit mechanism for the

application layer in low-cost RFID devices. In

Proceedings of ECCSC 2008, 4th European

Conference on Circuits and Systems for

Communications, pp.227-231.

Lees, M., 2003. Food authenticity and traceability.

Woodhead Publishing, pp. 221--225.

Kondo, N., 2010. Automation on fruit and vegetable

grading system and food traceability, In Trends in

Food Science & Technology, Volume 21, Issue 3, pp.

145-152

Niigaki, H., Fukui, K., 2008. Classification of Similar 3D

Objects with Different Types of Features from Multi-

view Images. -An Approach to Classify 100 Apples-,

In Proceedings of PSIVT '09, the 3rd Pacific-Rim

Symposium on Advances in Image and Video

Technology, pp. 1046-1057.

Jain, A. K., Flynn, P. J., Ross, A. A., 2007. Handbook of

biometrics, Springer.

Otsu, N., 1979. A thresholding selection method from

gray-level histogram, In IEEE Transactions on

Systems, Man and Cybernetics, Vol. 9 pp. 62-66

Niblack, W., 1986. An Introduction to Image Processing,

Premtice-Hall, Englewood Cliffs, NJ, pp. 115-116.

Sezgin, M., Sankur, B., 2004. Survey over image

thresholding techniques and quantitative performance

evaluation, In J. Electron. Imaging 13, 146

Monden, A., Yoshimoto, S., 2002. Fingerprint

Identification Using the Accidental Coincidence

Probability, In Proceedings of IAPR Workshop on

Machine Vision Applications (MVA), pp.124-127.

VISAPP 2012 - International Conference on Computer Vision Theory and Applications

704