MELON AUTHENTICATION BY AGRI-BIOMETRICS
Identifying Individual Fruits using a Single Image of Rind Pattern
Rui Ishiyama
1
, Yoichi Nakamura
2
, Akira Monden
2
, Lei Huang
3
and Seiji Yoshimoto
2
1
Media and Information Processing Research Laboratories, NEC Corporation,
1753, Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa 211-8666, Japan
2
Space Systems Division, NEC Corporation, 1-10, Nisshincho, Fuchu, Tokyo 183-8501, Japan
3
Software Purchasing Division, NEC Corporation, 5-7-1 Shiba, Minato-ku, Tokyo 108-8001, Japan
Keywords: Agri-Biometrics, Authentication, Fruits, Melon, Identification, Verification, Pose, Feature Points, Minutiae,
Fingerprint Matching, Image.
Abstract: We propose a new method of biometric authentication, called agri-biometrics that identifies an individual
fruit using a single image of its rind patterns. Our proposed method normalizes the rotations in depth of the
fruit and extracts a set of image features, which are compatible to the 'minutiae', from the normalized image;
thus, it enables us to apply a state-of-the-art technique of fingerprint matching to identify the rind patterns of
fruit. We conducted large-scale experiments to identify/verify 1,776 individual melons in practical
situations where the images were taken under different pose and illumination conditions on different days.
Our method in the experiments achieved excellent recognition of EER=0.06%. The agri-biometric
authentication we propose accomplishes 'verifiable' agri-food traceability and brand protection; once the
producers register pictures of their products into the database, anyone can verify the products on hand with
the camera of a mobile phone.
1 INTRODUCTION
1.1 Product Identification for
Traceability and Anti-Counterfeit
There have been growing demands for agri-products
that have diverse added values including those
originating from branded varieties, well-known
growing districts, premium grades, and organic
cultivation. Advertising these added values is now
important for marketing, and gaining the trust of
consumers is mandatory to sell products at high
prices. The problem is that consumers, and even
retailers and traders, cannot find these added values
by just looking at the products themselves. Thus,
product information has more impact on the price
than the product itself, and this explains why
counterfeiting is so enticing.
Increasing problems with counterfeited and fake
products are being reported as the supply chain
expands globally and internationally. These not only
misappropriate revenues from the sales of genuine
producers, but they can also have significant
consequences on consumers. Consumers are
currently having to pay a great deal of attention to
traceability, which is the ability to trace the history
of a product through all its stages of production,
processing, and distribution.
Traceability is established based on methods of
identifying and verifying individual examples of the
product. We not only need to search databases, but
also to verify that individual products on hand are
genuine and not counterfeits. The way products are
authenticated, identified, and verified determines the
accessibility and anti-counterfeit capabilities of
traceability systems.
The traditional way of identifying products is by
attaching tags that directly display product
information or serial numbers. Barcodes and RFIDs
have recently been used to improve accessibility
(Regattieri, 2007). As barcodes can be read by
cameras installed on standard mobile phones, these
offer greater access to consumers to obtain detailed
information on products through the Internet.
Although these technologies may provide
increasingly more information, which may ease
consumer confidence, the risk of counterfeiting is
not reduced.
RFIDs and hologram tags have been proposed as
698
Ishiyama R., Nakamura Y., Monden A., Huang L. and Yoshimoto S..
MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind Pattern.
DOI: 10.5220/0003842706980704
In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 698-704
ISBN: 978-989-8565-03-7
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
technologies to combat counterfeiting (Bernardi,
2008). Chromatography and DNA analysis
techniques have been conducted to inspect agri-
products themselves to prevent counterfeiting (Lees,
2003). Numerous anti-counterfeit technologies have
been utilized, and surveillance has been conducted
by public institutions. However, counterfeiting of
various agri-products is increasingly being reported.
1.2 Problems in Traceability of
Agri-Product
Existing methods encounter two main difficulties in
being effective deterrents against counterfeiting:
Cost of tagging
Anti-counterfeiting tags are inexorably
expensive and an enormous number of tags needs to
be attached to all agri-products. The risk of tags
being swapped cannot be avoided even after this
high cost is incurred.
Usability of verification
As inspecting tags and products require special
devices or skilled staff, only limited numbers of
products on the global market can be checked.
Consequently, counterfeiting is rarely discovered.
Consumers are not only unaware of anti-
counterfeiting measures but they do not want to pay
for these.
A novel method is required to solve these
problems so that agri-products can be authenticated
by anyone, anywhere, and at any time without
having to rely on costly tags or inspection
procedures.
1.3 Agri-Biometric Authentication
We propose a new methodology in this paper to
identify individual agri-products by having single
photographs taken of rind patterns (e.g., net, stripe,
and dot patterns on the rinds, see Figure 1) and by
matching these to an image database of
authenticated products. Since methods of
authenticating people using facial and fingerprint
features are called 'biometrics', we have called our
proposed method 'agri-biometrics'. The new method
authenticates the fruit bodies themselves through the
use of rind patterns, without the need to attach tags.
Rind patterns of fruit are generated depending on the
environment in which they are grown, and these are
unique to individual fruit. Even if fake fruit are
grown from the same seed and with the same
method of cultivation, creating an identical rind
pattern is supposed to be impossible. Thus, fake fruit
cannot be cultivated, at least not within reasonable
costs that would offset the expense of counterfeiting.
The key feature of the proposed method is that
only a single photograph is required that is taken
with handy standard cameras such those in mobile
phones to authenticate the individual fruit on hand
from the enormous amount of fruit on the market.
Producers in practical traceability systems register
images of shipped fruit into a database. As many
producers adopt automated systems for grading and
inspecting the quality of fruit (Kondo, 2010),
capturing images of individual fruit in a database
can easily be automated. If a traceability service to
match images with those in the database is provided
over the Internet, anyone can authenticate fruit using
his/her smartphone from everywhere and at any
time. As the whole market is monitored by everyone
at all times, counterfeiting is expected to be
effectively suppressed. Furthermore, as consumers
are able to check the products themselves, they
actually feel it is worth paying for added values.
Figure 1: Rind patterns of netted honeydew melon, water
melon, and green apple.
1.4 Previous Study and Proposed
Architecture
In the literature, a similar approach has been
reported. It identifies individual apples using
appearances of multiple images (Niigaki, 2009).
Since it requires numerous images to be taken for
each authentication to compensate for different
poses of apples, it is far from being a practical
application. We propose a new method that
normalizes pose variations to achieve authentication
using only a single image, and that utilizes
fingerprint matching technology to achieve
extremely accurate authentication. Figure 2 outlines
our new approach.
In our proposed method, a 3D model (sphere for
melons) approximates a fruit’s average shape to an
image and cancels out rotations in depth. This
simulates the same process as that with
fingerprinting, which also involves patterns on
curved 3D surfaces that are flattened onto a scanner;
the scanned image of the fingerprints does not
contain deformation due to rotations in depth.
The rind patterns of fruit differ greatly from
MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind
Pattern
699
fingerprints, but their features have the common
nature that feature points are located randomly for
each individual, which is different from facial
features. Our method extracts a feature set, which is
compatible to the 'minutiae' used to match
fingerprints, from a pose-normalized image. This
make it possible to utilize state-of-the-art methods of
fingerprint matching using minutiae features, whose
accuracy has been demonstrated to be sufficient
even for law enforcement applications (Jain, 2007).
In this study, which appears as the 1st report of
agri-biometrics research, netted honeydew melons
were chosen to be the targeted agri-product in this
study. There are numerous premium brands,
growing districts for melons, and the prices of
melons differ from $5 up to $100 depending on such
added value information. Thus the melon is
considered to be a typical example of the agri-
products having serious risks of counterfeits. We
actually identified thousands of melons in the
experiments to demonstrate high degrees of
accuracy in practical situations.
(Registration ) (Query)
Take a single image
Pose normalization
(Proposed method)
Feature extraction
(Compatible to minutiae)
Matching
(Using fingerprint matching algorithm)
Figure 2: Architecture of our proposed method to
authenticate the fruit using rind pattern image.
2 PROPOSED METHOD
A new method of authenticating fruit with agri-
biometrics that uses a single image is proposed in
this section. Our architecture utilizes the technique
of fingerprint matching to match the rind patterns of
fruit. It consists of three steps to make it feasible.
The first step solves the problem of variations in
object poses, which cannot be solved by
conventional techniques of fingerprint matching.
The second step extracts the feature set that is
minutiae-compatible from the rind pattern image.
The third step involves an extension of the
conventional technique of fingerprint matching.
2.1 Pose Normalization
The pose normalization step in our proposed method
is described in this subsection. As outlined in Figure
3, this step is used to take an image with the location
of its stem point as input, and it cancels out the
rotations in depth of the image. The contour of the
object is extracted in this step, and a standard 3D
shape model is fitted to the image. The input image
is mapped to the pose-normalized image with the
texture-mapping technique, which simulates an
image being taken by setting up the camera and the
fruit accurately in a predetermined normal pose.
The camera is modelled with weak perspective,
and a sphere is used as the standard 3D shape model
for melons (which is good approximation of the 3D
shape of melons, especially for the premium graded
honeydew melons we used). Let us denote the image
coordinates and the camera’s optical axis to
correspond to the x, y, and z-axes. The normal pose
is predetermined as the 'top view', where the melon's
axis piercing from its stem point to the base is
aligned with the z-axis.
Suppose that point P is on the sphere’s surface
and this corresponds to pixel (x
n
, y
n
) in the pose-
normalized image, and that
α
denotes the angle
between the line passing through the sphere’s center
to point P and the yz-plane. Also suppose that
β
denotes the angle between the line and xz-plane.
Here, the following equations are obtained.
x
n
= sin
α
, y
n
= cos
α
sin
β
.
(1)
If the sphere is rotated by angle
θ
around the y-
axis, P is moved to a pixel corresponding to a point
whose polar coordinates are (
α
+
θ
,
β
) on the sphere;
its image coordinates (x
t
, y
t
) are then obtained as:
x
t
= sin(
α
+
θ
), y
t
= cos(
α
+
θ
) sin
β
.
(2)
If the sphere is additionally rotated by angle
ϕ
around the z-axis, the image pixel corresponding
to P is moved to (x
s
, y
s
), which is obtained as:
x
s
= x
t
cos
- y
t
sin
ϕ
, y
s
= x
t
sin
ϕ
+ y
t
cos
ϕ
.
(3)
Here, the contour of the melon is detected as a
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
700
θ
ϕ
x
n
y
n
(a) Input image with
Stem point
(b) Circle detection
(
c
)
Estimate rotations in de
p
t
h
(
d
)
Pose-normalized Ima
g
e
(p
x
, p
y
)
x
s
y
s
Figure 3: Pose normalization step in our proposed method
to cancel out rotations in depth.
circle. For simplicity, we assume that the
background of the image is dark monotone, thus the
melon’s body region is detected by applying the
following well-known image processing algorithms:
Otsu’s digitization (Otsu, 1979) for the whole image,
Canny edge detector, and Hough transform for circle
finding (Figure 3(b)). In this study, the stem point is
manually inputted. If its automation is necessary for
the applications, numerous pattern-finding methods
can be applied.
The original image is translated and scaled so
that the detected circle is located at the origin and
has a unit radius (see Figure 3(c)). If the stem point
is located at (p
x
, p
y
), the rotations
θ
and
ϕ
, are
estimated as:
,sin
22
yx
pp +=
θ
xy
pp=
ϕ
tan
(4)
The pose-normalized image (Figure 3(d)) is
generated by calculating (x
s
, y
s
) for each pixel (x
n
,
y
n
) and the pixel values are mapped from the input
image. Although the input image has arbitrary
rotations in depth (Figure 3(a)), the rotations are
cancelled out in the pose-normalized image (Figure
3(d)).
2.2 Minutiae-Compatible Feature
Extraction
This subsection describes the feature extraction step
in our proposed method. As outlined in Figure 4, this
step is used to extract a set of feature points and their
directions. The feature set is compatible with the
‘minutiae’ that are widely used in fingerprint
matching, and it can be matched with the state-of-
the-art techniques of fingerprint matching.
First, the input image is digitized to extract the
contours of the melon’s netted rind pattern. Since the
shading on the melon’s surface differs locally and
depends on the illumination environments in which
the image was taken, locally adaptive threshold
(Niblack, 1986) was adopted in digitization (a
survey is given in (Sezgin, 2004)). This method
determines the thresholds for each of all pixels based
on the average and the variance in pixel intensities
of each neighbouring pixels. The threshold was
determined to be the local average plus the local
standard deviation multiplied by a predetermined
coefficient. The size of the neighbouring area and
the coefficient were fixed for the all images. The
fixed size and coefficient were determined in a
preliminary experiment using separated image
database to be sufficient to work well for extracting
the mesh rind pattern from any image taken in
general indoor environments.
After digitization, the image was filtered by
successive morphological operations of dilation and
erosion, and a median filter was used to remove
noise and to smooth the contours. There is an
example of the resulting image in Figure 4(b).
(
a) Pose-normalized image
(b) Digitized image
(c) Feature Extraction
(Points and directions)
(d) Minutiae-compatible
Feature se
t
Figure 4: Feature extraction step of our proposed method
to extract minutiae-compatible feature set from pose-
normalized rind pattern image.
Next, our proposed method was used to extract
image features that were compatible with the
‘minutiae’. A minutia consists of the location of a
feature point and the direction attached to it. Pixels
MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind
Pattern
701
on borders are traced to calculate the curvature to
extract such features. If the curvature takes a local
maximum that is larger than the predetermined
threshold, the pixel is extracted as a feature point.
The normal direction of the border at the feature
point is determined to be the ‘direction’ of the
feature point (see Figure 4(c)). Tracing all the
borders of the rind pattern yields a set of hundreds of
feature points and their directions, as shown in
Figure 4(d). The set is compatible with ‘minutiae’.
The feature points are extracted from a region
that looks like a doughnut, as seen in Figure 4(d).
The stem point region is omitted because it has few
feature points that looks similar all over individuals.
2.3 Matching Image Features with
Fingerprint Matching Technique
This subsection describes the feature matching step
in our proposed method, where the similarity in pairs
of images is evaluated by matching their feature sets
using a state-of-the-art technique of fingerprint
matching.
We propose utilizing the conventional algorithm
originally proposed for fingerprint identification
using the accidental coincidence probability
(Monden, 2002), which is referred to as ACP in the
following. The object pose in our applications is
always unstable and varying numbers of feature
points are missing in every match. ACP offers hope
in such situations, because it has been shown to
output a stable similarity score regardless of missing
features. We have to ensure that ACP is based on the
assumption that there is no correlation between the
patterns of different individuals. Fortunately like
fingerprints, the features of rind patterns of melons
(and many other fruit) conform to this assumption,
as fingerprints do.
How the ACP algorithm evaluates the similarity
score is summarized here. Suppose that two feature
sets, P
i
and P
t
consisting N
i and
N
t
feature points,
respectively, are matched by determining their
common corresponding feature points, whose
locations and directions are sufficiently similar after
applying global affine transformation between the
two sets. It is assumed that any pair of feature points
in either set is more distant than 2d. The threshold of
the distance to determine corresponding feature
points is d.
When P
i
is a random pattern, i.e. N
i
feature points
are randomly located in the area of S, the probability
that n feature points out of
N
i
will accidentally
correspond to any of
N
t
feature points in P
t
, i.e., the
distance is less than d, is estimated as:
=
i
p
i
tp
t
pit
N
N
nN
NN
n
N
NNNnp ),,;(
,
(5)
where
N
p
is an integer that is defined as:
=
2
d
S
N
p
π
(6)
Therefore, when P
t
is matched to any random
pattern P
i
, the probability that more than N
corresponding points will be accidentally determined
is estimated as:
=
=
),min(
),,;(),,;(
it
NN
Nn
pitpitfar
NNNnpNNNNp
(7)
The
p
far
in Equation 7 indicates ACP. Since the
smaller value for
p
far
leads to a higher likelihood of
the coincidence of patterns
P
t
and P
i
, we use (1-
p
far
) as the similarity score.
3 EXPERIMENTS
A total of 1,776 honeydew melons of the same strain,
i.e., it was difficult to identify individuals, were
collected to construct a large-scale image database
for the experiments (there are examples in Figure 5).
The experimental results obtained from
authenticating the melons indicated that the
proposed method of pose normalization made it
feasible to apply the technique of fingerprint
matching to match images of melons, i.e., 3D
objects. The proposed approach significantly
improved the accuracy of authentication.
3.1 Experimental Setup
The experiments simulated a realistic situation in
which a melon producer registered images of melons
into a database when they were shipped, and a
consumer took an image with the camera of his
mobile phone to authenticate the melon at a retail
store. Four main changes inevitably occur in the two
images to be matched for authentication in such
practical use:
Camera device
Pose of melon
Illumination environment
Melon’s colour and shape (subtle changes due
to passage of time in supply chain)
The image database consists of the two sets: the
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
702
registration set and query set which is taken two or
three days later in a different location, by a different
camera. Photographers were directed to take images
from the tops of stem points to construct the image
database, but no instruments were used to control the
pose of the melon or the camera. A melon was
directly placed on a desk, and an image was
captured with a hand-held camera. Consequently,
the camera axes of images were slanted by 10
degrees on average from the melon's vertical axis.
Figure 5 has example photographs in the registration
and query sets. The two images in the same column
are of the same individual honeydew melon.
The resolution of the images was reduced to 640
x 640 at the beginning of the feature extraction step
(see Section 2.2).
One query image of an individual melon was
matched with one registration image of the same
fruit in the experiments and 1,775 registration
images of other individual melons. Consequently,
1,776 genuine pairs and 1,775 x 1,776 = 3,154,200
imposter pairs were matched in the experiments to
evaluate the accuracy of authentication using the
false accept rate (FAR) and false reject rate (FRR).
To evaluate the rank recognition accuracy, an error
was recorded if any of the imposter pairs including
the query individual had a higher score than a
genuine pair, which should have been ranked first.
3.2 Experimental Results
We compared our proposed approach with a
conventional method that applies fingerprint
matching without the pose normalization step
described in Section 2. The recognition accuracies
for the two methods are compared in Table 1 and
Figure 7. Figure 7 plots the ROC curves whose
horizontal axis indicates FAR and vertical axis
indicates FRR. Table 1 also lists the first-rank
matching error percentages (percentages of genuine
pairs whose scores were less than any of the
imposter pairs including each individual query).
The method without pose normalization resulted
in a first-rank matching error of 3.1% and an EER
(Equal Error Rate: the error rate (expressed as a
percentage) when the authentication threshold was
set so that FAR and FRR were equal) of 1.5%.
These accuracies were much inferior to those
reported in the studies of fingerprint matching
techniques, which indicate that such direct
application of fingerprint matching is not
sufficiently rigorous.
In contrast, the first-rank matching error rate
and EER (expressed as a percentage) were
drastically reduced to 0.06% using our proposed
pose normalization. Even when the authentication
threshold was so rigorous that FAR=1.E-6 (one error
in a million), FRR still remained quite low (0.06%).
This implies that our proposed method offers anti-
counterfeit checking that is so accurate that it only
allows one in a million fake products, with only
0.06% error in the authentication of genuine
products.
Consequently, the experimental results
demonstrated that our proposed method reduces the
error in authentication down to less than 1/50 that
induced by the direct application of fingerprint
matching without the pose normalization, and it
offers a practical error rate that is much lower than
the results obtained from benchmark tests of
fingerprint matching (Jain, 2007).
Two or three days later
(a) Registration image set
(b) Query image set
Figure 5: Image database of netted melons for the
experiments.
Table 1: Recognition accuracy of proposed and
conventional methods.
EER
FRR
@FAR1.e-6
Top-rank
ID Erro
r
Proposed
0.06% 0.06% 0.06%
w/o pose-
normalization
1.5% 5.6%
3.10%
0.00
0.01
0.02
0.03
0.04
0.05
0.0
6
1.E-6 1.E-5 1.E-4 1.E-3 1.E-2 1.E-1 1.E+0
Proposed
w/o 3D
pose norm .
False Accept Rate (log-scale)
False Reject Rate
(FAR, FRR)=
(1.E-6, 0.06%)
(1.E-6, 5.6%)
EER=0.06%
EER=1.5%
0.00
0.01
0.02
0.03
0.04
0.05
0.0
6
1.E-6 1.E-5 1.E-4 1.E-3 1.E-2 1.E-1 1.E+0
Proposed
w/o 3D
pose norm .
False Accept Rate (log-scale)
False Reject Rate
(FAR, FRR)=
(1.E-6, 0.06%)
(1.E-6, 5.6%)
EER=0.06%
EER=1.5%
Figure 6: ROC curves to show the recognition
performances of our proposed method (solid line) and
conventional method (dotted line).
MELON AUTHENTICATION BY AGRI-BIOMETRICS - Identifying Individual Fruits using a Single Image of Rind
Pattern
703
4 CONCLUSIONS
A new method of agri-biometric authentication was
proposed to identify and verify individual pieces of
fruit, using a single image of their rind patterns. We
also proposed an architecture that normalized the
rotations in depth of the target in the image,
extracted minutiae-compatible features from a pose-
normalized image, and then utilized a fingerprint
matching technique to match the feature sets of
images. The proposed architecture achieved
excellent recognition accuracy in experiments using
images of 1,776 honeydew melons.
Our new method enabled a traceability system to
be attained that could protect fruit from being
counterfeited without having to use wasteful and
costly anti-counterfeiting tags. It would also enable
anyone in a global supply chain to authenticate
registered fruit with a standard camera from
anywhere and at any time.
We chose melons as the first target in our
research on methods of agri-biometrics
authentication, because their 3D shape is simple and
their rind patterns have an abundance of features.
Our proposed architecture can be applied to various
other fruit and agri-products. The two main
requirements to apply our method are to model and
fit a standard 3D shape model of the target to the
image and extract the feature points and their
directions from the rind pattern. Since the accuracy
of recognition depends on uniqueness and the
number of features in the rind pattern, we intend to
investigate cases of other agri-products in future
work to extend the applications of agri-biometrics.
REFERENCES
Regattieri, A., Gamberi, M., Manzini, R., 2007.
Traceability of food products: General framework and
experimental evidence. In Journal of Food
Engineering, Volume 81, Issue 2, July 2007, Pages
347-356.
Bernardi, P., 2008. An anti-counterfeit mechanism for the
application layer in low-cost RFID devices. In
Proceedings of ECCSC 2008, 4th European
Conference on Circuits and Systems for
Communications, pp.227-231.
Lees, M., 2003. Food authenticity and traceability.
Woodhead Publishing, pp. 221--225.
Kondo, N., 2010. Automation on fruit and vegetable
grading system and food traceability, In Trends in
Food Science & Technology, Volume 21, Issue 3, pp.
145-152
Niigaki, H., Fukui, K., 2008. Classification of Similar 3D
Objects with Different Types of Features from Multi-
view Images. -An Approach to Classify 100 Apples-,
In Proceedings of PSIVT '09, the 3rd Pacific-Rim
Symposium on Advances in Image and Video
Technology, pp. 1046-1057.
Jain, A. K., Flynn, P. J., Ross, A. A., 2007. Handbook of
biometrics, Springer.
Otsu, N., 1979. A thresholding selection method from
gray-level histogram, In IEEE Transactions on
Systems, Man and Cybernetics, Vol. 9 pp. 62-66
Niblack, W., 1986. An Introduction to Image Processing,
Premtice-Hall, Englewood Cliffs, NJ, pp. 115-116.
Sezgin, M., Sankur, B., 2004. Survey over image
thresholding techniques and quantitative performance
evaluation, In J. Electron. Imaging 13, 146
Monden, A., Yoshimoto, S., 2002. Fingerprint
Identification Using the Accidental Coincidence
Probability, In Proceedings of IAPR Workshop on
Machine Vision Applications (MVA), pp.124-127.
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
704