NEW METHODS FOR DISHWARE IDENTIFICATION
AND INSPECTION
Trung H. Duong and L. L. (Larry) Hoberock
School of Mechanical and Aerospace Engineering, Oklahoma State University, Stillwater, OK 74078, U.S.A.
Keywords: Dishware identification, Dishware inspection, Partitioning, Adaptive thresholding, Global thresholding.
Abstract: We propose automatically identifying dishes in mixed batches by using statistics of shape descriptors of
dish pieces. Experiments were conducted on 725 images of ceramic and plastic dishes taken in different
lighting conditions using different positions of 84 separate dishes of 5 different styles and shapes. In order
to find the minimum set of descriptors to produce fast, adaptable and efficient automatic dish recognition,
we employed several shape-based properties, including area, perimeter, ratio of length to width, extension,
and minimum bounding box, together with some properties based on gray level and color. For dish
inspection, we propose a new technique using partitioning and adaptive thresholding, combined with global
thresholding. For practical purposes, the algorithm should be fast, simple, and produce results invariant with
lighting conditions and dish rotation about the camera-dish axis. Such an algorithm is described in this
work. Matlab® R14 and Image Processing Toolbox V5.0 were used.
1 INTRODUCTION
Commercial dishwashing systems currently involve
human loading, sorting, inspecting, and unloading
dishes and silverware pieces before and after
washing in hot and humid environments. In such
difficult working conditions, leading to high turn-
over of low-paid employees, automation is desirable,
especially in large-scale kitchens of hospitals, navy
ships, schools, hotels and other dining facilities. Our
project is a part of developing an integrated machine
vision sorting and inspecting system for mixed dish
pieces and silverware exiting a flight-type
commercial dishwashing machine, coupled with
automatic loading and unloading.
Johnson (1993), working on the same dish set as
used in this project, employing area and radius of the
corner of the dish in using machine vision
identification of dish pieces exiting commercial
dishwashing machines. His method required an
invariant position of a dish under the camera axis,
which required a locating mechanism for each dish
piece. Even with pre-location, he reported poor
repeatability of results under small lighting
variations, such as those due to normal small
fluctuations in power supply voltage. For inspection
of dishes for cleanliness, his algorithm used simple
global thresholding, which is insufficient in accuracy
for actual implementation. We present a much
improved method for both dish identification and
inspection, yielding much better results.
Other investigators studied identification and
inspection of mixed silverware pieces exiting a
commercial dishwashing machine. Yeri (2002) used
blob analysis to recognize silverware pieces, using
indirect illuminaiton in a light tent to weaken
specular reflections. Lolla (2005) identified
silveware objects by their perimeters, symmetric and
asymmetric properties, and area moment of inertia
measurements. He used edge detection algorithms
together with template matching to inspect
recognized silverware pieces. Zhou (2008) proposed
an algorithm to recognize silverware pieces with
incomplete (truncated) images and a fusion-based
method for silverware inspection, producing very
good results.
The objective of this project is to develop
algorithms and procedures for on-line dishware
identification and inspection of certain types of dish
pieces exiting a commercial dishwasher. Figure 1
illustrates the overall process.
127
Duong T. and Hoberock L.
NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION.
DOI: 10.5220/0002169301270134
In Proceedings of the 6th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2009), page
ISBN: 978-989-674-000-9
Copyright
c
2009 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Figure 1: Processing Flow Chart.
2 EXPERIMENTAL APPROACH
2.1 Experimental Setup
The experimental setup, inherited from Zhou (2008)
is shown in Fig 2, with our modifications in lighting.
Figure 2: Experimental Setup.
After washing, it is anticipated that dish pieces
will be automatically placed on the conveyor.
However, in this project, dried dishes were placed
manually. While wet dishes can be easily handled
with this technique, dried dishes were much easier to
handle experimentally. A dish image is captured by
the camera when the dish is inside the light tent in
the camera field of view. In full implementation,
image taking will be triggered by appropriately
placed sensors, but in the work herein, this
triggering was done manually. This image is
transmitted to a computer for image processing. The
tent wall and curtains are used to eliminate
uncontrolled illumination from the outside
environment. In actual implementation, after
processing, a signal identifying the type of dish
piece will be sent to a sorting mechanism to sort the
dish into a stack of like dishes. Unidentified objects
will be automatically sent to a bin for such objects,
and if the dish is determined by the vision system to
be unclean, a signal will be sent to convey said dish
into a bin for re-washing, as indicated in Figure 1.
The camera used in this project was an area scan,
color digital industrial camera, Basler Co. model
A102kc, directly connected to an image processing
board in a personal computer, described later, for
real time image processing. Sensor size in the
camera was approximately 17 mm square.
Resolution was set to 1392 by 1038 pixels, which is
sufficient to discern a small dirty spot of SFS mm
square. The lens was a Fuji model CF35HA-1,
35mm focus length, with 14
o
26’x10
o
46’ aperture
view cone.
Let sensor size SS= 17mm, sensor resolution
SR= 1038 pixels, and focal length FL= 35mm.
Choose working distance WD= 600mm, which is
large enough to avoid distortion when the lens is
focused on an object within the working distance
(Zhou, 2008). Then we calculate the smallest feature
size, SFS, that the camera can detect (Edmund
Industrial Optics) by:
SR
FOV
SFS
×
=
2
(1)
where FOV= 200mm is the field of view, given
by
FL
WDSS
FOV
×
=
(2)
Subtituting FOV from (2) in (1) yields:
FLSR
WDSS
SFS
×
×
×
=
2
(3)
Using the above numerical values in (3) gives SFS=
0.38mm, which is sufficiently small for detection of
food particles in our project.
For inspection purposes, we desire an even
illumination across the dish piece, as well as a
minimum of specular reflections and shadows. The
white mat-finished cardboard forming the inside
surfaces of our light tent provided some diffuse
lighting of our dish pieces and reduction of specular
reflections, while the choice and placement of our
lights reduced shadowing. Acceptable lighting was
achieved by trial and error. After considerable
experimentation, we selected as our light sources
four 12V-20W DC halogen light bulbs surrounding
the dish piece and placed as indicated in Figure 2,
which provided sufficient illumination for both
ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics
128
identification and inspection. These lamps were
powered by a Switch Mode Power Supply (SMPS)
fed by 110V 60Hz building supply. Camera
exposure time was set at 20 µs, which was found by
experiment to yeild the best dish image details.
2.2 Dish Set
The dish set used in this project is shown in Fig 3.
Figure 3: Dish Set Consisting of 5 types and 3 Colors of
Dishes.
Our dish set was commercially available and
used by a large, 700 bed hospital in Oklahoma. It
consisted of 5 types and 3 colors of dishes. It was
selected not only because it is in wide commercial
use, but also because the colors, shape, and size of
different types of dishes are very similar. However,
each dish piece had uniform color, with no
decorative markings (i.e. they were “plain”). Other
commercially available “plain” dish sets present
much lower challenges for the identification
problem because their sizes are often easily
distinguishable. For easy reference, we name each
dish piece using size (large or small), and its
material or function (ceramic, plastic, or spacer).
Then LC and LP represent, respectively, the large
ceramic dish and the large plastic dish, while SC, SP
and SX represent, respectively, the small ceramic
dish, the small plastic dish, and the small plastic
spacer dish.
2.3 Pre-processing
We developed a pre-processing algorithm for
thresholding, computing areas, and choosing the
largest object (or particle) in a dish image, rejecting
all other particles. This process removed noise and
reliably retrieved a good dish image from the camera
image, which was then ready for identification and
inspection, illustrated in Figure 4.
Figure 4: Example of SX Pre-processing. Original camera
image (top-left); Binary mask for largest object (top-
right); Gray image (bottom-left); and Dish retrieved image
(bottom-right).
3 DISH IDENTIFICATION
The implemented automatic system should at least
be comparable in performance to what can be
achieved manually. Accordingly, from our
observations in actual commercial dishwashing
operations, our automatic system should be able to
accurately recognize and inspect 5 types of dishes in
real time at a minimum rate of 30 dishes per minute.
While this task is easily accomplished manually, it
poses a significant challenge for automation. We
desire an algorithm that works flawlessly under
varying dish positions beneath the camera and with
varying illumination.
3.1 Possible Approaches
A human can quickly recognize each type of dish
based on the weight, corner curvature, size, edge
pattern, color, and/or a mix of these properties. In
an attempt to imitate these capabilities, we
experimented with several approaches based on edge
detection, color recognition, and statistics of shape
descriptors, such as area and perimeter. When using
the area of a bounding box around the dish image
axes, we found poor repeatability, due to slight dish
non-alignment with the box (rotation), which
produced pixel counting errors. Methods based on
color or gray level intensity were found to be overly
sensitive to small lighting variations. Edge detection
methods, using Matlab and its Image Processing
Toolbox, were not only computationally expensive
and slow, but also proved difficult in selecting
appropriate threshold values (Duong, 2008).
NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION
129
3.2 Identification Algorithm
We propose using statistics of shape descriptors of
dish pieces to solve the identification problem.
Three easily calculated statistics of shape descriptors
are the dish image area, the ratio of dish image
length to width, and the ratio of the dish image area
to the area of the image oriented bounding box. The
area of the dish image is already available from the
pre-processing step.
In order to classify dish types, we used a training
set of 500 images, with 100 images for each dish
type, in varying position and orientation under the
camera, to estimate the distributions of dish image
properties. Examining the dish image area, we
observe from Figures 5 and 6 that two groupings
readily appear: large areas representing LC and LP,
and small areas representing SC, SP and SX. By
considering only dish area information, Table 1
shows that it is straightforward to identify SC, since
none of the image areas of SC overlap with any
other dish piece. However, there clearly is overlap in
image areas of LP and LC, and of SP and SX.
Figure 5: Small Dish Image Area Group.
Figure 6: Large Dish Image Area Group.
Table 1: The area contribution.
Dish type SX SP SC LP LC
Area
(10
4
pixels)
2.45-
2.58
2.53-
2.66
2.69-
2.83
4.96-
5.28
5.16-
5.51
Overlap
region
2.53- 2.58
SX or SP?
x
5.16- 5.28
LP or LC?
Accordingly, we employ other properties to
distinguish between them. Consider the ratio of dish
image area to the area of the oriented image
bounding box, called O_EXT, which can be thought
of as indicating the curvature of the dish corner. This
ratio is much faster to calculate, with more
repeatable results, than calculating the radius of
curvature of the dish corner, as attempted by
Johnson (1993). Consider further the ratio of
bounding box length to width, called O_REC, which
can be calculated quickly with repeatable results.
Using these two properties together, as indicated in
Figure 7, we can easily distinguish SP from SX.
Using area and length-to-width ratio, Figure 8
illustrates how LC is easily distinguished from LP.
The optimal lines to separate SP and SX, and LC
and LP are given in Table 2. To save time, the two
Figure 7: Separating SP and SX Dish Types.
Figure 8: Separating LC and LP Dish Types.
ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics
130
properties, O_REC and O_EXT, are computed only
if the area property is insufficient to make a reliable
decision. Otherwise, the algorithm stops at Step 1
below.
Table 2: Lines separating SP and SX, LC and LP.
Line to separate
SP and SX
Line to separate
LC and LP
Point 1 Point 2 Point 1 Point 2
Area
(10
4
pixels)
x x 5.22 5.13
O_REC 1.28 1.35 1.48 1.52
O_EXT 0.91 0.95 x x
Hence, our identification algorithm is as follows:
- Pre-processing: retrieve dish image as the
largest object in camera image.
- Step 1: Classify using dish area.
- Step 2: Separate SP and SX using O_REC
and O_EXT. Separate LC and LP using
O_REC and area.
3.3 Identification Results
Results were collected from 725 images of all types
of dish pieces, not including any of the 500 training
set images. All training and testing image sets were
produced from 84 dishes of all types, clean and
dirty, under different lighting conditions (produced
by changing the exposure time of the camera) and
under different dish positions and orientations under
the camera axis.
The results in Table 3 show accurate
identification for all images, with an average
computation time of 0.21 sec. This is deemed
acceptable to allow identification and inspection of
dishes at our target dish processing rate of 30 dish
pieces per minute. The variability from min to max
computation time is explained because the amount
of rotation among dish pieces varied with dish
position, causing variability in times to compute
classification parameters.
Table 3: Results of Dish Identification.
No. Correct
Time* (sec)
Min Average Max
LC 85 100% 0.18 0.22 0.57
LP 120 100% 0.18 0.33 0.59
SC 200 100% 0.17 0.18 0.23
SP 167 100% 0.17 0.20 0.49
SX 153 100% 0.16 1.24 0.48
All dishes 725 100% 0.16 0.21 0.59
(*) Matlab® R14, Image Processing Toolbox V5.0, Window
Vista, dual core 1.6GHz, 2GB RAM.
4 DISH INSPECTION
Automated dish inspection following commercial
dishwashing using image processing presents some
unique challenges. First, the intensity of dish images
is sensitive to changes in lighting, normal power
fluctuations, and camera sensitivity drift (Lolla,
2005). Second, even with reasonable attempts to
establish uniform illumination of dish pieces, uneven
illumination persists in the camera field of view.
This non-uniform color and gray intensity across a
clean dish varies as the position of the dish varies in
the field of view. Third, because of the non-flat
geometry of the dish surface, the gray intensity of
the image drops significantly at the dish side wall,
especially for a deep dish with steep sidewalls, such
as LC, SC and SX. Moreover, glare and shadows
increase the difficulty of discerning clean from dirty
dishes, even for human manual inspection. Fourth,
food particle images vary in gray level, depending
on food type, size, and location. Certain food
particles, such as dried egg yolk, can be especially
difficult to detect. Fifth, the definitions of a “clean
dish” and a “dirty dish” are subjective and ill-
defined (Zhou, 2008).
4.1 Previous Work
Zhou (2008) proposed a fusion based technique for
silverware inspection. His key idea was based on the
observation that shadows will move, but dirt will
not, between two images of a silverware piece
captured at two different positions under fixed
illumination. Zhou’s technique combines relevant
information from two images, which reduces noise
and recovers information in regions obscured by
lighting glare and shadows. His method could be
used in pre-processing before inspection, as long as
the computation time is sufficiently small. After
fusion of two images of one piece, Zhou applied
simple global thresholding to the three color (R, G
and B) channels. While this approach worked well
for silverware, it will not work for dish inspection,
because the gray level of a clean spot on the dish
wall is comparable to or less than a dirty spot on the
dish floor, and Zhou used only global thresholding.
Lolla (2005) used template edge matching for
silverware inspection. This approach is not only time
consuming, but also suffers from lacking the ability
to deal effectively with glare and shadows.
One approach we considered was to create
targeted illumination on the dish walls based on their
inclination angle, and then apply global thresholding
to the entire modified image. The problem was that
NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION
131
modifying illumination of the wall was time
consuming, and was difficult to adapt to inside
corners. A more promising approach was to treat the
dish floor region and dish wall region differently
during image processing. This is the essence of our
proposed method, in which we use partitioning and
adaptive thresholding, which is much simpler and
more efficient.
4.2 Inspection Algorithm
Our inspection method is inspired by observing how
human eyes inspect a dish piece. Humans focus
locally, and are very sensitive to relative changing in
color or intensity. The human visual system also
easily learns, and appears to eliminate glare and
shadows, as well as adapts the background intensity.
In our inspection algorithm, a “dirty spot” is
defined as a connected component that is (1)
sufficiently dark, and (2) darker than the
immediately surrounding area. The algorithm first
separates the dish image into a dish floor region and
a dish wall region. Then for each region, we
automatically locate all components that are darker
than the near surrounding area, employing adaptive
thresholding, and discard those components that are
not sufficiently dark, employing global thresholding.
After these steps, we combine the floor and wall
regions and carry out post-processing to remove
small noise.
The steps of our inspection algorithm are as
follows:
- Identify the dish piece, so that we have a
template for partitioning.
- Start from a gray image of a dish piece which
is the result of pre-processing.
- Partitioning: Detect and separate floor region
and wall region of a dish image by using the
appropriate floor template image.
- Adaptive and global thresholding: work with
the floor region and wall region separately.
Adaptive finds potential spots; global retains
only those with gray levels less than the
global threshold.
- Combine the two regions.
- Post processing: remove tinny spots that most
likely are produced by noise rather than dirt.
Our experimental results showed that it is
unnecessary to use a color image, such that even
though we took color images with our camera, we
employed only the corresponding gray scale image.
A sample result of the partitioning process based on
a dish floor template is shown in Figure 9.
Figure 9: Partitioning process (SC Dish): Original Camera
Image (top-left), Gray Scale Pre-processed Dish Image
(top-right), Dish Floor Region (bottom-left), and Dish
Wall Region (bottom-right).
For the dish in Figure 9, results of adaptive
thresholding combined with global thresholding for
floor region and wall region are shown in Figure 10.
Figure 10: Adaptive and Global Thresholding for Floor
Region (top-left) and Wall Region (top-right).
Combination of the Two Regions and Post-processing
(bottom).
For all tested images in our inspection process, the
values of parameters for adaptive thresholding and
global thresholding, found by trial and error, are
given in Tables 4 and 5 respectively.
ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics
132
Table 4: Adaptive Thresholding Parameters.
Dish floor region Dish wall region
Window
size
(pixels)
Threshold
value
(%)
Window
size
(pixels)
Threshold
value
(%)
SC 60 3 30 5
SX 50 5 30 3
SP 50 5 12 3
Table 5: Global thresholding parameters.
Dish floor region
(gray level 0-255)
Dish wall region
(gray level 0-255)
Lower
bound
Upper
bound
Lower
bound
Upper
bound
SC 20 86 20 70
SX 20 99 20 66
SP 20 99 20 54
4.3 Inspection Results
To obtain experimental results with our proposed
inspection method, we manually applied real food
particles to a variety of our dish pieces, varying
them in type size, shape, and location of food. We
used egg yolk, fruit juice, and a variety of sauces,
including tomato-based sauces, all of which were
allowed to air dry before inspection.
Figure 11 shows an example of results, with the
original SC dirty dish image on the left and detected
dirt spot boundaries (on the right) overlaid on the
original dish image. All dirt spots with variation of
gray intensity and location were correctly detected.
Figure 11: Origin SC (left) and all dirt spots detected
(right).
Notice that the glare (specular reflection) due to the
shiny surface of the ceramic dish did not produce
spurious results.
Inspection results of a dirty SX dish and a clean
SX dish are presented in Figures 12 and 13,
respectively. Notice in Figure 12 the dirt spot
exactly on the boundary of the floor region and wall
region of the dish image (near the centre of the left
image). This location of a dirty spot could be a
significant detection challenge, yet our algorithm did
detect it.
Figure 12: Original SX (left) and All Dirt Spots Detected
(right).
Figure 13: Origin Clean SX (left) and Correctly Detected
Non-dirt (right).
One inspection result of a dirty SX dish is shown in
Figure 14. All dirt spots, even some that are right on
the inner edges of multilayers of the dish, are
correctly detected. This type of dish with 3 shallow
layers was the most difficult case to tune the
parameters of the inspection algorithm.
Figure 14: Original SP (left) and All Dirt Spots Detected
(right).
NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION
133
A summary of inspection results for all types of dish
pieces, SC, SP, SX, LC, and LP is given in Table 6.
Table 6: Summary of Inspection Results for Small Dishes.
Total Correct Miss Failed Alarm
Clean SC 5 4 - 1
Dirty SC 5 5 0 -
Clean SP 5 4 - 1
Dirty SP 5 5 0 -
Clean SX 5 5 - 0
Dirty SX 5 5 0 -
Clean LC 5 5 - 0
Dirty LC 5 5 0 -
Clean LP 5 4 - 1
Dirty LP 5 5 0 -
All dishes 50
47
(94%)
0
(0%)
3
(6%)
“Failed Alarm” in Table 6 means an incorrect result
that a clean dish is classified as dirty. Such results
would reduce production in a real inspection process
line because the clean dish will be sent back to be re-
washed when this was not necessary. On the other
hand, such a result is deemed superior to a dirty dish
that is classified clean, which would be unacceptable
for hygienic and other reasons.
For the results in Table 6, the average
computation time for the inspection process was
1.28 second per dish (using Matlab® R14, Image
Processing Toolbox V5.0, Window Vista, dual core
1.6GHz, 2GB RAM). If we add this to the average
identification time reported earlier, the total average
time required for both identification and inspection
processes for the small area dish group is
approximately 1.47 second, which is acceptable for
our target of processing 30 dish pieces per minute (2
seconds per piece).
5 CONCLUSIONS
In this study we successfully implemented new and
novel dishware identification and inspection
algorithms. The experimental results show that these
algorithms work well under variation of lighting
conditions caused by variation of dish position under
the camera axis. The algorithms implemented on a
standardly available PC were sufficiently fast for
real time processing at a minimum rate of 30 dish
pieces per minute. We also experimented with other
dish sets, including plain circular and oval shaped
plates, and small bowls. With small changes in few
parameters, the algorithms work equally well.
On the other hand, partitioning and adaptive
thresholding combined with global thresholding as
presented here will not work for dishes that have
colored or molded patterns on the surface. However,
because most dish sets used in large scale dinning
operations are non-textured and mono-colored with
uniform background, our procedure should be
widely applicable.
ACKNOWLEDGEMENTS
This work has been supported by a scholarship from
Ministry of Education and Training of Vietnam
(MOET- www.moet.gov.vn) and a research
assistantship from Oklahoma State University.
REFERENCES
Bernsen, J., 1986. Dynamic thresholding of gray level
images. In Proc. Intl. Conf. on Pattern Recognition,
pp. 1251–1255.
Duong, H., T., Hoberock, L., L., 2008. Dish Recognition
Based on Statistics of Shape Descriptors. XXVIII
Oklahoma AIAA/ASME Symposium, Tulsa, Oklahoma.
Edmund Industrial Optics. Fundamental Parameters of an
imaging system. http://www.edmundoptics.com/
techsupport/DisplayArticle.cfm?articleid=287
Fisher, R., Perkins, S., 2003. Adaptive Thresholding.
http://homepages.inf.ed.ac.uk/rbf/HIPR2/adpthrsh.htm
Johnson, K., A., 1993. Machine Vision Sorting and
Inspection in Commercial Automatic Dishwashing.
Oklahoma State University M.S. Thesis.
Lolla, V., 2005. Identification and Inspection of
Silverware Using Machine Vision. Oklahoma State
University M.S. Thesis.
Yeri, S., 2002. Classification of Silverware Pieces Using
Machine Vision. Oklahoma State University Thesis.
Zhou, Y., and Hoberock, L. L., 2008. Flaw Detection
Using Image Registration and Fusion Techniques,
Proceedings of the ASME 2008 International Design
Engineering Technical Conference & Computers and
Information in Engineering Conference, Paper
DETC2008-49703, August 3 – 6, 2008, Brooklyn,
NY.
ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics
134