NEW METHODS FOR DISHWARE IDENTIFICATION

AND INSPECTION

Trung H. Duong and L. L. (Larry) Hoberock

School of Mechanical and Aerospace Engineering, Oklahoma State University, Stillwater, OK 74078, U.S.A.

Keywords: Dishware identification, Dishware inspection, Partitioning, Adaptive thresholding, Global thresholding.

Abstract: We propose automatically identifying dishes in mixed batches by using statistics of shape descriptors of

dish pieces. Experiments were conducted on 725 images of ceramic and plastic dishes taken in different

lighting conditions using different positions of 84 separate dishes of 5 different styles and shapes. In order

to find the minimum set of descriptors to produce fast, adaptable and efficient automatic dish recognition,

we employed several shape-based properties, including area, perimeter, ratio of length to width, extension,

and minimum bounding box, together with some properties based on gray level and color. For dish

inspection, we propose a new technique using partitioning and adaptive thresholding, combined with global

thresholding. For practical purposes, the algorithm should be fast, simple, and produce results invariant with

lighting conditions and dish rotation about the camera-dish axis. Such an algorithm is described in this

work. Matlab® R14 and Image Processing Toolbox V5.0 were used.

1 INTRODUCTION

Commercial dishwashing systems currently involve

human loading, sorting, inspecting, and unloading

dishes and silverware pieces before and after

washing in hot and humid environments. In such

difficult working conditions, leading to high turn-

over of low-paid employees, automation is desirable,

especially in large-scale kitchens of hospitals, navy

ships, schools, hotels and other dining facilities. Our

project is a part of developing an integrated machine

vision sorting and inspecting system for mixed dish

pieces and silverware exiting a flight-type

commercial dishwashing machine, coupled with

automatic loading and unloading.

Johnson (1993), working on the same dish set as

used in this project, employing area and radius of the

corner of the dish in using machine vision

identification of dish pieces exiting commercial

dishwashing machines. His method required an

invariant position of a dish under the camera axis,

which required a locating mechanism for each dish

piece. Even with pre-location, he reported poor

repeatability of results under small lighting

variations, such as those due to normal small

fluctuations in power supply voltage. For inspection

of dishes for cleanliness, his algorithm used simple

global thresholding, which is insufficient in accuracy

for actual implementation. We present a much

improved method for both dish identification and

inspection, yielding much better results.

Other investigators studied identification and

inspection of mixed silverware pieces exiting a

commercial dishwashing machine. Yeri (2002) used

blob analysis to recognize silverware pieces, using

indirect illuminaiton in a light tent to weaken

specular reflections. Lolla (2005) identified

silveware objects by their perimeters, symmetric and

asymmetric properties, and area moment of inertia

measurements. He used edge detection algorithms

together with template matching to inspect

recognized silverware pieces. Zhou (2008) proposed

an algorithm to recognize silverware pieces with

incomplete (truncated) images and a fusion-based

method for silverware inspection, producing very

good results.

The objective of this project is to develop

algorithms and procedures for on-line dishware

identification and inspection of certain types of dish

pieces exiting a commercial dishwasher. Figure 1

illustrates the overall process.

127

Duong T. and Hoberock L.

NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION.

DOI: 10.5220/0002169301270134

In Proceedings of the 6th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2009), page

ISBN: 978-989-674-000-9

Figure 1: Processing Flow Chart.

2 EXPERIMENTAL APPROACH

2.1 Experimental Setup

The experimental setup, inherited from Zhou (2008)

is shown in Fig 2, with our modifications in lighting.

Figure 2: Experimental Setup.

After washing, it is anticipated that dish pieces

will be automatically placed on the conveyor.

However, in this project, dried dishes were placed

manually. While wet dishes can be easily handled

with this technique, dried dishes were much easier to

handle experimentally. A dish image is captured by

the camera when the dish is inside the light tent in

the camera field of view. In full implementation,

image taking will be triggered by appropriately

placed sensors, but in the work herein, this

triggering was done manually. This image is

transmitted to a computer for image processing. The

tent wall and curtains are used to eliminate

uncontrolled illumination from the outside

environment. In actual implementation, after

processing, a signal identifying the type of dish

piece will be sent to a sorting mechanism to sort the

dish into a stack of like dishes. Unidentified objects

will be automatically sent to a bin for such objects,

and if the dish is determined by the vision system to

be unclean, a signal will be sent to convey said dish

into a bin for re-washing, as indicated in Figure 1.

The camera used in this project was an area scan,

color digital industrial camera, Basler Co. model

A102kc, directly connected to an image processing

board in a personal computer, described later, for

real time image processing. Sensor size in the

camera was approximately 17 mm square.

Resolution was set to 1392 by 1038 pixels, which is

sufficient to discern a small dirty spot of SFS mm

square. The lens was a Fuji model CF35HA-1,

35mm focus length, with 14

26’x10

46’ aperture

view cone.

Let sensor size SS= 17mm, sensor resolution

SR= 1038 pixels, and focal length FL= 35mm.

Choose working distance WD= 600mm, which is

large enough to avoid distortion when the lens is

focused on an object within the working distance

(Zhou, 2008). Then we calculate the smallest feature

size, SFS, that the camera can detect (Edmund

Industrial Optics) by:

FOV

SFS

(1)

where FOV= 200mm is the field of view, given

WDSS

FOV

(2)

Subtituting FOV from (2) in (1) yields:

FLSR

WDSS

SFS

(3)

Using the above numerical values in (3) gives SFS=

0.38mm, which is sufficiently small for detection of

food particles in our project.

For inspection purposes, we desire an even

illumination across the dish piece, as well as a

minimum of specular reflections and shadows. The

white mat-finished cardboard forming the inside

surfaces of our light tent provided some diffuse

lighting of our dish pieces and reduction of specular

reflections, while the choice and placement of our

lights reduced shadowing. Acceptable lighting was

achieved by trial and error. After considerable

experimentation, we selected as our light sources

four 12V-20W DC halogen light bulbs surrounding

the dish piece and placed as indicated in Figure 2,

which provided sufficient illumination for both

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

128

identification and inspection. These lamps were

fed by 110V 60Hz building supply. Camera

exposure time was set at 20 µs, which was found by

experiment to yeild the best dish image details.

2.2 Dish Set

The dish set used in this project is shown in Fig 3.

Figure 3: Dish Set Consisting of 5 types and 3 Colors of

Dishes.

Our dish set was commercially available and

used by a large, 700 bed hospital in Oklahoma. It

consisted of 5 types and 3 colors of dishes. It was

selected not only because it is in wide commercial

use, but also because the colors, shape, and size of

different types of dishes are very similar. However,

each dish piece had uniform color, with no

decorative markings (i.e. they were “plain”). Other

commercially available “plain” dish sets present

much lower challenges for the identification

problem because their sizes are often easily

distinguishable. For easy reference, we name each

dish piece using size (large or small), and its

material or function (ceramic, plastic, or spacer).

Then LC and LP represent, respectively, the large

ceramic dish and the large plastic dish, while SC, SP

and SX represent, respectively, the small ceramic

dish, the small plastic dish, and the small plastic

spacer dish.

2.3 Pre-processing

We developed a pre-processing algorithm for

thresholding, computing areas, and choosing the

largest object (or particle) in a dish image, rejecting

all other particles. This process removed noise and

reliably retrieved a good dish image from the camera

image, which was then ready for identification and

inspection, illustrated in Figure 4.

Figure 4: Example of SX Pre-processing. Original camera

image (top-left); Binary mask for largest object (top-

right); Gray image (bottom-left); and Dish retrieved image

(bottom-right).

3 DISH IDENTIFICATION

The implemented automatic system should at least

be comparable in performance to what can be

achieved manually. Accordingly, from our

observations in actual commercial dishwashing

operations, our automatic system should be able to

accurately recognize and inspect 5 types of dishes in

real time at a minimum rate of 30 dishes per minute.

While this task is easily accomplished manually, it

poses a significant challenge for automation. We

desire an algorithm that works flawlessly under

varying dish positions beneath the camera and with

varying illumination.

3.1 Possible Approaches

A human can quickly recognize each type of dish

based on the weight, corner curvature, size, edge

pattern, color, and/or a mix of these properties. In

an attempt to imitate these capabilities, we

experimented with several approaches based on edge

detection, color recognition, and statistics of shape

descriptors, such as area and perimeter. When using

the area of a bounding box around the dish image

axes, we found poor repeatability, due to slight dish

non-alignment with the box (rotation), which

produced pixel counting errors. Methods based on

color or gray level intensity were found to be overly

sensitive to small lighting variations. Edge detection

methods, using Matlab and its Image Processing

Toolbox, were not only computationally expensive

and slow, but also proved difficult in selecting

appropriate threshold values (Duong, 2008).

NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION

129

3.2 Identification Algorithm

We propose using statistics of shape descriptors of

dish pieces to solve the identification problem.

Three easily calculated statistics of shape descriptors

are the dish image area, the ratio of dish image

length to width, and the ratio of the dish image area

to the area of the image oriented bounding box. The

area of the dish image is already available from the

pre-processing step.

In order to classify dish types, we used a training

set of 500 images, with 100 images for each dish

type, in varying position and orientation under the

camera, to estimate the distributions of dish image

properties. Examining the dish image area, we

observe from Figures 5 and 6 that two groupings

readily appear: large areas representing LC and LP,

and small areas representing SC, SP and SX. By

considering only dish area information, Table 1

shows that it is straightforward to identify SC, since

none of the image areas of SC overlap with any

other dish piece. However, there clearly is overlap in

image areas of LP and LC, and of SP and SX.

Figure 5: Small Dish Image Area Group.

Figure 6: Large Dish Image Area Group.

Table 1: The area contribution.

Dish type SX SP SC LP LC

Area

(10

pixels)

2.45-

2.58

2.53-

2.66

2.69-

2.83

4.96-

5.28

5.16-

5.51

Overlap

region

2.53- 2.58

SX or SP?

5.16- 5.28

LP or LC?

Accordingly, we employ other properties to

distinguish between them. Consider the ratio of dish

image area to the area of the oriented image

bounding box, called O_EXT, which can be thought

of as indicating the curvature of the dish corner. This

ratio is much faster to calculate, with more

repeatable results, than calculating the radius of

curvature of the dish corner, as attempted by

Johnson (1993). Consider further the ratio of

bounding box length to width, called O_REC, which

can be calculated quickly with repeatable results.

Using these two properties together, as indicated in

Figure 7, we can easily distinguish SP from SX.

Using area and length-to-width ratio, Figure 8

illustrates how LC is easily distinguished from LP.

The optimal lines to separate SP and SX, and LC

and LP are given in Table 2. To save time, the two

Figure 7: Separating SP and SX Dish Types.

Figure 8: Separating LC and LP Dish Types.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

130

properties, O_REC and O_EXT, are computed only

if the area property is insufficient to make a reliable

decision. Otherwise, the algorithm stops at Step 1

below.

Table 2: Lines separating SP and SX, LC and LP.

Line to separate

SP and SX

Line to separate

LC and LP

Point 1 Point 2 Point 1 Point 2

Area

(10

pixels)

x x 5.22 5.13

O_REC 1.28 1.35 1.48 1.52

O_EXT 0.91 0.95 x x

Hence, our identification algorithm is as follows:

- Pre-processing: retrieve dish image as the

largest object in camera image.

- Step 1: Classify using dish area.

- Step 2: Separate SP and SX using O_REC

and O_EXT. Separate LC and LP using

O_REC and area.

3.3 Identification Results

Results were collected from 725 images of all types

of dish pieces, not including any of the 500 training

set images. All training and testing image sets were

produced from 84 dishes of all types, clean and

dirty, under different lighting conditions (produced

by changing the exposure time of the camera) and

under different dish positions and orientations under

the camera axis.

The results in Table 3 show accurate

identification for all images, with an average

computation time of 0.21 sec. This is deemed

acceptable to allow identification and inspection of

dishes at our target dish processing rate of 30 dish

pieces per minute. The variability from min to max

computation time is explained because the amount

of rotation among dish pieces varied with dish

position, causing variability in times to compute

classification parameters.

Table 3: Results of Dish Identification.

No. Correct

Time* (sec)

Min Average Max

LC 85 100% 0.18 0.22 0.57

LP 120 100% 0.18 0.33 0.59

SC 200 100% 0.17 0.18 0.23

SP 167 100% 0.17 0.20 0.49

SX 153 100% 0.16 1.24 0.48

All dishes 725 100% 0.16 0.21 0.59

(*) Matlab® R14, Image Processing Toolbox V5.0, Window

Vista, dual core 1.6GHz, 2GB RAM.

4 DISH INSPECTION

Automated dish inspection following commercial

dishwashing using image processing presents some

unique challenges. First, the intensity of dish images

is sensitive to changes in lighting, normal power

fluctuations, and camera sensitivity drift (Lolla,

2005). Second, even with reasonable attempts to

establish uniform illumination of dish pieces, uneven

illumination persists in the camera field of view.

This non-uniform color and gray intensity across a

clean dish varies as the position of the dish varies in

the field of view. Third, because of the non-flat

geometry of the dish surface, the gray intensity of

the image drops significantly at the dish side wall,

especially for a deep dish with steep sidewalls, such

as LC, SC and SX. Moreover, glare and shadows

increase the difficulty of discerning clean from dirty

dishes, even for human manual inspection. Fourth,

food particle images vary in gray level, depending

on food type, size, and location. Certain food

particles, such as dried egg yolk, can be especially

difficult to detect. Fifth, the definitions of a “clean

dish” and a “dirty dish” are subjective and ill-

defined (Zhou, 2008).

4.1 Previous Work

Zhou (2008) proposed a fusion based technique for

silverware inspection. His key idea was based on the

observation that shadows will move, but dirt will

not, between two images of a silverware piece

captured at two different positions under fixed

illumination. Zhou’s technique combines relevant

information from two images, which reduces noise

and recovers information in regions obscured by

lighting glare and shadows. His method could be

used in pre-processing before inspection, as long as

the computation time is sufficiently small. After

fusion of two images of one piece, Zhou applied

simple global thresholding to the three color (R, G

and B) channels. While this approach worked well

for silverware, it will not work for dish inspection,

because the gray level of a clean spot on the dish

wall is comparable to or less than a dirty spot on the

dish floor, and Zhou used only global thresholding.

Lolla (2005) used template edge matching for

silverware inspection. This approach is not only time

consuming, but also suffers from lacking the ability

to deal effectively with glare and shadows.

One approach we considered was to create

targeted illumination on the dish walls based on their

inclination angle, and then apply global thresholding

to the entire modified image. The problem was that

NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION

131

modifying illumination of the wall was time

consuming, and was difficult to adapt to inside

corners. A more promising approach was to treat the

dish floor region and dish wall region differently

during image processing. This is the essence of our

proposed method, in which we use partitioning and

adaptive thresholding, which is much simpler and

more efficient.

4.2 Inspection Algorithm

Our inspection method is inspired by observing how

human eyes inspect a dish piece. Humans focus

locally, and are very sensitive to relative changing in

color or intensity. The human visual system also

easily learns, and appears to eliminate glare and

shadows, as well as adapts the background intensity.

In our inspection algorithm, a “dirty spot” is

defined as a connected component that is (1)

sufficiently dark, and (2) darker than the

immediately surrounding area. The algorithm first

separates the dish image into a dish floor region and

a dish wall region. Then for each region, we

automatically locate all components that are darker

than the near surrounding area, employing adaptive

thresholding, and discard those components that are

not sufficiently dark, employing global thresholding.

After these steps, we combine the floor and wall

regions and carry out post-processing to remove

small noise.

The steps of our inspection algorithm are as

follows:

- Identify the dish piece, so that we have a

template for partitioning.

- Start from a gray image of a dish piece which

is the result of pre-processing.

- Partitioning: Detect and separate floor region

and wall region of a dish image by using the

appropriate floor template image.

- Adaptive and global thresholding: work with

the floor region and wall region separately.

Adaptive finds potential spots; global retains

only those with gray levels less than the

global threshold.

- Combine the two regions.

- Post processing: remove tinny spots that most

likely are produced by noise rather than dirt.

Our experimental results showed that it is

unnecessary to use a color image, such that even

though we took color images with our camera, we

employed only the corresponding gray scale image.

A sample result of the partitioning process based on

a dish floor template is shown in Figure 9.

Figure 9: Partitioning process (SC Dish): Original Camera

Image (top-left), Gray Scale Pre-processed Dish Image

(top-right), Dish Floor Region (bottom-left), and Dish

Wall Region (bottom-right).

For the dish in Figure 9, results of adaptive

thresholding combined with global thresholding for

floor region and wall region are shown in Figure 10.

Figure 10: Adaptive and Global Thresholding for Floor

Region (top-left) and Wall Region (top-right).

Combination of the Two Regions and Post-processing

(bottom).

For all tested images in our inspection process, the

values of parameters for adaptive thresholding and

global thresholding, found by trial and error, are

given in Tables 4 and 5 respectively.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

132

Table 4: Adaptive Thresholding Parameters.

Dish floor region Dish wall region

Window

size

(pixels)

Threshold

value

(%)

Window

size

(pixels)

Threshold

value

(%)

SC 60 3 30 5

SX 50 5 30 3

SP 50 5 12 3

Table 5: Global thresholding parameters.

Dish floor region

(gray level 0-255)

Dish wall region

(gray level 0-255)

Lower

bound

Upper

bound

Lower

bound

Upper

bound

SC 20 86 20 70

SX 20 99 20 66

SP 20 99 20 54

4.3 Inspection Results

To obtain experimental results with our proposed

inspection method, we manually applied real food

particles to a variety of our dish pieces, varying

them in type size, shape, and location of food. We

used egg yolk, fruit juice, and a variety of sauces,

including tomato-based sauces, all of which were

allowed to air dry before inspection.

Figure 11 shows an example of results, with the

original SC dirty dish image on the left and detected

dirt spot boundaries (on the right) overlaid on the

original dish image. All dirt spots with variation of

gray intensity and location were correctly detected.

Figure 11: Origin SC (left) and all dirt spots detected

(right).

Notice that the glare (specular reflection) due to the

shiny surface of the ceramic dish did not produce

spurious results.

Inspection results of a dirty SX dish and a clean

SX dish are presented in Figures 12 and 13,

respectively. Notice in Figure 12 the dirt spot

exactly on the boundary of the floor region and wall

region of the dish image (near the centre of the left

image). This location of a dirty spot could be a

significant detection challenge, yet our algorithm did

detect it.

Figure 12: Original SX (left) and All Dirt Spots Detected

(right).

Figure 13: Origin Clean SX (left) and Correctly Detected

Non-dirt (right).

One inspection result of a dirty SX dish is shown in

Figure 14. All dirt spots, even some that are right on

the inner edges of multilayers of the dish, are

correctly detected. This type of dish with 3 shallow

layers was the most difficult case to tune the

parameters of the inspection algorithm.

Figure 14: Original SP (left) and All Dirt Spots Detected

(right).

NEW METHODS FOR DISHWARE IDENTIFICATION AND INSPECTION

133

A summary of inspection results for all types of dish

pieces, SC, SP, SX, LC, and LP is given in Table 6.

Table 6: Summary of Inspection Results for Small Dishes.

Total Correct Miss Failed Alarm

Clean SC 5 4 - 1

Dirty SC 5 5 0 -

Clean SP 5 4 - 1

Dirty SP 5 5 0 -

Clean SX 5 5 - 0

Dirty SX 5 5 0 -

Clean LC 5 5 - 0

Dirty LC 5 5 0 -

Clean LP 5 4 - 1

Dirty LP 5 5 0 -

All dishes 50

(94%)

(0%)

(6%)

“Failed Alarm” in Table 6 means an incorrect result

that a clean dish is classified as dirty. Such results

would reduce production in a real inspection process

line because the clean dish will be sent back to be re-

washed when this was not necessary. On the other

hand, such a result is deemed superior to a dirty dish

that is classified clean, which would be unacceptable

for hygienic and other reasons.

For the results in Table 6, the average

computation time for the inspection process was

1.28 second per dish (using Matlab® R14, Image

Processing Toolbox V5.0, Window Vista, dual core

1.6GHz, 2GB RAM). If we add this to the average

identification time reported earlier, the total average

time required for both identification and inspection

processes for the small area dish group is

approximately 1.47 second, which is acceptable for

our target of processing 30 dish pieces per minute (2

seconds per piece).

5 CONCLUSIONS

In this study we successfully implemented new and

novel dishware identification and inspection

algorithms. The experimental results show that these

algorithms work well under variation of lighting

conditions caused by variation of dish position under

the camera axis. The algorithms implemented on a

standardly available PC were sufficiently fast for

real time processing at a minimum rate of 30 dish

pieces per minute. We also experimented with other

dish sets, including plain circular and oval shaped

plates, and small bowls. With small changes in few

parameters, the algorithms work equally well.

On the other hand, partitioning and adaptive

thresholding combined with global thresholding as

presented here will not work for dishes that have

colored or molded patterns on the surface. However,

because most dish sets used in large scale dinning

operations are non-textured and mono-colored with

uniform background, our procedure should be

widely applicable.

ACKNOWLEDGEMENTS

This work has been supported by a scholarship from

Ministry of Education and Training of Vietnam

(MOET- www.moet.gov.vn) and a research

assistantship from Oklahoma State University.

REFERENCES

Bernsen, J., 1986. Dynamic thresholding of gray level

images. In Proc. Intl. Conf. on Pattern Recognition,

pp. 1251–1255.

Duong, H., T., Hoberock, L., L., 2008. Dish Recognition

Based on Statistics of Shape Descriptors. XXVIII

Oklahoma AIAA/ASME Symposium, Tulsa, Oklahoma.

Edmund Industrial Optics. Fundamental Parameters of an

imaging system. http://www.edmundoptics.com/

techsupport/DisplayArticle.cfm?articleid=287

Fisher, R., Perkins, S., 2003. Adaptive Thresholding.

http://homepages.inf.ed.ac.uk/rbf/HIPR2/adpthrsh.htm

Johnson, K., A., 1993. Machine Vision Sorting and

Inspection in Commercial Automatic Dishwashing.

Oklahoma State University M.S. Thesis.

Lolla, V., 2005. Identification and Inspection of

Silverware Using Machine Vision. Oklahoma State

University M.S. Thesis.

Yeri, S., 2002. Classification of Silverware Pieces Using

Machine Vision. Oklahoma State University Thesis.

Zhou, Y., and Hoberock, L. L., 2008. Flaw Detection

Using Image Registration and Fusion Techniques,

Proceedings of the ASME 2008 International Design

Engineering Technical Conference & Computers and

Information in Engineering Conference, Paper

DETC2008-49703, August 3 – 6, 2008, Brooklyn,

NY.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

134