Color Feature-based Pillbox Image Color Recognition
Peng Guo
1
, Ronald J. Stanley
1
, Justin G. Cole
2
, Jason Hagerty
1,2
and William V. Stoecker
2
1
Missouri University of Science and Technology, 201 N. State St, Rolla, MO, U.S.A.
2
Stoecker & Associates, Rolla, MO, U.S.A.
{pgp49, stanleyj, jrh55c, wvs, jgcpp9}@mst.edu, jgcole@iu.edu
Keywords: Pillbox Image, Color Recognition, Support Vector Machine, Image Processing.
Abstract: Patients, their families and caregivers routinely examine pills for medication identification. Key pill
information includes color, shape, size and pill imprint. The pill can then be identified using an online pill
database. This process is time-consuming and error prone, leading researchers to develop techniques for
automatic pill identification. Pill color may be the pill feature that contributes most to automatic pill
identification. In this research, we investigate features from two color planes: red, green and blue (RGB),
and hue saturation and value (HSV), as well as chromaticity and brightness features. Color-based
classification is explored using MatLab over 2140 National Library of Medicine (NLM) Pillbox reference
images using 20 feature descriptors. The pill region is extracted using image processing techniques
including erosion, dilation and thresholding. Using a leave-one-image-out approach for classifier
training/testing, a support vector machine (SVM) classifier yielded an average accuracy over 12 categories
as high as 97.90%.
1 INTRODUCTION
The use of prescription drugs is increasing generally,
especially among older persons, who are often
burdened with polypharmacy. (Gu, Dillon, & Burt,
2010; Schumock et al., 2015; Thielke et al., 2010).
Almost 60% of adults took prescription pills in 2012,
a figure which was only 50% in 2000 (Kantor, Rehm,
Haas, Chan, & Giovannucci, 2015). The possibility
of pill misidentification, and possible adverse drug
events, has therefore increased. Automatic pill
identification would help reduce the possibility of
pill misidentification. Because there are so many
different medications and generic varieties of each
medication, it would be extremely difficult for
anyone to identify all pills, without specific
background knowledge. The National Library of
Medicine (NLM) hosted a Pill Image Recognition
Challenge as part of its research and development in
Computational Photography Project for Pill
Identification (C3PI). In this research, we analyze
the pill images presented in this challenge, using
color features and a support vector machine (SVM)
learning algorithm.
The research began with the pill recognition
aspect of the project. Utilizing the NLM curated
Pillbox images, which also included metadata
information of the physical attributes of each pill,
enabled the development of a baseline recognition
algorithm which performed well under controlled
conditions. Generalization for the algorithm required
accounting for real-world factors such as lighting
condition, camera resolution, and non-homogeneous
backgrounds.
Previous work in this domain (Madsen et al,
2013) and (Wan et al., 2015) yielded good results,
but were limited to using images of similar quality as
the Pillbox images. In this study, we expand the
number of target color to be recognized from 7 to 12
and increase the number of Pillbox images to
approximately 2100.
Using an SVM classifier, we were able to
achieve a recognition accuracy based on 12 color
categories of 97.90%.
2 METHODOLOGY
The workflow of this research consisted of first
determining perceived five color component values
(red, green, blue, yellow, white) and twelve
perceived actual color values. There were 2151 high
resolution pill images in the Pillbox database as of
December 2014. The Pillbox images are of very high
quality, with high resolution, controlled illumination,
and uniform background. The high quality of the
188
Guo P., J. Stanley R., G. Cole J., Hagerty J. and V. Stoecker W.
Color Feature-based Pillbox Image Color Recognition.
DOI: 10.5220/0006136001880194
In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 188-194
ISBN: 978-989-758-225-7
Copyright
c
2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
database allows segmentation using a simple
threshold to separate the pill object from the
background for almost all pills. Using the
segmented pill, the color classification process was
performed using a four-step approach which is
outlined in Figure 1 and summarized as:
Step 1: Segment the pill from the Pillbox image.
Step 2: Determine the best axis to divide the
segmented pill image into two halves.
Step 3: Extract features from each half of the
segmented pill region.
Step 4: Classify the segments into a color
category (classification)
Figure 1: Pill color classifier development.
2.1 Segmentation of Pill Region
The Pillbox images are all dimensional similar; each
being 768×1024. Demonstrated in Figure 2, the
colored background, front and back of each pill are
included in the Pillbox images. Since the
background color can influence color detection it is
not used for pill color analysis. As such, a reliable
segmentation algorithm of pill from background is
needed.
Figure 2: Example of a Pillbox image.
The algorithm to segment the pill image is as
follows:
Step 1: Load red, green and blue (RGB) image
Step 2: Take a sample of the background pixels
(10 lines of pixels, shown in the Figure 2)
and calculate the mean intensity of
background.
Step 3: Threshold the image based on the
calculated mean intensity value
Step 4: Repeat Step 2-3 for Green and Blue color
channels
Step 5: Use erosion to eliminate noise and
dilation to fill holes
Step 6: Combine the binary mask that was
generated for each color plane using a
union operation.
Because the pill color is uniform on both sides of
the pill in this data set, only one side of the
segmented pill region (top-left side was arbitrarily
chosen) is used for feature analysis. The
segmentation of pill region is shown in Figure 3,
with background color set to be black (pixel value is
0 in RGB color space) for convenience of feature
calculation shown later.
Figure 3: Segmentation of pill region.
2.2 Creation of Vertical Segments of
Pill Region
In this process, the segmented pill region obtained in
the previous step is divided into two vertical
segments, left segment and right segment. The
reason for creating two vertical segments instead of
analyzing the whole segmented pill region is that
some pills have two or more colors (see Figure 4).
To avoid classification errors that would occur with
whole-pill color analysis, the pill is divided into
vertical two segments.
After the pill region of interest is obtained
through segmentation, a vertical central axis needs
to be located. The central axis is simply defined as
the middle column in the segmented pill region
because of the accuracy of the segmentation. For a
segmentation mask image of size, the middle
column is defined as column
/2
. The central axis
location and vertical segments can be viewed in
Figure 5.
Segment pill region in original image
Creating vertical segments of left and right part
Feature extraction
Color classification
Color Feature-based Pillbox Image Color Recognition
189
Figure 4: A two-colored Pillbox image.
Figure 5: Central axis and left/right vertical segments.
2.3 Feature Extraction
For each vertical segment of the pill region image,
there are five different categories of color features
computed from the segmented pills, including: (1)
RGB intensity, (2) HSV intensity, (3) Chromaticity,
(4) Left-Right averages, and (5) Brightness. An
overview of the features extracted for each feature
category is presented in Table 1. The algorithms for
computing each feature are given in detail following
Table 1.
Table 1: 20 Color Features.
Feature
Category
Labe
l
Measure Description
RGB Mean
And Std. Dev.
F1 Red intensity Red, Green and
Blue color space
statistics
F2 Green
intensity
F3 Blue
intensity
F4 Std. Dev.
Red intensity
F5 Std. Dev.
Green
intensity
F6 Std. Dev.
Blue
intensity
HSV Mean
And Std. Dev.
F7 Hue Mean Hue, Saturation
and Value color
space statistics
F8 Value Mean
F9 Saturation
Mean
F10 Hue Std.
Dev.
F11 Value Std.
Dev.
F12 Saturation
Std. Dev.
Chromaticity
Means
F13 Red
chromaticity
Mean
Chromaticity
measures color
quality and
overall values of
brightness.
Red chromaticity
= R/(R+G+B);
green and blue
are similar.
Chromaticity
combines the
features of
saturation (color
purity) and
hue (color
specificity) in a
single value.
F14 Green
chromaticity
Mean
F15 Blue
chromaticity
Mean
F16 Yellow
chromaticity
Mean
Combination F17 RGA Averages of
colors for each
two-color
combination.
F18 GBA
F19 BRA
Brightness F20 Intensity

3
2.3.1 RGB Intensity
The RGB color model is additive in the sense that
the three primary color spectra are added together,
wavelength for wavelength, to make the final color
spectrum (Boughen & Gross, 2003; Poynton, 2003).
After the pill region image is obtained, the color
image is decomposed into red, green and blue
images (figure 6). The background pixels are set to 0.
All nonzero pixel values indicate pill region pixels;
for this region, SumRed, SumGreen and SumBlue
represent the summation of intensity values for pill
region pixels. The number of pill region pixels is
found by NumRed, NumGreen and NumBlue.
(a) (b) (c)
Figure 6: The grayscale images of pill region. a) red
channel, b) green channel, c) blue channel.
In each of the three channels, the mean of red
intensity (F1), green intensity (F2) and blue intensity
(F3) are calculated according to the equations 1-3.
 


(1)
 


(2)
VISAPP 2017 - International Conference on Computer Vision Theory and Applications
190



(3)
The standard deviation for the three channels of
the RGB color space, denoted as Std Red (F4), Std
Green (F5) and Std Blue (F6) are calculated using
the Matlab® function “std.”
2.3.2 HSV Intensity
The hue, saturation and value (HSV) color space
represents colors in a way that is similar to the way
that humans perceive colors. HSV attempts to the
separate chroma and luminance such a particular hue
is the same independent of luminance. The
conversion from RGB to HSV color space is
provided by Matlab® with the function “rgb2hsv”
which is detailed on Matlab official document
webpage (MathWorks, 2016). The given pixel value
represents the current pixel parameter (hue,
saturation or value), as the entire image is distributed
in these three channels. Figure 7 gives an example of
a pill region in HSV color space.
(a) (b) (c)
Figure 7: Illustration of grayscale image of HSV color
space. a) hue, b) saturation, c) value.
The pixel value sums for each of the three
channels, SumHue, SumSat and SumVal, can be
obtained by summing the non-zero pixels (black
pixels also have 0 values in HSV color space). The
counting of pill region pixels NumHue, NumSat and
NumVal is similar to RGB pixel counting. Hence,
the HSV feature are also extracted using equations
4-6, similar to RGB feature extraction.



(4)



(5)



(6)
Note that the HSV calculations are also
accomplished with the Matlab® function “std,” and
are processed in all three HSV color space channels.
2.3.3 Chromaticity and Brightness Features
For each channel of the HSV color space image,
along with red, yellow, and blue chromaticity (See
Equations 7-10) features were calculated for every
pill. As an objective specification of the quality of a
color regardless of its luminance, chromaticity can
give us another view of the color being recognized
and help in the color classification. And for this
category of feature, four chromaticity features are
defined: red, green, blue and yellow chromaticity.
Each of these features represents a different aspect
of color according to the color space we have here.
Chromaticity features were among the most useful
features in previous works on pill color recognition
(Lee, Park, Jain, & Lee, 2012; Madsen, Payne,
Hagerty, Szanto, Moss, Wronkiewicz, Stoecker,
2013; Wan, Woods, Salgado-Montejo, Velasco, &
Spence, 2015).

(7)

(8)

(9)

2
(10)
The brightness feature F20 is calculated as the
average of the brightness in the red, green and blue
channels (equation 11). Brightness is needed to
better classify achromic colors, such as gray and
white


3
(11)
3 CLASSIFICATION
The left and right pill region vertical segments each
have a specific color category. In the classification
process, the left and right vertical segments are
trained and tested separately in order to avoid the
issue of color mixture when treating the pill region
as a whole. However, the left part and right part in
classification steps are the same regardless of the
variation of color; hence, for the experiments, we
consider just the left part as an example.
The labels for vertical segments were assigned
manually by two of the authors (J.G.C. and P.G.)
according to the 12 FDA colors (Julie N. Barrows
Arthur L. Lipman, 2009) listed in table 3 below. The
Color Feature-based Pillbox Image Color Recognition
191
entire database is labelled in 12 groups with one
color in each. Take red for example, all the red pills
are labelled as “positive (1)” for red color; all other
pills are labelled “negative (0)” to complete the red
color labels; the same labelling is done for all 12
colors.
Table 3: FDA Pill Color Classification Labels.
FDA Pill Colors
Black Pink Gray Turquoise
Blue Purple Green White
Brown Red Orange Yellow
In the first stage, twenty features (F1-F20) were
extracted and used for color-based classification
based on a leave-one-out method. A Support Vector
Machine (SVM) classifier was investigated to take
the input of twenty feature columns for individual
pill classification (Cortes & Vapnik, 1995).
As 12 color are treated as the target color in this
study, 12 different binary classifiers are built, the
classification for a single color is carried out with
the following steps:
Step 1: Train the SVM classifier algorithm
using a leave-one-image-out approach. The
classifier is trained based on the left vertical
segment feature vectors for all except the
left-out pill image, which is used for testing.
Step 2: Classify the pill left-out pill test
image using the SVM classifier.
Step 3: Assign class labels (1 for target color
confirmed, 0 for not target color) to the test
segment image.
Step 4: Repeat steps 1-3 for all the segmented
images in the experimental data set.
Step 5: To finish classifications for all the 12
colors, Step 1 to Step 4 above are repeated
for all color labels (total 12 iterations).
For the SVM classifier, the LIBSVM(Chang &
Lin, 2011) implementation is employed in this paper.
This SVM tries to find an optimal hyperplane for
linear inseparable classes which acts as a decision
function to classify data in high dimensions. A linear
kernel is used for the SVM to update the penalty
parameter by a leave-one-image-out method, as
explained in the paper of (Guo et al., 2015), (De et
al., 2013). The implementation is completed with
Matlab® and presented in (Guo et al., 2015).
4 EXPERIMENTS PERFORMED
Twelve color categories are defined for the
classification target. The entire database of 2140
pillbox images are vertically segmented into two
groups, each group of pill region segments (2140 for
either left or right segments) will be assigned one of
twelve color labels automatically by the classifier.
The twelve target FDA color categories in Table 3
are manually assigned as the training and testing
targets by the author. In the classification process, a
leave-one-out approach is employed where 2139
images are used for training and the single left-out
image is tested.
In the scoring used for classification accuracy,
the percentage of rightly classified images is
calculated for every color category. If the class label
automatically assigned to the test image is the same
as the manual class label, then the image is
considered to be correctly labelled. Otherwise, the
image is considered to be incorrectly labelled.
4.1 Experimental Results and Analysis
As previously stated, we obtained the vertical
segment image classifications using the SVM
classifier with a leave-one-image-out approach
based on all the twenty features generated. Then the
vertical segment classifications are compared with
the target color truth label and finally calculated the
percentage of right classification accuracy. We
evaluated performance of these pill image
classifications using the three approaches that is
presented in Section 3. Table 4 shows the
classification results obtained with the SVM
classifier, for all twelve color categories.
Table 4: Classification results obtained for all color
categories.
Color
Red Green Blue Yellow Black White
Accuracy
98.46% 99.44% 99.21% 99.07% 100% 95.14%
Color
Gray Pink Cyan Purple Brown Orange
Accuracy
90% 100% 99.47% 99.39% 96.78% 97.85%
As can be observed in Table 4, the highest
accuracy of classification for all the color categories
are the black and pink color classification, which
both were 100% correct. Additionally, cyan has an
accuracy of 99.47% (2129/2140), green
classification is found to be 99.44% correct
(2128/2140), 2127/2140 of purple pills are classified
correctly, followed by blue follows with a
classification accuracy of 99.21% (2123/2140). For
yellow, 2120 pills are correctly recognized giving an
accuracy of 99.07% (2120/2140). Red is recognized
VISAPP 2017 - International Conference on Computer Vision Theory and Applications
192
in 98.46% (2107/2140) in classification and Orange
as 97.85% (2094/2140).
However, white and gray give confusing
classification results of 95.14% (2036/2140) and 90%
(1926/2140) since manual labelling consistency is
difficult for these colors with identical hues. After
adjusting Pillbox image labelling using the output
results, manually misclassified images can be
corrected, which improves classification accuracy to
as high as 99%. However, until labelling is further
investigated, the lower white and gray results are
used in calculating overall accuracy.
5 CONCLUSIONS
Under the idea of basic component colors for
classification, the features are extracted as the basis
of red, green and blue color related features.
Furthermore, the chromaticity, since it combines
saturation and hue (measuring color proportion over
all values of luminance), provides a simple model
for the color perceived by humans. Because of
uncertainty regarding labelling of white and gray,
performance ranged from over 98% for nine of the
colors to 97.85% for orange, 95.14% for white, and
90% for gray. Perfect accuracy (100%) is yielded as
the classification result for both cyan and pink color
Pillbox images. Overall, the classification accuracy
obtained from all the 12 color categories is 97.90%
which is higher than results obtained by (Madsen et
al, 2013) and (Wan et al., 2015).
Future research can be focused on principal
feature analysis to find the most significant features,
accuracy obtained from different current feature
groups. Additional data could enable classification
by unsupervised learning algorithms such as deep
learning. And, as to make the algorithm more
generalized and applicable in real world conditions,
a noise level study on the pill images should be
performed, taking into account non-optimal
condition such as uneven illumination, image blur
and heterogeneous backgrounds, etc.
As the first step in computing the visual content
in pill color recognition, we have already made
progress in reference image-based classification.
Additional research should also focus on imprint
identification, score marks and analysis of
consumer-quality pill identification, to enable
identification of pills under real-world conditions.
ACKNOWLEDGEMENTS
The images used in this research are obtained from
the NLM (National Library of Medicine) of National
Institutes of Health Pillbox image data base.
REFERENCES
Boughen, N., & Gross, J. (2003). Lightwave 3D 7.5
Lighting. book, Plano, TX, USA: Wordware
Publishing Inc.
Chang, C., & Lin, C. (2011). LIBSVM: A Library for
Support Vector Machines. ACM Transactions on
Intelligent Systems and Technology (TIST), 2, 1–39.
Cortes, C., & Vapnik, V. (1995). Support-vector networks.
Machine Learning, 20(3), 273–297.
De, S., Stanley, R. J., Lu, C., Long, R., Antani, S., Thoma,
G., & Zuna, R. (2013). A fusion-based approach for
uterine cervical cancer histology image classification.
Computerized Medical Imaging and Graphics, 37(7–
8), 475–487.
Gu, Q., Dillon, C. F., & Burt, V. L. (2010). Prescription
drug use continues to increase: U.S. prescription drug
data for 2007-2008. NCHS Data Brief, (42), 1–8.
Retrieved from
Guo, P., Banerjee, K., Stanley, R., Long, R., Antani, S.,
Thoma, G., … Stoecker, W. (2015). Nuclei-Based
Features for Uterine Cervical Cancer Histology Image
Analysis with Fusion-based Classification. IEEE
Journal of Biomedical and Health Informatics, (c).
Julie N. Barrows Arthur L. Lipman, C. J. B. (2009). Color
Additives: FDA’s Regulatory Process and Historical
Perspectives. book.
Kantor, E. D., Rehm, C. D., Haas, J. S., Chan, A. T., &
Giovannucci, E. L. (2015). Trends in Prescription
Drug Use Among Adults in the United States From
1999-2012. Jama, 314(17), 1818–1831.
Lee, Y. B., Park, U., Jain, A. K., & Lee, S. W. (2012).
Pill-ID: Matching and retrieval of drug pill images. In
Pattern Recognition Letters (Vol. 33, pp. 904–910).
Madsen, D. J., Payne, K. S., Hagerty, J. R., Szanto, N.,
Wronkiewicz, M., Moss, R.H., Stoecker. W.V. (2013).
Automatic Pill Identification from Pillbox Images. In:
8th International Conference on Computer Vision
Theory and Applications (VISAPP), 21-24 February
2013. 378-384.
Poynton, C. A. (2003). Digital video and HDTV:
algorithms and interfaces. Morgan Kaufmann
Publishers.
Schumock, G. T., Li, E. C., Suda, K. J., Wiest, M. D.,
Stubbings, J., Matusiak, L. M., … Vermeulen, L. C.
(2015). National trends in prescription drug
expenditures and projections for 2015. American
Journal of Health-System Pharmacy: AJHP: Official
Journal of the American Society of Health-System
Pharmacists, 72(9), 717–736.
Color Feature-based Pillbox Image Color Recognition
193
Thielke, S. M., Simoni-Wastila, L., Edlund, M. J.,
DeVries, A., Martin, B. C., Braden, J. B., … Sullivan,
M. D. (2010). Age and sex trends in long-term opioid
use in two large American health systems between
2000 and 2005. Pain Medicine (Malden, Mass.),
11(2), 248–56.
Wan, X., Woods, A. T., Salgado-Montejo, A., Velasco, C.,
& Spence, C. (2015). Assessing the expectations
associated with pharmaceutical pill colour and shape.
Food Quality and Preference, 45, 171–182.
VISAPP 2017 - International Conference on Computer Vision Theory and Applications
194