Color Feature-based Pillbox Image Color Recognition

Peng Guo

, Ronald J. Stanley

, Justin G. Cole

, Jason Hagerty

1,2

and William V. Stoecker

Missouri University of Science and Technology, 201 N. State St, Rolla, MO, U.S.A.

Stoecker & Associates, Rolla, MO, U.S.A.

{pgp49, stanleyj, jrh55c, wvs, jgcpp9}@mst.edu, jgcole@iu.edu

Keywords: Pillbox Image, Color Recognition, Support Vector Machine, Image Processing.

Abstract: Patients, their families and caregivers routinely examine pills for medication identification. Key pill

information includes color, shape, size and pill imprint. The pill can then be identified using an online pill

database. This process is time-consuming and error prone, leading researchers to develop techniques for

automatic pill identification. Pill color may be the pill feature that contributes most to automatic pill

identification. In this research, we investigate features from two color planes: red, green and blue (RGB),

and hue saturation and value (HSV), as well as chromaticity and brightness features. Color-based

classification is explored using MatLab over 2140 National Library of Medicine (NLM) Pillbox reference

images using 20 feature descriptors. The pill region is extracted using image processing techniques

including erosion, dilation and thresholding. Using a leave-one-image-out approach for classifier

training/testing, a support vector machine (SVM) classifier yielded an average accuracy over 12 categories

as high as 97.90%.

1 INTRODUCTION

The use of prescription drugs is increasing generally,

especially among older persons, who are often

burdened with polypharmacy. (Gu, Dillon, & Burt,

2010; Schumock et al., 2015; Thielke et al., 2010).

Almost 60% of adults took prescription pills in 2012,

a figure which was only 50% in 2000 (Kantor, Rehm,

Haas, Chan, & Giovannucci, 2015). The possibility

of pill misidentification, and possible adverse drug

events, has therefore increased. Automatic pill

identification would help reduce the possibility of

pill misidentification. Because there are so many

different medications and generic varieties of each

medication, it would be extremely difficult for

anyone to identify all pills, without specific

background knowledge. The National Library of

Medicine (NLM) hosted a Pill Image Recognition

Challenge as part of its research and development in

Computational Photography Project for Pill

Identification (C3PI). In this research, we analyze

the pill images presented in this challenge, using

color features and a support vector machine (SVM)

learning algorithm.

The research began with the pill recognition

aspect of the project. Utilizing the NLM curated

Pillbox images, which also included metadata

information of the physical attributes of each pill,

enabled the development of a baseline recognition

algorithm which performed well under controlled

conditions. Generalization for the algorithm required

accounting for real-world factors such as lighting

condition, camera resolution, and non-homogeneous

backgrounds.

Previous work in this domain (Madsen et al,

2013) and (Wan et al., 2015) yielded good results,

but were limited to using images of similar quality as

the Pillbox images. In this study, we expand the

number of target color to be recognized from 7 to 12

and increase the number of Pillbox images to

approximately 2100.

Using an SVM classifier, we were able to

achieve a recognition accuracy based on 12 color

categories of 97.90%.

2 METHODOLOGY

The workflow of this research consisted of first

determining perceived five color component values

(red, green, blue, yellow, white) and twelve

perceived actual color values. There were 2151 high

resolution pill images in the Pillbox database as of

December 2014. The Pillbox images are of very high

quality, with high resolution, controlled illumination,

and uniform background. The high quality of the

188

Guo P., J. Stanley R., G. Cole J., Hagerty J. and V. Stoecker W.

Color Feature-based Pillbox Image Color Recognition.

DOI: 10.5220/0006136001880194

In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 188-194

ISBN: 978-989-758-225-7

database allows segmentation using a simple

threshold to separate the pill object from the

background for almost all pills. Using the

segmented pill, the color classification process was

performed using a four-step approach which is

outlined in Figure 1 and summarized as:

Step 1: Segment the pill from the Pillbox image.

Step 2: Determine the best axis to divide the

segmented pill image into two halves.

Step 3: Extract features from each half of the

segmented pill region.

Step 4: Classify the segments into a color

category (classification)

Figure 1: Pill color classifier development.

2.1 Segmentation of Pill Region

The Pillbox images are all dimensional similar; each

being 768×1024. Demonstrated in Figure 2, the

colored background, front and back of each pill are

included in the Pillbox images. Since the

background color can influence color detection it is

not used for pill color analysis. As such, a reliable

segmentation algorithm of pill from background is

needed.

Figure 2: Example of a Pillbox image.

The algorithm to segment the pill image is as

follows:

Step 1: Load red, green and blue (RGB) image

Step 2: Take a sample of the background pixels

(10 lines of pixels, shown in the Figure 2)

and calculate the mean intensity of

background.

Step 3: Threshold the image based on the

calculated mean intensity value

Step 4: Repeat Step 2-3 for Green and Blue color

channels

Step 5: Use erosion to eliminate noise and

dilation to fill holes

Step 6: Combine the binary mask that was

generated for each color plane using a

union operation.

Because the pill color is uniform on both sides of

the pill in this data set, only one side of the

segmented pill region (top-left side was arbitrarily

chosen) is used for feature analysis. The

segmentation of pill region is shown in Figure 3,

with background color set to be black (pixel value is

0 in RGB color space) for convenience of feature

calculation shown later.

Figure 3: Segmentation of pill region.

2.2 Creation of Vertical Segments of

Pill Region

In this process, the segmented pill region obtained in

the previous step is divided into two vertical

segments, left segment and right segment. The

reason for creating two vertical segments instead of

analyzing the whole segmented pill region is that

some pills have two or more colors (see Figure 4).

To avoid classification errors that would occur with

whole-pill color analysis, the pill is divided into

vertical two segments.

After the pill region of interest is obtained

through segmentation, a vertical central axis needs

to be located. The central axis is simply defined as

the middle column in the segmented pill region

because of the accuracy of the segmentation. For a

segmentation mask image of size, the middle

column is defined as column



/2



. The central axis

location and vertical segments can be viewed in

Figure 5.

Segment pill region in original image

Creating vertical segments of left and right part

Feature extraction

Color classification

Color Feature-based Pillbox Image Color Recognition

189

Figure 4: A two-colored Pillbox image.

Figure 5: Central axis and left/right vertical segments.

2.3 Feature Extraction

For each vertical segment of the pill region image,

there are five different categories of color features

computed from the segmented pills, including: (1)

RGB intensity, (2) HSV intensity, (3) Chromaticity,

(4) Left-Right averages, and (5) Brightness. An

overview of the features extracted for each feature

category is presented in Table 1. The algorithms for

computing each feature are given in detail following

Table 1.

Table 1: 20 Color Features.

Feature

Category

Labe

Measure Description

RGB Mean

And Std. Dev.

F1 Red intensity Red, Green and

Blue color space

statistics

F2 Green

intensity

F3 Blue

intensity

F4 Std. Dev.

Red intensity

F5 Std. Dev.

Green

intensity

F6 Std. Dev.

Blue

intensity

HSV Mean

And Std. Dev.

F7 Hue Mean Hue, Saturation

and Value color

space statistics

F8 Value Mean

F9 Saturation

Mean

F10 Hue Std.

Dev.

F11 Value Std.

Dev.

F12 Saturation

Std. Dev.

Chromaticity

Means

F13 Red

chromaticity

Mean

Chromaticity

measures color

quality and

overall values of

brightness.

Red chromaticity

= R/(R+G+B);

green and blue

are similar.

Chromaticity

combines the

features of

saturation (color

purity) and

hue (color

specificity) in a

single value.

F14 Green

chromaticity

Mean

F15 Blue

chromaticity

Mean

F16 Yellow

chromaticity

Mean

Combination F17 RGA Averages of

colors for each

two-color

combination.

F18 GBA

F19 BRA

Brightness F20 Intensity



2.3.1 RGB Intensity

The RGB color model is additive in the sense that

the three primary color spectra are added together,

wavelength for wavelength, to make the final color

spectrum (Boughen & Gross, 2003; Poynton, 2003).

After the pill region image is obtained, the color

image is decomposed into red, green and blue

images (figure 6). The background pixels are set to 0.

All nonzero pixel values indicate pill region pixels;

for this region, SumRed, SumGreen and SumBlue

represent the summation of intensity values for pill

region pixels. The number of pill region pixels is

found by NumRed, NumGreen and NumBlue.

(a) (b) (c)

Figure 6: The grayscale images of pill region. a) red

channel, b) green channel, c) blue channel.

In each of the three channels, the mean of red

intensity (F1), green intensity (F2) and blue intensity

(F3) are calculated according to the equations 1-3.

  





(1)

  





(2)

VISAPP 2017 - International Conference on Computer Vision Theory and Applications

190

 





(3)

The standard deviation for the three channels of

the RGB color space, denoted as Std Red (F4), Std

Green (F5) and Std Blue (F6) are calculated using

the Matlab® function “std.”

2.3.2 HSV Intensity

The hue, saturation and value (HSV) color space

represents colors in a way that is similar to the way

that humans perceive colors. HSV attempts to the

separate chroma and luminance such a particular hue

is the same independent of luminance. The

conversion from RGB to HSV color space is

provided by Matlab® with the function “rgb2hsv”

which is detailed on Matlab official document

webpage (MathWorks, 2016). The given pixel value

represents the current pixel parameter (hue,

saturation or value), as the entire image is distributed

in these three channels. Figure 7 gives an example of

a pill region in HSV color space.

(a) (b) (c)

Figure 7: Illustration of grayscale image of HSV color

space. a) hue, b) saturation, c) value.

The pixel value sums for each of the three

channels, SumHue, SumSat and SumVal, can be

obtained by summing the non-zero pixels (black

pixels also have 0 values in HSV color space). The

counting of pill region pixels NumHue, NumSat and

NumVal is similar to RGB pixel counting. Hence,

the HSV feature are also extracted using equations

4-6, similar to RGB feature extraction.

 





(4)

 





(5)

 





(6)

Note that the HSV calculations are also

accomplished with the Matlab® function “std,” and

are processed in all three HSV color space channels.

2.3.3 Chromaticity and Brightness Features

For each channel of the HSV color space image,

along with red, yellow, and blue chromaticity (See

Equations 7-10) features were calculated for every

pill. As an objective specification of the quality of a

color regardless of its luminance, chromaticity can

give us another view of the color being recognized

and help in the color classification. And for this

category of feature, four chromaticity features are

defined: red, green, blue and yellow chromaticity.

Each of these features represents a different aspect

of color according to the color space we have here.

Chromaticity features were among the most useful

features in previous works on pill color recognition

(Lee, Park, Jain, & Lee, 2012; Madsen, Payne,

Hagerty, Szanto, Moss, Wronkiewicz, Stoecker,

2013; Wan, Woods, Salgado-Montejo, Velasco, &

Spence, 2015).











(7)











(8)











(9)









2    

(10)

The brightness feature F20 is calculated as the

average of the brightness in the red, green and blue

channels (equation 11). Brightness is needed to

better classify achromic colors, such as gray and

white

 



(11)

3 CLASSIFICATION

The left and right pill region vertical segments each

have a specific color category. In the classification

process, the left and right vertical segments are

trained and tested separately in order to avoid the

issue of color mixture when treating the pill region

as a whole. However, the left part and right part in

classification steps are the same regardless of the

variation of color; hence, for the experiments, we

consider just the left part as an example.

The labels for vertical segments were assigned

manually by two of the authors (J.G.C. and P.G.)

according to the 12 FDA colors (Julie N. Barrows

Arthur L. Lipman, 2009) listed in table 3 below. The

Color Feature-based Pillbox Image Color Recognition

191

entire database is labelled in 12 groups with one

color in each. Take red for example, all the red pills

are labelled as “positive (1)” for red color; all other

pills are labelled “negative (0)” to complete the red

color labels; the same labelling is done for all 12

colors.

Table 3: FDA Pill Color Classification Labels.

FDA Pill Colors

Black Pink Gray Turquoise

Blue Purple Green White

Brown Red Orange Yellow

In the first stage, twenty features (F1-F20) were

extracted and used for color-based classification

based on a leave-one-out method. A Support Vector

Machine (SVM) classifier was investigated to take

the input of twenty feature columns for individual

pill classification (Cortes & Vapnik, 1995).

As 12 color are treated as the target color in this

study, 12 different binary classifiers are built, the

classification for a single color is carried out with

the following steps:

Step 1: Train the SVM classifier algorithm

using a leave-one-image-out approach. The

classifier is trained based on the left vertical

segment feature vectors for all except the

left-out pill image, which is used for testing.

Step 2: Classify the pill left-out pill test

image using the SVM classifier.

Step 3: Assign class labels (1 for target color

confirmed, 0 for not target color) to the test

segment image.

Step 4: Repeat steps 1-3 for all the segmented

images in the experimental data set.

Step 5: To finish classifications for all the 12

colors, Step 1 to Step 4 above are repeated

for all color labels (total 12 iterations).

For the SVM classifier, the LIBSVM(Chang &

Lin, 2011) implementation is employed in this paper.

This SVM tries to find an optimal hyperplane for

linear inseparable classes which acts as a decision

function to classify data in high dimensions. A linear

kernel is used for the SVM to update the penalty

parameter by a leave-one-image-out method, as

explained in the paper of (Guo et al., 2015), (De et

al., 2013). The implementation is completed with

Matlab® and presented in (Guo et al., 2015).

4 EXPERIMENTS PERFORMED

Twelve color categories are defined for the

classification target. The entire database of 2140

pillbox images are vertically segmented into two

groups, each group of pill region segments (2140 for

either left or right segments) will be assigned one of

twelve color labels automatically by the classifier.

The twelve target FDA color categories in Table 3

are manually assigned as the training and testing

targets by the author. In the classification process, a

leave-one-out approach is employed where 2139

images are used for training and the single left-out

image is tested.

In the scoring used for classification accuracy,

the percentage of rightly classified images is

calculated for every color category. If the class label

automatically assigned to the test image is the same

as the manual class label, then the image is

considered to be correctly labelled. Otherwise, the

image is considered to be incorrectly labelled.

4.1 Experimental Results and Analysis

As previously stated, we obtained the vertical

segment image classifications using the SVM

classifier with a leave-one-image-out approach

based on all the twenty features generated. Then the

vertical segment classifications are compared with

the target color truth label and finally calculated the

percentage of right classification accuracy. We

evaluated performance of these pill image

classifications using the three approaches that is

presented in Section 3. Table 4 shows the

classification results obtained with the SVM

classifier, for all twelve color categories.

Table 4: Classification results obtained for all color

categories.

Color

Red Green Blue Yellow Black White

Accuracy

98.46% 99.44% 99.21% 99.07% 100% 95.14%

Color

Gray Pink Cyan Purple Brown Orange

Accuracy

90% 100% 99.47% 99.39% 96.78% 97.85%

As can be observed in Table 4, the highest

accuracy of classification for all the color categories

are the black and pink color classification, which

both were 100% correct. Additionally, cyan has an

accuracy of 99.47% (2129/2140), green

classification is found to be 99.44% correct

(2128/2140), 2127/2140 of purple pills are classified

correctly, followed by blue follows with a

classification accuracy of 99.21% (2123/2140). For

yellow, 2120 pills are correctly recognized giving an

accuracy of 99.07% (2120/2140). Red is recognized

VISAPP 2017 - International Conference on Computer Vision Theory and Applications

192

in 98.46% (2107/2140) in classification and Orange

as 97.85% (2094/2140).

However, white and gray give confusing

classification results of 95.14% (2036/2140) and 90%

(1926/2140) since manual labelling consistency is

difficult for these colors with identical hues. After

adjusting Pillbox image labelling using the output

results, manually misclassified images can be

corrected, which improves classification accuracy to

as high as 99%. However, until labelling is further

investigated, the lower white and gray results are

used in calculating overall accuracy.

5 CONCLUSIONS

Under the idea of basic component colors for

classification, the features are extracted as the basis

of red, green and blue color related features.

Furthermore, the chromaticity, since it combines

saturation and hue (measuring color proportion over

all values of luminance), provides a simple model

for the color perceived by humans. Because of

uncertainty regarding labelling of white and gray,

performance ranged from over 98% for nine of the

colors to 97.85% for orange, 95.14% for white, and

90% for gray. Perfect accuracy (100%) is yielded as

the classification result for both cyan and pink color

Pillbox images. Overall, the classification accuracy

obtained from all the 12 color categories is 97.90%

which is higher than results obtained by (Madsen et

al, 2013) and (Wan et al., 2015).

Future research can be focused on principal

feature analysis to find the most significant features,

accuracy obtained from different current feature

groups. Additional data could enable classification

by unsupervised learning algorithms such as deep

learning. And, as to make the algorithm more

generalized and applicable in real world conditions,

a noise level study on the pill images should be

performed, taking into account non-optimal

condition such as uneven illumination, image blur

and heterogeneous backgrounds, etc.

As the first step in computing the visual content

in pill color recognition, we have already made

progress in reference image-based classification.

Additional research should also focus on imprint

identification, score marks and analysis of

consumer-quality pill identification, to enable

identification of pills under real-world conditions.

ACKNOWLEDGEMENTS

The images used in this research are obtained from

the NLM (National Library of Medicine) of National

Institutes of Health Pillbox image data base.

REFERENCES

Boughen, N., & Gross, J. (2003). Lightwave 3D 7.5

Lighting. book, Plano, TX, USA: Wordware

Publishing Inc.

Chang, C., & Lin, C. (2011). LIBSVM : A Library for

Support Vector Machines. ACM Transactions on

Intelligent Systems and Technology (TIST), 2, 1–39.

Cortes, C., & Vapnik, V. (1995). Support-vector networks.

Machine Learning, 20(3), 273–297.

De, S., Stanley, R. J., Lu, C., Long, R., Antani, S., Thoma,

G., & Zuna, R. (2013). A fusion-based approach for

uterine cervical cancer histology image classification.

Computerized Medical Imaging and Graphics, 37(7–

8), 475–487.

Gu, Q., Dillon, C. F., & Burt, V. L. (2010). Prescription

drug use continues to increase: U.S. prescription drug

data for 2007-2008. NCHS Data Brief, (42), 1–8.

Retrieved from

Guo, P., Banerjee, K., Stanley, R., Long, R., Antani, S.,

Thoma, G., … Stoecker, W. (2015). Nuclei-Based

Features for Uterine Cervical Cancer Histology Image

Analysis with Fusion-based Classification. IEEE

Journal of Biomedical and Health Informatics, (c).

Julie N. Barrows Arthur L. Lipman, C. J. B. (2009). Color

Additives: FDA’s Regulatory Process and Historical

Perspectives. book.

Kantor, E. D., Rehm, C. D., Haas, J. S., Chan, A. T., &

Giovannucci, E. L. (2015). Trends in Prescription

Drug Use Among Adults in the United States From

1999-2012. Jama, 314(17), 1818–1831.

Lee, Y. B., Park, U., Jain, A. K., & Lee, S. W. (2012).

Pill-ID: Matching and retrieval of drug pill images. In

Pattern Recognition Letters (Vol. 33, pp. 904–910).

Madsen, D. J., Payne, K. S., Hagerty, J. R., Szanto, N.,

Wronkiewicz, M., Moss, R.H., Stoecker. W.V. (2013).

Automatic Pill Identification from Pillbox Images. In:

8th International Conference on Computer Vision

Theory and Applications (VISAPP), 21-24 February

2013. 378-384.

Poynton, C. A. (2003). Digital video and HDTV :

algorithms and interfaces. Morgan Kaufmann

Publishers.

Schumock, G. T., Li, E. C., Suda, K. J., Wiest, M. D.,

Stubbings, J., Matusiak, L. M., … Vermeulen, L. C.

(2015). National trends in prescription drug

expenditures and projections for 2015. American

Journal of Health-System Pharmacy : AJHP : Official

Journal of the American Society of Health-System

Pharmacists, 72(9), 717–736.

Color Feature-based Pillbox Image Color Recognition

193

Thielke, S. M., Simoni-Wastila, L., Edlund, M. J.,

DeVries, A., Martin, B. C., Braden, J. B., … Sullivan,

M. D. (2010). Age and sex trends in long-term opioid

use in two large American health systems between

2000 and 2005. Pain Medicine (Malden, Mass.),

11(2), 248–56.

Wan, X., Woods, A. T., Salgado-Montejo, A., Velasco, C.,

& Spence, C. (2015). Assessing the expectations

associated with pharmaceutical pill colour and shape.

Food Quality and Preference, 45, 171–182.

VISAPP 2017 - International Conference on Computer Vision Theory and Applications

194