AN ORDER CLUSTERING SYSTEM USING ART2 NEURAL

NETWORK AND PARTICLE SWARM OPTIMIZATION

METHODN

R. J. Kuo

Department of Industrial Management, National Taiwan University of Science and Technology

No. 43, Section 4, Keelung Road, Taipei, Taiwan

M. J. Wang, T. W. Huang

Department of Industrial Engineering and Management, National Taipei University of Technology

No. 1, Section 3, Chung-Hsiao East Road, Taipei, Taiwan

Tung-Lai Hu

Department of Business Management, National Taipei University of Technology

No. 1, Section 3, Chung-Hsiao East Road, Taipei, Taiwan

Keywords: Clustering analysis, SMT production system, ART2 neural network, Particle swarm optimization algorithm,

K-means.

Abstract: Surface mount technology (SMT) production system set up is quite time consuming for industrial personal

computers (PC) because of high level of customization. Therefore, this study intends to propose a novel

two-stage clustering algorithm for grouping the orders together before scheduling in order to reduce the

SMT setup time. The first stage first uses the adaptive resonance theory 2 (ART2) neural network for

finding the number of clusters and then feed the results to the second stage, which uses particle swarm K-

means optimization (PSKO) algorithm. An internationally well-known industrial PC manufacturer provided

the related evaluation information. The results show that the proposed clustering method outperforms other

three clustering algorithms. Through order clustering, scheduling products belonging to the same cluster

together can reduce the production time and the machine idle time.

1 INTRODUCTION

Unlike regular personal computer (PC)

manufacturing, high mix low volume, or high

customization, is one of the characteristics of the

industrial PC industry. It is also the biggest

challenge facing the industrial PC industry. In this

industry, printed circuit board (PCB) assembly is a

fundamental manufacturing process, in which

surface mount technology (SMT) plays a very

important role. By applying an SMT production

system, not only can there be more components on

the limited space of a PCB, but also the production

efficiency and product stability can be enhanced.

However, the high mix low volume production style

is still a difficulty for the SMT production system. In

this production system, production line-change is a

very serious bottleneck since before line-change,

production labor must prepare for the materials and

also bind these materials, which is very time

consuming. If production labor can not complete

these tasks before the next order starts, it would both

cause idle time for these expensive machines and

decrease the production capability utilization.

Currently, this problem is frequently encountered in

the SMT production system, and the only way it can

be handled is to prepare for more materials and

increase the binding labor. Thus, how to reduce the

setup time for SMT line-change operation is a very

important issue.

Kuo R., Wang M., Huang T. and Hu T. (2009).

AN ORDER CLUSTERING SYSTEM USING ART2 NEURAL NETWORK AND PARTICLE SWARM OPTIMIZATION METHODN.

In Proceedings of the 11th International Conference on Enterprise Information Systems - Artiﬁcial Intelligence and Decision Support Systems, pages

55-60

DOI: 10.5220/0001860300550060

 SciTePress

Therefore, this study applies a clustering

technique to cluster bills of materials (BOMs) for all

orders or products. After clustering, production

engineers can to schedule similar orders together for

production. This can simplify the material-

preparation time in order to reach reducing SMT

line-change time. For the clustering technique, this

study proposes a novel two-stage clusering

algorithm. The first stage employs an adaptive

resonance theory 2 (ART2) neural network to find

the number of clusters and feed it to the second

stage. The second stage applies particle swarm K-

means optimization (PSKO) algorithm to reach the

final solution.

In the model evaluation, data provided by

Advantech Company which is an internationally

well-known industrial PC manufacturer, shows that

the proposed two-stage clustering algorithm is better

than three other algorithms in accuracy. This can

dramatically reduce the SMP production system

setup time and increase the utilization of production

capability.

2 BACKGROUND

This section briefly presents the general literature

survey for the clustering analysis and particle swarm

optimization algorithm.

2.1 Clustering Analysis

There have been many algorithms being applied in

clustering analysis. With artificial intelligence (AI)

and soft computing, many clustering algorithms

based on these techniques and theories have been

proposed. Thus, the following discussions will focus

on AI related clustering algorithms (Xu and

Wunsch, 2005).

An artificial neural network (ANN) is a system

that has been derived through models from

neurophysiology. ANN-based clustering has been

dominated by self-organizing feature maps (SOM)

and adaptive resonance theory (ART) (Kohonen,

1990, Carpenter and Grossberg, 1987, and Carpenter

and Grossberg, 1987).

Fuzzy clustering is different from hard clustering

since its restriction is relaxed and the object can

belong to all of the clusters with a certain degree of

membership. The methods of fuzzy clustering

include fuzzy C-means (Hoppner et al., 1999) and

fuzzy c-shells (Bezdek and Hathaway, 1992).

Clustering can also be regarded as a category of

optimization problems that use evolutionary

algorithms, like genetic algorithms (GA) or ant

colony optimization algorithms. But the main

drawback of these clustering algorithms is the

process of parameter selection. Many techniques

supporting this method such as genetic K-means

algorithm (Krishna and Murty, 1999), Tabu search

clustering (Al-Sultan, 1995), simulated annealing

clustering (Brown and Huntley, 1992 and Smyth,

1998) and ant colony clustering algorithm (Kuo et

al., 2005b).

Due to limitations of some clustering algorithms,

which need to know the number of clusters before

implementing clustering, a two-stage framework is

proposed here to overcome this problem. The basic

idea of the two-stage clustering algorithm is to first

find the number of clusters using an automatically

clustering algorithm, such as an ART2 neural

network, and then feed this number to the other

clustering algorithm to find the final solution.

Kuo et al. (2002) proposed a two-stage method

which integrates both the SOM and K-means, with

results indicating that the proposed method is much

better than using only SOM or K-means. Then, Kuo

and his colleagues modified the GKA by Krishna

and Murty (1999) and used SOM’s solution as the

initial solution for the modified GKA (Kuo et al.,

2004). The results showed that this method was

better than the previously published method, SOM +

K-means. Kuo and his colleagues also presented a

method integrating of ART2 and a genetic algorithm

(Kuo et al., 2005a), and one that used a different

coding method (Kuo et al., 2006).

2.2 Particle Swarm Optimization

The PSO algorithm shares many similarities with

evolutionary computation techniques such as

Genetic Algorithms (GA). The system is initialized

with a population of random solutions and searches

for optima by updating generations. However, unlike

GA, the PSO algorithm has no evolutionary

operators, such as crossover and mutation. In the

PSO algorithm, the potential solutions, called

particles, move through the problem space by

following the current optimum particles. The main

developments in the PSO algorithm can be generally

divided into three phases as follows.

Xiao et al. (2003) proposed a hybrid clustering

approach based on an SOM neural network. The

proposed algorithm uses a PSO algorithm to evolve

the weight for the SOM neural network. The weights

are trained by the SOM neural network in the first

stage, and in the second stage they are optimized by

the PSO algorithm. The experimental results show

AN ORDER CLUSTERING SYSTEM USING ART2 NEURAL NETWORK AND PARTICLE SWARM

OPTIMIZATION METHODN

that the hybrid method tries to tune the original

SOM such that it can achieve a better tradeoff

between the average quantization error and the

topographic error.

Merwe and Engelbrecht (2003) proposed two

new approaches using a PSO algorithm to cluster

data. The first method, called the PSO clustering

algorithm, shows how the PSO algorithm can be

used to find the centroids of a user-specified number

of clusters. The second method, called the Hybrid

PSO algorithm, first uses K-means clustering to seed

the initial swarm and then uses PSO algorithm to

refine the clusters formed by K-means. These new

PSO algorithms were evaluated on six data sets, and

compared with the performance of K-means

clustering. Results show that both PSO clustering

techniques have much potential. Chen and Ye also

proposed a clustering analysis algorithm based on

the PSO algorithm (Chen and Ye, 2004). But the

concept of that algorithm is similar to others in the

literature (Merwe and Engelbrecht, 2003), and its

main difference is only in the fitness function.

Finally, the experimental results obtained using four

artificial data sets show the algorithm proposed here

has better performance than K-means and fuzzy C-

means.

3 METHODOLOGY

This section presents the proposed order clustering

system which includes data collection and

transformation, principle component analysis, and

clustering analysis. The following subsections

briefly discuss these three components.

3.1 Data Collection

and Transformation

In order to classify the product group for the up

coming orders belonging to it, all the previous orders

must first be collected and make clustering analysis

made for them. Basically, for most companies, these

data can be retrieved from the enterprise resources

planning system. Then, the bill of materials (BOM)

for each order is used as the feature for each order,

or product.

3.2 Principle Component Analysis

Due to the vast number of materials used for each

order, it is very time consuming to use these data for

clustering analysis, and it is more feasible to

condense these data in advance. A statistical method,

principle component analysis, is employed for this

purpose so that each order will have a limited

number of features. This can result in shorter

computational time and without influencing the

computational results.

3.3 Clustering Analysis

After implementing principle component analysis,

the data obtained is applied for clustering analysis

using a two-stage clustering with ART2 neural

network and a proposed particle swarm K-means

optimization algorithm as well.

3.3.1 ART2 Neural Network

ART2 architectures are designed for processing

analog as well as binary input patterns. The ART2

neural network consists of F1 and F2 layers. There

are seven nodes in the F1 layer (W, X, U, V, P, Q).

The input signal is processed by the F1 layer and

then is passed from the bottom to top value (b

). The

result of the bottom-to-top value is an input signal of

F2 layer. The nodes of F2 layer compete with each

other to produce a winning unit and the winning unit

returns the signal to the F1 layer. The match value is

then calculated with top to bottom value (t

) in the

F1 layer and compared with the vigilance value. If

the match value is greater than the vigilance value,

then the weight of b

and t

is updated. Otherwise,

the reset signal is sent to the F2 layer and the

winning unit is inhibited. After inhibition, the other

winning unit will be found in the F2 layer. If all of

the F2 layer nodes are inhibited, the F2 layer will

produce a new node and generate the initial

corresponding weights to the new node. The detailed

learning procedure can be found in [Grossbert

1976].

3.3.2 PSKO Algorithm

Particle swarm optimization, like GA, is a

population-based stochastic search process. The

algorithm maintains a population of particles, where

each particle represents a potential solution to an

optimization problem. By referring to Krishna and

Murty＇s concept (1999) of integrating GA with K-

means, this article proposes to integrate K-means

with PSO algorithm (Merwe and Engelbrecht,

2003). It is called the particle swarm K-means

optimization (PSKO) algorithm and the

computational procedures are as follows.

Step 1: Set up parameters including population

size (number of particles), maximum

ICEIS 2009 - International Conference on Enterprise Information Systems

velocity, V

max

, inertia weight, W, and two

learning factors, c

and c

Step 2: Initialize each particle randomly with

initial position X

and velocity V

. In

PSO clustering, we need in advance to

know the number of clusters, k. In this

study ART2 neural network will provide

this information to the PSKO algorithm.

Since each particle is a vector containing

k cluster centroids, each cluster＇s

position, X

, can be represented as:

),,,,(

1 ikijiid

zzzX ⋅⋅⋅

⋅

⋅=

(1)

where z

denotes the jth cluster’s

centroid for the ith particle.

Step 3: Calculate fitness value for each particle.

[

]

||||

∑∑

=∈∀

−=

jnx

zxvaluefitness

(2)

where x denotes the data vector, n

denotes the number of data for jth cluster

of the ith particle, and

||||

zx −

denotes

the Euclidean distance of data vector to

all cluster centroids.

Step 4: Update the local best, P

, and global best,

Step 5: According to the best position of P

and

, update the velocity and position for

each particle using Equations (3) and (4).

It should be noted that V can not be

larger than V

max

)()(

2211 idgdidid

old

new

XPrandcXPrandcVWV −⋅⋅+−⋅⋅+⋅=

(3)

new

old

new

VXX +=

(4)

where c

and c

are learning factors,

respectively, while rand

and rand

denote random numbers between (0, 1).

Step 6: Calculate the Euclidean distance of every

piece of data x to all cluster centorids for

each particle.

Step 7: For each particle, assign each piece of

data, x, to the cluster with the closest

cluster centroid.

Step 8: Recalculate the cluster centroid vectors for

every particle, using

∑

∈∀

(5)

Step 9: Stop if the specified number of iterations

is satisfied; otherwise, go back to Step 3.

4 MODEL EVALUATION

RESULTS

This section will apply the proposed clustering

method for order clustering in order to reduce the

SMT production system setup time. Advantech

Company provided the related data for assessment.

The setup of production line change for different

orders has become a critical bottleneck in SMT

production system because that implementing

material-preparation and material-binding tasks by

hand for production line change is very time

consuming. If this can not be ready before next order

starts production, it will cause machines idle time

and result in low production capability utilization.

Therefore, we apply a two-stage clustering method

(Kuo et al. 2002) to group similar products or orders

together and find the same materials (shared

materials) that will be used for all the products in the

same group. This enables the production engineers

to put the same needed materials in the same

material positions for SMT retrieval equipment, so

production engineers can arrange for similar

products to be produced together. Thus, the material

preparation operation can be simplified in order to

reduce the SMT setup time.

4.1 Data Collection

A material report was provided by Advantech

Company. Using ACCESS SQL query, there is a

total of 2826 different products, or orders, and a total

of 2517 materials used. In order to cluster these

products, it is necessary to filter the noise of product

and material data and then transform them into a

2826×2517 two-dimension matrix.

4.2 Principle Component Analysis

That there are 2517 features for product feature

matrix and it is very time consuming for clustering

analysis if there are many features. In order to

reduce the computational time and also maintain the

solution quality, this study uses Matlab 6.5.1 to

make the principle component analysis for the

product feature matrix in order to extract the

dimensions for the principle component factors. The

selection of factor is according to the Eigne value

which should be larger than 1. After analysis, there

are a total of 200 principle component factors. The

cumulative explanation variance is 81.98%.

AN ORDER CLUSTERING SYSTEM USING ART2 NEURAL NETWORK AND PARTICLE SWARM

OPTIMIZATION METHODN

4.3 Clustering Analysis

After reducing the original feature matrix with size

of 2826×2517 to 2826×200 through principle

component analysis, it is necessary to know the

number of clusters in advance. Based on our

previous research (Kuo et al., 2005), this study

applies a two-stage clustering method, ART2+PSKO

which is a kind of method for processing gray value

in the first stage, ART2 automatically finds the

number of clusters, while the second stage uses the

PSKO algorithm to find the final solution.

4.3.1 Determination of Cluster Number

The ART2 algorithm was coded by using Matlab

6.5.1. Since in ART2 algorithm the number of

clusters is determined by the vigilance value, once it

is well determined ART2 can automatically cluster

the data. Thereafter, the second stage employs K-

means, PSO clustering, hybrid PSO, and PSKO

algorithms to find the final solutions. Since the

vigilance value will dramatically affect the

clustering outcome, this study uses Wilk’s Lambda

value as the indicator for determining the number of

clusters. Wilk’s Lambda is frequently applied by

MANOVA as the indicator for determining the

number of clusters. If Wilk’s Lambda value

suddenly increases in two different numbers of

clusters, then the number of clusters before variance

can be treated as the best number of clusters.

Theoretically, Wilk’s Lambda value is defined as:

Lambda '

total

within

sWilk =

(6)

where SS

WITHIN

and SS

total

are the within-cluster and

total variances, respectively.

Table 1 lists the corresponding Wilk’s Lambda

values for different vigilance values. The result

shows that 35 is the best number of clusters.

Table 1: Wilk’s Lambda value of ART2.

Vigilance

Value

Cluster

Number

Wilk’s

Lambda value

0.9372 43 0.67823

0.9347 41 0.685

0.932 39 0.69358

0.9295 37 0.69955

0.972 35 0.70688

0.926 33 0.72989

4.3.2 Comparison of Different Clustering

Algorithms

Based on the ART2 algorithm’s result, four different

clustering algorithms are used to further find the

final solution for comparison. Table 2 depicts the

Sum of Euclidean Distances (SEDs) of four

algorithms, which indicates that the ART2+PSKO

algorithm has the smallest SED value, 83373.859.

Table 2: SED values for each clustering algorithm.

Clustering

Algorithms

ART2

K-means

ART2

PSO

clusterin

ART2

Hybri

PSO

ART2

PSKO

SED 90276 94407 90153 83373

4.4 Shared Materials

According to the clustering results of ART2+PSKO,

the products are grouped into thirty five clusters.

Totally, there are eleven clusters whose numbers of

shared materials are over one hundred.

4.5 Performance Evaluation

This study simulates the production line-change

efficiency through on-field collection of the

production plan. We first use the regular scheduling

method to schedule the jobs for two days. Then, this

study further groups similar products belonging to

similar clusters to be scheduled together for

production in order to reduce the SMT line-change

time. According to these simulation results, we

found that using the proposed ART2 ＋ PSKO

clustering algorithm really can efficiently reduce

material binding time by 10.9% due to considering

shared materials. Also, total production time

decreased 7.3%, and the most dramatic improvement

was the 89.3% reduction of machine idle time.

5 CONCLUSIONS

This study has demonstrated that the use of a

clustering technique can reduce both production

time and machine idle time since similar products or

orders are scheduled for production together. In

addition, this study also proposed a novel clustering

algorithm that integrates both the K-means

algorithm and the PSO algorithm for order

clustering. Integration of the PSO clustering

ICEIS 2009 - International Conference on Enterprise Information Systems

algorithm and the K-means algorithm gives the

particles both global and local search capabilities.

The results show that its performance is better than

those of other three clustering algorithms for

Advantech Company’s order clustering problem.

REFERENCES

K. Al-Sultan, “A Tabu search approach to the clustering

problem,” Pattern Recognition, Vol. 28, No. 9, 1995,

pp. 1443–1451.

J. Bezdek and R. Hathaway, “Numerical convergence and

interpretation of the fuzzy c-shells clustering

algorithms,” IEEE Transaction on Neural Networks,

Vol. 3, No. 5, 1992, pp. 787–793.

D. Brown and C. Huntley, “A practical application of

simulated annealing to clustering,” Pattern

Recognition, vol. 25, no. 4, 1992, pp. 401–412.

G. Carpenter and S. Grossberg, “A massively parallel

architecture for a self-organizing neural pattern

recognition machine,” Computer Vision Graphic and

Image Process, Vol. 37, 1987, pp. 54–115.

G. Carpenter and S. Grossberg, “ART2:Self-organization

of stable category recognition codes for analog input

patterns,” Applied Optics, Vol.26, No.23, 1987,

pp.4919–4930.

C. Y. Chen and F. Ye, “Particle swarm optimization

algorithm and its application to clustering analysis,”

2004 IEEE International Conference on Networking,

2004, pp. 789 – 794.

M. Clerc, “The swarm and the queen: towards a

deterministic and adaptive particle swarm

optimization,” Evolutionary Computation, 1999.

R. Eberhart and J. Kennedy, “A new optimizer using

particle swarm theory,” Proceedings of the Sixth

International Symposium on Micro Machine and

Human Science, 1995, pp.39-43.

S. Grossberg, “Adaptive pattern recognition and universal

encoding II: Feedback, expectation, olfaction, and

illusions,” Biological Cybernetics, Vol. 23, 1976,

pp.187-202.

F. Höppner, F. Klawonn and R. Kruse, Fuzzy Cluster

Analysis: Methods for Classification, Data Analysis,

and Image Recognition. New York: Wiley, 1999.

L. Kaufman and P. Rousseeuw, Finding Groups in Data:

An Introduction to Cluster Analysis: Wiley, 1990.

T. Kohonen, “The self-organizing map,” Proc. IEEE, Vol.

78, No. 9, 1990, pp.1464–1480.

K. Krishna and M. N. Murty, “Genetic K-means

algorithm,” IEEE Transactions on Systems, Man and

Cybernetics, Vol. 29, No. 3, 1999, pp. 433–439.

R. J. Kuo, L.M. Ho and C. M. Hu, “Integration of Self-

Organizing Feature Map and K-Means Algorithm for

Market Segmentation,” International Journal of

Computers and Operations Research, 29, 2002, pp.

1475-1493.

R. J. Kuo, Chang, K., and Chien S.Y., “Integration of self-

organizing feature maps and genetic algorithm based

clustering method for market segmentation,” Journal

of Organizational Computing and Electronic

Commerce, Vol. 14, No. 1, pp.43-60, 2004.

R. J. Kuo, J. L. Liao and C. Tu, "Integration of ART2

neural network and genetic K-means algorithm for

analyzing Web browsing paths in electronic

commerce," Decision Support Systems Vol. 40, Issue:

2, 2005a, pp. 355-374.

R. J. Kuo, Wang, H. S., Hu, Tung-Lai, and Chou, S. H.,

“Application of Ant K-means on Clustering Analysis

in Data Mining,“ International Journal of Computers

and Mathematics with Applications,50, pp.1709-1724,

November-December, 2005b.

R. J. Kuo, An, Y.L., Wang, H.S., and Chung, W.J.,

“Integration of Self-Organizing Feature Maps Neural

Network and Genetic K-means Algorithm for Market

Segmentation," International Journal of Expert

Systems with Applications, 30(2), pp.313-324,

February, 2006.

D. W. Van der Merwe and A. P. Engelbrecht, “Data

clustering using particle swarm optimization,” The

2003 Congress on Evolutionary Computation, 2003,

pp. 215 - 220.

Y. Shi and R. Eberhart, “A modified particle swarm

optimizer,” Proceedings of the IEEE International

Conference on Evolutionary Computation, 1998a, pp.

69-73.

Y. Shi and R. Eberhart, “Parameter selection in particle

swarm optimization,” In Evolutionary Programming

VI/: Proc. EP98, New York: Springer-Verlag, 1998b.

P. Smyth, “Model selection for probabilistic clustering

using cross validated likelihood,” Statistics and

Computer, Vol. 10, 1998, pp. 63–72.

X. Xiao, E. R. Dow, R. Eberhart, Z. B. Miled and R. J.

Oppelt, “Gene clustering using self-organizing maps

and particle swarm optimization,” Proceedings of the

International Parallel and Distributed Processing

Symposium, 2003, pp.22-28.

R. Xu and D. Wunsch, “Survey of clustering algorithms,”

IEEE Transactions on Neural Networks, Vol.16, Issue

3, 2005, pp. 645 – 678.

AN ORDER CLUSTERING SYSTEM USING ART2 NEURAL NETWORK AND PARTICLE SWARM

OPTIMIZATION METHODN