Extreme Learning Machine with Enhanced Variation of Activation
Functions
Jacek Kabziński
Institute of Automatic Control, Lodz University of Technology, Stefanowskiego 18/22, Lodz, Poland
Keywords: Machine Learning, Feedforward Neural Network, Extreme Learning Machine, Neural Approximation.
Abstract: The main aim of this paper is to stress the fact that the sufficient variability of activation functions (AF) is
important for an Extreme Learning Machine (ELM) approximation accuracy and applicability. A slight
modification of the standard ELM procedure is proposed, which allows increasing the variance of each AF,
without losing too much from the simplicity of random selection of parameters. The proposed modification
does not increase the computational complexity of an ELM training significantly. Enhancing the variation of
AFs results in reduced output weights norm, better numerical conditioning of the output weights calculation,
smaller errors for the same number of the hidden neurons. The proposed approach works efficiently together
with the Tikhonov regularization of ELM.
1 INTRODUCTION
Extreme Learning Machine (ELM) is widely
accepted assignment of the learning algorithm for a
single-hidden- layer, feedforward neural network.
The main concepts behind the ELM are that: (i) the
weights and biases of the hidden nodes are generated
randomly and are not adjusted and (ii) the output
weights are determined analytically. ELM may be
applied for modeling (regression) as well as
classification problems, and various modifications of
the basic algorithm are possible. The literature
concerning ELMs is numerous, but some recent
review papers extensively describe the recent trends
and modifications of the standard algorithm (Huang,
Huang, Song, and You, 2015), (Liu, Lin, Fang, and
Xu, 2015), (Lin, Liu, Fang, and Xu, 2015). Very short
learning times and the simplicity of the algorithm are
the most attractive features ELMs. On the other hand,
several publications report that ELM may: create ill-
condition numerical problems (Chen, Zhu, and Wang,
2013), introduce overfitting, require too big number
of neurons (Liu et al., 2015), that the influence of
number of neurons and parameters of random weights
selection on the resulting performance is unclear and
requires analysis (Parviainen and Riihimäki, 2013).
The necessity of improving the numerical properties
of ELM was noticed in several recent publications
(Akusok, Bjork, Miche, and Lendasse, 2015).
In this contribution a standard ELM applied for
regression problems with batch data processing is
considered. Numerical properties of the standard
algorithm are commented and insufficient variation
of the AFs is recognized as the reason of huge output
weights and large modeling errors. The modification
of the input weights and biases selection procedure is
proposed to improve the ELM accuracy and
applicability.
2 STANDARD ELM MODELING
2.1 Components of ELM
2.1.1 Training Data
The training data for a n-input ELM compose a batch
of N samples:
(
,
)
,
∈
,
∈,=1,…,
, (1)
where
denote the inputs and
the desired outputs,
that form the target (column) vector:
=
[
⋯
]
. (2)
It is commonly accepted that the inputs are
normalized to the interval [0,1] each.
Kabzinski, J.
Extreme Learning Machine with Enhanced Var iation of Activation Functions.
DOI: 10.5220/0006066200770082
In Proceedings of the 8th International Joint Conference on Computational Intelligence (IJCCI 2016) - Volume 3: NCTA, pages 77-82
ISBN: 978-989-758-201-1
Copyright
c
2016 by SCITEPRESS – Science and Technology Publications, Lda. All rights reser ved
77
2.1.2 Hidden Neurons
The single, hidden layer of M neurons transforms the
input data into a different representation, called the
feature space. The most popular are “projection-based
neurons”. Each n-dimensional input is projected by
the input layer weights
=
[
,
…
,
]
,=
1,.. and the bias
into the k-th neuron input and
next a nonlinear transformation ℎ
, called activation
function (AF), is applied to obtain the neuron output.
The matrix form of the hidden layer performance on
the batch of N samples is represented by a ×
matrix:
=
ℎ
(
+
)
⋯ℎ
(
+
)
⋮⋱⋮
ℎ
(
+
)
⋯ℎ
(
+
)
. (3)
Although it is not obligatory that the hidden layer
must contain only one kind of neurons, it is usually
the case. Any piecewise differentiable function may
be used as activation function: sigmoid, hyperbolic
tangent, threshold are among the most popular.
Another type of neurons used in ELM is “distance-
based” neurons, such as Radial Basis Functions
(RBF) or multi-quadratic functions. Each neuron uses
the distance from the centroid (represented by
) as
the input to the nonlinear transformation (Guang-Bin
Huang and Chee-Kheong Siew, 2004). The formal
representation of ELM also generates the matrix
similar to (3):
=
ℎ
(‖
−
‖
,
)
⋯ℎ
(‖
−
‖
,
)
⋮⋱⋮
ℎ
(‖
−
‖
,
)
⋯ℎ
(‖
−
‖
,
)
.(4)
The acceptable number of neurons may be found
using validation data, Leave-One-Out validation
procedure, random adding or removing the neurons
(Akusok et al., 2015) or ranking the neurons (Miche
et al., 2010), (Feng, Lan, Zhang, and Qian, 2015),
(Miche, van Heeswijk, Bas, Simula, and Lendasse,
2011).
The weights and the biases of the hidden neurons are
generated on random. The uniform distributions in
[-1,1] for the weights and in [0, 1] for the biases is the
most popular choice. If the data are normalized to
have the zero mean and the unit variance the normal
distribution may be used to generate the neuron
parameters (Akusok et al., 2015).
2.1.3 Output Weights
The output of an ELM is obtained by applying the
output weights to the hidden neurons, therefore the
outputs for all samples in the batch are:
=. (5)
The output weights are found by minimizing the
approximation error:
=
‖
−
‖
. (6)
The optimal solution is:

=
, (7)
where
is the Moore–Penrose generalized inverse
of matrix .
The matrix possesses N rows and M columns. If the
number of training samples N is bigger than the
number of hidden neurons M, ≥, and the matrix
H has full column rank, then
=
(
)

. (8)
If the number of training samples N is smaller than
the number of hidden neurons M, <, and the
matrix H has full row-rank, then
=
(

)

, (9)
but the last case is impractical in modeling.
2.2 Numerical Aspects of ELM
The calculation of the output weights is the most
sensitive stage of the ELM algorithm and may be a
reason for various numerical difficulties. Each of two
ways may be selected to calculate

: specialized
algorithms for calculation of pseudoinverse (8), or
simply, calculation of following matrices:
Λ=
,Ω=,

=Λ

Ω. (10)
The computational complexity of both approaches is
similar, but the second one is characterized by smaller
memory requirements (Akusok et al., 2015). For the
both approaches working with moderate condition
number of
is crucial for the algorithm stability
and is necessary to get moderate values of

. Huge
output weights may reinforce the round-off errors of
arithmetical operations performed by the network and
make the application impossible.
The well-known therapy for numerical problems in
ELM caused by ill-conditioned matrix H is Tikhonov
regularization of the least-square problem (Huang et
al., 2015), (Akusok et al., 2015). Instead of
minimizing (6), the weighted problem is considered,
with the performance index:
NCTA 2016 - 8th International Conference on Neural Computation Theory and Applications
78
=
‖
‖
+
‖
−
‖
, (11)
where >0 is a design parameter. The optimal
solution is:

=
+


. (12)
Inevitably, this modification degrades the modeling
accuracy. The proper choice of C depend on the
problem structure and it is difficult to formulate
general rules. The structure of the matrix H depends
on the number of neurons, samples, and inputs, and
on the shape of activations functions. The target
vector has no influence on H.
To demonstrate that ill-conditioning of the matrix
may really cause problems, a simple example is
demonstrated. Consider the sigmoid activation
functions with input weights and biases taken
randomly according to the uniform distribution in the
interval [-1,1], samples selected according to the
same distribution in the unit cube and calculate the
(
). The mean of condition coefficients
obtained in 20 experiments, for several numbers of
inputs is presented in fig. 1.
The number of hidden neurons was always smaller
than a half of a number of samples. The condition
number of
increases rapidly with a number of
neurons and samples. Performing any calculations
with the numbers as big as 10
14
in the matrix Λ

is
not reasonable.
As it is visible, the problem of ill-conditioning
escalates for small-dimensional modeling,
paradoxically. The regularization is not able to solve
this difficulty efficiently. To keep the condition
number in reasonable constraints
<10

must be
used, as it is presented in fig. 2. It is much bigger than
typically applied values
=50×ℎ.
3 ENHANCED VARIATION OF
ACTIVATION FUNCTIONS
The ELM is not able to learn the features from the
data, as a fully trained neural network does. The
randomly chosen weights happen to specify a linear
mapping to the space of neuron arguments. There-
fore the nonlinear mapping of the data into a feature
space should be able to extract the features sufficient
for predicting the target variable of a regression task.
It is not achieved at all times if the typical rules of
neuron construction (described in section 2) are
applied.
Figure 1: Logarithm of the condition number of
as a
function of batch size and the number of hidden neurons for
different number of inputs.
Figure 2: Logarithm of the condition number of
as a
function of batch size and the number of hidden neurons for
different number of inputs and the regularisation parameter
=10
.
For instance consider a case with 4 neurons, 2 inputs,
100 samples, weights and biases selection according
to the uniform distribution in [-1,1]. The mean
condition number of
in 2000 experiments is ∼
10
and the exemplary plots of activation functions
(from the last experiment) are presented in fig. 3.
It may be noticed that the activation functions are
almost linear and that the range of theirs outputs is
limited.
The importance of sufficient AFs variability was
noticed previously (Parviainen and Riihimäki, 2013),
(Kabziński, 2015). The challenge is to correct the
random mechanism of weights and biases creation to
increase variability without losing too much from the
simplicity of random selection.
0
100
200
300
0
20
40
60
80
100
2
4
6
8
10
12
14
16
18
20
22
No.of Samples
No of Inputs= , 1, 2, 5, 10, 20, 40
No. of Neuron
s
log10(mean(cond(H*H
T
)))
0
100
200
300
0
20
40
60
80
100
2
3
4
5
6
7
8
9
No.of Samples
No of Inputs= , 1, 2, 5, 10, 20, 40
No. of Neurons
log10(m ean( cond (H*H
T
)))
Extreme Learning Machine with Enhanced Variation of Activation Functions
79
Figure 3: Plots of exemplary sigmoid activation functions
obtained from a standard ELM.
For sigmoid AFs the following procedure may be
proposed:
The first step to enlarge the variation of a sigmoid
activation function is to increase the range of uniform
random selection of input weights. It must be suitably
fixed to guarantee that the sigmoid operation neither
remains linear nor too strongly saturates in the input
domain. Therefore the weights are selected from
uniform random distribution in the interval
[
−,
]
.
The values 5<<10 seems suitable.
The selected weights for the k-th neuron are divided
into positive and negative.
Next, the biases may be selected to ensure that the
range of a sigmoid function is sufficiently large. The
minimal value of the sigmoid function
ℎ
(
)
=
(

)
(13)
in the unit cube is achieved at the vertex selected
according to the following rules:
,
>0⇒
=0,
,
<0⇒
=1=1,…,,
(14)
and equals
ℎ
,
=
(
∑
,
:
,


)
, (15)
while the maximal value is achieved at the point given
by:
,
>0⇒
=1,
,
<0⇒
=0=1,…,
(16)
and equals
ℎ
,
=
(
∑
,
:
,


)
. (17)
Assuming that the minimal value of sigmoid AF
should be smaller than the given
, and the maximal
value should be bigger than
, 0<
<
<1
yields the range for the bias selection. It follows from
the inequalities
ℎ
,
<
,ℎ
,
>
, (18)
that the bias
should be selected on random,
uniformly from
∈[,], where
=−
∑
,
:
,

−
−1, (19)
=−
−1−
∑
,
:
,

. (20)
Of course, this approach does not guarantee that the
range of the AF will be
[
,
]
, as the parameters are
still random variables, but at least has chance to be.
A similar strategy may be applied for other
“projection based” AFs.
4 EFFECTIVENESS OF ELM
WITH ENHANCED VARIATION
OF ACTIVATION FUNCTIONS
To demonstrate the effectiveness of the proposed
method, consider a two-dimensional function
=sin(2
(
+
)
,
,
∈[0,1]. (21)
200 samples selected on random constitute the
training set, and 100 samples the testing sets. The
surface (21) was modeled by the standard ELM (S-
ELM), ELM with the enhanced variation of AFs (EV-
ELM) with parameters
=0.1,
=0.9, standard
ELM with regularization (R-ELM) and ELM with
regularization and the enhanced variation of AFs
(EV-R-ELM) with parameters =10
,
=
0.1,
=0.9.
In fig. 4,5 the modeling errors and the conditioning of
S-ELM and EV-ELM are compared. Application of
EV-ELM allows smaller modeling errors and far
better numerical properties of the obtained model.
The output weights are ~10
times smaller in EV-
ELM then in S-ELM.
The reason for this improvement is visible in fig. 6
where the plots of 4 AFs (selected from 60) are
presented. Fig. 6 and 3 demonstrate that in EV-ELM
AFs are nonlinear and cover wider subinterval in
[0,1], however not all of them vary from
to
.
The advantage of EV-ELM is even more obvious if it
is applied together with the regularization. Fig. 7, 8
demonstrate the comparison between R-ELM and
0
0.5
1
0
0.5
1
0.4
0.5
0.6
0.7
0.8
x
y
0
0.5
1
0
0.5
1
0.4
0.5
0.6
0.7
0.8
x
y
0
0.5
1
0
0.5
1
0.7
0.8
0.9
x
y
0
0.5
1
0
0.5
1
0.55
0.6
0.65
x
y
NCTA 2016 - 8th International Conference on Neural Computation Theory and Applications
80
Figure 4: Testing (dotted) and training (solid) errors for S-
ELM (brighter - green) and EV-ELM (darker – blue).
EV-R-ELM. The regularization alone is able to keep
the output weights on a moderate level, but the
modeling becomes unacceptably inaccurate. The EV-
R-ELM improves the modeling accuracy and the
conditioning together. Similar experiments were
conducted for different modeling problems: with the
training output corrupted by noise, with multiple
inputs, with numerous extrema, and always the
conclusions were similar.
5 CONCLUSIONS
The main aim of this paper was to stress the fact that
the sufficient variability of AFs is important for an
ELM model accuracy and applicability. A slight
modification of the standard ELM procedure is
proposed, which allows enhancing the variance of
each AF, without losing too much from the simplicity
of random selection of parameters.
First, it is proposed to change the parameter of the
random distribution of the input weights, next to
modify the random distribution for the basses, taken
already established weights into account. The
proposed modification does not increase the
computational complexity of an ELM training
significantly – only two simple calculations for each
bias are added. Enhancing the variation of AFs results
in reduced output weights norm, better numerical
conditioning of the output weights calculation,
smaller errors for the same number of the hidden
neurons. The proposed approach works efficiently
together with the Tikhonov regularization applied in
ELM.
Figure 5: Logarithm of the norm of the output weights
(stars) and ((
) (circles) for S-ELM (brighter
- green) and EV-ELM (darker – blue).
Figure 6: Selected AFs (4 from 60) from the EV-ELM
model.
Figure 7: Testing (dotted) and training (solid) errors for R-
ELM (brighter - green) and EV-R-ELM (darker – blue).
0 10 20 30 40 50 60
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
No of hidden neurons
ELM errors
train err.
test er r.
train err.
test er r.
0 10 20 30 40 50 60
-5
0
5
10
15
20
No of hidden neurons
log(norm(OW )
log( c ond( H*H
T
)
log(norm(OW )
log( c ond( H*H
T
)
0
0.5
1
0
0.5
1
0
0.5
1
x
1
X
2
0
0.5
1
0
0.5
1
0
0.5
1
x
1
X
2
0
0.5
1
0
0.5
1
0
0.5
1
x
1
X
2
0
0.5
1
0
0.5
1
0
0.5
1
x
1
X
2
0 10 20 30 40 50 60
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
No of hidden neurons
ELM errors
train err .
test er r.
train err .
test er r.
Extreme Learning Machine with Enhanced Variation of Activation Functions
81
Figure 8: Logarithm of the norm of the output weights
(stars) and ((
) (circles) for R-ELM (brighter
- green) and EV-R-ELM (darker – blue).
REFERENCES
Akusok, A., Bjork, K. M., Miche, Y., and Lendasse, A.
(2015). High-Performance Extreme Learning Machi-
nes: A Complete Toolbox for Big Data Applications.
IEEE Access, 3, 1011–1025. http://doi.org/10.1109/
ACCESS.2015.2450498
Chen, Z. X., Zhu, H. Y., and Wang, Y. G. (2013). A
modified extreme learning machine with sigmoidal
activation functions. Neural Computing and
Applications, 22(3-4), 541–550. http://doi.org/10.1007/
s00521-012-0860-2
Feng, G., Lan, Y., Zhang, X., and Qian, Z. (2015). Dynamic
adjustment of hidden node parameters for extreme
learning machine. IEEE Transactions on Cybernetics,
45(2), 279–288. http://doi.org/10.1109/TCYB.2014.23
25594
Guang-Bin Huang, and Chee-Kheong Siew. (2004).
Extreme learning machine: RBF network case. In
ICARCV 2004 8th Control, Automation, Robotics and
Vision Conference, 2004. (Vol. 2, pp. 1029–1036).
IEEE. http://doi.org/10.1109/ICARCV.2004.1468985
Huang, G., Huang, G. Bin, Song, S., and You, K. (2015).
Trends in extreme learning machines: A review. Neural
Networks, 61, 32–48. http://doi.org/10.1016/j.neunet.
2014.10.001
Kabziński, J. (2015). Is Extreme Learning Machine
Effective for Multisource Friction Modeling? (in
Artificial Intelligence Applications and Innovations,
Springer pp. 318–333). http://doi.org/10.1007/978-3-
319-23868-5_23
Lin, S., Liu, X., Fang, J., and Xu, Z. (2015). Is extreme
learning machine feasible? A theoretical assessment
(Part II). IEEE Transactions on Neural Networks and
Learning Systems, 26(1), 21–34. http://doi.org/10.
1109/TNNLS.2014.2336665
Liu, X., Lin, S., Fang, J., and Xu, Z. (2015). Is extreme
learning machine feasible? A theoretical assessment
(Part I). IEEE Transactions on Neural Networks and
Learning Systems, 26(1), 7–20. http://doi.org/10.1109/
TNNLS.2014.2335212
Miche, Y., Sorjamaa, A., Bas, P., Simula, O., Jutten, C., and
Lendasse, A. (2010). OP-ELM: Optimally pruned
extreme learning machine. IEEE Transactions on
Neural Networks, 21(1), 158–162. http://doi.org/10.
1109/TNN.2009.2036259
Miche, Y., van Heeswijk, M., Bas, P., Simula, O., and
Lendasse, A. (2011). TROP-ELM: A double-regulari-
zed ELM using LARS and Tikhonov regularization.
Neurocomputing, 74(16), 2413–2421. http://doi.org/
10.1016/j.neucom.2010.12.042
Parviainen, E., and Riihimäki, J. (2013). A connection
between extreme learning machine and neural network
kernel. In Communications in Computer and
Information Science (Vol. 272 CCIS, pp. 122–135).
0 10 20 30 40 50 60
-5
0
5
10
15
20
No of hidden neurons
log(norm(OW)
log(c ond( H*H
T
)
log(norm(OW)
log(c ond( H*H
T
)
NCTA 2016 - 8th International Conference on Neural Computation Theory and Applications
82