0 5 10 15
10
−20
10
−10
10
0
10
10
Iteration #
Residuals
Residuals for example AC3, standard Newton
Residual
Normalized Residual
Figure 4: Residual and normalized residual trajectories for
COMPl
e
ib example
AC3
, using standard Newton algorithm.
residual and normalized residual trajectories obtained
using the modified Newton algorithm. The first two
step sizes have values about 2.9·10
−4
and 1.17·10
−3
,
and the correspondingresiduals are very close, 1.7315
and 1.7298. The algorithm then uses a step size 1,
which increases the residual to the value 1.94 · 10
4
.
The next iteration takes a step size of about 1.997, re-
ducing the residual to 6.18. The previous behavior is
repeated once more (with different values), but then
the algorithm converges fastly. The standard Newton
algorihm has a smoother behaviour from iteration 1
on (similar to that for example
AC3
in Figure 4), but
it starts with a larger value, 1.19 · 10
7
, and needs one
additional iteration.
0 5 10 15
10
−20
10
−15
10
−10
10
−5
10
0
10
5
Iteration #
Residuals
Residuals for example AC15, modified Newton
Residual
Normalized Residual
Figure 5: Residual and normalized residual trajectories for
COMPl
e
ib example
AC15
, using modified Newton algo-
rithm.
Even if, after setting the step to 1, the modified algo-
rithm produces a residual almost as large as for the
first iteration of the standard Newton algorithm, usu-
ally the next step produces a much lower residual, and
so, the number of iterations decreases. Such a behav-
ior has been seen, e.g., for COMPl
e
ib example
JE1
(n = 30, m = 3), for which, at step 3 of the Newton
algorithm, the residual rose to 3.31·10
10
(from 5.38 at
the previous iteration), but the next residual value was
1.17· 10
4
. The standard algorithm recorded the max-
imal residual of 5.43 · 10
11
at the first iteration. The
number of iterations for convergence were 11 and 17,
respectively.
Slow convergence. The modified Newton algo-
rithm usually converges faster than the standard New-
ton algoritm. However, there are examples for which
the standard algorithm requires less iterations than the
modified algorithm. This happened for COMPl
e
ib
examples
AC5
,
AC18
,
JE2
,
JE3
,
DIS5
,
WEC1
,
NN11
,
CM4 IS
,
CM5 IS
, and
FS
, for which the modified al-
gorithm needed additional 5, 5, 3, 3, 4, 6, 1, 6, 9, and
1 iterations, respectively. To understand the reason
of such a behaviour, consider example
AC18
. Algo-
rithm mN needed 19 iterations, while Algorithm sN
needed only 14. The reason is that Algorithm mN
uses 7 small, comparable steps, but the residual for
all these steps is reduced only by a factor of about
3.5 (from 4.4 · 10
3
to 1.25 · 10
3
). (On the other
hand, using step sizes equal to 1 reduced the resid-
ual faster.) A similar behavior appears for COMPl
e
ib
example
WEC1
, for which Algorithm mN needs 23 it-
erations, while Algorithm sN needs only 17. The
modified algorithm uses 9 small steps (with values in
[0.21,0.33]), but the residual is reduced only with a
factor of about 3 (from 3.1· 10
7
to 1.02 · 10
7
).
This slow convergence is not discovered by the
implemented stagnation detection procedure. A pos-
sible way to improve such a behaviour is to monitor
the succesive reduction factors, defined as ratios be-
tween two consecutive residuals. If the current resid-
ual is high (e.g., higher than 1000) and the mean value
of reduction factors over a certain number of consec-
utive preceding iterations (say, 4) is lower than a cer-
tain value (e.g., 4), then a step size of 1 might improve
the convergence rate. This procedure has not yet been
tried. The rationale for choosing the values above is
as follows. First, such a procedure is not needed when
the residuals are small, since then few iterations will
be sufficient to achieve convergence. Moreover, for
Algorithm mN, the reduction factors may have a large
variation from one iterate to the next one, and there-
fore a moving window of more than two or three iter-
ations is needed to have a useful value for the mean.
Finally, the mean value of 4 is a reasonable selection,
since it was observed that this is the usual value of the
ratios for the initial part of the standard Newton al-
gorithm. Indeed, this was the case for over 90 out of
150 tested COMPl
e
ib examples. Other 30 examples
had ratios larger than 4, while the remaining exam-
ples had either lower values or converged too quickly.
Figure 6 and Figure 7 show the ratios between
ICINCO2014-11thInternationalConferenceonInformaticsinControl,AutomationandRobotics
408