Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing

Anikó Csébfalvi, György Csébfalvi

2012

Abstract

17 years ago, Hooker (1995) presented a pioneering work with the following title: "Testing Heuristics: We Have It All Wrong". If we ask the question now: "Do we have it all wrong?" the answer will be undoubtedly yes. The problem of the fair comparison remained essentially the same in the heuristic community. When we use stochastic methods in the optimization (namely heuristics or metaheuristics with several tunable parameters and starting seeds) then the usual presentation practice: "one problem - one result" is extremely far from the fair comparison. From statistical point of view, the minimal requirement is a so-called "small-sample" which is a set of results generated by independent runs and an appropriate "small-sample-test" according to the theory of the experimental design and evaluation and the protocol used for example, in the drug development processes. The viability and efficiency of the proposed statistically correct "bias-free" nonparametric methodology is demonstrated using a well-known nonlinear structural optimization example on the set of state-of-the-art heuristics. In the motivating example we used the presented solutions as a small-sample generated by a "hyperheuristic" and we test its quality against ANGEL, where the "supernatural" hybrid metaheuristic ANGEL combines ant colony optimization (AN), genetic algorithm (GE) and a gradient-based local search (L) strategy. ANGEL is an "essence of the different but at the same time similar heuristic approaches". The extremely simple and practically tuning-free ANGEL presents a number of interesting aspects such as extremely good adaptability and the ability to cope with totally different large real applications from the highly nonlinear structural optimization to the long-term optimization of the geothermal energy utilization.

References

  1. Adeli, H., Kamal, O., (1991). Efficient optimization of plane trusses, Advances in Engineering Software; 13 (3), 116-122.
  2. Barr, R.S., Goldenm, B. L., Kelly, J. P., Resende, M. G. C., Stewart, W. R., (1995). Design and Reporting on Computational Experiments with Heuristic Methods. Journal of Heuristics, 1, 9-32.
  3. Csébfalvi, A., (2007). Angel method for discrete optimization problems. Periodica Polytechnica: Civil Engineering, 51/2 37-46, doi: 10.3311/pp.ci.2007- 2.06, http://www.pp.bme.hu/ci.
  4. Csébfalvi, A., (2009). A hybrid meta-heuristic method for continuous engineering optimization. Periodica Polytechnica: Civil Engineering, 53 (2), 93-100, doi: 10.3311/pp.ci.2009-2.05.
  5. Csébfalvi, A., (2011). Multiple constrained sizing-shaping truss-optimization using ANGEL method. Periodica Polytechnica: Civil Engineering, 55 (1), 81-86, doi: 10.3311/pp.ci.2011-1.10, http://www.pp.bme.hu/ci.
  6. Csébfalvi, A., Schreiner, J., (2011) A Net Present Value Oriented Hybrid Method to Optimize the Revenue of Geothermal Systems with Respect to Operation and Expansion, In: Tsompanakis, Y., Topping, B.H.V. (Eds.) Proceedings of the Second International Conference on Soft Computing Technology in Civil, Structural and Environmental Engineering. Chania, Greece, Civil-Comp Press, Stirling, UK.
  7. Dobbs, M. W., Nelson, R. B., (1976). Application of optimality criteria to automated structural design. AIAA Journal, 14 (10), 1436-43.
  8. El-Sayed, M. E., Jang, T. S., (1994). Structural optimization using unconstrained non-linear goal programming algorithm. Computers and Structures, 52 (4), 723-727.
  9. Galante, M., (1992). Structures optimization by a simple genetic algorithm. In Proceedings of Numerical Methods in Engineering and Applied Sciences, CIMNE: Barcelona, Spain, 862-870.
  10. Haftka, R. T., Kamat, M. P., (1985). Elements of Structural Optimization, Martinus Nighoff: Dordrecht.
  11. Haug E. J., Arora, J. S., (1979) Applied Optimal Design, Wiley: New York.
  12. Hooker, J. N., (1995). Testing Heuristics: We Have It All Wrong. Journal of Heuristics, 1, 33-42.
  13. Kaveh, A., Talatahari, S., (2009) Particle swarm optimizer, ant colony strategy and harmony search scheme hybridized for optimization of truss structures. Computers and Structures, 87 (5-6), 267-283.
  14. Khan, M. R., Willmert, K. D., Thornton, W. A., (1979). An optimality criterion method for large-scale structures. AIAA Journal 17 (7), 753-61.
  15. Koohestani, K., Kazemzadeh Azad, S., (2009). An Adaptive Real-Coded Genetic Algorithm for Size and Shape Optimization of Truss Structures. In Topping, B.H.V., Tsompanakis Y. (Eds.), Proceedings of the First International Conference on Soft Computing Technology in Civil, Structural and Environmental Engineering, Civil-Comp Press, Stirlingshire, UK.
  16. Lee, K. S., Geem, Z. W., (2004). A new structural optimization method based on the harmony search algorithm, Computers and Structures, 82 (9-10), 781- 798.
  17. Lemonge, A. C. C., (1999). Application of Genetic algorithms in Structural Optimization Problems. Ph.D. Thesis, Program of Civil Engineering-COPPE, Federal University of Rio de Janeiro, Brazil.
  18. Lemonge, A. C. C., Barbosa, H. J. C., (2004). An adaptive penalty scheme for genetic algorithms in structural optimization. International Journal of Numerical Methods in Engineering, 59, 703-736.
  19. Li, L. J., Huang, Z. B., Liu, F., Wu, Q. H., (2007). A heuristic particle swarm optimizer for optimization of pin connected structures. Computers and Structures, 85(7-8), 340-349.
  20. Rizzi, P., (1976). Optimization of multiconstrained structures based on optimality criteria. In Proceedings of 17th Structures, Structural Dynamics, and Materials Conference, King of Prussia, PA.
  21. Schmit, L. A. Jr, Farshi, B., (1974). Some approximation concepts for structural synthesis. AIAA Journal, 12 (5), 692-9.
  22. Schmit, L. A., Miura, H., (1976). Approximation concepts for efficient structural synthesis. NASA CR-2552, Washington, DC: NASA.
  23. Venkayya, V. B., (1971). Design of optimum structures. Computers and Structures, 1 (1-2), 265-309.
Download


Paper Citation


in Harvard Style

Csébfalvi A. and Csébfalvi G. (2012). Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing . In Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012) ISBN 978-989-8565-33-4, pages 306-309. DOI: 10.5220/0004168403060309


in Bibtex Style

@conference{ecta12,
author={Anikó Csébfalvi and György Csébfalvi},
title={Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing},
booktitle={Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012)},
year={2012},
pages={306-309},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004168403060309},
isbn={978-989-8565-33-4},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 4th International Joint Conference on Computational Intelligence - Volume 1: ECTA, (IJCCI 2012)
TI - Fair Comparison of Population-based Heuristic Approaches - The Evils of Competitive Testing
SN - 978-989-8565-33-4
AU - Csébfalvi A.
AU - Csébfalvi G.
PY - 2012
SP - 306
EP - 309
DO - 10.5220/0004168403060309