The Behavior of Deep Statistical Comparison Approach for Different Criteria of Comparing Distributions

Tome Eftimov, Peter Korošec, Barbara Koroušić Seljak

2017

Abstract

Deep Statistical Comparison (DSC) is a recently proposed approach for the statistical comparison of meta-heuristic stochastic algorithms for single-objective optimization. The main contribution of the DSC is a ranking scheme, which is based on the whole distribution, instead of using only one statistic, such as average or median, which are commonly used. Contrary to common approach, the DSC gives more robust statistical results, which are not affected by outliers or misleading ranking scheme. The DSC ranking scheme uses a statistical test for comparing distributions in order to rank the algorithms. DSC was tested using the two-sample Kolmogorov-Smirnov (KS) test. However, distributions can be compared using different criteria, statistical tests. In this paper, we analyze the behavior of the DSC using two different criteria, the two-sample Kolmogorov-Smirnov (KS) test and the Anderson-Darling (AD) test. Experimental results from benchmark tests consisting of single-objective problems, show that both criteria behave similarly. However, when algorithms are compared on a single problem, it is better to use the AD test because it is more powerful and can better detect differences than the KS test when the distributions vary in shift only, in scale only, in symmetry only, or have the same mean and standard deviation but differ on the tail ends only. This influence is not emphasized when the approach is used for multiple-problem analysis.

Paper Citation

in Harvard Style

Eftimov T., Korošec P. and Koroušić Seljak B. (2017). The Behavior of Deep Statistical Comparison Approach for Different Criteria of Comparing Distributions.In Proceedings of the 9th International Joint Conference on Computational Intelligence - Volume 1: IJCCI, ISBN 978-989-758-274-5, pages 73-82. DOI: 10.5220/0006499900730082

in Bibtex Style

@conference{ijcci17,
author={Tome Eftimov and Peter Korošec and Barbara Koroušić Seljak},
title={The Behavior of Deep Statistical Comparison Approach for Different Criteria of Comparing Distributions},
booktitle={Proceedings of the 9th International Joint Conference on Computational Intelligence - Volume 1: IJCCI,},
year={2017},
pages={73-82},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006499900730082},
isbn={978-989-758-274-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 9th International Joint Conference on Computational Intelligence - Volume 1: IJCCI,
TI - The Behavior of Deep Statistical Comparison Approach for Different Criteria of Comparing Distributions
SN - 978-989-758-274-5
AU - Eftimov T.
AU - Korošec P.
AU - Koroušić Seljak B.
PY - 2017
SP - 73
EP - 82
DO - 10.5220/0006499900730082