Studying the Relationship Between Crossover Features and Performance

on MNK-Landscapes Using Regression Models

Teruhisa Nakashima

, Hern

an Aguirre

and Kiyoshi Tanaka

Department of Electrical and Computer Engineering, Shinshu University, Japan

{23w2805a, ahernan, ktanaka}@shinshu-u.ac.jp

Keywords:

Crossover Features, Performance, Regression Models, Evolutionary Multi-Objective Optimization,

MNK-Landscapes.

Abstract:

Crossover is a key component of evolutionary algorithms and has been the focus of numerous studies. Its

effectiveness depends on the operator’s properties to mix information, the speciﬁc characteristics of the prob-

lem, and the diversity of the population, inﬂuenced by the dynamics of the algorithm. This study focuses on

binary representations and introduces a method to examine the relationship between crossover features and

the performance of a multi-objective evolutionary algorithm on problem subclasses with random and neighbor

patterns of variable interactions. The aim is to identify the crossover features relevant to performance in each

problem subclass through regression models.

1 INTRODUCTION

In the real world, many optimization problems require

optimizing multiple objectives simultaneously. Nu-

merous types of algorithms have been proposed to ad-

dress these problems, one of which is Multi-objective

Evolutionary Algorithms (MOEAs). MOEAs are op-

timization method inspired on biological evolution,

where genetic operations are repeatedly applied to in-

dividuals with solutions represented as genes to per-

form optimization.

To enhance the solution-searching capability of a

MOEA, it is necessary to tune its components. One

such component is crossover, a genetic operator that

generates offspring by combining the genetic infoma-

tion from parent individuals. The role of crossover in

evolutionary algorithms has been the subject of sev-

eral studies (Spears, 2000). Various crossover meth-

ods have been proposed, and their properties for mix-

ing information have been investigated, along with

their effects on performance, particularly on single

objective optimization. In general, the effectiveness

of crossover depends on the properties of the operator,

the characteristics of the problem, and the population

diversity, which is inﬂuenced by the dynamics of the

algorithm.

MOEAs evolve a population of solutions aiming

to ﬁnd a set of Pareto non-dominated solutions, which

https://orcid.org/0009-0009-7637-0925

https://orcid.org/0000-0003-4480-1339

are usually spread in a broad region of decision and

objective space. In evolutionary multi-objective op-

timization, there are several studies focusing on the

diversity of solutions present in the population and

the disruptive nature of crossing parent individuals

that could be far apart in decision space (Ishibuchi

et al., 2014), (Sato et al., 2013), (Sato et al., 2007).

In MOEAs, crossover methods are often compared

based on the performance achieved by the algorithm

on a few problems, without much consideration of the

operator’s properties.

This work focuses on binary representations and

presents a method to study and assess the relation-

ship between crossover features and the performance

of a MOEA on subclasses of problems, for which we

know some of their properties. Our aim is to identify

the main crossover features that are relevant to per-

formance in each problem subclass using regression

models of the features.

We ﬁrst deﬁne features that capture the

information-mixing characteristics of crossover

operators, and then apply a crossover feature extrac-

tion method that is independent of the characteristics

of the particular problem or the dynamics of the al-

gorithm. For a given crossover operator, we quantify

these features on a sample of solutions generated

by the operator. Separately, for each crossover for

which features were quantiﬁed, we run several times

a multi-objective evolutionary algorithm on instances

of a problem subclass to collect information about

the performance of the algorithm. Then, we create re-

Nakashima, T., Aguirre, H. and Tanaka, K.

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models.

DOI: 10.5220/0012941200003837

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Joint Conference on Computational Intelligence (IJCCI 2024), pages 84-95

ISBN: 978-989-758-721-4; ISSN: 2184-3236

gression models to describe the relationship between

the features and the performance of an algorithm.

We study the proposed approach by solving

several instances of MNK-Landscapes(Aguirre and

Tanaka, 2004)(Aguirre and Tanaka, 2007) with N =

20 bits, M = 2 objectives, and K = 2 and 4 interacting

variables with nearest neighbor and random models

of epistasis. For each combination of M, N, K, and

type of epistasis model, we ﬁnd an optimal model that

determines the important features for each subclass of

problems and explains performance in terms of them.

We show that the models, tested on unseen problem

instances, correctly predict better performance by the

top-ranked operators. The proposed method could be

useful to select an appropriate crossover operator.

2 METHOD

2.1 Overview

This work presents a method to study and assess the

relationship between crossover features and the per-

formance of a MOEA on subclasses of problems. The

main steps of the method are as follows:

Step 1. Deﬁne features of the information-mixing

characteristics of crossover operators, specify a

set of crossover operators and compute their fea-

tures.

Step 2. Deﬁne a problem subclass and several in-

stances of it. With each speciﬁed crossover, run

a MOEA a number of times on several instances

of a problem subclass. Then, compute for each

crossover a performance value in the problem sub-

class.

Step 3. Learn a regression model to capture the re-

lationship between the crossover features and the

performance values.

Step 4. Validate the predictive accuracy of the regres-

sion model on unseen instances of the problem

subclass.

In the following we detail the characteristics of the

method.

2.2 Crossover Feature Extraction

The feature extraction approach initially generates a

sample offspring population from the same pair of

parent individuals using a crossover method. The ge-

netic information of the offspring population is then

used to compute certain features of the operator.

Let us denote X a crossover operator, and p

, p

two parent individuals that are deﬁned as binary vec-

tors of length N, i.e. p

= (p

(i,1)

,··· , p

(i,N)

) ∈{0,1}

where N is the number of variables. A crossover op-

eration on the parents is denoted X (p

) and gen-

erates two binary vectors ∈{0, 1}

as offspring.

To efﬁciently track the origin of the genes in the

offspring and analyze features, we specify p

and

so that their corresponding j-th variables p

(1, j)

and p

(2, j)

contain different values. Namely, we

prepare parent individuals p

= (0,0,0,··· , 0) and

= (1,1, 1, ··· , 1) with N bits, perform crossover

X (p

) M times to generate a sample offspring

population {x

,··· ,x

}, and represent the i-th

offspring as x

= (x

(i,1)

(i,2)

,··· ,x

(i,N)

). The ob-

tained sample population is then analyzed for “search

ability” and “sequence inﬂuence,” extracting four fea-

tures(Caruana et al., 1989).

“Search ability” refers to the capability to efﬁ-

ciently search through a multitude of solution candi-

dates, which varies between the early and late stages

of the search. In the early stage, the ability to ex-

pand the search range to discover good solutions is

needed. In the later stage, the ability to reﬁne already

good solution candidates is required. Two features,

“Unique Rate” and “Parent Bias,” are calculated to

observe these abilities.

“Sequence inﬂuence” refers to the impact of man-

aging solutions as a concatenation of genes, which

usually involves arranging the genes in contiguous lo-

cations in a one-dimensional array. This arrangement

can cause various inﬂuences. Two features, namely

“Adjacent Bias” and “Position Bias,” are calculated

to observe these inﬂuences.

The next section explains in detail the features.

2.3 Features

2.3.1 Unique Rate

Unique Rate (UR) represents the extent to which new

individuals can be generated from the same parent

individuals by counting the number of unique indi-

viduals in the sample offspring population. A higher

value indicates fewer duplicates and easier genera-

tion of new individuals during the search. Let us

denote the vector containing the uniqueness status

of the offspring in the sampled population as u =

,··· ,u

) ∈ {0, 1}

, where u

= 1 indicates x

unique and u

= 0 otherwise,







1 (i = 1)

0 (i ̸= 1 ∧x

∈ {x

,··· ,x

i−1

})

1 (i ̸= 1 ∧x

/∈ {x

,··· ,x

i−1

}).

(1)

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models

The number of unique offspring divided by the total

number of offspring 2M gives the Unique Rate,

UR =

∑

i=1

×100. (2)

2.3.2 Parent Bias

Parent Bias (PaB) represents the similarity of off-

spring to the two parent individuals. A higher value

indicates greater similarity to one of the parents, sug-

gesting an inheritance bias towards one of parent

characteristics during the search. When offspring re-

ceive genes equally from both parents, the total num-

ber of genes received from each parent should be half

the number of bits N. To calculate parent bias, we

use the absolute value of the difference between the

actual number of genes from one parent and N/2. In

our case, it is easy to count the bits set to 1 in the off-

spring, which are known to come from parent p

. The

average of these differences in the sampled offspring

population gives the Parent Bias,

PaB =

∑

i=1

∑

j=1

(i, j)

)−

×100. (3)

2.3.3 Adjacent Bias

Adjacent Bias (AB) represents the continuity of gene

inheritance from the same parent, indicating the

tendency of neighboring genes in the offspring to

originate from the same or different parents. A

higher value shows greater continuity. Given the

offspring x

, we count at the j-th bit position, j =

1,··· ,N, the length l

of the uninterrupted sequence

(i, j−l

)

,··· ,x

(i, j−1)

(i, j)

of bits inherited from the

same parent, i.e., x

(i,k)

∈ p

∨ x

(i,k)

∈ p

, ∀k ∈ {j −

,··· , j −1, j}. Let us denote the vector of coun-

ters l

for the offspring x

as v

= (v

(i,1)

,··· ,v

(i,N)

(i, j)

= l

∈ {0,··· ,N −1}, j = 1,··· , N. Since all

bits from p

are 0s and from p

are 1s, the counter of

continuous inheritance v

(i, j)

for the i-th offspring at

the j-th bit position is easily computed as follows,

(i, j)







0 ( j = 1)

(i, j−1)

+ 1 ( j ̸= 1 ∧x

(i, j−1)

= x

(i, j)

)

0 ( j ̸= 1 ∧x

(i, j−1)

̸= x

(i, j)

(4)

The sum of the element of v

divided by the maxi-

mum possible sum of v

across all offspring gives the

Adjacent Bias,

AB =

∑

i=1

∑

j=1

(i, j)

N(N−1)

×100. (5)

AB = 0 if all 2M offspring are perfectly uni-

form. That is, ∀i = 1,··· ,2M, for each bit position

j = 2, ··· , N −1, the bit x

(i, j)

came from one parent

and the neighboring bits x

(i, j−1)

and x

(i, j+1)

from the

other parent. On the other hand, AB = 100 if all 2M

offspring are clones of one of the parents. That is,

∀i = 1, ··· , 2M, at each bit position j = 1,··· ,N, the

bit x

(i, j)

came from the same parent.

2.3.4 Position Bias

Upon examination of a particular gene position (loci)

within the sampled offspring population, if on aver-

age half of the offspring inherits its gene from parent

and the other half from parent p

, we say that the

operator is positionally unbiased.

Position Bias (PoB) measures the frequency of the

parents’ genes in their offspring at every gene position

and computes the deviation from the expected posi-

tionally unbiased frequency. A higher value shows

stronger positional bias. Since two offspring are gen-

erated per crossover, only one offspring is used for

calculation. The gene frequency at each gene posi-

tion is determined by the sum of gene values ‘1’ from

parent p

at that position. The difference between this

sum and half the number of crossover operations, av-

eraged across gene positions is the positional bias, as

shown below,

PoB =

∑

j=1

∑

i=1

(2i, j)

)−

×100. (6)

2.4 Performance Value

In our study we consider several crossover opera-

tors, which for convenience we identify as X

, k =

1,··· ,α. To analyze the relationship between

crossover features and performance, we calculate a

performance value (PV) on a problem subclass for

each crossover operator. MOEAs are stochastic algo-

rithms. To provide an estimate of their performance

on an instance of the problem subclass, they are usu-

ally run several times on the same instance. This

leads to a range of values for the performance met-

ric, where the mean performance of the MOEA lies

within a certain probability. This range is expected to

vary from instance to instance, though the instances

belong to the same problem subclass. Thus, to com-

pute PV we take into account the diverse ranges of

the performance metric across multiple problem in-

stances within the subclass.

As mentioned above, the MOEA set with

crossover X

is run several times on the same in-

stance. To measure performance, we use the Hyper-

volume (HV)(Zitzler et al., 2003) computed on the set

of non-dominated solutions in the population at the ﬁ-

nal generation of the MOEA’s run. Let us denote the

ECTA 2024 - 16th International Conference on Evolutionary Computation Theory and Applications

problem instances with the index i = 1,···,β, and the

runs of the MOEA with the index j = 1,··· ,ρ. Then,

(i, j)

denotes the HV achieved by the MOEA with

crossover X

in the j-th run on the i-th problem in-

stance.

First, the average HV value of all runs on the i-th

problem instance for crossover X

was calculated by

∑

j=1

(i, j)

. (7)

Next, the average HV value of all crossover operators

on the i-th problem instance was computed by

∑

k=1

. (8)

To address the differences in ranges of HV value

across problem instances, we standardized the HV

value (SHV) for crossover X

on the i-th problem in-

stance by

SHV

−HV

∑

k=1

(HV

−HV

)

. (9)

The average SHV for each crossover across all prob-

lem instances was taken as the performance value PV

for the problem subclass,

= SHV

∑

i=1

SHV

. (10)

2.5 Model Learning

The performance of the MOEA set with the crossover

operator X

,k = 1,··· ,α, on a problem subclass and

the features measured for the same operator are col-

lected as data points. We extract the features unique

rate UR

, parent bias PaB

, adjacent bias AB

, and

positional bias PoB

using equations (2)-(6). We re-

peat this process several times using different random

seeds and take their average as the feature value of

the crossover, i.e., UR

, PaB

, AB

, and PoB

Thus, for each problem subclass, we have α data

points (PV

,UR

,PaB

,AB

,PoB

) correspond-

ing to the same number of crossover types used in this

study.

To examine the relationship between crossover

features and problem performance, PV was used as

the dependent variable, and the four features UR, PaB,

AB, and PoB were used as the explanatory variables.

For each problem subclass, linear regression models

PV = f (U R,PaB,AB,PoB) were created for all 15

combinations of the explanatory variables. Models

in which the coefﬁcients of all explanatory variables

had a p-value of 0.05 or lower were retrained, splitting

the data randomly into training (80%) and validation

(20%) datasets to calculate the Root Mean Square Er-

ror (RMSE) and Mean Absolute Error (MAE). This

process was repeated 50 times, using different ran-

dom seeds for the splitting of the data, to obtain the

average RMSE and MAE. The model f

∗

with the low-

est RMSE and MAE on the validation datasets was

considered the linear regression model that best cap-

tures the relationship between the problem subclass

and crossover features.

2.6 Model Veriﬁcation

To verify the regression model’s accuracy, we

use the selected model f

∗

to predict the per-

formance value of each crossover, i.e.,

∗

(UR

,PaB

,AB

,PoB

). We sort according to

and determine the sets of the 5 best and worst

performing crossovers, deﬁned as follows

best

= {X

| 1 ≤i ≤5}

worst

= {X

| α −5 ≤i ≤ α},

where (k

,··· ,k

) is the the ordered list of crossover

indexes such that

> ··· >

. In

addition, we also determine the mean performing

crossover, i.e., those with the predicted performance

value

closest to zero. We sort according to

the absolute value |

| and determine the 5 mean

performing crossovers as follows

mean

= {X

| 1 ≤i ≤5},

where (k

,··· ,k

) is the the ordered list of crossover

indexes such that |

| < |

| < ··· < |

For each crossover operator X

∈ X

best

, X

∈

mean

and X

∈ X

worst

, we optimize 10 different

unseen instances 50 times each with different ran-

dom seeds using the same MOEA conﬁguration. For

the i-th unseen instance, we compute the HV for

each run of the MOEA with crossover X , HV

{HV

(i,1)

,··· ,HV

(i,50)

}, and perform pairwise t-tests

between the HV results by each crossover X

∈ X

best

and each crossover X

∈ X

worst

. Thus, in total we

perform 25 t-tests per instance and summarize their

results by counting the number of times the average

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models

hypervolume HV

by X

∈X

best

is better, no signiﬁ-

cantly different, or worse than HV

by X

∈ X

worst

Similarly, we perform pairwise t-tests for each X

∈

best

and each X

∈ X

mean

and summarize their re-

sults.

3 MNK-LANDSCAPES

NK-Landscapes (Kauffman, 1993) are well-known

mathematical models of rugged ﬁtness landscapes

with a conﬁgurable number of variables and land-

scape ruggedness, speciﬁed by the parameters N and

K, respectively. They were extended to the multi-

objective domain by adding another parameter M,

which deﬁnes the number of objectives, and consider-

ing different variable epistasis models for each objec-

tive. This multi-objective version is known as MNK-

Landscapes (Aguirre and Tanaka, 2004) and is math-

ematically represented as:

(x) =

∑

j=1

i, j

, z

(i, j)

,...,z

(i, j)

| {z }

bits interacting with x

), (11)

where f

denotes the i-th ﬁtness function, i =

1,2,...,M, x = (x

,...,x

) ∈ {0,1}

, f

i, j

: B

→

R gives the ﬁtness contribution of bit x

to f

, and

(i, j)

,...,z

(i, j)

are the K

bits interacting with bit x

the string x. This means that the ﬁtness value of the

bit x

depends not only on its own value but also on

the other K

bits that it interacts with. In other words,

the subfunction f

i, j

(i, j)

,...,z

(i, j)

) is a look-

up table of 2

ﬁtness values, one per each com-

bination of the K

+ 1 variables. These ﬁtness val-

ues are random real numbers in the range [0.0,1.0]

drawn from a uniform distribution U (0,1). A larger

creates a more rugged landscape, which makes the

benchmark problem more difﬁcult to solve (Aguirre

and Tanaka, 2004)(Aguirre and Tanaka, 2007)(Pe-

likan, 2010)(Daolio et al., 2015)(Martins et al., 2021).

The K

bits that interact with bit x

deﬁne an epis-

tasis model of variable interactions, which is speciﬁed

either by the nearest neighbor bits of x

or random

bits, i.e. z

(i, j)

= x

∈{x

,...,x

}and

j = U(1,...,N).

Note that when the epistasis model is random, the

subset of bits that interact with x

is different for each

ﬁtness function. However, even if the epistasis model

for bit x

is the same in all objectives, as is the case

in the nearest neighbor model, the ﬁtness contribution

of x

is different for each objective function due to the

random generation of f

i, j

The properties of MNK-Landscapes in regards to

each conﬁgurable parameter, number of variables, ob-

Table 1: Crossover Operators.

Crossover Parameter

One Point

Two Point

Multi Point {3,4,··· ,19}

Uniform

Half Uniform

Exponential {0.05,0.1, ··· , 0.9}

Binomial {0.05,0.1,··· ,0.9}

Table 2: Crossover Feature Extraction.

Number of Bits N 20

Crossover Executions M 50

Experiments 100

Crossover Types α 82

Table 3: MNK-Landscapes.

Objectives O 2

Variables N 20

Interacting Variables K 2, 4

Epistasis Model Random, Near

Instances for Training β 30

Instances for Veriﬁcation 10

Table 4: MOEA.

Generations 500

Population Size 100

Crossover See Table1

Crossover Probability 1.0

Mutation Bit ﬂip

Bit Mutation Rate 0.1

Runs for Training γ 10

Runs for Validation 50

jectives, and epistasis, and their effect on the perfor-

mance of MOEA have been studied by Aguirre et al in

(Aguirre and Tanaka, 2004) and (Aguirre and Tanaka,

2007).

In this work, we analyze MNK-Landscapes with

random and nearest neighbor models of epistasis with

the same number K of interacting variables in all ob-

jective functions, i.e. K

= K,i = 1,...,M.

4 EXPERIMENTAL DETAILS

We investigate the relationship between crossover

features and performance on subclasses of problems

with interacting variables under nearest neighbor and

random models of epistasis.

The crossover used in this study are listed in Ta-

ble 1. They include the commonly used One and

ECTA 2024 - 16th International Conference on Evolutionary Computation Theory and Applications

Two Point crossovers. Uniform crossover (Syswerda,

1989) clones the parents and swaps bits with a prob-

ability of 0.5 to create offspring. Half Uniform

crossover swaps exactly half of the non matching bits

between the parents. Multi Point crossover uses the

number of crossing points as a parameter, whereas

Exponential and Binomial crossover operators (Storn

and Price, 1995)(Price et al., 2005) use a crossover

rate per variable as a parameter. The Binomial

crossover applies the speciﬁed crossover rate to all

variables, similar to Uniform crossover. On the other

hand, the Exponential crossover applies the speciﬁed

crossover rate to the ﬁrst variable and then reduces

it exponentially for the subsequent variables. Vari-

ations of these latter three operators are considered

by setting them with different values of their param-

eters, as shown in Table 1. Considering the 7 basic

operators in Table 1 and their parameters, we have 41

types of crossover operators. Additionally, we also

considered shufﬂed (Caruana et al., 1989) versions of

these operators. Thus, in total, we have α = 82 types

of crossover operators. For convenience, we denote

these crossover operators as X

, k = 1, ··· , α.

Feature extraction was performed for these

crossovers assuming individuals have N=20 variables.

With each crossover operator X

and the predeﬁned

parents p

and p

set as explained in Section 2, we

create a sample offspring population of size 2M ap-

plying X

) M = 50 times and compute the fea-

tures. We repeat this process 100 times, each time

with a distinct sample offspring population, to cal-

culate the average feature values for each crossover.

The parameters used for feature extraction are sum-

marized in Table 2.

The benchmark problem for performance eval-

uation was MNK-Landscapes(Aguirre and Tanaka,

2007). In this study, we use landscapes with M = 2

objectives, N = 20 bits, and K = 2 and 4 interacting

variables with nearest neighbor (Near) and random

(Random) models of epistasis. To compute the per-

formance value of the crossovers and train the mod-

els, we solve β = 30 problem instances for each of the

four problem subclasses determined by the combina-

tions of K and epistasis models. To verify the mod-

els we use other 10 instances per problem subclass.

Overall we use 160 problem instances in this study.

The parameters of the problems used are summarized

in Table 3.

The MOEA conﬁguration used for optimizing

these problems employed a NSGA-II algorithm (Deb

et al., 2002) with a population size of 100 running

for 500 generations. The crossover probability was

set to 1.0, and the mutation used was Bit Flip Mu-

tation with a mutation rate of 0.1. We add to this

conﬁguration one of the above X

crossover operators

and compile results on the algorithm’s performance

for each crossover, k = 1,··· ,82. Our implementa-

tion uses the framework proposed by (Blank and Deb,

2020). The MOEAs are run 10 times in each instance

to compute the performance values of the crossovers

and train the models. On the other hand, the MOEAs

are run 50 times in each instance to validate the mod-

els. Table 4 summarizes the parameters used for the

multi-objective optimizer.

The reference point for the HV is (0.0, 0.0).

5 RESULTS AND DISCUSSION

5.1 Selected Models

We ﬁrst examine the general characteristics of the se-

lected models. The coefﬁcients, standard errors, t-

values, p-values, conﬁdence intervals, coefﬁcient of

determination R

, and adjusted R

for the selected

models f

∗

with the lowest RMSE and MAE for each

problem subclass are shown in Tables 5 - 8.

Looking at the explanatory variables selected in

the models, we ﬁnd that for problems with random

variable interaction, Parent Bias alone can predict per-

formance for K=2 and K=4. On the other hand, for

problems with near variable interaction, Unique Rate,

and Adjacent Bias are selected for K=2 and K=4, with

Parent Bias also being included for K = 4. Since

problems with near variable interaction have objec-

tive function values inﬂuenced by adjacent genes, the

selection of Adjacent Bias in the regression model in-

dicates that problem characteristics are reﬂected in the

crossover features.

The R

and adjusted R

values are in the range

0.83-0.94, indicating that in all problem subclasses a

large proportion of the variation in the performance

value PV is predictable from the explanatory vari-

ables.

Below, we analyze the regression models in detail,

grouping by type of variable interaction.

5.1.1 Random Variable Interaction

The models for the two problem subclasses with ran-

dom variable interaction are simple linear regression

models with Parent Bias as the explanatory variable.

Comparing the two models in Tables 5, and 6, the

model for K = 4 has a higher R

and adjusted R

with

lower RMSE and MAE than the model for K=2, indi-

cating higher prediction accuracy and fewer outliers.

Plots showing the regression lines and data for

each crossover are presented in Figures 1, and 2. In

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models

Table 5: Best Regression Model for M = 2, N = 20, K = 2, and random variable interactions on 30 instances.

coef std error t-value p-value 95%CI R

Adj. R

RMSE MAE

Const −0.7548 0.040 −19.029 0.000 −0.834,−0.676

0.875 0.873 0.202 0.134

PaB 0.0181 0.001 23.640 0.000 0.017,0.020

Table 6: Best Regression Model for M = 2, N = 20, K = 4, and random variable interactions on 30 instances.

coef std error t-value p-value 95%CI R

Adj. R

RMSE MAE

Const −0.9904 0.034 −29.487 0.000 −1.057,−0.924

0.944 0.943 0.174 0.128

PaB 0.0237 0.001 36.632 0.000 0.022,0.025

Table 7: Best Regression Model for M = 2, N = 20, K = 2, and near variable interactions on 30 instances.

coef std error t-value p-value 95%CI R

Adj. R

RMSE MAE

Const −2.1858 0.158 −13.844 0.000 −2.500,−1.872

0.894 0.891 0.220 0.150UR 0.0137 0.001 9.457 0.000 0.011,0.017

AB 0.0396 0.002 23.665 0.000 0.036,0.043

Table 8: Best Regression Model for M = 2, N = 20, K = 4, and near variable interactions on 30 instances.

coef std error t-value p-value 95%CI R

Adj. R

RMSE MAE

Const −2.2277 0.151 −14.738 0.000 −2.529,−1.927

0.923 0.920 0.220 0.147

UR 0.0127 0.001 9.019 0.000 0.010,0.015

PaB 0.0073 0.002 3.057 0.003 0.003,0.012

AB 0.0330 0.004 8.705 0.000 0.025,0.041

both ﬁgures, the data and the models indicate that

higher Parent Bias corresponds to better performance.

The regression coefﬁcient for Parent Bias is larger for

K = 4 and the slope of the regression line is steeper,

suggesting a greater inﬂuence of the explanatory vari-

able in problems with larger K. Note that the best-

performing crossovers have Parent Bias around 90%.

This suggests that the most successful crossovers are

those that produce offspring that are one or two bits

different from one of the parents. To verify this, col-

umn Random in Table 9 shows the features and per-

formance values of the 5 crossovers predicted to be

the best by the models (higher

PV ) for these problem

subclasses. Since in both models the only explanatory

variable is Parent Bias, the same crossovers are pre-

dicted to perform better both in K=2 and K=4. Note

that 4 of the 5 crossovers share the same highest Par-

ent Bias value, i.e., Exponential (E), Binomial (B) and

their shufﬂed versions (SE and SB) with the smallest

rate of 0.05, which is equivalent to accepting, on av-

erage, only one bit from the second parent. The ﬁfth

crossover is shufﬂed exponential (SE) with parame-

ter 0.10, which, on average, accepts 2 bits from the

second parent.

From the same ﬁgures, note that the error be-

tween the predicted

PV and actual PV is larger

for crossovers with smaller Parent Bias, particularly

when Parent Bias is zero. Crossovers with a Parent

Bias of zero include Half Uniform (HU) and Multi

Point with 19 crossover points (MP19), along with

their shufﬂed versions (SHU and SMP19). Table 10

summarizes the features, actual PV , and predicted

performance values for these crossovers. Note that

HU and MP19 achieve a PV well below the mean

(PV < 0.0), and their shufﬂed versions improve their

actual PV. However, only SHU performs signiﬁcantly

better than the mean for K=2, i.e. PV = 0.422 > 0.0,

and slightly better than the mean for K=4, i.e. PV =

0.081 > 0.0. HU, SHU, and SMP19 share similar fea-

tures. However, their performance is very different

and cannot be explained neither by Parent Bias nor

by the other features of these crossovers. MP19 is

a special case where its extremely poor performance,

PV = −1.399 and PV = −1.174, can be explained by

its low Unique Rate.

5.1.2 Near Variable Interaction

The models for the two problem subclasses with near

variable interaction are multiple regression models

with Unique Rate and Adjacent Bias as explanatory

variables for K = 2, and Parent Bias added for K = 4.

From Tables 7 and 8, note that the regression coef-

ﬁcient for Adjacent Bias is three times larger than the

coefﬁcient for Unique Rate. The models predict better

performance for crossovers with a high Adjacent Bias

that can pass adjacent bits from the same parent to

ECTA 2024 - 16th International Conference on Evolutionary Computation Theory and Applications

Figure 1: Regression Model for K = 2, Random. Figure 2: Regression Model for K = 4, Random.

Table 9: The 5 best performing crossovers K = 2,4 with Random and Near variable interactions.

Random Near

SE0.05 B0.05 SB0.05 E0.05 E0.1 E0.9 E0.8 E0.7 E0.6 E0.5

Unique Rate 37.04 37.08 37.08 36.68 37.82 79.40 82.64 73.76 64.98 56.32

Parent Bias 90.0 90.0 90.0 90.0 89.9 60.54 65.41 75.03 81.21 85.08

Adjacent Bias 59.81 60.18 60.16 60.50 60.00 54.90 52.37 54.65 56.91 58.16

Position Bias 90.0 90.0 90.0 90.0 89.9 19.86 57.81 73.57 80.98 85.03

K=2

PV 0.717 0.958 0.887 0.772 0.814 0.939 0.0.868 0.923 0.779 0.901

PV 0.871 0.871 0.871 0.871 0.869 0.927 0.879 0.853 0.829 0.771

K=4

PV 1.011 1.100 1.158 1.261 1.025 0.955 0.949 1.089 1.101 0.850

PV 1.143 1.143 1.143 1.143 1.141 0.960 0.938 0.971 0.987 0.960

Table 10: Crossover with Parent Bias of 0.

HU SHU MP19 SMP19

Unique Rate 99.94 99.98 2 99.96

Parent Bias 0 0 0 0

Adjacent Bias 7.90 7.88 0 7.76

Position Bias 11.32 11.23 0 11.69

K=2

PV −0.182 0.422 −1.399 −0.666

PV −0.755 −0.755 −0.755 −0.755

K=4

PV −0.922 0.081 −1.174 −0.718

PV −0.990 −0.990 −0.990 −0.990

the offspring while maintaining a good Unique Rate

to avoid creating the same offspring several times.

Similar to random variable interactions, compar-

ing the two models for near variable interactions, the

model for K = 4 has a higher R2 and adjusted R2

with lower RMSE and MAE than the model for K=2.

To further investigate the better prediction accuracy

for larger K, Table 11 shows the results of a subopti-

mal model for K=4, which excludes Parent Bias and

includes only Unique Rate and Adjacent Bias, the

explanatory variables included in the best model for

K=2. From Table 11, note that R

of the suboptimal

model reduces to 0.914 compared to R

= 0.923 of

the optimal model for K=4 shown in Table 8. How-

ever, the R

of the suboptimal model for K=4 it is still

better than the R

= 0.894 of the optimal model for

K=2 shown in Table 7, although both use the same

explanatory variables. Summarizing, prediction ac-

curacy is better for K=4 than for K=2, whether the

variable interactions are random or near.

Since these are multiple linear regression models,

the models are visualized using residual plots, shown

in Figures 3, and 4. In Figure 3, as the predicted

performance

PV exceeds -0.5, the range of possi-

ble residuals decreases, indicating higher accuracy for

higher

PV values. Conversely, around

PV = -0.5, the

range of possible residuals is larger, with both positive

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models

Figure 3: Residual Plot for K = 2, Near.

Figure 4: Residual Plot for K = 4, Near.

and negative distributions, indicating both underpre-

dictions and overpredictions. In Figure 4, the range

of possible residuals is smaller than for K = 2, indi-

cating higher prediction accuracy. This result is also

reﬂected in the R

values in Tables 7, and 8. However,

for K = 4, there is a data point with residual exceeding

1.0.

In Figures 3 and 4, the two leftmost points cor-

respond to 18 and 19 point crossovers. Their pre-

dicted performance values

PV are underestimated,

but looking at the actual PV are two of the worst

performing operators. Similarly, the point with the

largest positive residual, > 0.6 in K=2 and > 1.0 in

K=4, correspond to shufﬂe half uniform crossover.

The predicted performance value

PV of this opera-

tor is also largely underestimated. However, looking

at the actual PV it is above average (> 0.0) but far

from being among the best. The most overestimated

crossovers, those that appear with negative residu-

als, correspond to multi point crossovers with 10 to

17 crossing points. These crossovers are also in the

group of worst-performing operators.

Column Near in Table 9 shows the features and

performance values of the 5 crossovers predicted to

be the best (higher

PV ) by the models for the near

variable interaction problem subclasses. Although the

number of explanatory variables for K=2 and K=4 is

different, the same crossovers are predicted as the best

in both models. Note that all 5 operators are exponen-

tial crossovers, with the parameter >= 0.5. However,

their rank order is different. For K=4, to compute

the model also takes into account the value of the Par-

ent Bias feature, in addition to Unique Rate and Ad-

jacent Bias used for K=2.

Table 11: Suboptimal Regression Model for K=4, Near

30 instances.

coefﬁcient p-value

Constant −2.2529 0.000

Unique Rate 0.0133 0.000

Adjacent Bias 0.0436 0.000

0.914

Adj. R

0.911

RMSE 0.233

MAE 0.169

Figure 5: HV Boxplot on validation instances for K = 2,

Random.

5.2 Model Validation with Unseen Data

Each regression model was learned to estimate the

performance of crossover operators in a problem sub-

class. In this section, we examine the predictive

power of the models for individual unseen instances

in each problem subclass. As explained in Section

2.6, we identify three groups of crossovers, namely

the best 5, worst 5, and mean 5 performing crossovers,

ECTA 2024 - 16th International Conference on Evolutionary Computation Theory and Applications

Table 12: X

and X

t-Test Results K = 2, Ran-

dom.

ID X

> X

∼ X

< X

31 24 1 0

32 21 4 0

33 25 0 0

34 21 4 0

35 25 0 0

36 4 19 2

37 20 5 0

38 0 22 3

39 16 9 0

40 17 8 0

Table 13: X

and X

t-Test Results K = 4, Ran-

dom.

ID X

> X

∼ X

< X

31 23 2 0

32 25 0 0

33 21 4 0

34 23 2 0

35 25 0 0

36 23 2 0

37 23 2 0

38 22 3 0

39 25 0 0

40 21 4 0

Table 14: X

and X

t-Test Results K = 2, Near.

ID X

> X

∼ X

< X

31 25 0 0

32 23 2 0

33 25 0 0

34 25 0 0

35 24 1 0

36 25 0 0

37 25 0 0

38 25 0 0

39 25 0 0

40 25 0 0

Table 15: X

and X

t-Test Results K = 4, Near.

ID X

> X

∼ X

< X

31 25 0 0

32 25 0 0

33 25 0 0

34 25 0 0

35 25 0 0

36 25 0 0

37 25 0 0

38 25 0 0

39 25 0 0

40 25 0 0

Table 16: X

and X

t-Test Results K = 2, Ran-

dom.

ID X

> X

∼ X

< X

31 25 0 0

32 21 4 0

33 25 0 0

34 25 0 0

35 22 3 0

36 1 23 1

37 17 8 0

38 0 19 6

39 10 15 0

40 11 14 0

Table 17: X

and X

t-Test Results K = 4, Ran-

dom.

ID X

> X

∼ X

< X

31 18 7 0

32 25 0 0

33 15 10 0

34 23 2 0

35 25 0 0

36 21 4 0

37 19 6 0

38 25 0 0

39 18 7 0

40 15 10 0

Table 18: X

and X

t-Test Results K = 2, Near.

ID X

> X

∼ X

< X

31 25 0 0

32 6 19 0

33 11 14 0

34 18 7 0

35 14 11 0

36 23 2 0

37 10 15 0

38 19 6 0

39 19 6 0

40 13 12 0

Table 19: X

and X

t-Test Results K = 4, Near.

ID X

> X

∼ X

< X

31 22 3 0

32 25 0 0

33 16 9 0

34 22 3 0

35 24 1 0

36 25 0 0

37 25 0 0

38 22 3 0

39 23 2 0

40 22 3 0

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models

and perform pairwise t-tests between the best and the

other crossovers. To illustrate the variance among in-

stances, Figure 5 shows box plots of the HV achieved

in the 50 runs of the 5 best and 5 worst crossover

operators, with 500 data points in total, in each un-

seen instance of problem subclass K = 2 with random

variable interaction. Note that the distributions of the

points and their medium values vary greatly from in-

stance to instance.

The summary of the Welch’s t-test performed to

compare the mean hypervolume on 50 runs achieved

by the 5 best X

∈ X

best

and 5 worst X

∈ X

worst

crossovers is shown in Tables 12-15. Results are

shown for 10 unseen instances in each problem sub-

class, i.e., instances not used to train the models.

The unseen instances are identiﬁed with a number be-

tween 31 and 40.

The t-tests try to assess whether the models could

be useful to classify good crossovers from bad ones

in particular instances of the problem subclasses. If

there is a clear differentiation between the best and

worst crossovers for a particular instance, X

> X

should approach 25 and X

< X

should aproach 0,

indicating that the mean hypervolume by most X

∈

best

is better than by X

∈ X

worst

. As shown in Ta-

bles 13,14, and 15, X

< X

is zero for all instances,

indicating the model is useful. However, in Table 12,

instances 36 and 38 have non-zero values for X

< X

To determine if this result is due to the model or the

instances, Figure 6 shows separate box plots for the

HV achieved by the best 5 and the worst 5 crossovers

for instances 33, 36, 38, and 40. In instance 33, there

is a perfect score of X

> X

= 25 in favor of the best

crossover operators, which is in accordance with the

box plots that show distributions of HV values clearly

centered around different means. In instance 40, the

scores X

> X

= 17 and X

∼X

= 8 indicate that the

best crossovers are mostly better than and sometimes

similar to the worst crossovers. The box plots show

that the distributions of the two groups are different.

In instances 36 and 38, the scores X

∼ X

= 19

and 22, respectively, show that, mostly, there is no

statistical difference in the HV means between the

best and worst crossovers. The box plots conﬁrm this,

and also show that the distributions of HV values are

very similar, with a small standard deviation around

the mean. This suggests that these instances are sim-

pler, where crossover features make no difference in

performance and therefore the majority of crossovers

will achieve a similar HV.

Tables 16-19 show the results of the pairwise t-

tests between the best X

∈ X

best

and the mean X

∈

mean

performing crossovers. Note that in almost

all problem subclasses and instances, X

’s perfor-

mance is better or similar to X

. As expected, due

to the variability in the instances, there are more cases

where the performance of the predicted best and mean

crossovers in the subclass perform similarly in a given

instance.

A summary of the mean performing crossovers in

each problem subclass is shown in Table 20. It can be

seen that in random variable interactions, the group of

crossovers considered as mean performing is the same

for K=2 and K=4. In near variable interactions, three

crossovers are common for K=2 and K=4. Also note

that the groups of mean performing crossovers are dif-

ferent for random and near variable interactions.

These results show that the models are useful for

classifying good performing crossovers from average

and bad performing crossover in particular instances.

Figure 6: HV Boxplot by best and worst crossovers on se-

lected instances for K = 2, Random.

Table 20: Mean performing crossovers.

Random Near

K=2 K=4 K=2 K=4

B0.7

√ √ √

SB0.7

√ √

B0.3

√ √

SB0.3

√ √

MP4

√

MP5

√ √

MP6

√

SOP

√ √

STP

√ √

SB0.2

√

6 CONCLUSION

This work presented a method to study the relation-

ship between crossover features and the performance

ECTA 2024 - 16th International Conference on Evolutionary Computation Theory and Applications

of a multi-objective evolutionary algorithm on sub-

classes of problems by using regression models. Sev-

eral crossover operators were studied, extracting their

features with a method independent of the charac-

teristics of the problem instances or the dynamics

of an evolutionary algorithm. A performance value

for a problem subclass, relative to other operators,

was computed for each crossover running a multi-

objective evolutionary algorithm on several instances.

We used MNK-Landscapes with random and near-

variable interactions to deﬁne problem subclasses.

We veriﬁed that the models identiﬁed relevant fea-

tures for each problem subclass and can explain a

large proportion of the performance variance.

In the future, we would like to validate the

method with other multi-objective evolutionary al-

gorithm conﬁgurations and use problems with more

variables and objectives. Also, it would be interesting

to add problem features to obtain more general mod-

els.

REFERENCES

Aguirre, H. E. and Tanaka, K. (2004). Insights on properties

of multiobjective MNK-landscapes. In Proceedings

of the 2004 Congress on Evolutionary Computation

(IEEE Cat. No.04TH8753), volume 1, pages 196–203

Vol.1. IEEE.

Aguirre, H. E. and Tanaka, K. (2007). Working princi-

ples, behavior, and performance of moeas on mnk-

landscapes. European Journal of Operational Re-

search, 181(3):1670–1690.

Blank, J. and Deb, K. (2020). pymoo: Multi-objective opti-

mization in python. IEEE Access, 8:89497–89509.

Caruana, R., Eshelman, L., and Schaffer, J. (1989). Rep-

resentation and hidden bias ii: Eliminating deﬁning

length bias in genetic search via shufﬂe crossover. vol-

ume 1, pages 750–755.

Daolio, F., Liefooghe, A., Verel, S., Aguirre, H., and

Tanaka, K. (2015). Global vs local search on multi-

objective nk-landscapes: Contrasting the impact of

problem features. In Proceedings of the 2015 Annual

Conference on Genetic and Evolutionary Computa-

tion, GECCO ’15, pages 369–376, New York, NY,

USA. Association for Computing Machinery.

Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. (2002).

A fast and elitist multiobjective genetic algorithm:

Nsga-ii. IEEE Transactions on Evolutionary Compu-

tation, 6(2):182–197.

Ishibuchi, H., Tanigaki, Y., Masuda, H., and Nojima, Y.

(2014). Distance-based analysis of crossover opera-

tors for many-objective knapsack problems. In Bartz-

Beielstein, T., Branke, J., Filipi

c, B., and Smith, J.,

editors, Parallel Problem Solving from Nature – PPSN

XIII, pages 600–610, Cham. Springer International

Publishing.

Kauffman, S. A. (1993). The Origins of Order: Self-

Organization and Selection in Evolution. Oxford Uni-

versity Press.

Martins, M. S. R., Yafrani, M. E., Delgado, M., L

uders,

R., Santana, R., Siqueira, H. V., Akcay, H. G., and

Ahiod, B. (2021). Analysis of bayesian network learn-

ing techniques for a hybrid multi-objective bayesian

estimation of distribution algorithm: a case study on

mnk landscape. Journal of Heuristics, 27(4):549–573.

Pelikan, M. (2010). Nk landscapes, problem difﬁculty, and

hybrid evolutionary algorithms. In Proceedings of the

12th Annual Conference on Genetic and Evolution-

ary Computation, GECCO ’10, pages 665–672, New

York, NY, USA. Association for Computing Machin-

ery.

Price, K., Storn, R. M., and Lampinen, J. A. (2005). Dif-

ferential Evolution: A Practical Approach to Global

Optimization (Natural Computing Series). Springer-

Verlag, Berlin, Heidelberg.

Sato, H., Aguirre, H., and Tanaka, K. (2013). Variable

space diversity, crossover and mutation in moea solv-

ing many-objective knapsack problems. Annals of

Mathematics and Artiﬁcial Intelligence, 68(4):197–

224.

Sato, H., Aguirre, H. E., and Tanaka, K. (2007). Local dom-

inance and local recombination in moeas on 0/1 mul-

tiobjective knapsack problems. European Journal of

Operational Research, 181(3):1708 – 1723.

Spears, W. (2000). Evolutionary Algorithms: The Role of

Mutation and Recombination. Natural Computing Se-

ries. Springer Berlin Heidelberg.

Storn, R. and Price, K. (1995). Differential evolution: A

simple and efﬁcient adaptive scheme for global opti-

mization over continuous spaces. Journal of Global

Optimization, 23.

Syswerda, G. (1989). Uniform crossover in genetic algo-

rithms. Proceedings of the Third International Con-

ference on Genetic Algorithms, pages 2–9.

Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C., and

da Fonseca, V. (2003). Performance assessment

of multiobjective optimizers: an analysis and re-

view. IEEE Transactions on Evolutionary Computa-

tion, 7(2):117–132.

Studying the Relationship Between Crossover Features and Performance on MNK-Landscapes Using Regression Models