Strategies for Runtime Prediction and Mathematical Solvers Tuning
Michael Barry and Ren
´
e Schumann
Smart Infrastructure Laboratory, Institute of Information Systems,
University of Applied Sciences Western Switzerland (HES-SO) Valais/Wallis, Rue de Technopole 3, 3960 Sierre, Switzerland
Keywords:
Mixed Integer Problems, Mathematical Solvers, Tuning, Runtime Prediction, Optimization, Machine Learn-
ing, Genetic Algorithm, Novelty Search.
Abstract:
Mathematical solvers have evolved to become complex software and thereby have become a difficult subject
for Runtime Prediction and parameter tuning. This paper studies various Machine Learning methods and data
generation techniques to compare their effectiveness for both Runtime Prediction and parameter tuning. We
show that machine Learning methods and Data Generation strategies that perform well for Runtime Prediction
do not necessary result in better results for solver tuning. We show that Data Generation algorithms with an
emphasis on exploitation combined with Random Forest is successful and random trees are effective for Run-
time Prediction. We apply these methods to a hydro power model and present results from two experiments.
1 INTRODUCTION
Training data selection strategies aim to improve ML
method performance by carefully selecting which
training instances to include in the training data. This
can be helpful in applications (He et al., 2008) where
the data is imbalanced, i.e. some behaviors are not
represented equally, or where the data must be gener-
ated by some costly process. Training Data Genera-
tion methods effectively create a bias towards training
instances that have a better learning outcome. How-
ever, the extent to which an ML method can exploit
this bias depends on the ML method. We argue that
the combination of various Data Generation and ML
methods must be considered, otherwise a bias is given
and the performance of some ML methods may be
underestimated. In this paper, we study the combi-
nation of Data Generation and ML method and apply
it to a problem of parameter tuning for mathematical
solvers. Mathematical solvers are complex software
tools that use a variety of strategies to solve math-
ematical problems. Common mathematical solvers
will have a good general performance, but may be
fine tuned using a set of parameters for specific prob-
lems (IBM, 2016). Identifying which strategies work
best for a given problem is not intuitive and solving
completely is impractical. By using ML, the run-
time of a given problem and solver parameters can
be predicted. The best predicted runtime can then
be used to solve the problem. Such a system can
address two problems: Runtime Prediction and Run-
time Optimization. Building a predictor (Hutter et al.,
2014) will inevitably require a training set consisting
of solved mathematical models. Building such a train-
ing set can be computationally expensive. However,
if the predictor can be generalized to other models,
or different configurations of the same model, invest-
ing computation time into a training phase pays off in
the long run. This forms the training Data Generation
problem, where a good training Data Generation strat-
egy can improve a Machine Learning methods perfor-
mance or reducing the size (therefore the computation
time) of the training set. This paper aims to identify
ideal pairs of Data Generation and ML methods ap-
plied to the problem of tuning mathematical solvers.
We describe relevant background literature in Section
2 and formalize the problem in Section 3 and 4. We
describe the the applied mathematical model in Sec-
tion 6 and our method in Section 7. We present results
from two experiments, where we describe the setup in
Section 8 and results in 9.
2 BACKGROUND
Runtime prediction using ML is a well studied field.
There are many comparisons of general ML methods
applied to Runtime Prediction e.g. (Kotthoff et al.,
2012; Hutter et al., 2014). However, such meth-
ods have only recently been applied to tuning math-
Barry, M. and Schumann, R.
Strategies for Runtime Prediction and Mathematical Solvers Tuning.
DOI: 10.5220/0007387606690676
In Proceedings of the 11th International Conference on Agents and Artificial Intelligence (ICAART 2019), pages 669-676
ISBN: 978-989-758-350-6
Copyright
c
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
669
ematical solvers (Kotthoff et al., 2012; Hutter et al.,
2014). Currently, mostly off-the-shelf ML methods
have been applied and extensively studied in (Kot-
thoff et al., 2012; Hutter et al., 2014). However,
most advances in the field focus on tuning an existing
framework for the specific problem of tuning mathe-
matical solvers and feature selection (Xu et al., 2011).
Algorithm Runtime Prediction falls into a category of
ML problems where data must be produced by run-
ning the algorithm itself. This allows a method to
carefully select which data is to be produced. Such
a method is commonly used in active learning and
has been applied in areas that suffer from insufficient
or unbalanced data (Ertekin et al., 2007). Although
most applications of Runtime Prediction for mathe-
matical solvers use a random data generation strategy,
recent work (Barry et al., 2018) has indicated the suc-
cess when applying a novelty search approach. How-
ever, the study was only applied using Artificial Neu-
ral Networks We wish to extend this work by com-
paring various data generation methods applied to a
variety of Machine Learning methods. Runtime Pre-
diction has applications in job scheduling, but also
Runtime Optimization (also referred to as algorithm
tuning or configuration) (Hutter et al., 2014). Good
Runtime Predictions allows a user to select the algo-
rithm parameters that result in the desired algorithm
behavior, most commonly a faster runtime, but also
solution quality (L
´
opez-Ib
´
anez and St
¨
utzle, 2014). In
this paper we intend to extend the field of Runtime
Prediction by combining currently used ML methods
with various Data Generation strategies.
3 RUNTIME PREDICTION
Runtime prediction in general concerns predicting an
algorithms (or softwares) runtime based on a descrip-
tion of the problem and the algorithm applied. In the
context of mathematical solvers, predictions are made
based on a model description m and a set of solver
parameters s applied to the solver. In addition to the
resulting runtime y, we can describe each run z as a
tuple:
z = (m, s, y) (1)
Where the resulting runtime y is a function f of the
model description m and a set of solver parameters s.
y = f (m, s) (2)
Trained on a set of example runs [z
1
, ..., z
n
], a pre-
dictor D can give an estimated runtime ˆy
i
.
ˆy
i
= D
z
1
,...,z
n
(m
i
, s
i
) (3)
As a result, the problem of Runtime Prediction
is a common learning problem where a valid method
should minimize the total error E between the predic-
tion ˆy
i
and actual value y for a set of unseen instances
[z
n
+ 1, ..., z
s
]:
minE
prediction
= min
s
i=n+1
| D
z
1
,...,z
n
(m
i
, s
i
) f (m
i
, s
i
) |
(4)
4 SOLVER TUNING
The problem of solver tuning can be described as find-
ing the set of solver parameters ¯s with the lowest run-
time for a given model m.
¯s = argmin
s
f (m, s
1
, ..., s
n
) (5)
However, running the solver for the entire set
[s
1
, ..., s
n
] to identify ¯s is computationally expensive
and requires n runs to increase the solvers perfor-
mance for a single run. Instead, using a predictor, one
can find the set of solver parameters ˆs with the lowest
predicted runtime for a given model m.
ˆs = argmin
s
D
z
1
,...,z
n
(m, s
i
) (6)
The aim in solver tuning is to reduce the error be-
tween the runtime of the estimate best solver configu-
ration ˆs and the actual best solver configuration ¯s for
a given model m.
minE
tuning
= min( f (m, ˆs) f (m, ¯s)) (7)
5 TRAINING DATA
For both problems we wish to choose a predictor D
and training set [z
1
, ..., z
n
] that produces the smallest
error E. The more computation time is given to the
training data generation stage, the more data is avail-
able and therefore we expect a higher accuracy for
the predictor. However, as producing the training set
is computationally expensive, we have the conflicting
objective to reduce the size n of the training set as
much as possible.
6 APPLICATION
The aim of our approach is to create a predictor that
can predict with a high accuracy the runtime for a
given model instance and solver parameters. Sec-
ondly, we use the predictions to select the best solver
parameters for a given model instance. We apply
ICAART 2019 - 11th International Conference on Agents and Artificial Intelligence
670
these methods for solving a Hydro Power Operations
model, that aims to identify when to utilize its water
reserves to optimize its revenue based on energy mar-
ket prices. The Mixed Integer Problem (MIP) model
is used as it is highly configurable, varying greatly
in complexity based on its configuration. Details are
given in our previous work (Barry et al., 2015; Barry
and Schumann, 2015). Model configurations include:
Plant Topology: The topologies of a hydro power
plant describes details such as the size and number
of the reservoir, turbines and water inflows.
Time Period: Different time periods results in vary-
ing data such as how much water flows into the
reservoir and expected market prices.
To solve the model we use the mathematical solver
CPLEX (Version 12.7) (GAM, 2017), as its perfor-
mance is commonly accepted and represents an in-
dustry standard. CPLEX has a large set of parame-
ters to configure. Hutter et al. (Hutter et al., 2010)
identified a total of 76 parameters that can increase
the performance of CPLEX. We reduce this set to the
following parameters as initial results and literature
(Hutter et al., 2009) show that they are good candi-
dates to increase a solvers performance:
startalg: [1,2,3,4,5] Selects the algorithm used for
the relaxation of the MIP
subalg: [1,2,3,4,5] Selects the algorithm used for the
linear subproblem.
heurfreq: [-1,0] Selects how often CPLEX applies
heuristics. This can be set manually to a specific
value, but we only consider turning it off (-1) or
setting it to automatic (0).
mipsearch: [1,2] Selects the search strategy for the
branch and cut phase.
cuts: [-1,1,2,3] Selects how aggressively cuts are
generated, ranging from not at all (-1) to very ag-
gressive (3). Values 4 and 5 were omitted as they
did not yield any speedup for any instances.
7 METHOD
The Data Generation strategy selects sets of model in-
stances and solver parameters, which are then passed
to the CPLEX solver. The resulting runtime for each
set of inputs is then used to train the predictor. The
overall Runtime Prediction and Runtime Optimiza-
tion process is shown in Figure 1. For a given model
instance and solver parameters, the predictor predicts
the runtime. To optimize the runtime, the predictor
predicts the runtime for every set of possible solver
parameters. The best runtime with the corresponding
solver parameters is then selected to run the CPLEX
solver for the given model instance.
Figure 1: The overall Runtime Prediction and Runtime Op-
timization process.
7.1 Data Generation Strategies
To select which sets of model instances and solver pa-
rameters to use as training instances we consider three
training Data Generation strategies: Random, Opti-
mal Runtime and Novelty. Ideally, the strategy pro-
duces a set of training instances that is:
Small: Each training instance must be passed to the
CPLEX solver to ascertain the solvers runtime,
which is computationally expensive. Reducing
the size of the training data reduces the compu-
tation time for the training phase.
Representative: The training data is a sample of the
entire search space. For a predictor trained on this
sample to be accurate, the sample must be repre-
sentative of the entire search space.
7.1.1 Evolutionary Algorithm
All algorithms for the Data Generation are based
on a Genetic Algorithm which can be summarized
as shown in the listing of Algorithm 1. Genetic
algorithms are distinctive from one and another based
on the implementation of the representation, initial-
ization, evaluation, selection, breeding (crossover
and mutation operators) and termination criteria.
In this paper, the first two algorithms (Minimal
Runtime GA and Novelty GA) differ only by the
evaluation function. The third algorithm only acts
as a representation of random selection, where each
generation a random set of individuals are added. As
parameters for these algorithms we use a population
size of 50, a mutation to crossover rate of 90% and a
selection rate of 20 %. The parameters were chosen
after some initial testing. We describe the general
GA below, with more details given for the spe-
cific evaluation function in sections 7.1.2 and 7.1.3.
Strategies for Runtime Prediction and Mathematical Solvers Tuning
671
Algorithm 1: Genetic Algorithm.
1: P = initialise()
2: while Termination() do
3: evaluate(P)
4: selection(P)
5: breed(P)
For more details on GAs, we refer to (Chawdhry et al.,
2012).
Representation: Each individual represents a model
and solver configuration as a string of numbers:
{m
1
, ... m
n
, s
1
,...,s
k
}, where each m is a model
configuration, s is a solver configuration, n is the
total number of model configuration and k is the
total number of solver configurations.
Initialization: Random initialization is used, where
each gene m and s are assigned a random value.
Selection: A roulette wheel selection is used where
each individual is assigned a pocket proportional
to their fitness. A survivor is chosen based on
which pocket the ball (in this case a random num-
ber) falls into.
Breeding: Breeding involves creating a new indi-
vidual through a crossover or mutation operator.
Random mutation is used, where a survivor is
copied except for one gene. This gene is mu-
tated by randomly selected a new value. Ran-
dom Crossover involves constructing an individ-
ual from two parents, where each gene is copied
from one of the two parents.
Termination criteria: The number of unique solver
runs are used as a termination criteria as running
the solver is the computationally most expensive
component of the algorithm.
7.1.2 Minimal Runtime GA
The Minimum Runtime Genetic Algorithm (MRGA)
uses an evaluation function that assigns high fitness
values to individuals in the population that result in
a low runtime for the solver. As a result, the popu-
lation evolves towards individuals which apply a set
of solver parameters that achieve a low runtime when
applied to a particular model. To ensure that the al-
gorithm does not simply prefer less complex model
configurations, we normalize the runtime on the max-
imum runtime of the individuals k nearest neighbors
(k = 5). The evaluation function F
MRGA
is shown in
equation 8, where N is the set of k nearest individuals
to the individual represented by model configuration
m and solver parameters s.
F
MRGA
(m, s) =
f (m, s)
max f (m
k
, s
k
)
: z
k
N (8)
The distance calculation used to determine the
nearest neighbors is the number if genes that differ.
For example, individuals {1,2,3,4,5} and{1,4,3,2,5}
have a distance of 2. The assumption is that individu-
als with similar model and solver configurations will
result in a similar runtime.
7.1.3 Novelty GA
Another GA is used, but using an evaluation function
based on the novelty of a solution. Novelty is de-
scribed as how different an individual behaves to the
rest of the population. In our case, it considers how
much the runtime differs to other neighboring indi-
viduals. For this we calculate the sum of differences
in their runtime between the individual and its nearest
neighbors. The novelty evaluation function F
Novelty
is
shown in equation 9.
F
Novelty
(m, s) =
5
k=5
f (m
k
, s
k
) f (m
i
, s
i
) : z
k
N (9)
This means it will also favor parameter settings
which result in a high runtime, if it represents an as-
pect that has not been covered by other individuals.
As a result, this algorithm will produce a population
that represents all rare points in the search space.
7.1.4 Random Algorithm
Random training Data Generation is most commonly
used in literature (Hutter et al., 2014; Kotthoff et al.,
2012), as it is the simplest to implement. We create an
algorithm, based on a GA, that adds new random in-
dividuals in each generation to provide a comparison
for varying training data set sizes.
7.2 ML Strategies
Given a set of solver parameters and a model in-
stance, the predictor provides an estimate of the run-
time without having to run the computationally ex-
pensive CPLEX solver. The predictor is built us-
ing one of the following ML strategies (Hutter et al.,
2014; Eibe et al., 2016): Artificial Neural Networks
(ANN), Ridge Regression (RR), Gaussian Process
Regression (GPR), Random Tree(RT) and Random
Forrest (RF). We chose this set of ML strategies as
they are commonly used in ML and are also explored
in (Hutter et al., 2014), but theoretically any ML
method could be used. We use the implementations
from the commonly used WEKA (Eibe et al., 2016)
ML library and use WEKA default parameters for
ICAART 2019 - 11th International Conference on Agents and Artificial Intelligence
672
each method. However, as ANNs require inputs to
define their structure, we chose the parameters that
performed generally well in a small pre-study. The
structure applied consists of one hidden layer with
as many hidden neurons as input neurons. We use
a learning rate of 0.01, momentum of 0.1 and 2000
training iterations. The predictor gives the runtime
as output. For inputs of the predictor, we use each
solver parameters combined with a set of complexity
measures. Complexity measures gives an indication
of the size and structure of the mathematical model
after it has been configured according to an individu-
als model description. We do not use the model con-
figurations directly as input, as this would limit the
predictor, allowing it to only predict runtime for com-
binations of already known configurations. Using the
complexity measures allows it to generalize. The fol-
lowing complexity measrues are uses: The number of
variables, functions, Non-zero functions and binary
variables. These are obtained from the model statis-
tics of the CPLEX solver.
8 EXPERIMENT SETUP
Our experiments analyze the combination of all Data
Generation and ML strategies. For each combination,
we analyze its predictors performance for different
sizes of training instances. Therefore, for every 100
instances selected by the Data Generation strategy we
train a predictor. We can then observe the predictors
performance as the training sets size increases. We
use a randomly selected set of model instances as test
sets. To make comparisons between methods with-
out being affected by this random selection, as well
as each algorithms non-deterministic nature, we re-
peat each experiment 150 times and present here me-
dian values. Runtime is measured in CPLEX ticks,
which is a stable measurement of how much compu-
tation CPLEX performs to produce a result. We use
this measurement as it is independent of any other
processes, allowing us to reliably run several mod-
els in parallel. We apply a maximum threshold of
4,000,000 CPLEX ticks for each run. We also mea-
sure the size of the training sets used for producing
the predictor, as this indicates the number of mod-
els needed to be solved by CPLEX and represents the
computationally most expensive process in the train-
ing phase. The experiments are run on a dedicated
Linux server, using GAMS (GAM, 2017) as the mod-
eling language and our own implementation of the ex-
periment setup and Data Generation strategies in Java.
We create two experiments, the first focusing on us-
ing the Machine Learning methods for Runtime Pre-
diction and the second for Runtime Optimization. To
measure performance for the purpose of Runtime Pre-
diction we calculate the absolute difference between
the predicted and actual runtime for each ML method.
We select two model instances and all possible solver
parameters for those instances as test data. This re-
sults in 800 test instances for each repeat. All strate-
gies are tested using the same test data in each repeat.
We make observations until a size of 2000 training in-
stances, creating a predictor every 100 model solves.
In total, for 150 repeats, 20 predictors per repeat and
800 test instances per predictor we make 2’400’000
observations per method. For the Runtime Optimiza-
tion it is important to observe the runtime when us-
ing the best solver parameters suggested by the pre-
dictor. We compare the results between the different
sets of ML and training Data Generation strategies,
as well as the default settings of the CPLEX solver.
Again, we select two model configurations as test in-
stances. However, as the predictor is used to select
the best solver parameters, this results in only two test
instances. However, due to 150 repetitions and 20 in-
dividually trained predictors per repeat, we observe
6’000 data points per method. We also wish to mea-
sure the best possible runtime as the maximum po-
tential possible of any method. However, this is only
possible by a brute force approach and is not prac-
tical. However, as we repeat each experiment many
times for scientific certainty, we almost do so regard-
less. Therefore, we verify that we have searched the
space completely with an exhaustive search and in-
clude the best possible times in our results for com-
parison.
9 RESULTS
In our results we expect a well performing predictor
to have an overall low prediction error when used for
Runtime Prediction (Experiment 1) as shown in Fig-
ure 3. We also expect to see a low variance in these
prediction errors and a stable learning curve. Fur-
thermore, when these predictors are used for Runtime
Optimization (Experiment 2) as shown in 2, we ex-
pect to see an overall low runtime. Overall we see
that the Data Generation method has little effect on
the prediction accuracy and only nominal effects on
the error variance. However, we see that the learning
method has a impact on the prediction errors. RFs are
the best method for Runtime Prediction. More obser-
vations are described below. In Figure 3 we see that
performances in Runtime Optimization do not reflect
the performances for Runtime Prediction and there-
fore should be treated as a separate problem. We see
Strategies for Runtime Prediction and Mathematical Solvers Tuning
673
Figure 2: Graphic showing the error in Runtime Predictions.
that that the Data Generation and Machine Learning
method has a large effect on the predictors perfor-
mance. RF demonstrates the best and stable perfor-
mance and is ideally combined with strategy that em-
phasizes performance, such as MRGA. Using such a
strategy, the performance of the CPLEX solver can
be increase by around 40% when compared to de-
fault parameters. However, potential exists for a much
large performance boost and that the minimum run-
time GA is not the best performing Data Generation
method for all Machine Learning methods. The state
of the art currently uses ANN with a random Data
Generation strategy. We use this as a baseline to com-
pare our results. For the Runtime Optimization, we
also include the industry standard as a second base-
line, which consists of using the default settings of the
solver. For Runtime Predictions, the potential best is
a 0% error with no variance. The errors are normal-
ized by a maximum runtime threshold, which is ap-
plied to the solver. Therefore, as long as the predictor
learns this threshold, we would not expect an error
higher than 100%. The potential best for the Runtime
Optimization can only be determined by solving the
search space completely. The potential maximum is
the threshold applied to the solver. The fact that Data
Generation strategies have a larger effect for Runtime
Optimization than prediction can be explained by the
high variance in the prediction errors. The outliers in
the predictor errors indicate that there is a small sub-
set in individuals that are difficult to predict. They
are a result of an imbalanced search space due to spe-
cialized parameters sets. Specialized parameter sets
ICAART 2019 - 11th International Conference on Agents and Artificial Intelligence
674
Figure 3: Graphic showing the runtime resulting from using a predictor to suggest solver parameters.
achieve a low runtime for specific models, but not for
others. As such, specialized parameter settings can
produce a low runtime when used for the correct mod-
els and are key for Runtime Optimization. However,
they are also more difficult to predict. Therefore, a
predictors ability to perform in Runtime Optimization
is limited by the understanding of specialized param-
eter sets. As such, Data Generation such as MRGA
that search for the specialized parameter sets will gen-
erally perform best for Runtime Optimization. The
effect of the unbalanced search space is also notice-
able for the Machine Learning methods. For exam-
ple, RF are resistant to imbalanced training data and
in addition to having a generally low prediction error.
In comparison, the regression methods are affected by
the imbalance except when a MRGA Data Generation
method is used. The trade off between the training set
size and performance is clearly shown in our results.
Regression methods, when using MRGA, outperform
RF when only small data sets are available. Deci-
sion trees in particular perform well, before suffer-
ing from over fitting. When novelty search is used for
the purpose of Runtime Optimization, we can observe
a convergence of performances. Novelty search em-
phasis all rare sets of individuals in the search space
and therefore gives a more equal representation of all
such rare sets. This shows that the Machine Learn-
ing performance is only determined by its ability to
handle imbalanced data. The threshold applied to the
CPLEX can be seen in Figure 2 as the the random for-
est and decision tree learn that no estimation should
exceed the threshold. However, other methods esti-
Strategies for Runtime Prediction and Mathematical Solvers Tuning
675
mate runtime that exceed the threshold.
10 CONCLUSION
We have successfully tested several Data Generation
strategies and ML strategies for predicting and opti-
mizing the runtime of mathematical solvers. In sum-
mary, we can draw the following conclusions from
our study. Runtime Prediction requires robust Ma-
chine Learning strategies, that are not effect by un-
balanced data. Therefore, RF, RT and ANN are vi-
able methods, but RF seems to be best suited. Data
generation methods have almost no effect on a pre-
dictors accuracy. Predictors that perform well in pre-
dicting the runtime of a solver does not necessarily
result in the best performing predictor for Runtime
Optimization. The performance of a predictor used
for Runtime Optimization depends on the Machine
Learning method used as well as the Data Genera-
tion strategy. In our experiments RF performs best
for all Data Generation strategies, but it is best com-
bined with a MRGA. Other ML methods are strongly
dependent on the Data Generation strategy. Regres-
sion methods are only effective if the unbalance in
the training data is countered with strategies similar to
the MRGA. ANN benefit best from a Novelty Strat-
egy. Random selection strategies are discouraged, but
can be used in combination with RF. For small data
sizes, GPR with MRGA is recommended. Overall, a
performance increase of around 40% can be achieved
when compared to the default parameters. However,
the maximum potential for such methods is around
70%.
REFERENCES
(2017). GAMS/CPLEX 10 solver manual.
Barry, M., Abgottspon, H., and Schumann, R. (2018).
Solver tuning and model configuration. In Joint Ger-
man/Austrian Conference on Artificial Intelligence
(K
¨
unstliche Intelligenz), pages 141–154. Springer.
Barry, M., Schillinger, M., Weigt, H., and Schumann, R.
(2015). Configuration of hydro power plant mathe-
matical models. In Energy Informatics: Proceedings
of the Energieinformatik 2015, volume 9424 of Lec-
ture Notes in Computer Sciences. Springer.
Barry, M. and Schumann, R. (2015). Dynamic and con-
figurable mathematical modelling of a hydropower
plant research in progress paper. In Presented at
the 29. Workshop ”Planen, Scheduling und Konfiguri-
eren, Entwerfen” (PuK 2015).
Chawdhry, P. K., Roy, R., and Pant, R. K. (2012). Soft
computing in engineering design and manufacturing.
Springer Science & Business Media.
Eibe, F., Hall, M., Witten, I., and Pal, J. (2016). The
weka workbench. Online Appendix for “Data Mining:
Practical Machine Learning Tools and Techniques, 4.
Ertekin, S., Huang, J., Bottou, L., and Giles, L. (2007).
Learning on the border: active learning in imbalanced
data classification. In Proceedings of the sixteenth
ACM conference on Conference on information and
knowledge management, pages 127–136. ACM.
He, H., Bai, Y., Garcia, E. A., and Li, S. (2008). Adasyn:
Adaptive synthetic sampling approach for imbal-
anced learning. In Neural Networks, 2008. IJCNN
2008.(IEEE World Congress on Computational In-
telligence). IEEE International Joint Conference on,
pages 1322–1328. IEEE.
Hutter, F., Hoos, H., and Leyton-Brown, K. (2010). Au-
tomated configuration of mixed integer programming
solvers. Integration of AI and OR Techniques in Con-
straint Programming for Combinatorial Optimization
Problems, page 190.
Hutter, F., Hoos, H. H., Leyton-Brown, K., and St
¨
utzle, T.
(2009). Paramils: an automatic algorithm configura-
tion framework. Journal of Artificial Intelligence Re-
search, 36:267–306.
Hutter, F., Xu, L., Hoos, H. H., and Leyton-Brown, K.
(2014). Algorithm runtime prediction: Methods &
evaluation. Artificial Intelligence, 206:79–111.
IBM (2016). CPLEX Performance Tuning for Mixed Inte-
ger Programs.
Kotthoff, L., Gent, I. P., and Miguel, I. (2012). An evalu-
ation of machine learning in algorithm selection for
search problems. AI Communications, 25(3):257–
270.
L
´
opez-Ib
´
anez, M. and St
¨
utzle, T. (2014). Automatically im-
proving the anytime behaviour of optimisation algo-
rithms. European Journal of Operational Research,
235(3):569–582.
Xu, L., Hutter, F., Hoos, H. H., and Leyton-Brown, K.
(2011). Hydra-mip: Automated algorithm configura-
tion and selection for mixed integer programming. In
RCRA workshop on experimental evaluation of algo-
rithms for solving problems with combinatorial explo-
sion at the international joint conference on artificial
intelligence (IJCAI), pages 16–30.
ICAART 2019 - 11th International Conference on Agents and Artificial Intelligence
676