A Novel Approach to Model Design and Tuning through Automatic Parameter Screening and Optimization - Theory and Application to a Helicopter Flight Simulator Case-study

Matteo Hessel, Francesco Borgatelli, Fabio Ortalli

2014

Abstract

The aim of this paper is to describe a novel methodology for model-design and tuning in computer simulations, based on automatic parameter screening and optimization. Simulation requires three steps: mathematical modelling, numerical solution, and tuning of the model’s parameters. We address Tuning because, at the state-of-the-art, the development of life-critical simulations requires months to appropriately tune the model. Our methodology can be split in Screening (identification of the relevant parameters to simulate a system) and Optimization (search of optimal values for those parameters). All techniques are fully general, because they leverage ideas from Machine-Learning and Optimization Theory to achieve their goals without directly analysing the simulator’s mathematical model. Concerning screening, we show how Machine-Learning algorithms, based on Neural Networks and Logistic Regression, can be used for ranking the parameters according to their relevance. Concerning optimization, we describe two algorithms: an adaptive hill-climbing procedure and a novel strategy, specific for model tuning, called sequential masking. Eventually, we show the performances achieved and the impact on the time and effort required for tuning a helicopter flight-simulator, proving that the proposed techniques can significantly speed-up the process.

References

  1. Fisher, R.A., 1935, The design of experiments. Oxford, England: Oliver & Boyd. xi 251 pp.
  2. Rosenblatt F., 1958, The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review 65: 386-408.
  3. Nelder J., Wedderburn R., 1972 Generalized Linear Models, Journal of the Royal Statistical Society. Series A (General) 135 (3): 370-384.
  4. Rumelhart D.E., Hinton G.E., Williams R.J., 1986, Learning representations by back-propagating errors. Nature 323 (6088): 533-536. doi:10.1038/323533a0.
  5. Cybenko G., 1989 Approximations by superpositions of sigmoidal functions, Mathematics of Control, Signals, and Systems, 2 (4), 303-314.
  6. Le Cessie S., Van Houwelingen J.C., 1992, Ridge estimators in Logistic Regression. Applied Statistics.
  7. Bettonvil B., Kleijnen J.P.C., 1997, Searching for important factors in simulation models with many factors: Sequential bifurcation, European Journal of Operational Research, Volume 96, Issue 1, Pages 180- 194.
  8. Haykin S., 1998, Neural Networks: A Comprehensive Foundation (2 ed.). Prentice Hall. ISBN 0-13-273350- 1.
  9. Harrel F., 2001 Regression Modeling Strategies, SpringerVerlag.
  10. Kern S., Muller S.D., Hansen N., Büche D., Ocenasek J., Koumoutsakos P., 2004, learning probability distributions in continuous evolutionary strategies - a comparative review, Journal of Natural Computing Volume 3 Issue 1, Pages 77 - 112.
  11. Bishop C., 2006 Pattern Recognition and Machine Learning, Springer Science+Business Media, LLC, pp 217-218.
  12. Hinton G.E., Osindero S., Yee-Whye The, 2006, A fast learning algorithm for deep belief nets, Neural Computation, 18(7):1527-1554.
  13. Morgan P.J., Cleave-Hogg D, Desousa S., LamMcCulloch J., 2006, Applying theory to practice in undergraduate education using high fidelity simulation, Med Teach, vol. 28, no. 1, pp. e10-e15.
  14. Zhang A., 2007 One-factor-at-a-time Screening Designs for Computer Experiments, SAE Technical Paper 2007-01-1660, doi:10.4271/2007-01-1660.
  15. Last M., Luta G., Orso A., Porter A., Young S., 2008, Pooled ANOVA, Computational Statistics & Data Analysis, Volume 52, Issue 12, Pages 5215-5228.
  16. Hall M., Eibe F., Holmes G., Pfahringer B., Reutemann P., Witten I., 2009 The WEKA Data Mining Software: An Update; SIGKDDExplorations, Volume11, Issue1.
  17. Lewis J. H., and Jiang S. B., 2009, A theoretical model for respiratory motion artifacts in free-breathing CT scans, Phys Med Biol, vol. 54, no. 3, pp. 745-755.
  18. Bergmeir C., Benìtez J.M., 2012, Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS, Journal of Statistical Software, Volume 46, Issue 7.
  19. Vidal F.P., Villard P., Lutton E., 2013, Automatic tuning of respiratory model for patient-based simulation, MIBISOC'13 - International Conference on Medical Imaging using Bio-inspired and Soft Computing.
Download


Paper Citation


in Harvard Style

Hessel M., Borgatelli F. and Ortalli F. (2014). A Novel Approach to Model Design and Tuning through Automatic Parameter Screening and Optimization - Theory and Application to a Helicopter Flight Simulator Case-study . In Proceedings of the 4th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH, ISBN 978-989-758-038-3, pages 24-35. DOI: 10.5220/0005022600240035


in Bibtex Style

@conference{simultech14,
author={Matteo Hessel and Francesco Borgatelli and Fabio Ortalli},
title={A Novel Approach to Model Design and Tuning through Automatic Parameter Screening and Optimization - Theory and Application to a Helicopter Flight Simulator Case-study},
booktitle={Proceedings of the 4th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH,},
year={2014},
pages={24-35},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005022600240035},
isbn={978-989-758-038-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 4th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH,
TI - A Novel Approach to Model Design and Tuning through Automatic Parameter Screening and Optimization - Theory and Application to a Helicopter Flight Simulator Case-study
SN - 978-989-758-038-3
AU - Hessel M.
AU - Borgatelli F.
AU - Ortalli F.
PY - 2014
SP - 24
EP - 35
DO - 10.5220/0005022600240035