# A MINIMUM RELATIVE ENTROPY PRINCIPLE FOR ADAPTIVE CONTROL IN LINEAR QUADRATIC REGULATORS

### Daniel A. Braun, Pedro A. Ortega

#### Abstract

The design of optimal adaptive controllers is usually based on heuristics, because solving Bellman’s equations over information states is notoriously intractable. Approximate adaptive controllers often rely on the principle of certainty-equivalence where the control process deals with parameter point estimates as if they represented “true” parameter values. Here we present a stochastic control rule instead where controls are sampled from a posterior distribution over a set of probabilistic input-output models and the true model is identified by Bayesian inference. This allows reformulating the adaptive control problem as an inference and sampling problem derived from a minimum relative entropy principle. Importantly, inference and action sampling both work forward in time and hence such a Bayesian adaptive controller is applicable on-line. We demonstrate the improved performance that can be achieved by such an approach for linear quadratic regulator examples.

#### References

- A° ström, K. and Wittenmark, B. (1995). Adaptive Control. Prentice Hall, 2nd edition.
- Bradtke, S. (1993). Reinforcement learning applied to linear quadratic control. Advances in Neural Information Processing Systems 5.
- Campi, M. and Kumar, P. (1996). Optimal adaptive control of an lqg system. Proc. 35th Conf. on Decision and Control, pages 349-353.
- Engel, Y., Mannor, S., and Meir, R. (2005). Reinforcement learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pages 201-208.
- Haruno, M., Wolpert, D., and Kawato, M. (2001). Mosaic model for sensorimotor learning and control. Neural Computation, 13:2201-2220.
- Haykin, S. (2001). Kalman filtering and neural networks. John Wiley and Sons.
- Julier, S.J., U. J. and Durrant-Whyte, H. (1995). A new approach for filtering nonlinear systems. Proc. Am. Control Conference, pages 1628-1632.
- Kappen, B., Gomez, V., and Opper, M. (2009). Optimal control as a graphical model inference problem. arXiv:0901.0633.
- Ortega, P. and Braun, D. (2010). A bayesian rule for adaptive control based on causal interventions. In Proceedings of the third conference on artificial general intelligence, pages 121-126. Atlantis Press.
- Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, UK.
- Stengel, R. (1993). Optimal control and estimation. Dover Publications.
- Todorov, E. (2009). Efficient computation of optimal actions. Proceedings of the National Academy of Sciences U.S.A., 106:11478-11483.
- Todorov, E. . and Jordan, M. (2002). Optimal feedback control as a theory of motor coordination. Nat. Neurosci., 5:1226-1235.
- Toussaint, M., Harmeling, S., and Storkey, A. (2006). Probabilistic inference for solving (po)mdps. Technical report, EDI-INF-RR-0934, University of Edinburgh, School of Informatics.
- Wittenmark, B. (1975). Stochastic adaptive control methods: a survey. International Journal of Control, 21:705-730.

#### Paper Citation

#### in Harvard Style

A. Braun D. and A. Ortega P. (2010). **A MINIMUM RELATIVE ENTROPY PRINCIPLE FOR ADAPTIVE CONTROL IN LINEAR QUADRATIC REGULATORS** . In *Proceedings of the 7th International Conference on Informatics in Control, Automation and Robotics - Volume 3: ICINCO,* ISBN 978-989-8425-02-7, pages 103-108. DOI: 10.5220/0002938801030108

#### in Bibtex Style

@conference{icinco10,

author={Daniel A. Braun and Pedro A. Ortega},

title={A MINIMUM RELATIVE ENTROPY PRINCIPLE FOR ADAPTIVE CONTROL IN LINEAR QUADRATIC REGULATORS},

booktitle={Proceedings of the 7th International Conference on Informatics in Control, Automation and Robotics - Volume 3: ICINCO,},

year={2010},

pages={103-108},

publisher={SciTePress},

organization={INSTICC},

doi={10.5220/0002938801030108},

isbn={978-989-8425-02-7},

}

#### in EndNote Style

TY - CONF

JO - Proceedings of the 7th International Conference on Informatics in Control, Automation and Robotics - Volume 3: ICINCO,

TI - A MINIMUM RELATIVE ENTROPY PRINCIPLE FOR ADAPTIVE CONTROL IN LINEAR QUADRATIC REGULATORS

SN - 978-989-8425-02-7

AU - A. Braun D.

AU - A. Ortega P.

PY - 2010

SP - 103

EP - 108

DO - 10.5220/0002938801030108