APPLYING Q-LEARNING TO NON-MARKOVIAN ENVIRONMENTS

Jurij Chizhov, Arkady Borisov

2009

Abstract

This paper considers the problem of intelligent agent functioning in non-Markovian environments. We advice to divide the problem into two subproblems: just finding non-Markovian states in the environment and building an internal representation of original environment by the agent. The internal representation is free from non Markovian states because insufficient number of additional dynamically created states and transitions are provided. Then, the obtained environment might be used in classical reinforcement learning algorithms (like SARSA(λ)) which guarantee the convergence by Bellman equation. A great difficulty is to recognize different “copies” of the same states. The paper contains a theoretical introduction, ideas and problem description, and, finally, an illustration of results and conclusions.

References

  1. Butz, M.V., Goldberg, D.E., Lanzi, P.L., 2005. Paper: Gradient Descent Methods in Learning Classifier Systems: Improving CXS Performance in Multistep problems, IEEE Transactions, Vol. 9, Issue 5.
  2. Sutton, R., Barto, R., 1998. Reinforcement Learning. An Introduction. Cambridge, MA: MIT Press.
  3. Russell, S., Norvig, R. 2003, Artificial Intelligence: A Modern Approach, Prentice Hall. New Jersey, 2nd ed.
  4. Padgham, L., Winikoff, M., 2004. Developing Intelligent Agent Systems. A Practical Guide. John Wiley & Sons.
  5. Kwee, I., Hutter, M., Schmidhuber J., 2001. Paper: Market-Based Reinforcement Learning in Partially Observable Worlds.
  6. Lin, L-J., 1993, PhD thesis: Reinforcement Learning for Robots Using Neural Networks, Carnegie Mellon University, Pittsburgh, CMU-CS-93-103.
Download


Paper Citation


in Harvard Style

Chizhov J. and Borisov A. (2009). APPLYING Q-LEARNING TO NON-MARKOVIAN ENVIRONMENTS . In Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 1: ICAART, ISBN 978-989-8111-66-1, pages 306-311. DOI: 10.5220/0001755603060311


in Bibtex Style

@conference{icaart09,
author={Jurij Chizhov and Arkady Borisov},
title={APPLYING Q-LEARNING TO NON-MARKOVIAN ENVIRONMENTS},
booktitle={Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,},
year={2009},
pages={306-311},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001755603060311},
isbn={978-989-8111-66-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,
TI - APPLYING Q-LEARNING TO NON-MARKOVIAN ENVIRONMENTS
SN - 978-989-8111-66-1
AU - Chizhov J.
AU - Borisov A.
PY - 2009
SP - 306
EP - 311
DO - 10.5220/0001755603060311