Expert-guided Symmetry Detection in Markov Decision Processes

Giorgio Angelotti, Giorgio Angelotti, Nicolas Drougard, Nicolas Drougard, Caroline Chanel, Caroline Chanel

2022

Abstract

Learning a Markov Decision Process (MDP) from a fixed batch of trajectories is a non-trivial task whose outcome’s quality depends on both the amount and the diversity of the sampled regions of the state-action space. Yet, many MDPs are endowed with invariant reward and transition functions with respect to some transformations of the current state and action. Being able to detect and exploit these structures could benefit not only the learning of the MDP but also the computation of its subsequent optimal control policy. In this work we propose a paradigm, based on Density Estimation methods, that aims to detect the presence of some already supposed transformations of the state-action space for which the MDP dynamics is invariant. We tested the proposed approach in a discrete toroidal grid environment and in two notorious environments of OpenAI’s Gym Learning Suite. The results demonstrate that the model distributional shift is reduced when the dataset is augmented with the data obtained by using the detected symmetries, allowing for a more thorough and data-efficient learning of the transition functions.

Paper Citation

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Expert-guided Symmetry Detection in Markov Decision Processes
SN - 978-989-758-547-0
AU - Angelotti G.
AU - Drougard N.
AU - Chanel C.
PY - 2022
SP - 88
EP - 98
DO - 10.5220/0010783400003116

in Harvard Style

Angelotti G., Drougard N. and Chanel C. (2022). Expert-guided Symmetry Detection in Markov Decision Processes. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-547-0, pages 88-98. DOI: 10.5220/0010783400003116

in Bibtex Style

@conference{icaart22,
author={Giorgio Angelotti and Nicolas Drougard and Caroline Chanel},
title={Expert-guided Symmetry Detection in Markov Decision Processes},
booktitle={Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2022},
pages={88-98},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010783400003116},
isbn={978-989-758-547-0},
}