Data Augmentation Through Expert-Guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning

Giorgio Angelotti, Giorgio Angelotti, Nicolas Drougard, Nicolas Drougard, Caroline Chanel, Caroline Chanel

2023

Abstract

Offline estimation of the dynamical model of a Markov Decision Process (MDP) is a non-trivial task that greatly depends on the data available in the learning phase. Sometimes the dynamics of the model is invariant with respect to some transformations of the current state and action. Recent works showed that an expert-guided pipeline relying on Density Estimation methods as Deep Neural Network based Normalizing Flows effectively detects this structure in deterministic environments, both categorical and continuous-valued. The acquired knowledge can be exploited to augment the original data set, leading eventually to a reduction in the distributional shift between the true and the learned model. Such data augmentation technique can be exploited as a preliminary process to be executed before adopting an Offline Reinforcement Learning architecture, increasing its performance. In this work we extend the paradigm to also tackle non-deterministic MDPs, in particular, 1) we propose a detection threshold in categorical environments based on statistical distances, and 2) we show that the former results lead to a performance improvement when solving the learned MDP and then applying the optimized policy in the real environment.

Paper Citation

in Harvard Style

Angelotti G., Drougard N. and Chanel C. (2023). Data Augmentation Through Expert-Guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-623-1, pages 115-124. DOI: 10.5220/0011633400003393

in Bibtex Style

@conference{icaart23,
author={Giorgio Angelotti and Nicolas Drougard and Caroline Chanel},
title={Data Augmentation Through Expert-Guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2023},
pages={115-124},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011633400003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Data Augmentation Through Expert-Guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning
SN - 978-989-758-623-1
AU - Angelotti G.
AU - Drougard N.
AU - Chanel C.
PY - 2023
SP - 115
EP - 124
DO - 10.5220/0011633400003393