Cross-Domain Generalization with Reverse Dynamics Models in Offline Model-Based Reinforcement Learning

Yana Stoyanova, Maryam Tavakol

2025

Abstract

Recent advancements in offline reinforcement learning (RL) have enabled automation in many real-world applications, where online interactions are often infeasible or costly, especially in high-stakes problems like healthcare or robotics. However, most algorithms are developed and evaluated in the same environment, which does not reflect the ever-charging nature of our world. Hence, beyond dealing with the distributional shift between the learning policy and offline data, it is crucial to account for domain shifts. Model-based offline RL (MBORL) methods are generally preferred over model-free counterparts for their ability to generalize beyond the dataset by learning (forward) dynamics models to generate new trajectories. Nevertheless, these models tend to overgeneralize in out-of-support regions due to limited samples. In this paper, we present a safer approach to balance conservatism and generalization by learning a reverse dynamics model instead, that can adapt to environments with varying dynamics, known as cross-domain generalization. We introduce CARI (Context-Aware Reverse Imaginations), a novel approach that incorporates context-awareness to capture domain-specific characteristics into the reverse dynamics model, resulting in more accurate models. Experiments on four variants of Hopper and Walker2D demonstrate that CARI consistently matches or outperforms state-of-the-art MBORL techniques that utilize a reverse dynamics model for cross-domain generalization.

Paper Citation

in Harvard Style

Stoyanova Y. and Tavakol M. (2025). Cross-Domain Generalization with Reverse Dynamics Models in Offline Model-Based Reinforcement Learning. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 57-68. DOI: 10.5220/0013097200003890

in Bibtex Style

@conference{icaart25,
author={Yana Stoyanova and Maryam Tavakol},
title={Cross-Domain Generalization with Reverse Dynamics Models in Offline Model-Based Reinforcement Learning},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2025},
pages={57-68},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013097200003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Cross-Domain Generalization with Reverse Dynamics Models in Offline Model-Based Reinforcement Learning
SN - 978-989-758-737-5
AU - Stoyanova Y.
AU - Tavakol M.
PY - 2025
SP - 57
EP - 68
DO - 10.5220/0013097200003890
PB - SciTePress