LSTM, ConvLSTM, MDN-RNN and GridLSTM Memory-based Deep Reinforcement Learning

Fernando Fradique Duarte, Nuno Lau, Artur Pereira, Luís Reis

2023

Abstract

Memory-based Deep Reinforcement Learning has been shown to be a viable solution to successfully learn control policies directly from high-dimensional sensory data in complex vision-based control tasks. At the core of this success lies the Long Short-Term Memory or LSTM, a well-known type of Recurrent Neural Network. More recent developments have introduced the ConvLSTM, a convolutional variant of the LSTM and the MDN-RNN, a Mixture Density Network combined with an LSTM, as memory modules in the context of Deep Reinforcement Learning. The defining characteristic of the ConvLSTM is its ability to preserve spatial information, which may prove to be a crucial factor when dealing with vision-based control tasks while the MDN-RNN can act as a predictive memory eschewing the need to explicitly plan ahead. Also of interest to this work is the GridLSTM, a network of LSTM cells arranged in a multidimensional grid. The objective of this paper is therefore to perform a comparative study of several memory modules, based on the LSTM, ConvLSTM, MDN-RNN and GridLSTM in the scope of Deep Reinforcement Learning, and more specifically as the memory modules of the agent. All experiments were validated using the Atari 2600 videogame benchmark.

Paper Citation

in Harvard Style

Fradique Duarte F., Lau N., Pereira A. and Reis L. (2023). LSTM, ConvLSTM, MDN-RNN and GridLSTM Memory-based Deep Reinforcement Learning. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-623-1, pages 169-179. DOI: 10.5220/0011664900003393

in Bibtex Style

@conference{icaart23,
author={Fernando Fradique Duarte and Nuno Lau and Artur Pereira and Luís Reis},
title={LSTM, ConvLSTM, MDN-RNN and GridLSTM Memory-based Deep Reinforcement Learning},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2023},
pages={169-179},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011664900003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - LSTM, ConvLSTM, MDN-RNN and GridLSTM Memory-based Deep Reinforcement Learning
SN - 978-989-758-623-1
AU - Fradique Duarte F.
AU - Lau N.
AU - Pereira A.
AU - Reis L.
PY - 2023
SP - 169
EP - 179
DO - 10.5220/0011664900003393