Accelerate Training of Reinforcement Learning Agent by Utilization of Current and Previous Experience

Chenxing Li, Chenxing Li, Yinlong Liu, Zhenshan Bing, Fabian Schreier, Fabian Schreier, Jan Seyler, Shahram Eivazi, Shahram Eivazi

2023

Abstract

In this paper, we examine three extensions to the Q-function Targets via Optimization (QT-Opt) algorithm and empirically studies their effects on training time over complex robotic tasks. The vanilla QT-Opt algorithm requires lots of offline data (several months with multiple robots) for training which is hard to collect in practice. To bridge the gap between basic reinforcement learning research and real world robotic applications, first we propose to use hindsight goals techniques (Hindsight Experience Replay, Hindsight Goal Generation) and Energy-Based Prioritization (EBP) to increase data efficiency in reinforcement learning. Then, an efficient offline data collection method using PD control method and dynamic buffer are proposed. Our experiments show that both data collection and training the agent for a robotic grasping task takes about one day only, besides, the learning performance maintains high level (80% successful rate). This work serves as a step towards accelerating the training of reinforcement learning for complex real world robotics tasks.

Paper Citation

in Harvard Style

Li C., Liu Y., Bing Z., Schreier F., Seyler J. and Eivazi S. (2023). Accelerate Training of Reinforcement Learning Agent by Utilization of Current and Previous Experience. In Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART, ISBN 978-989-758-623-1, pages 698-705. DOI: 10.5220/0011745600003393

in Bibtex Style

@conference{icaart23,
author={Chenxing Li and Yinlong Liu and Zhenshan Bing and Fabian Schreier and Jan Seyler and Shahram Eivazi},
title={Accelerate Training of Reinforcement Learning Agent by Utilization of Current and Previous Experience},
booktitle={Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,},
year={2023},
pages={698-705},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011745600003393},
isbn={978-989-758-623-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART,
TI - Accelerate Training of Reinforcement Learning Agent by Utilization of Current and Previous Experience
SN - 978-989-758-623-1
AU - Li C.
AU - Liu Y.
AU - Bing Z.
AU - Schreier F.
AU - Seyler J.
AU - Eivazi S.
PY - 2023
SP - 698
EP - 705
DO - 10.5220/0011745600003393