Tangled Program Graphs with Indexed Memory in Control Tasks with Short Time Dependencies

Tanya Djavaherpour, Ali Naqvi, Stephen Kelly

2024

Abstract

This paper addresses the challenges of shared temporal memory for evolutionary reinforcement learning agents in partially observable control tasks with short time dependencies. Tangled Program Graphs (TPG) is a genetic programming framework which has been widely studied in memory intensive tasks from video games, time series forecasting, and predictive control domains. In this study, we aim to improve external indexed memory usage in TPG by minimizing the impact of destructive agents during cultural transmission. We test various memory resetting strategies—per agent, per episode, and a no-memory control group—and evaluate their effectiveness in mitigating destructive effects while maintaining performance. Results from Acrobot, Pendulum, and CartPole tasks show that resetting memory more often can significantly boost TPG performance while preserving computational efficiency. These findings highlight the importance of memory management in Reinforcement Learning (RL) and suggest opportunities for further optimization for more complex visual RL environments, including adaptive memory resetting and evolved probabilistic memory operations.

Paper Citation

in Harvard Style

Djavaherpour T., Naqvi A. and Kelly S. (2024). Tangled Program Graphs with Indexed Memory in Control Tasks with Short Time Dependencies. In Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA; ISBN 978-989-758-721-4, SciTePress, pages 296-303. DOI: 10.5220/0013016800003837

in Bibtex Style

@conference{ecta24,
author={Tanya Djavaherpour and Ali Naqvi and Stephen Kelly},
title={Tangled Program Graphs with Indexed Memory in Control Tasks with Short Time Dependencies},
booktitle={Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA},
year={2024},
pages={296-303},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013016800003837},
isbn={978-989-758-721-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computational Intelligence - Volume 1: ECTA
TI - Tangled Program Graphs with Indexed Memory in Control Tasks with Short Time Dependencies
SN - 978-989-758-721-4
AU - Djavaherpour T.
AU - Naqvi A.
AU - Kelly S.
PY - 2024
SP - 296
EP - 303
DO - 10.5220/0013016800003837
PB - SciTePress