Rethinking Population-Assisted Off-policy Reinforcement Learning

Zheng，Bowen; Cheng，Ran

10.1145/3583131.3590512

2023-07-15

Genetic and Evolutionary Computation Conference (GECCO)

GECCO 2023 - Proceedings of the 2023 Genetic and Evolutionary Computation Conference

624-632

JUL 15-19, 2023

null,Lisbon,PORTUGAL

1601 Broadway, 10th Floor, NEW YORK, NY, UNITED STATES

ASSOC COMPUTING MACHINERY

While off-policy reinforcement learning (RL) algorithms are sample efficient due to gradient-based updates and data reuse in the replay buffer, they struggle with convergence to local optima due to limited exploration. On the other hand, population-based algorithms offer a natural exploration strategy, but their heuristic black-box operators are inefficient. Recent algorithms have integrated these two methods, connecting them through a shared replay buffer. However, the effect of using diverse data from population optimization iterations on off-policy RL algorithms has not been thoroughly investigated. In this paper, we first analyze the use of off-policy RL algorithms in combination with population-based algorithms, showing that the use of population data could introduce an overlooked error and harm performance. To test this, we propose a uniform and scalable training design and conduct experiments on our tailored framework in robot locomotion tasks from the OpenAI gym. Our results substantiate that using population data in off-policy RL can cause instability during training and even degrade performance. To remedy this issue, we further propose a double replay buffer design that provides more on-policy data and show its effectiveness through experiments. Our results offer practical insights for training these hybrid methods.

evolutionary reinforcement learning neuroevolution off-policy learning

First ; Corresponding

English

CPCI-S

Program for Guangdong Introducing Innovative and Entrepreneurial Teams[2017ZT07X386]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Information Systems

WOS:001031455100070

2-s2.0-85167728613

Scopus

Department of Computer Science and Engineering,Southern University of Science and Technology,Guangdong,Shenzhen,China

Zheng，Bowen,Cheng，Ran. Rethinking Population-Assisted Off-policy Reinforcement Learning[C]. 1601 Broadway, 10th Floor, NEW YORK, NY, UNITED STATES:ASSOC COMPUTING MACHINERY,2023:624-632.