Evolutionary Reinforcement Learning (ERL), which integrates Evolutionary Algorithms (EAs) and Reinforcement Learning (RL) for optimization, has demonstrated remarkable performance advancements. By fusing the strengths of both approaches, ERL has emerged as a promising research direction.
We want to create a survey that researchers from different academic backgrounds can quickly understand and get started with.
This repository is under construction ...
26 July 2024: 🔥🔥🔥 The paper has been accepted by the top journal in evolutionary computation (IEEE Transactions on Evolutionary Computation, TEvC).
20 June 2024: Our survey paper (new version) has been updated on arXiv. Arxiv: Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms This survey is now categorized to better align with the conventions of researchers in different fields, providing a quicker and more accessible introduction for researchers.
20 January 2024: I am updating the survey on this topic and will fully update the site when the survey is updated.
If you discover any related works that I have missed, please submit an issue. I will update it in the survey and on the website. Thank you very much!
-
If you want to get started, I recommend referring to the Accepted Papers with the Released Code, as it provides an easy way to explore research papers and their associated code implementations. Furthermore, building your algorithm on top of state-of-the-art algorithms will greatly enhance your productivity and efficiency.
-
If you are interested in sequential decision-making problems, it is recommended to focus primarily on EA-Assisted Optimization of RL and Synergistic Optimization of EA and RL. If you are interested in other optimization problems, it is suggested to pay attention to RL-Assisted Optimization of EA. I primarily focus on the former.
If you do find our survey or the repository helpful (or if you would be so kind as to offer us some encouragement), please consider kindly giving a star, and citing our paper.
@ARTICLE{10637292,
author={Li, Pengyi and Hao, Jianye and Tang, Hongyao and Fu, Xian and Zhen, Yan and Tang, Ke},
journal={IEEE Transactions on Evolutionary Computation},
title={Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms},
year={2024},
keywords={Optimization;Sociology;Evolutionary computation;Decision making;Surveys;Reinforcement learning;Genomics;Evolutionary Algorithms;Reinforcement Learning;Evolutionary
Reinforcement Learning},
doi={10.1109/TEVC.2024.3443913}}
Other Surveys:
- Combining evolution and deep reinforcement learning for policy search: a survey
- Deep reinforcement learning versus evolution strategies: A comparative survey
- A survey on evolutionary reinforcement learning algorithms
- Reinforcement learning versus evolutionary computation: A survey on hybrid algorithms
- Evolutionary computation and the reinforcement learning problem
- Evolutionary reinforcement learning: A survey
We provide three main research directions with various branches as follows: (In total, it includes approximately 90 works.)
- ⭐ EA-Assisted Optimization of RL
- ⭐ RL-Assisted Optimization of EA
- ⭐ Synergistic Optimization of EA and RL
Detailed information can be found in our survey paper.
ELIFE 2020 Reinforcement Learning beyond The Bellman Equation: Exploring Critic Objectives using Evolution
https://direct.mit.edu/isal/proceedings/isal2020/32/441/98464 Code: https://github.com/ajleite/RLBeyondBellman
ICLR 2021 Genetic Soft Updates for Policy Evolution in Deep Reinforcement Learning
OpenReview: https://openreview.net/forum?id=TGFO0DbD_pk Code: Not Found
ICLR 2023 Improving Deep Policy Gradients with Value Function Search
OpenReview: https://openreview.net/forum?id=6qZC7pfenQm Code: Not Found
CORL 2018 Scalable deep reinforcement learning for vision-based robotic manipulation
Link: https://proceedings.mlr.press/v87/kalashnikov18a Code: https://github.com/quantumiracle/QT_Opt
ICML 2019 RL4RealLife Workshop Q-learning for continuous actions with cross-entropy guided policies
Link: https://arxiv.org/abs/1903.10605 Code: Not Found
Preprint 2022 Evolutionary Action Selection for Gradient-based Policy Learning
Link https://arxiv.org/abs/2201.04286 Code: Not Found
Preprint 2021 Soft Actor-Critic with Cross-entropy Policy Optimization
Link: https://arxiv.org/abs/2112.11115 Code: https://github.com/wcgcyx/SAC-CEPO
CORL 2021 GRAC: Self-guided and Self-regularized Actor-critic
Link: https://arxiv.org/abs/2009.08973 Code: https://github.com/stanford-iprl-lab/GRAC
ICML 2022 Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification
Link: https://arxiv.org/abs/2111.11188 Code: https://github.com/ling-pan/OMAR
Preprint 2020 Deep Multi-agent Reinforcement Learning for Decentralized Continuous Cooperative Control
Link: https://beipeng.github.io/files/2003.06709.pdf Code: https://github.com/oxwhirl/comix
GECCO 2018 Online Meta-learning by Parallel Algorithm Competition
Link: https://arxiv.org/abs/1702.07490 Code: Not Found
Preprint 2017 Population Based Training of Neural Networks
Link: https://arxiv.org/abs/1711.09846 Code: https://github.com/voiler/PopulationBasedTraining
ICLR 2021 Sample-efficient Automated Deep Reinforcement Learning
Link: https://arxiv.org/abs/2009.01555 Code: https://github.com/automl/SEARL
IEEE IRC 2022 GA+DDPG+HER: Genetic Algorithm-based Function Optimizer in Deep Reinforcement Learning for Robotic Manipulation Tasks
Link: https://arxiv.org/abs/2203.00141 Code: https://github.com/aralab-unr/ga-drl-aubo-ara-lab
Preprint 2021 Towards Automatic Actor-critic Solutions to Continuous Control
Link: https://arxiv.org/abs/2106.08918 Code: https://github.com/jakegrigsby/deep_control
Preprint 2020 Online Hyper-parameter Tuning in Offpolicy Learning via Evolutionary Strategies
Link: https://arxiv.org/abs/2006.07554 Code: Not Found
ICLR 2021 Evolving Reinforcement Learning Algorithms
Link: https://arxiv.org/abs/2101.03958 Code: https://github.com/google/brain_autorl/tree/main/evolving_rl
NeurIPS 2022 Discovered Policy Optimisation
Link: https://arxiv.org/abs/2210.05639 Code: https://github.com/luchris429/discovered-policy-optimisation
ICLR 2024 Discovering Temporally-Aware Reinforcement Learning Algorithms
Link: https://arxiv.org/abs/2402.05828 Code: https://github.com/EmptyJackson/groove
ICLR 2024 Behaviour Distillation
Link: https://openreview.net/forum?id=qup9xD8mW4 Code: https://github.com/FLAIROx/behaviour-distillation
ICML 2023 Adversarial Cheap Talk
Link: https://arxiv.org/abs/2211.11030 Code: https://github.com/luchris429/adversarial-cheap-talk
IROS 2021 PNS: Population-guided Novelty Search for Reinforcement Learning in Hard Exploration Environments
Link: https://arxiv.org/abs/1811.10264 Code: Not Found
Nature 2021 Go explore: A New Approach for Hard-exploration Problems
Link: https://arxiv.org/abs/1901.10995 Code: https://github.com/uber-research/go-explore
NeurIPS 2018 Genetic-gated Networks for Deep Reinforcement Learning
Link: https://arxiv.org/abs/1903.01886 Code: Not Found
GECCO 2021 Evo-rl: Evolutionary-driven Reinforcement Learning
Link: https://arxiv.org/abs/2007.04725 Code: Not Found
AAAI 2023 Robust Multi-agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers
Link: https://arxiv.org/abs/2305.05909 Code: https://github.com/zzq-bot/ROMANCE
Preprint 2023 Communication-robust Multiagent Learning by Adaptable Auxiliary Multi-agent Adversary Generation
Link: https://arxiv.org/abs/2305.05116 Code: Not Found
ICLR 2020 Evolutionary Population Curriculum for Scaling Multi-agent Reinforcement Learning
Link: https://arxiv.org/abs/2003.10423 Code: https://github.com/qian18long/epciclr2020
IROS 2020 MAPPER: Multi-agent Path Planning with Evolutionary Reinforcement Learning in Mixed Dynamic Environments
Link: https://arxiv.org/abs/2007.15724 Code: Not Found
NeurIPS 2021 Symbolic Regression Via Neural-guided Genetic Programming Population Seeding
Link: https://arxiv.org/pdf/2111.00053.pdf Code: https://github.com/dso-org/deep-symbolic-optimization
Knowl Based Syst 2021 Rule-based Reinforcement Learning Methodology To Inform Evolutionary Algorithms For Constrained Optimization Of Engineering Applications
Link: https://www.sciencedirect.com/science/article/abs/pii/S095070512100099X Code: https://github.com/mradaideh/neorl
NeurIPS 2023 Deepaco: Neuralenhanced Ant Systems For Combinatorial Optimization,
Link: https://arxiv.org/abs/2309.14032 Code: https://github.com/henry-yeh/DeepACO
ICLR 2023 ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation
Arxiv: https://arxiv.org/abs/2210.17375 Code: https://github.com/yeshenpy/ERL-Re2
ELSEVIER Information Sciences A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning
Arxiv: https://arxiv.org/abs/2201.00129 Code: https://github.com/Yuxing-Wang-THU/Surrogate-assisted-ERL
ICLR 2021 submission PGPS: Coupling Policy Gradient with Population-based Search
OpenReview: https://openreview.net/forum?id=PeT5p3ocagr Code: https://github.com/NamKim88/PGPS/blob/master/Main.py
ICLR 2018 Policy Optimization By Genetic Distillation
Link: https://arxiv.org/abs/1711.01012 Code: https://www.catalyzex.com/paper/policy-optimization-by-genetic-distillation/code
AAMMAS 2021 Guiding Evolutionary Strategies With Off-policy Actor-critic
Link: https://robintyh1.github.io/papers/Tang2021CEMACER.pdf Code: Not Found
SSCI 2021 Population Based Reinforcement Learning
Link: https://ieeexplore.ieee.org/document/9660084 Code: https://github.com/jjccero/pbrl
IEEE Acess 2020 Efficient Novelty Search Through Deep Reinforcement Learning
Link: https://ieeexplore.ieee.org/document/9139203 Code: https://github.com/shilx001/NoveltySearch_Improvement
Comput. Intell. Neurosci 2021 Diversity Evolutionary Policy Deep Reinforcement Learning
Link: https://www.hindawi.com/journals/cin/2021/5300189/ Code: Not Found
Arxiv Preprint 2020 QD-RL: Efficient Mixing Of Quality And Diversity In Reinforcement Learning,
Link: https://www.researchgate.net/publication/342198149_QD-RL_Efficient_Mixing_of_Quality_and_Diversity_in_Reinforcement_Learning Code: https://openreview.net/forum?id=5Dl1378QutR
GECCO 2021 Policy Gradient Assisted Map-elites
Link: https://www.semanticscholar.org/paper/Policy-gradient-assisted-MAP-Elites-Nilsson-Cully/67038237383a8f4802a9595636a6fb73f748dc5b Code: https://github.com/ollebompa/PGA-MAP-Elites
GECCO 2022 Approximating Gradients For Differentiable Quality Diversity In Reinforcement Learning
Link: https://arxiv.org/abs/2202.03666 Code: https://github.com/icaros-usc/dqd-rl
ICLR 2024 Sample-efficient Quality-diversity By Cooperative Coevolution
Link: https://openreview.net/forum?id=JDud6zbpFv Code: https://openreview.net/forum?id=JDud6zbpFv
ICLR 2023 Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery
Link: https://openreview.net/forum?id=6BHlZgyPOZY Code: https://github.com/instadeepai/qd-skill-discovery-benchmark
GECCO 2022 Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning
Link: https://arxiv.org/pdf/2202.03666.pdf Code: https://github.com/icaros-usc/dqd-rl
ICLR 2019 CEM-RL: Combining evolutionary and gradient-based methods for policy search
Arxiv: https://arxiv.org/abs/1810.01222 Code: https://github.com/apourchot/CEM-RL
Waiting for updates
ESOA 2006 Reinforcement learning for online control of evolutionary algorithms
Link: https://link.springer.com/chapter/10.1007/978-3-540-69868-5_10 Code: Not Found
PPSN 2020 Learning step-size adaptation in CMA-ES
Link: https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/20-PPSN-LTO-CMA.pdf Code: https://github.com/automl/LTO-CMA
ECAI 2020 Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework
Link: https://ecai2020.eu/papers/1237_paper.pdf Code: https://github.com/automl/DAC
TETCI 2022 Variational reinforcement learning for hyper-parameter tuning of adaptive evolutionary algorithm
Link: https://www.researchgate.net/publication/365582495_Variational_Reinforcement_Learning_for_Hyper-Parameter_Tuning_of_Adaptive_Evolutionary_Algorithm Code: Not Found
IEEE Comput. Intell. Mag., 2023 Controlling sequential hybrid evolutionary algorithm by q-learning
Link: https://ieeexplore.ieee.org/document/10035716/ Code: https://github.com/xiaomeiabc/Controlling-Sequential-Hybrid-Evolutionary-Algorithm-by-Q-Learning
NeurIPS 2022 Multiagent dynamic algorithm configuration
Arxiv: https://arxiv.org/abs/2210.06835 Code: https://github.com/lamda-bbo/madac
Appl. Soft Comput., 2021 Q-learning-based parameter control in differential evolution for structural optimization
Link: https://www.sciencedirect.com/science/article/abs/pii/S1568494621003872 Code: Not Found
Energy Reports, 2021 Reinforcement learning-based differential evolution for parameters extraction of photovoltaic models
Link: https://www.sciencedirect.com/science/article/pii/S2352484721000974 Code: Not Found
L4DC 2020 Model-predictive control via cross-entropy and gradient-based optimization
Link: https://proceedings.mlr.press/v120/bharadhwaj20a/bharadhwaj20a.pdf Code: https://github.com/homangab/gradcem
CORL 2021 Learning off-policy with online planning
Arxiv: https://arxiv.org/abs/2008.10066 Code: https://github.com/hari-sikchi/LOOP
ICML 2022 Temporal difference learning for model predictive control
Arxiv: https://arxiv.org/abs/2203.04955 Code: https://github.com/nicklashansen/tdmpc
ICML 2024 EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search
Link: https://openreview.net/forum?id=75Hes6Zse4 Code: https://github.com/yeshenpy/EvoRainbow
ICML 2024 Value-Evolutionary-Based Reinforcement Learning
Link: https://openreview.net/forum?id=XobPpcN4yZ Code: https://github.com/yeshenpy/VEB-RL
ICLR 2023 ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation
Arxiv: https://arxiv.org/abs/2210.17375 Code: https://github.com/yeshenpy/ERL-Re2
ICLR 2021 submission PGPS: Coupling Policy Gradient with Population-based Search
OpenReview: https://openreview.net/forum?id=PeT5p3ocagr Code: https://github.com/NamKim88/PGPS/blob/master/Main.py
AAMAS 2022 Off-policy evolutionary reinforcement learning with maximum mutations (Maximum Mutation Reinforcement Learning for Scalable Control)
Link: https://nbviewer.org/github/karush17/karush17.github.io/blob/master/_pages/temp4.pdf Code: https://github.com/karush17/esac
ELSEVIER Information Sciences A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning
Arxiv: https://arxiv.org/abs/2201.00129 Code: https://github.com/Yuxing-Wang-THU/Surrogate-assisted-ERL
Preprint Evolutionary action selection for gradient-based policy learning
Arxiv: https://arxiv.org/abs/2201.04286v1 Code: Not Found
NeurIPS 2020 Competitive and cooperative heterogeneous deep reinforcement learning
Arxiv: https://arxiv.org/abs/2011.00791 Code: Not Found
AMMAS Guiding Evolutionary Strategies with Off-Policy Actor-Critic
Link: https://dl.acm.org/doi/10.5555/3463952.3464104 Code: Not Found
AAAI 2020 PDERL: Proximal Distilled Evolutionary Reinforcement Learning
Arxiv: https://arxiv.org/abs/1906.09807 Code: https://github.com/crisbodnar/pderl
LOD 2020 Gradient Bias to Solve the Generalization Limit of Genetic Algorithms Through Hybridization with Reinforcement Learning
Link: https://dl.acm.org/doi/abs/10.1007/978-3-030-64583-0_26 Code: https://github.com/ricordium/Gradient-Bias
ICML 2019 Collaborative Evolutionary Reinforcement Learning
Arxiv: https://arxiv.org/abs/1905.00976 Code: https://github.com/intelai/cerl
Preprint FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
Link: https://arxiv.org/pdf/1907.00526v2.pdf Code: Not Found
NeurIPS 2018 Evolution-Guided Policy Gradient in Reinforcement Learning
Arxiv: https://arxiv.org/abs/1810.01222 Code: https://github.com/apourchot/CEM-RL
ICML 2023 RACE: Improve Multi-Agent Reinforcement Learning with Representation Asymmetry and Collaborative Evolution
Link: https://icml.cc/virtual/2023/poster/23791 Code: https://github.com/yeshenpy/RACE
GECCO 2023 Novelty Seeking Multiagent Evolutionary Reinforcement Learning
Link: https://dl.acm.org/doi/abs/10.1145/3583131.3590428 Code: Not Found
IJCNN 2023 Evolution Strategies Enhanced Complex Multiagent Coordination
Link: https://ieeexplore.ieee.org/document/10191313 Code: Not Found
ICONIP 2022 Cooperation and Competition: Flocking with Evolutionary Multi-Agent Reinforcement Learning
Arxiv: https://link.springer.com/chapter/10.1007/978-3-031-30105-6_23 Code: Not Found
GECCO 2021 MAEDyS: multiagent evolution via dynamic skill selection
Link: https://dl.acm.org/doi/abs/10.1145/3449639.3459387 Code: Not Found
ICML 2020 Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
Arxiv: https://arxiv.org/abs/1906.07315 Code: Anonymous Code or https://github.com/ShawK91/MERL
NeurIPS 2021 Evolution gym: A large-scale benchmark for evolving soft robots
Link: https://dl.acm.org/doi/abs/10.1145/3449639.3459387 Code: http://evogym.csail.mit.edu
Preprint Leveraging hyperbolic embeddings for coarse-to-fine robot design
Link: https://arxiv.org/abs/2311.00462 Code: https://github.com/drdh/HERD
TEC 2023 Rapidly evolving soft robots via action inheritance
Link: https://ieeexplore.ieee.org/document/10296048 Code: https://github.com/HandingWangXDGroup/AIEA
Nature Communications 2021 Embodied Intelligence via Learning and Evolution
Link: https://arxiv.org/abs/2102.02202 Code: https://github.com/agrimgupta92/derl
ICLR 2021 Task-Agnostic Morphology Evolution
Link: https://arxiv.org/abs/2102.13100 Code: https://github.com/jhejna/morphology-opt
IEEE Transactions on Cybernetics 2024 Interpretable-AI Policies using Evolutionary Nonlinear Decision Trees for Discrete Action Systems
Link: https://ieeexplore.ieee.org/document/9805655 Code: https://github.com/yddhebar/NLDT
GECCO 2022 Interpretable ai for policy-making in pandemics
Link: https://arxiv.org/abs/2204.04256 Code: Not found
SSCI 2021 A co-evolutionary approach to interpretable reinforcement learning in environments with continuous action spaces
Link: https://ieeexplore.ieee.org/document/9660048 Code: Not found
SIGAPP 2023 Quality diversity evolutionary learning of decision trees
Link: https://arxiv.org/abs/2204.04256 Code: Not found
Preprint Social Interpretable Reinforcement Learning
Link: https://arxiv.org/abs/2401.15480 Code: Not found
Access 2021 Symbolic regression methods for reinforcement learning
Link: https://arxiv.org/abs/2204.04256 Code: Not found
Evolutionary computation 1995 Classifier fitness based on accuracy
Link: https://dl.acm.org/doi/10.1162/evco.1995.3.2.149 Code: https://github.com/hosford42/xcs
Natural Computing 2002 Classifiers that approximate functions
Link: https://link.springer.com/article/10.1023/A:1016535925043 Code: Not found
Evolutionary Intelligence 2015 XCSF with tile coding in discontinuous action-value landscapes
Link: https://link.springer.com/article/10.1007/s12065-015-0129-7 Code: Not found
Evolutionary Computation 2013 Dynamical genetic programming in XCSF
Link: https://pubmed.ncbi.nlm.nih.gov/22564070/ Code: Not found