RL Algorithms¶

This table displays the rl algorithms that are implemented in the stable baselines project, along with some useful characteristics: support for recurrent policies, discrete/continuous actions, multiprocessing.

Name	Refactored [1]	Recurrent	`Box`	`Discrete`	Multi Processing
A2C	✔️	✔️	✔️	✔️	✔️
ACER	✔️	✔️	❌ [4]	✔️	✔️
ACKTR	✔️	✔️	✔️	✔️	✔️
DDPG	✔️	❌	✔️	❌	✔️ [3]
DQN	✔️	❌	❌	✔️	❌
HER	✔️	❌	✔️	✔️	❌
GAIL [2]	✔️	✔️	✔️	✔️	✔️ [3]
PPO1	✔️	❌	✔️	✔️	✔️ [3]
PPO2	✔️	✔️	✔️	✔️	✔️
SAC	✔️	❌	✔️	❌	❌
TD3	✔️	❌	✔️	❌	❌
TRPO	✔️	❌	✔️	✔	✔️ [3]

[1]	Whether or not the algorithm has be refactored to fit the `BaseRLModel` class.

[2]	Only implemented for TRPO.

[3]	(1, 2, 3, 4) Multi Processing with MPI.

[4]	TODO, in project scope.

Note

Non-array spaces such as Dict or Tuple are not currently supported by any algorithm, except HER for dict when working with gym.GoalEnv

Actions gym.spaces:

Box: A N-dimensional box that contains every point in the action space.
Discrete: A list of possible actions, where each timestep only one of the actions can be used.
MultiDiscrete: A list of possible actions, where each timestep only one action of each discrete set can be used.
MultiBinary: A list of possible actions, where each timestep any of the actions can be used in any combination.

Note

Some logging values (like ep_rewmean, eplenmean) are only available when using a Monitor wrapper See Issue #339 for more info.

Reproducibility¶

Completely reproducible results are not guaranteed across Tensorflow releases or different platforms. Furthermore, results need not be reproducible between CPU and GPU executions, even when using identical seeds.

In order to make computations deterministic on CPU, on your specific problem on one specific platform, you need to pass a seed argument at the creation of a model and set n_cpu_tf_sess=1 (number of cpu for Tensorflow session). If you pass an environment to the model using set_env(), then you also need to seed the environment first.

Note

Because of the current limits of Tensorflow 1.x, we cannot ensure reproducible results on the GPU yet. This issue is solved in Stable-Baselines3 “PyTorch edition”

Note

TD3 sometimes fail to have reproducible results for obscure reasons, even when following the previous steps (cf PR #492). If you find the reason then please open an issue ;)

Credit: part of the Reproducibility section comes from PyTorch Documentation