sdpkjc/abcdrl

Modular Single-file Reinfocement Learning Algorithms Library

abcdrl's Issues

🗺️ Roadmap

v0.2.0

• Using Poetry & PEP-517 compliant #58
• Publish on PyPi #58
• Implemented DQN by TensorFlow2 #59
• #69
• Adapted to openrlbenchmark

v0.3.0

• Unit tests
• Overhaul buffer
• Implemented by JAX (faster abcdrl)

Future

• ppo_atari
• sac_atari
• redq
• drqv2_dmc
• MARL Algs | MADDPG?
• Offline RL Algs | CQL?

[Feature Request] Update wandb logger

Description

wandb/wandb#5008 completes support for gymnasium env monitoring.

Suggested Solution

Update wandb version to v0.13.11, and modify our custom logger to use the monitor provided by wandb directly. This will simplify the code and provide its robustness.

This adjustment might fix the #70 bug.

Alternatives

No response

Additional Context

No response

Checklist

• I have checked that there is no similar issue in the repo

[Bug]: optimize_memory_usage error in ReplayBuffer of dqn_atari fail

Description

1. In dqn_atari_torch and dqn_atari_tf, the optimize_memory_usage option is set to false by default. It should be set to true.
2. After I changed the code to set it to true, I noticed that the performance was abnormal.

To Reproduce

Relevant log output / Error message

No response

System Info

No response

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• I have provided a minimal working example to reproduce the bug
• I've used the markdown code blocks for both code and stack traces.

Additional context

if self.size != self.buffer_size:
    idxs = np.random.choice(self.size, size=batch_size, replace=True)

➡️

if self.size != self.buffer_size:
    idxs = np.random.choice(self.size - 1, size=batch_size, replace=True)

With that change, we might be able to fix the problem and wait for testing.

[Feature Request] arguments parser from python-fire to tyro

Description

Deprecated python-fire and use tyro.
Makes arguments management clearer.

Suggested Solution

...

Alternatives

No response

Additional Context

No response

Checklist

• I have checked that there is no similar issue in the repo

Prepare for v0.1.0

• Support wandb tag in script #10
• Issues & Pull requests Template #13
• Licence Ref #15
• Wrapper overhaul #16
• Add mypy type-hint check in pre-commit #17
• Scripts and examples overhaul #20
• DQN in Atari env #25
• Benchmark #33
• tutorial of run and modify alg #52

[Feature Request] Bump tyro to v0.5 & Update bool config docs

Description

Good PR ➡️ brentyi/tyro#50

We need to follow up on this and unify the way the bool flag is set.

Suggested Solution

1. Bump tyro to v0.5+
2. Update docs and benchmark.py

Alternatives

No response

Additional Context

No response

Checklist

• I have checked that there is no similar issue in the repo

[Bug]: capture_video fault in mujoco env with gymnasium0.27.0

Description

Left video on gymnasium0.27.0, Right video on gymnasium0.26.3

To Reproduce

python abcdrl/ddpg_torch.py --track True --capture-video True

Relevant log output / Error message

No response

System Info

No response

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• I have provided a minimal working example to reproduce the bug
• I've used the markdown code blocks for both code and stack traces.

Additional context

No response

[Bug]: Didn't stop, When enable capture_video

Description

The training stops while capturing videos and the experiment gets stuck.

This is because the video_recorder of Gymnasium is not close. Maybe this is a bug in the gym, so we can consider closing it manually for now.

To Reproduce

Exec this cmd:

python abcdrl/dqn.py  --num-envs 2  --learning-starts 8 --buffer-size 10 --batch-size 4 --total-timesteps 30  --capture-video True

output:

{'log_type': 'collect', 'sample_step': 24, 'logs': {'episodic_length': 12, 'episodic_return': 12.0}}
Moviepy - Building video /Users/zhaoyanxiao/Dev/abcdrl/videos/CartPole-v1__dqn/rl-video-episode-0.mp4.
Moviepy - Writing video /Users/zhaoyanxiao/Dev/abcdrl/videos/CartPole-v1__dqn/rl-video-episode-0.mp4

Moviepy - Done !
Moviepy - video ready /Users/zhaoyanxiao/Dev/abcdrl/videos/CartPole-v1__dqn/rl-video-episode-0.mp4
{'log_type': 'collect', 'sample_step': 26, 'logs': {'episodic_length': 13, 'episodic_return': 13.0}}
# Didn't stop

Relevant log output / Error message

No response

System Info

No response

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• I have provided a minimal working example to reproduce the bug
• I've used the markdown code blocks for both code and stack traces.

Additional context

No response