enlite-ai/maze

Maze Applied Reinforcement Learning Framework

Applied Reinforcement Learning with Python

MazeRL is an application oriented Deep Reinforcement Learning (RL) framework, addressing real-world decision problems. Our vision is to cover the complete development life cycle of RL applications ranging from simulation engineering up to agent development, training and deployment.

This is a preliminary, non-stable release of Maze. It is not yet complete and not all of our interfaces have settled yet. Hence, there might be some breaking changes on our way towards the first stable release.

Spotlight Features

Below we list a few selected Maze features.

• Design and visualize your policy and value networks with the Perception Module. It is based on PyTorch and provides a large variety of neural network building blocks and model styles. Quickly compose powerful representation learners from building blocks such as: dense, convolution, graph convolution and attention, recurrent architectures, action- and observation masking, self-attention etc.
• Create the conditions for efficient RL training without writing boiler plate code, e.g. by supporting best practices like pre-processing and normalizing your observations.
• Maze supports advanced environment structures reflecting the requirements of real-world industrial decision problems such as multi-step and multi-agent scenarios. You can of course work with existing Gym-compatible environments.
• Use the provided Maze trainers (A2C, PPO, Impala, SAC, Evolution Strategies), which are supporting dictionary action and observation spaces as well as multi-step (auto-regressive policies) training. Or stick to your favorite tools and trainers by combining Maze with other RL frameworks.
• Out of the box support for advanced training workflows such as imitation learning from teacher policies and policy fine-tuning.
• Keep even complex application and experiment configuration manageable with the Hydra Config System.

Get Started

• Make sure PyTorch is installed and then get the latest released version of Maze as follows:

  pip install -U maze-rl
  

  Read more about other options like the installation of the latest development version.

  ⚡ We encourage you to start with Python 3.7, as many popular environments like Atari or Box2D can not easily be installed in newer Python environments. Maze itself supports newer Python versions, but for Python 3.9 you might have to install additional binary dependencies manually

• Alternatively you can work with Maze in a container with pre-installed Jupyter lab: Run docker run -p 8888:8888 enliteai/maze:playground and open localhost:8888 in your browser. This loads Jupyter

• To see Maze in action, check out a first example.

• Try your own Gym env or visit our Maze step-by-step tutorial.

Installation

First Example

Step by Step Tutorial

Documentation

• Clone this project template repo to start your own Maze project.

Learn more about Maze

The documentation is the starting point to learn more about the underlying concepts, but most importantly also provides code snippets and minimum working examples to get you started quickly.

• The Workflow section guides you through typical tasks in a RL project

  • Training
  • Rollouts
  • Collection and Visualizing Rollouts
  • Imitation Learning and Fine-Tuning

• Policy and Value Networks introduces you to the Perception Module, how to customize action spaces and the underlying action probability distributions and two styles of policy and value networks construction:

  • Template models are composed directly from an environment's observation and action space, allowing you to train with suitable agent networks on a new environment within minutes.

  • Custom models gives you the full flexibility of application specific models, either with the provided Maze building blocks or directly with PyTorch.

• Learn more about core concepts and structures such as the Maze environment hierarchy, the Maze event system providing a convenient way to collect statistics and KPIs, enable flexible reward formulation and supporting offline analysis.

• Structured Environments and Action Masking introduces you to a general concept, which can greatly improve the performance of the trained agents in practical RL problems.

License

Maze is freely available for research and non-commercial use. A commercial license is available, if interested please contact us on our company website or write us an email.

We believe in Open Source principles and aim at transitioning Maze to a commercial Open Source project, releasing larger parts of the framework under a permissive license in the near future.

maze's Issues

Configuration problems in the step-by-step tutorial

I've just been trying out maze and tried out the step-by-step tutorial.

In Step 5 (5. Training the MazeEnv) the instructions are incomplete or wrong.

I was able to get it running in the end, but it took (us) quite some time. I'm not sure if this is a bug in maze or hydra, of if just some newer version of either library changes the behavior a little bit. But you should update the documentation such that it works out of the box for new users of the library.

---

The setup (under Ubuntu 2020.04):

>> mkdir maze5 && cd maze5
>> pyenv local 3.8.8
>> python -m venv .venv
>> source .venv/bin/activate
>> pip install maze-rl torch
>> pip list
Package                 Version
----------------------- -----------
hydra-core              1.1.0
hydra-nevergrad-sweeper 1.1.5
maze-rl                 0.1.7
torch                   1.9.0
...

Then just copy-pasted the files from the https://github.com/enlite-ai/maze-examples/tree/main/tutorial_maze_env/part03_maze_env repo and adjusted the _target paths in the config yamls (e.g. from _target_: tutorial_maze_env.part03_maze_env.env.maze_env.maze_env_factory to _target_: env.maze_env.maze_env_factory).

Problem 1:

When you run the suggested training command, Hydra will just complain that it can't find the configuration files.

>> maze-run -cn conf_train env=tutorial_cutting_2d_basic wrappers=tutorial_cutting_2d_basic \
    model=tutorial_cutting_2d_basic algorithm=ppo
In 'conf_train': Could not find 'model/tutorial_cutting_2d_basic'

Available options in 'model':
        flatten_concat
        flatten_concat_shared_embedding
        pixel_obs
        pixel_obs_rnn
        rllib
        vector_obs
        vector_obs_rnn
Config search path:
        provider=hydra, path=pkg://hydra.conf
        provider=main, path=pkg://maze.conf
        provider=schema, path=structured://

Fix:

You can just define the config directory for hydra with maze-run -cd conf -cn conf_train .... Then Hydra will find the 3 config files and load them correctly.

Problem 2:

After loading the config files, hydra tries to load the modules defined in the _target fields.
And that fails immediatly with:

  ...
  File "***/maze5-uWAZh5bh/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 104, in _resolve_target
    return _locate(target)
  File "***/maze5-uWAZh5bh/lib/python3.8/site-packages/hydra/_internal/utils.py", line 563, in _locate
    raise ImportError(f"Error loading module '{path}'") from e

ImportError: Error loading module 'env.maze_env.maze_env_factory'

Fix:

For some reason Hydra doesn't know the path to the directory from where we call maze-run.
And therefore it doesn't find the env directory containing the maze_env file.

This is fixable by just setting the environment variable: export PYTHONPATH="$PYTHONPATH:$PWD/".

Hello from Hydra :)

Thanks for using Hydra!
I see that you are using Hydra 1.1 already which is great. One thing that is really recent is the ability to configure the config searchpath from the primary config.
You can learn about it here.

This can probably eliminate the need of your users to even know what a ConfigSearchpathPlugin is.

Feel free to jump into the Hydra chat if you have any questions.