aidan-curtis/firehose

We are trying to put out big fires

firehose

We are trying to put out big fires with Deep Reinforcement Learning.

Check out our project website for live demos of our learned agents in action: https://williamshen-nz.github.io/firehose/

Check out our project paper here: https://williamshen-nz.github.io/firehose/firehose-paper.pdf

Firehose is an open-source deep reinforcement learning (DRL) framework for training and evaluating wildfire management agents in realistic environments. Firehose allows researchers to easily train and evaluate a variety of RL agents on diverse environments, extend our state/action spaces and reward functions, and design hand-crafted baselines. Firehose is driven in the backend by Cell2Fire, a state-of-the-art wildfire simulator. We have only kept the essentially Cell2Fire code in this repo to make it less cluttered.

You can find the majority of the code for Firehose in the cell2fire/firehose module, cell2fire/evaluate_model.py script and cell2fire/rl_experiment_vectorized.py script. The .ipynb notebooks were used to check results, generate videos, etc.

Example RL Agent

• Left = Min-L2 Baseline
• Middle = Max-L2 Baseline
• Right = RL Agent

Installation

Use a virtual environment it'll make your life easier

1. Download Eigen and store it somewhere you like if you don't already have it: http://eigen.tuxfamily.org/index.php?title=Main_Page#Download
   • If you already have it on your machine just locate it and note down the path.
2. Compile and setup cell2fire
   1. cd Cell2Fire/cell2fire/Cell2FireC
   2. Edit Makefile to have the correct path to Eigen
   3. make
   4. cd ../ && pip install -r requirements.txt
   5. cd ../ && pip install -r requirements.txt (yes repeat it twice to go up directory)
3. Yay! Follow instructions below to run the environment and train an agent.

Usage

Some pretrained models are provided in the pretrained_models/ directory. Use the evaluate_model.py script to evaluate them.

Run the gym env

python cell2fire/gym_env.py

Evaluate the naive policy

This writes a video to a videos/ folder that will be created

python cell2fire/evaluate_model.py --algo naive

Train RL agents parallelized

Look at the script for the CLI args or run it with the --help flag

python cell2fire/rl_experiment_vectorized.py

Miscellaneous

Supercloud Installation

TLDR: just run the commands below once you have ssh'ed into Supercloud

cd ~
git clone https://github.com/aidan-curtis/firehose.git
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip
cd eigen-3.4.0/ && cmake .. -DCMAKE_INSTALL_PREFIX=/home/gridsan/${USER}/eigen

cd ~/firehose/cell2fire/Cell2FireC
make -f Makefile_supercloud

After this, you can drop into an interactive shell LLsub -i, load the conda env module load anaconda/2022a, and install the required dependencies with pip install -r requirements.txt

Command to check it is all working: python cell2fire/evaluate_model.py --disable-video --disable-render

Text Installation Instructions:

1. ssh into Supercloud and cd ~/ into your home directory if not already there
2. Clone the firehose repo, use https if you don't have ssh keys setup otherwise Github complains
   • git clone https://github.com/aidan-curtis/firehose.git
3. Download Eigen, unpack and build. Get the latest URL from here: https://eigen.tuxfamily.org/index.php?title=Main_Page
   1. wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
   2. unzip eigen-3.4.0.zip
   3. cd eigen-3.4.0/ && cmake .. -DCMAKE_INSTALL_PREFIX=/home/gridsan/${USER}/eigen
4. Modify the Makefile to have the correct path to Eigen
   • EIGENDIR = /home/gridsan/${USER}/eigen/include/eigen3/
5. Compile cell2fire: cd ~/firehose/cell2fire/Cell2FireC && make -f Makefile_UBUNTU
6. To run the experients, run sbatch --array=1-10 deploy_experiments.sh to run a batch of 10 experiments

Videos

The cell2fire/videos/video_editor.ipynb notebook has examples on using movie-py to edit videos instead of using ffmpeg through the command line.

Stack 2 videos side by side

ffmpeg -i left.mp4 -i right.mp4 -filter_complex hstack=inputs=2 merged-2.mp4

Stack 3 videos side by side

ffmpeg -i left.mp4 -i middle.mp4 -i right.mp4 -filter_complex "[0:v][1:v][2:v]hstack=inputs=3[v]" -map "[v]" merged-3.mp4

firehose's Issues

Evaluation Improvements

https://github.com/aidan-curtis/firehose/blob/main/cell2fire/evaluate_model.py

• Right now the evaluation script doesn't support random ignition points. Support random ignition points
• Support all the args that train script does.
• Support multiple eval iterations and collecting data into JSON or whatever.
• Jupyter notebook for plotting and making life easy
• Update gym env to output data we want

Action space

Task 1: Start taking action after initial ignition (random ignition/fixed ignition points)
Task 2: Can only take action before ignition (random ignition/fixed ignition points)
Task 3: Take actions before and after ignition ()

Actions:

• Change the type of vegetation (Certain brush size)
• Water (Certain brush size)

Option 1:
Network architecture: CNN --> (image -> discrete action space)
N teams - Start at fixed positions in a map
Choose to either to change vegetation/spray water/moveto

Option 2:
Network architecture: FCN (image-->image)
Action is an image that specifies fire and vegetation, but also

Requirement.txt lacking versions

Good morning,

I'm trying to use firehose but I've got problem with env/libraries compatibility.
My problem starts when I'm typing:
python cell2fire/evaluate_model.py --algo naive

I need stable-baseline3 1.5.0 max for register_policy to exist.
I also need gym 0.21 for video compatibility but I can't install gym 0.21 unless I've got the right versions for all previously libraries.

Can you share the versions you use for:
numpy pandas matplotlib seaborn opencv-python imread tqdm deap gym stable-baselines3 tensorboard ?

Thanks in advance !

Random Ignition not Random Across Processes

In vectorized env. Is it because of the lambda we got. If they are different instances of FireEnv then they should be random.

I guess random rng stuff might be going on. This isn't a big problem anyways but we should fix

Presentation Tick Off

• The main idea of the presentation is to communicate:
  • Why are you solving what you are solving
    • No need of literature review?
    • Why should we care about this problem?
    • Good rule of thumb is not to have the motivation be more than 1/5 or 1/6 of the entire presentation unless absolutely necessary.
  • Description of the method
    • Clearly outline the observation space, the action space, the reward structure, learning algorithm.
    • If your environment is visual — we encourage you to include snapshots of how the environment looks like.
  • What is the baseline method that you are comparing against
  • What results you got / why did the method you tried failed / other issues you ran into
    • What changes you made and how did it change the results
  • What did you learn from the course project?
  • In 1-2 sentences describe if we should expect major differences between the presentation and the final report.
    • No need to include future work that you will be not be doing in the course project.
• Try to use more images / plots and less text.
• Try to animate slides wherever possible — so that all the content doesn’t appear at once.
• Don’t put derivations on the slides — we will go over any derivations from the project report.
• Remember many people will be seeing the presentations for the first time.
  • Prioritize explaining main experiments / results well instead of explaining everything quickly.
  • The presentation doesn't need to exhaustively go into everything you tried — the report is the right place for it.

Segfault

Might help us debug other stuff

Program received signal SIGSEGV, Segmentation fault.
0x000055555555ca23 in Cell2Fire::outputGrid (this=0x7fffffffd060) at /usr/include/c++/9/bits/stl_vector.h:1040
1040          operator[](size_type __n) _GLIBCXX_NOEXCEPT
(gdb) bt
#0  0x000055555555ca23 in Cell2Fire::outputGrid (this=0x7fffffffd060) at /usr/include/c++/9/bits/stl_vector.h:1040
#1  0x000055555555e545 in Cell2Fire::Results (this=0x7fffffffd060) at Cell2Fire.cpp:1118
#2  0x0000555555564620 in Cell2Fire::Step (this=0x7fffffffd060, generator=..., rnumber=<optimized out>, rnumber2=<optimized out>, rnumber3=<optimized out>) at Cell2Fire.cpp:1341
#3  0x00005555555676e5 in main._omp_fn.0(void) () at Cell2Fire.cpp:1527
#4  0x00007ffff7c4d8e6 in GOMP_parallel () from /lib/x86_64-linux-gnu/libgomp.so.1
#5  0x000055555555b25a in main (argc=<optimized out>, argv=<optimized out>) at Cell2Fire.cpp:1419

At commit 433c86c

Set this line to true. Ran, added any input action. On 2nd go will segfault.
https://github.com/aidan-curtis/firehose/blob/main/cell2fire/Cell2FireC/Cell2Fire.cpp#L1509

On Ubuntu. Used gdb: https://stackoverflow.com/questions/3718998/fixing-segmentation-faults-in-c

Fix `pre_run_steps` and `num_steps_after_action`

Fix the indexing. Probably use recursion rather than looping.

Call stack will be num_steps_after_action deep so it won't cause any problems unless we have infinite recursion

Logits don't satisfy Simplex

Warning! We assume masking is supported for VecEnv
----------------------------------------
| rollout/                |            |
|    ep_len_mean          | 70         |
|    ep_rew_mean          | -23.7      |
| time/                   |            |
|    fps                  | 407        |
|    iterations           | 102        |
|    time_elapsed         | 8197       |
|    total_timesteps      | 3342336    |
| train/                  |            |
|    approx_kl            | 0.14997472 |
|    clip_fraction        | 0.598      |
|    clip_range           | 0.2        |
|    entropy_loss         | -3.58      |
|    explained_variance   | 0.975      |
|    learning_rate        | 0.0003     |
|    loss                 | -0.0384    |
|    n_updates            | 1010       |
|    policy_gradient_loss | -0.0328    |
|    value_loss           | 0.287      |
----------------------------------------
Traceback (most recent call last):
  File "cell2fire/rl_experiment_vectorized.py", line 288, in <module>
    train(args=parser.parse_args())
  File "cell2fire/rl_experiment_vectorized.py", line 219, in train
    trainer.train()
  File "cell2fire/rl_experiment_vectorized.py", line 209, in train
    raise e
  File "cell2fire/rl_experiment_vectorized.py", line 204, in train
    model.learn(
  File "/home/gridsan/wshen/.local/lib/python3.8/site-packages/sb3_contrib/ppo_mask/ppo_mask.py", line 569, in learn
    self.train()
  File "/home/gridsan/wshen/.local/lib/python3.8/site-packages/sb3_contrib/ppo_mask/ppo_mask.py", line 429, in train
    values, log_prob, entropy = self.policy.evaluate_actions(
  File "/home/gridsan/wshen/.local/lib/python3.8/site-packages/sb3_contrib/common/maskable/policies.py", line 280, in evaluate_actions
    distribution.apply_masking(action_masks)
  File "/home/gridsan/wshen/.local/lib/python3.8/site-packages/sb3_contrib/common/maskable/distributions.py", line 152, in apply_masking
    self.distribution.apply_masking(masks)
  File "/home/gridsan/wshen/.local/lib/python3.8/site-packages/sb3_contrib/common/maskable/distributions.py", line 62, in apply_masking
    super().__init__(logits=logits)
  File "/state/partition1/llgrid/pkg/anaconda/anaconda3-2022a/lib/python3.8/site-packages/torch/distributions/categorical.py", line 64, in __init__
    super(Categorical, self).__init__(batch_shape, validate_args=validate_args)
  File "/state/partition1/llgrid/pkg/anaconda/anaconda3-2022a/lib/python3.8/site-packages/torch/distributions/distribution.py", line 55, in __init__
    raise ValueError(
ValueError: Expected parameter probs (Tensor of shape (64, 1600)) of distribution MaskableCategorical(probs: torch.Size([64, 1600]), logits: torch.Size([64, 1600])) to satisfy the constraint Simplex(), but found invalid values:
tensor([[2.7397e-07, 2.0007e-07, 6.2320e-07,  ..., 2.7629e-07, 8.0404e-05,
         7.5556e-07],
        [2.4421e-07, 3.3548e-07, 6.6760e-07,  ..., 1.7289e-07, 2.6347e-06,
         4.3772e-07],
        [1.4054e-07, 1.0664e-07, 3.6739e-07,  ..., 1.5730e-07, 3.2728e-06,
         4.0184e-07],
        ...,
        [3.9183e-07, 2.9925e-07, 3.5064e-07,  ..., 2.0548e-07, 2.0982e-07,
         5.6737e-07],
        [6.9835e-06, 4.1832e-06, 1.2083e-05,  ..., 7.9078e-06, 4.3440e-04,
         1.1752e-05],
        [1.5757e-06, 5.5615e-06, 4.7831e-07,  ..., 6.5367e-07, 2.5431e-04,
         5.1062e-07]], device='cuda:0', grad_fn=<SoftmaxBackward0>)
8 7
9
7

https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/master/sb3_contrib/common/maskable/distributions.py#L41-L65

Next steps

• Reward for Naive - naive implemented
  • just need to plot against frequency of updates and how long we wait. (not really for now)
• Same for random (not really useful so won't bother)
• Added support for num steps before simulation and num steps per action in C++ code
• Action costs - e.g. helicopter, how far last action was
• Reward Engineering
• Variable Radius Optimization - added radius 2 actions
• RGB image as input to the model with CNN - **Aidan finished this and will start training

crashing cell2fire process

cell2fire crashes: check inodes and delete files

hypothesis:

cell2fire process finishes stepping but we are still trying to step
=> this causes error and process to die and break