zhengliz/anl

Train an RL agent to localize actively (PyTorch)

Active Neural Localization

This is a PyTorch implementation of the ICLR-18 paper:

Active Neural Localization
Devendra Singh Chaplot, Emilio Parisotto, Ruslan Salakhutdinov
Carnegie Mellon University

Project Website: https://devendrachaplot.github.io/projects/Neural-Localization

This repository contains:

• Code for the Maze2D Environment which generates random 2D mazes for active localization.
• Code for training an Active Neural Localization agent in Maze2D Environment using A3C.

Dependencies

• PyTorch (v0.3)

Usage

Training

For training an Active Neural Localization A3C agent with 16 threads on 7x7 mazes with maximum episode length 30:

python a3c_main.py --num-processes 16 --map-size 7 --max-episode-length 30 --dump-location ./saved/ --test-data ./test_data/m7_n1000.npy

The code will save the best model at ./saved/model_best and the training log at ./saved/train.log. The code uses ./test_data/m7_n1000.npy as the test data and makes sure that any maze in the test data is not used while training.

Evaluation

After training, the model can be evaluated using:

python a3c_main.py --num-processes 0 --evaluate 1 --map-size 7 --max-episode-length 30 --load ./saved/model_best --test-data ./test_data/m7_n1000.npy

Pre-trained models

The pretrained_models directory contains pre-trained models for map-size 7 (max-episode-length 15 and 30), map-size 15 (max-episode-length 20 and 40) and map-size 21 (max-episode-length 30 and 60). The test data used for training these models is provided in the test_data directory.

For evaluating a pre-trained model on maze size 15x15 with maximum episode length 40:

python a3c_main.py --num-processes 0 --evaluate 1 --map-size 15 --max-episode-length 40 --load ./pretrained_models/m15_l40 --test-data ./test_data/m15_n1000.npy

Rollout trajectory with pre-trained model

python generate_trajectory.py --num-processes 0 --evaluate 1 --map-size 7 --max-episode-length 30 --load ./pretrained_models/m7_l30 --test-data ./test_data/m7_n1000.npy

Perturb trajectory and rollout

python perturb_trajectory.py --num-processes 0 --evaluate 1 --map-size 7 --max-episode-length 30 --load ./pretrained_models/m7_l30 --test-data ./test_data/m7_n1000.npy

Generating test data

The repository contains test data of map-sizes 7, 15 and 21 with 1000 mazes each in the test_data directory.

For generating more test data:

python generate_test_data.py --map-size 7 --num-mazes 100 --test-data-location ./test_data/ --test-data-filename my_new_test_data.npy

This will generate a test data file at test_data/my_new_test_data.npy containing 100 7x7 mazes.

All arguments

All arguments for a3c_main.py:

  -h, --help            show this help message and exit
  -l L, --max-episode-length L
                        maximum length of an episode (default: 30)
  -m MAP_SIZE, --map-size MAP_SIZE
                        m: Size of the maze m x m (default: 7), must be an odd
                        natural number
  --lr LR               learning rate (default: 0.001)
  --num-iters NS        number of training iterations per training thread
                        (default: 10000000)
  --gamma G             discount factor for rewards (default: 0.99)
  --tau T               parameter for GAE (default: 1.00)
  --seed S              random seed (default: 1)
  -n N, --num-processes N
                        how many training processes to use (default: 8)
  --num-steps NS        number of forward steps in A3C (default: 20)
  --hist-size HIST_SIZE
                        action history size (default: 5)
  --load LOAD           model path to load, 0 to not reload (default: 0)
  -e EVALUATE, --evaluate EVALUATE
                        0:Train, 1:Evaluate on test data (default: 0)
  -d DUMP_LOCATION, --dump-location DUMP_LOCATION
                        path to dump models and log (default: ./saved/)
  -td TEST_DATA, --test-data TEST_DATA
                        Test data filepath (default: ./test_data/m7_n1000.npy)

Cite as

Chaplot, Devendra Singh, Parisotto, Emilio and Salakhutdinov, Ruslan. Active Neural Localization. In International Conference on Learning Representations, 2018. (PDF)

Bibtex:

@inproceedings{chaplot2018active,
  title={Active Neural Localization},
  author={Chaplot, Devendra Singh and Parisotto, Emilio and Salakhutdinov, Ruslan},
  booktitle={International Conference on Learning Representations},
  year={2018}
}

Acknowledgements

The implementation of A3C is borrowed from https://github.com/ikostrikov/pytorch-a3c.