linecode/droidlet

A modular embodied agent architecture and platform for building embodied agents

This repository, corresponding tutorials and docs are still being refined (and not ready yet).

droidlet helps you rapidly build agents (real or virtual) that perform a wide variety of tasks specified by humans. The agents can use natural language, memory and humans in the loop.

droidlet is an early research project for AI researchers to explore ideas around grounded dialogue, interactive learning and human-computer interfaces.

droidlet is in active development and is fairly unstable in design, API, performance and correctness. It is not meant for any production use.

Reach out to us at [email protected], to discuss your use case or just share your thoughts!

Getting Started

You want to do one of three things:

1. Robots: Reproduce and extend the PyRobot based agent on real robots such as LocoBot or photo-realistic simulators such as AIHabitat.
2. Minecraft: Reproduced and extend the minecraft based game agent
3. New Agent: write your own agent from scratch, starting from our base_agent abstraction

Robots	Minecraft	New Agent
1. Clone the source code git clone --recursive https://github.com/facebookresearch/droidlet.git cd droidlet
2. Check system requirements - Linux - Python 3 (Anaconda recommended) - NVIDIA GPU (8GB+) - PyRobot-compatible robot or sim - Habitat-sim instructions below	- Linux - Python 3 (Anaconda recommended) - NVIDIA GPU (4GB+) - Minecraft - more instructions below	- Linux - Python 3 (Anaconda recommended)
3. Install dependencies conda create -n droidlet_env python=3.7 \ pytorch==1.7.1 torchvision==0.8.2 \ cudatoolkit=11.0 -c pytorch conda activate droidlet_env pip install -r \ agents/locobot/requirements.txt python setup.py develop	pip install -r \ agents/craftassist/requirements.txt	pip install -r requirements.txt
4. Instructions for running the Locobot agent	Instructions for running the Craftassist agent
5. Tutorials, runnable in Google Colab (more coming soon). The tutorials introduce the `base_agent` architecture and take you through the 4 components of an Agent
6. API Documentation
7. Agent-specific API Documentation
Locobot agent API	CraftAssist agent API	Not Applicable

Documentation, Tutorials and Papers

Two papers cover the design of droidlet:

1. droidlet: modular, heterogenous, multi-modal agents covers the overall design of droidlet as an embodied AI platform that is extensible to physical robots and simulators.
2. CraftAssist: A Framework for Dialogue-enabled Interactive Agents covers the design of the dialogue parser and the task system of an earlier version of droidlet that is specific to the game Minecraft

Citation

If you use droidlet in your work, please cite our arXiv paper:

@misc{pratik2021droidlet,
      title={droidlet: modular, heterogenous, multi-modal agents}, 
      author={Anurag Pratik and Soumith Chintala and Kavya Srinet and Dhiraj Gandhi and Rebecca Qian and Yuxuan Sun and Ryan Drew and Sara Elkafrawy and Anoushka Tiwari and Tucker Hart and Mary Williamson and Abhinav Gupta and Arthur Szlam},
      year={2021},
      eprint={2101.10384},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

License

droidlet is MIT licensed.

Other Links

Datasets

Download links to the datasets described in section 6 of Technical Whitepaper are provided here:

• The house dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/house_data.tar.gz
• The segmentation dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/instance_segmentation_data.tar.gz
• The dialogue dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/dialogue_data.tar.gz

In the root of each tarball is a README that details the file structure contained within.