myhololens/craftassist

A virtual assistant bot in Minecraft

The goal of this project is to build an intelligent, collaborative assistant bot in the game of Minecraft¹ that can perform a wide variety of tasks specified by human players. Its primary purpose is to be a tool for artifical intelligence researchers interested in grounded dialogue and interactive learning. This project is in active development.

A detailed outline and documentation is available in this paper

This release is motivated by a long-term research agenda described here.

Installation & Getting Started

Do this section before cloning the repo.

Dependencies

Make sure the following packages have already been installed before moving on:

• CMake
• Python3
• Glog
• Boost
• Eigen
• gcc version: 7.4.0 on ubuntu 18.04
• For Mac users:
  • LLVM version < 10 to successfully use clang. Working with multiple versions of Xcode.

Python Requirements: Using A Conda Environment

To build a conda environment that supports this release:

# Create a new env and install pip
conda create -n minecraft_env
conda activate minecraft_env
conda install pip

# The requirements.txt contains some reqs that need to be installed with conda 
# and others that need to be installed with pip.  This switches between these 
# two installation methods
while read requirement;
    do conda install --yes $requirement || pip install $requirement;
done < requirements.txt
pip install ipdb
conda install pytorch torchvision -c pytorch

Then activate this environment whenever you want to run the agent.

Install git-lfs

# OSX
brew install git-lfs
git lfs install

# On Ubuntu
sudo add-apt-repository ppa:git-core/ppa
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install

Checking out the repo

Use this command, or your submodules will not be pulled, and your clone will take a very long time:

git lfs clone --recursive [email protected]:facebookresearch/craftassist.git

Now cd craftassist before proceeding to the following sections.

Building client and server

To build Cuberite and the C++ Minecraft client:

make

Run the Cuberite instance

Run the following command

python ./python/cuberite_process.py

to start an instance of cuberite instance listening on localhost:25565

Connecting your Minecraft game client (so you can see what's happening)

Buy and download the official Minecraft client.

You can inspect the world and view the Minecraft agent's actions by logging into the running Cuberite instance from the game client.

To connect the client to the running Cuberite instance, click in the Minecraft client:

Multiplayer > Direct Connect > localhost:25565

Error: Unsupported Protocol Version

Minecraft has recently release v1.13, and our Cuberite system supports at most v1.12

Please follow these instructions to add a 1.12.x profile and use it to connect.

Running the interactive V0 agent

Assuming you have set up the Cuberite server and the client, in a separate tab, run:

python ./python/craftassist/craftassist_agent.py

You should see a new bot player join the game.

Chat with the bot by pressing t to open the dialogue box, and Enter to submit.

Use the w, a, s, and d keys to navigate, left and right mouse clicks to destroy and place blocks, and e to open your inventory and select blocks to place.

Running tests

./python/craftassist/test.sh

Datasets

Download links to the datasets described in section 6 of Technical Whitepaper are provided here:

• The house dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/house_data.tar.gz
• The segmentation dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/instance_segmentation_data.tar.gz
• The dialogue dataset: https://craftassist.s3-us-west-2.amazonaws.com/pubr/dialogue_data.tar.gz

In the root of each tarball is a README that details the file structure contained within.

Citation

If you would like to cite this repository in your research, please cite the CraftAssist paper.

@misc{gray2019craftassist,
    title={CraftAssist: A Framework for Dialogue-enabled Interactive Agents},
    author={Jonathan Gray and Kavya Srinet and Yacine Jernite and Haonan Yu and Zhuoyuan Chen and Demi Guo and Siddharth Goyal and C. Lawrence Zitnick and Arthur Szlam},
    year={2019},
    eprint={1907.08584},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}

License

CraftAssist is MIT licensed.

¹ Minecraft features: © Mojang Synergies AB included courtesy of Mojang AB