A platform for letting researchers connect an intelligent AI directly to realtime communication networks and 3D worlds.
In partnership with SuperReality, a free open source teaching platform for everyone.
Interactive Endpoint Connector tools for Browsers, JS, TypeScript
- Discord
- XREngine - Bot Package
- XR Engine API
Connector tools for Python 3, Machine Learing tools such as TensorFlow, PyTorch, etc.
Interaction API goals: Listen, Speak, Interact, Move, Emote
- XR Engine Testing Bot - Sequence of commands
- Echo Bot - Echo what's said in a room by everyone, bots, people, 1 person
- Rasa - Chat bot mainly used for managing models.
-
- dialogpt.small
- dialogpt.medium
- dialogpt.large
-
- gptneo.small
- gptneo.large
- gptneo.xlarge
Under the hood you'll find an instance of Chrome (using Puppeteer) which can be run in headless or GUI mode. The bot uses control surfaces from the user API to interact, and also has some extra access to world state.
DigitalBeing requires the following to run:
- Install python dependencies in requirements.txt
pip install -r requirements.txt
- Download and link spacy model for rasa. Spacy is used for pre-processing of the utterances, tokenization, and featurization.
python3 -m spacy download en_core_web_md python3 -m spacy link en_core_web_md en
- Install node.js dependencies
npm install
- Create a local .env file with configuration variable for example.
DISCORD_API_TOKEN=<Your Discord Bot API Token>
This will create a discord bot client which will listen for incomming messages when the bot gets mentioned in your server chat. - Use the parameters file for the agent inside
server/agent_params.py
here you can select which agents to launch along with some other flags. you only need to edit the following parameter to launch the specific agents. Right now not all agents are working but the following list of agents have been tested and are working.SELECTED_AGENTS = [ 'dialogpt.small', 'dialogpt.medium', 'dialogpt.large', 'gptneo.small', 'gptneo.large', 'gptneo.xlarge' ]
- Run the host bot framework
npm run start-gui
- Run the bot with
npm start
-- by default it will connect to a test room on our dev server and open a port on gRPC - Run example.py to connect the default hello world implementation
- Copy and paste the code from example.py into your project and hook your models into it.
- If you use WSL in Windows, you might get an error about Chromium, file or directory not found.
- To fix this install Chromium by using
- wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- sudo apt -y install ./google-chrome-stable_current_amd64.deb
You can run it using docker, if you don't have node installed or need to test.
# Build the image
docker build digital_being .
# Run the image
docker run -d digital_being