singnet/rocca

Rational OpenCog Controlled Agent (ROCCA). Use OpenCog to control a rational agent in OpenAI Gym and Malmo environments.

Rational OpenCog Controlled Agent

Description

Rational OpenCog Controlled Agent, or ROCCA, is a project aiming at creating an opencog agent that acts rationally in OpenAI Gym environments (including Minecraft via minerl).

At its core it relies on PLN (Probabilistic Logic Networks) for both learning and planning. In practice most of the learning is however handled by the pattern miner, which can be seen as a specialized form of PLN reasoning. Planning, the discovery of cognitive schematics, is handled by PLN and its temporal reasoning rule base. Decision is currently a hardwired module, heavily inspired by OpenPsi with a more rational sampling procedure (Thompson Sampling for better exploitation vs exploration tradeoff).

Status

For now learning is only able to abstract temporal patterns based on directly observable events. That is the agent is able to notice that particular action sequences in certain contexts tend to be followed by rewards, however it is not, as of right now, able to reason about action sequences that it has never observed. This requires Temporal Deduction, currently under development.

Once Temporal Deduction is complete we still have a lot of things to add such as

1. More sophisticated temporal and then spatial inference rules.
2. ECAN, for Attention Allocation, to dynamically restrict the atomspace to subsets of items to process/pay-attention-to.
3. Record attention spreading to learn/improve Hebbian links.
4. Concept creation and schematization (crystallized attention allocation).
5. Record internal processes, not just attention spreading, as percepta to enable deeper forms of instrospective reasoning.
6. Plan internal actions, not just external, to enable self-growth.

Requirements

OpenCog tools

• cogutil
• atomspace
• ure
• spacetime
• pln
• miner
• [optional] cogserver
• [optional] attention
• [optional] opencog

Third party tools

• Python 3
• python-orderedmultidict https://pypi.org/project/orderedmultidict/
• fastcore https://fastcore.fast.ai
• OpenAI Gym https://gym.openai.com/
• MineRL https://minerl.io
• nbdev https://nbdev.fast.ai

Install

In the root folder enter the following command:

pip install -e .

For the tools used for development:

pip install -r requirements-dev.txt

How to use

A gym agent defined under the rocca/agents folder is provided that can used to implement agents for given environments. See the examples under the examples folder.

There are Jupyter notebooks provided for experimentation as well.

Develop

If you write code in notebooks that is exported (has the #export comment on top of the cell), remember to invoke nbdev_build_lib to update the library. Remember to use black for formatting, you can invoke black . from the project root to format everything.

You can also use the Makefile for your convenience, invoking make rocca will do both of the above in sequence.

Development container

The .devcontainer folder has configuration for VS Code devcontainer functionality. You can use it to setup a development environment very quickly and regardless of the OS you use.

Tests

Static type checking

Using type annotations is highly encouraged. One can type check the entire Python ROCCA code by calling

tests/mypy.sh

from the root folder.

To only type check some subfolder, you may call mypy.sh from that subfolder. For instance to type check the examples subfolder

cd examples
../tests/mypy.sh

Unit tests

Simply run pytest in the root folder.

References

There is no ROCCA paper per se yet. In the meantime here is a list of related references

• An Inferential Approach to Mining Surprising Patterns in Hypergraphs, Nil Geisweiller et Ben Goertzel
• Partial Operator Induction with Beta Distributions, Nil Geisweiller
• Thompson Sampling is Asymptotically Optimal in General Environments, Jan Leike et al
• Draft about temporal reasoning in PLN, Nil Geisweiller
• References about OpenCog including PLN

rocca's Issues

Add translation layer for Pong-v0

It would be cool to add an RBG->Atomese translation layer (once the opencog-gym API is finalized) of https://gym.openai.com/envs/Pong-v0/. It shouldn't be too difficult (just recognize the relevant objects by their colors) and we can try to reproduce OpenNars procedural reasoning on Pong (see https://github.com/opennars/OpenNARS-for-Applications, ./NAR pong, etc).

Implement shell env

We want a simple env where perceptions are directly provided by the user (via the python shell I suppose) and the actions provided by the agent (or possibly intercepted by the user as well).

It seems the environment would inherit from GymWrapper or such, and the agent would inherit from OpencogAgent or such.

You may see https://github.com/singnet/rocca/blob/master/examples/chase.py as an example of env and agent. Alternatively that could probably be done as a jupyter notebook, which I know less about. See https://github.com/singnet/rocca/blob/master/01_minerl_hello_world.ipynb and https://github.com/singnet/rocca/blob/master/02_minerl_navigate_agent.ipynb jupiter notebook examples.

Implement toy gym env for testing planning

Description

We want to be able to test opencog-gym on the most trivial environments, even though Gym provides so many environments I couldn't find one as simple as the one described below:

• Name: Chase-v0
• Description: board with 2 squares, Left and Right. At each step, if no food is present on the board, a food pellet appears on the opposite square of the one containing previous pellet, initially starting on the Left square. The agent can be positioned on either the Left or Right square and has 4 actions, go-left and go-right, stay, eat. Upon eating the pellet the agent receives a reward of 1.

While implementing the env, do not convert the observations/actions into atomese, make the env as if it was coming out of the gym project. The steps to convert observations/actions to atomese will be handled separately by the wrapper once available.

Remarks

1. Obviously if such env (or similar) already exists among the thousands of third party environments, then no need to implement it, but take into account the fact that we will want to create variations of it as well, to test different and progressively more complex cases.
2. For now I think it's OK if the env lives within that repository, if we start growing the list of env and we think they are worth sharing with the gym community, then we can move them in their own repo.