amarok1412/rori_core

modulable open-source chatterbot platform

RORI v4.0.0

RORI is a modulable open-source chatterbot platform. The first version was written in 2011 (2.0 in September 2012). I rewrote it in Rust in 2017, and I'm currently migrating the whole communication to use GNU Ring.

A complete RORI chain needs 4 things:

1. An entry point is a application which get commands from an user and send it to rori_server. For example, a chat where the entry point reads what users says.
2. An endpoint is a application which performs actions requested by RORI. For example, it can execute a shell command or write something in a chat.
3. A module is a script activated when a condition is fulfilled and send actions for endpoints to RORI.
4. The rori_server which get data from entries, call modules, and send data to endpoints.

Why RORI?

I run a lot of chatterbots on multiple services (IRC, Discord, Websites, my computer). Some bots do the exact same thing but run on a different service. The idea is to avoid to rewrite the core of each chatterbot and use the same base. Now, I just have to write an interface to communicate with this core on each services.

This is some examples of what I will do with RORI (as soon as the migration is finished):

• Ask RORI to launch music on the best device (on my computer, or stream on a discord server for example).
• Ask RORI to be alarmed at 7:40.
• Ask RORI to send messages to a friend.
• Ask RORI to shutdown a device.
• Send a picture to RORI and ask to store this pict in the best folder.
• Ask RORI to send me a notification before a rendez-vous.

Acronym

RORI is for RORI On RIng. Where RORI is for Really Obvious Really Intelligent.

How it works

Please, see wiki

• Definitions
• API documentation
• Authentification system and discovery
• Interaction handling (server side)
• Modules
• Custom datatypes handling
• Emotions
• Bridges

Build instructions

make dependencies will install the following packages:

1. ring-daemon: https://ring.cx for the communication
2. cargo: https://crates.io/ for rust
3. sqlite3 for the database
4. libdbus for the communication between the client and ring-daemon
5. libncurses for the UI
6. openssl to generate keys for the API
7. python > 3.6 for modules.
8. pip3 and modules: wikipedia feedparser appdirs

Run your instance

./launch-rori.sh will:

1. Generate keys for the API
2. Generate modules table into the database
3. Then run RORI to generate the config file and the database

config.json looks something like:

{
  "ring_id":"xxxxxxxxxxxxxxxxx",
  "api_listener":"0.0.0.0:1412",
}

Another way is to use docker... this is for now, how I run it:

make docker # build the image
make docker-run

Providing https support

The recommended way is to use nginx + Let's Encrypt in front of the service to provide https support. The app will not support code for that.

Contribute

Please, feel free to contribute to this project in submitting patches, corrections, opening issues, etc.

If you don't know what you can do, you can look the good-first-issue label, or still creates modules.

For more infos and ideas read CONTRIBUTING.md (this file doesn't exists for now) and CODE_OF_CONDUCT.md.

rori_core's Issues

Add contributing notes and process

• Add CONTRIBUTING.md
• Add CODE_OF_CONDUCT.md
• Add templates for issues and templates

Get coverage outputs for tests

This is some tools to get coverage:

1. https://github.com/SimonKagstrom/kcov/
2. https://github.com/xd009642/tarpaulin/

Then get results on https://coveralls.io/

Authentification system and discovery

Definitions

User

A user for RORI is someone who uses the system under an identity. For example, Alice (real person) can uses her phone, her computer linked to RORI under the identity alice@rori (user for RORI) and an IRC client as an anonymous user.

Device

A device for RORI is a ring_id. This device can be linked to a specific user or to the anonymous user.

Multi-devices

A user can uses as many as devices they want. The thing is, RORI can choose to interact with only one device of a user. For example, if Alice takes a picture and asks RORI to store the photo in a specific folder on her computer, we only have to interact with the endpoint on the computer, not all devices.

Name-service

Each RORI embeds its own name-service. Doc is here. It's used to perform the translation between a nickname and a ring_id.

Note: Differences with GNU/Ring

User

A user for RING is a ring_id (in fact, a key pair... but here it's not important)

Device

A device for RING is a daemon using the ring_id of a user.

Multi-devices

If a user has multiple devices, they should be able to get messages and calls on all devices. Not like RORI. That's why one ring_id is sufficient here.

Registration processus

See diagram here

Discovery

Now, we have an user alice, with 2 devices (pc, android). RORI's name server will provides some endpoints. /name/alice will return the ring id of the pc or android. /name/alice_pc will return the id of pc and /name/alice_android the id of android.

Unregister and remove devices

Alice have the ability to revoke a device or to remove her account. /unregister will remove all of its devices and remove /name/alice too. So someone will be able to re-register alice.
/revoke_device will revoke the current device and de-link it from her account (or with a parameter, another device).

Note

The add_device must only be able when recognized as alice. Nobody can add a device to another account.
An attacker can directly modify the database RORI side. This is not solved here. So alice must trust RORI.

Current status:

• RORI automatically accepts requests and add contact from anyone.
• provide a name server
• register accounts
• register device
• revoke device from device
• unregister
• revoke device from another device
• register device from another device
• comments
• Link 2 devices under the same nickname
• mocks daemon
• unit tests database
• unit tests server
• unit tests manager
• unit tests API
• Write wiki page

rori/command specify reply metadata

API: implement /addr/xxxxxx

See https://tuleap.ring.cx/plugins/mediawiki/wiki/ring/index.php?title=Name_server_protocol

RORI needs to implement a reverse lookup to get an username (not a device name) from a ring id.

• Implement it
• Test it
• Document it.

Mutli users on one device

A ring_id should be able to host multiples users, like the discord bridge client. This means a lot of changes to do:

• Allow multiples users in database
• Update API to be able to support multiple users without breaking compability
• Supports author metadatas for bridges when receiving a message
• Update rori/commands to be compatible with changes
• Add a set_bridge order
• Update RORI.py to be compatible with these changes.
• Fix tests and improve coverage
• Update documentation!

Fix coverage for Travis

For now, build is broken on travis (libdbus-1-dev and ubuntu broken) + tarpaulin still broken for rust stable with this project

Use https endpoint for test_api

Replace:

let body = reqwest::get("http://127.0.0.1:1412/name/rori").unwrap()
                  .text().unwrap();

by:

let body = reqwest::get("https://127.0.0.1:1412/name/rori").unwrap()
                  .text().unwrap();

(Take care of the certificate verification)

And remove new_http from api.rs

Improve documentation for https API endpoint

Add a wiki page on the identity generation for Tls.

Tls for the API

For now, the API (src/api.rs) is not available through https. Could be nice to be directly compatible with TLS.

Note: to be fully functional, must wait for https://gerrit-ring.savoirfairelinux.com/#/c/9094/.

API: add get_rori_color for users

Actually, a RORI client looks like this:

Because all official clients should have the same design, the API should give the current wanted colors for RORI to homogeneize the result on all platforms (for example red when using, black when sleeping)

Add Scheduler to schedule modules

Add a scheduler to allow scheduling modules like alarm, news, mail, etc

Description

A Scheduler start a module with the parameters that the module needs. Jobs can be scheduled to run once or to repeat. The input is given by a database. Modules must be able to add/remove/update tasks in the scheduler.

Tasks

• Design API for modules
• Create table in the db (id|module|argument|at|secs|mins|hours|day (string)|repeat (bool)
• Code add/remove/update/etc => to test
• If module crash, scheduler must not panic
• Clean todos
• Clean warnings
• Add tests
• Add documentation

Bad datatype compatible with anonymous account

Environment

latest

Reproduce steps:

Testing the music command with a non music compatible client via an anonymous user

Expected result:

The music should failed.

Actual result:

Instead with got a success message (Yeay music!)

Use the module form for internal modules

/register and othere internal modules should use the rori::module::Module structure.

A WIP is in progress here

Improve Docker usage

Actually make docker-run works, but is not sufficient, this is some problems to fix:

• For now, we need to run separately generate_modules.py and its bad

• Files are removed, should be able to keep ring files, rori.db, config.json, api keys.

• Fedora 27 is still broken due to dbus issues.

Improve /order results

For now, orders like /register gives something like format!("{} is now known as {}_{}", ring_id, username, devicename) in a plain/text message.

Two things:

1. A JSON message should be returned because it's easy to parse (and to document)
2. the payload should have rori/order for type.

• Support custom datatypes in interactions
• Improve /register
• Improve /unregister
• Improve /add_device
• Improve /rm_device
• Improve /link
• Set rori/order datatype
• documentation

Note: the problem is, payloads are rewritten by the daemon... and custom datatypes are not take into account but forced to text/plain...

Devices should be linked to supported types

Description

For now, modules are for text/plain datatypes.

But it's not perfect for a lot of modules. For example, if a client want to support orders for controlling the music on this device for example, we do not want to see some text on the user side.

Implementation

Currently we use a similar systems for rori orders like /register. What RORI will see is:

{
    "text/plain":"msg"
}

The idea is to support custom messages.

For example a client can be able to send messages like:

{
    "music":"informations"
}

and RORI will try to launch a music module. But a module can also craft a message on a custom datatypes. The thing is, all clients will not be able to support all datatypes. So they should announce supported types for better result.

Full example

1. Device announce to RORI that this device supports music datatypes.
2. Device send to RORI:

{
    "text/plain":"play music"
}

3. RORI launch the music module which craft for Device (cause compatible) this message:

{
    "music":"play random"
}

4. Device receives the order and launch a random music on the device.

TODO List

• link devices to datatypes
• Add the possibility to add datatypes
• Add the ability to remove datatypes
• unit tests
• Music example
• Update wiki
• Update README.md

Add Linux Desktop Client

TBD

Replace dbus by the REST API

The interaction with the daemon should use the REST API in the next version

Format feature for get_localized_sentence

This is bad:
self.rori.get_localized_sentence('current_meteo', self.sentences) + city + self.rori.get_localized_sentence('is', self.sentences) + description + ".\n"

Should be able to directly create a formatted string

Move rori_modules folder to https://github.com/AmarOk1412/rori_modules/

rori_modules directory is just a present in this repository for testing. But has to be moved to https://github.com/AmarOk1412/rori_modules/.

UI for `rori_core`

TBD. NOTE: will be a ncurses UI

RORI base emotions

RORI emotions uses the Shaver et al. (2001) system.

Primary emotions

Primary emotions are stored into rori.db and are:

• Love
• Joy
• Surprise
• Anger
• Sadness
• Fear
  And will be under the form of an integer between 0 and 100.

Combinations

These six emotions can be combinated in any modules to form a full spectrum of emotions:
https://en.wikipedia.org/wiki/Contrasting_and_categorization_of_emotions#Parrott's_emotions_by_groups

Current status:

• Add table into rori.db
• Add utils to change emotions in modules
• Add an example
• wiki page
• Update README.md
• Clean PR

Modules specification

Activation

RORI's side

For RORI this is what a Module is:

pub struct Module {
    pub condition: Box<Condition>,
    pub name: String,
    pub path: String,
    pub priority: u64,
    pub enabled: bool,
}

Basically, this structure contains the name of the module, it's priority (see the module activation loop in #5), if the module is enabled and more interesting:

1. the path to the module.
2. a condition which needs to be fulfilled to activate the module.

Condition example

Condition is related to the type of a module. For example, RORI has text modules, like the history one (a module which store all text messages in the database). So, the linked structure for this module will contains a TextCondition, an implementation for the Condition trait. This TextCondition is defined as follow:

pub struct TextCondition {
    condition: String,
}

impl Condition for TextCondition {
    fn is_fulfilled_by(&self, interaction: &Interaction) -> bool {
        let re = Regex::new(&*self.condition).unwrap();
        re.is_match(&*interaction.body.to_lowercase())
    }
}

and the history module has for condition: .*.
So each new text messages, it will get the interaction and will be activated.

Architecture

A module is decomposed in several parts:

Informations in the database

First thing first, a module must be added to the modules table in rori.db:

CREATE TABLE modules (
                id          INTEGER PRIMARY KEY,
                name        TEXT,
                priority    INTEGER,
                enabled     BOOLEAN,
                type        TEXT,
                condition   TEXT,
                path        TEXT,
                metadatas   TEXT
                )

So:

id          name        priority    enabled     type        condition   path        metadatas 
----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------
X           history     0           1           text        .*          history               

To add modules into the database there is the generate_modules.py and add_module.py scripts in the repository

Python Script

Modules are python3 scripts designed as the following:

class Module:
    def __init__(self, sentences):
        self.rori = RORI.RORI()
        self.stop_processing = False
        self.sentences = sentences

    def process(self, interaction):
        pass

    def continue_processing(self):
        return not self.stop_processing

For example, this is what we got for the hello_world module:

import datetime
import random
import re
from rori import DBManager, Module

class Database(DBManager):
    def select_message_from_today(self, author):
        dbcur = self.conn.cursor()
        current_day = str(datetime.datetime.now()).split(' ')[0]
        today_messages = "SELECT body From History Where author_ring_id=\"{0}\" AND tm>= Datetime('{1}');".format(author, current_day)
        return dbcur.execute(today_messages).fetchall()

class Module(Module):
    def process(self, interaction):
        alreadySeen = False
        nbSeen = 0
        for message in Database().select_message_from_today(interaction.author_ring_id):
            m = re.findall(r"^(salut|bonjour|bonsoir|hei|hi|hello|yo|o/)( rori| ?!?)$", message[0], flags=re.IGNORECASE)
            if len(m) > 0:
                nbSeen += 1
                if nbSeen > 1:
                    alreadySeen = True
                    break
        if alreadySeen:
            randomstr = random.choice(["already", "already2", ""])
            string_to_say = self.rori.get_localized_sentence(randomstr, self.sentences)
            res = self.rori.send_for_best_client("text", interaction.author_ring_id, string_to_say)
        else:
            randomstr = random.choice(["salut", "bonjour", "longtime", "o/"])
            string_to_say = self.rori.get_localized_sentence(randomstr, self.sentences)
            res = self.rori.send_for_best_client("text", interaction.author_ring_id, string_to_say)
        self.stop_processing = True

So, if someone wants to implement a module, they just have to create a process method and if they want to stop the processing at the next priority, set self.stop_processing to True

rsc.json

self.sentences is a json object which contains some sentences translated in different languages. For now, the choice of the language is a flag in the RORI class, but we should handle this in a better way in the future. This is for example the current rsc.json of the hello_world module:

{
  "salut": {
    "en":"Oh. It's *you.* ",
    "fr_FR":"Oh... c'est toi."
  },
  "bonjour": {
    "en":"Hello",
    "fr_FR":"Bonjour"
  },
  "longtime": {
    "en":"Hello, it's been a long time",
    "fr_FR":"Salut ! Ça faisait un petit bout de temps..."
  },
  "o/": {
    "en":"o/",
    "fr_FR":"o/"
  },
  "already": {
    "en":"Hi, but I already see you",
    "fr_FR":"Re-bonjour"
  },
  "already2": {
    "en":"Are you amnesic?",
    "fr_FR":"Rebonjour !"
  }
}

Loader

Modules are launched by the launch_module.py script defined as following:

from rori import Interaction, Module
import os.path

def load_module(path):
    path = path.replace("/", ".")
    if path[len(path)-1] == ".":
        path = path[:-1]
    # TODO pass via importlib module to avoid the exec line
    exec("import %s.module as module" % path, globals())

def exec_module(path, interaction):
    module_path = path
    load_module(module_path)
    if path[-1] == "/":
        path = path[:-1]
    path = "rori_modules/" + path + "/rsc.json"
    sentences = "{}"
    if os.path.isfile(path):
        with open(path, 'r') as f:
            sentences = f.read()
    m = module.Module(sentences)
    m.process(Interaction(interaction))
    return m.continue_processing()

So, if the path is history, the module should be described in rori_modules/history/module.py

Utils

Some helpers are furnished for writing modules. It is in rori_modules/rori/*:

1. interaction is here to store interactions like in the Rust part.
2. module is the base class of modules. Users should inherit their modules from here.
3. RORI contains helper to manipulate RORI
4. utils contains functions to manipulate the database.

100% coverage

It's not an utopia, it's a minimum.

• Ignore tests/ and main.rs from coverage report
• 100% on server.rs https://coveralls.io/builds/17139646/source?filename=src/rori/server.rs

Migrate old modules

See https://github.com/AmarOk1412/rori_modules/

For the first release

• history
• music_launcher
• music_stop
• hello_world
• goodbye_world
• age
• license
• creator
• who
• AFINN111
• name
• humor
• sing
• blackscreen
• meteo
• mutesound
• enablesound
• uptime
• lock
• alarm
• Write script to init database
• Add modules for /link, /add_device, /register, /unregister, /rm_device

If some free time

• wait

• nervermind

• get_subject

• write

Later

• is_a_color
• add_word
• rm_word

Improve build system

This is some tasks to do:

• Configure Travis CI to build RORI and run tests
• Add a clear Makefile to do basic commands and install dependencies
• Add a Dockerfile to embed RORI in a container
• Add build instructions in the Wiki and a link to that page in the README.md
• Add scripts to install all environnement

Support additional metadatas to pass to modules

For now, messages only contains datatypes and message body as specified here: https://github.com/AmarOk1412/rori_core/wiki/Custom-datatypes-handling

The problem is, with client like the Discord one, we can't really know informations about:

• Which channel does it comes from
• Who is sending this, because we have the bridge to pass through
• And for the answering, to which message we answer. This will allow threads,

For now, Ring doesn't support that. So, we need:

1. For now, add additional metadatas we need after the datatype (datatype;key=value&key2=value2) as a workaround to continue the client. NOTE: this breaks the text/plain for current client.
2. Patch OpenDHT + Ring with a better solution
3. Update this, with the final method as soon as possible.

API changes

The Interaction object will have a Vec<String> metadatas to contains

Modules ideas

• Soupe du jour from soupson.ca
• TODO List
• Calendar

Add tools to get informations on sentence for modules

It could be cool to be able to get (from modules) some helpers like:

• Nouns in a sentence or the subject (see Natural Language Toolkit like the v3?)
• Localization of a client if possible? For example for the meteo module. Or preferences.
• Add the possibility to build conversations like a flow.
• See conversations as a graph

Interaction handling

Definitions:

Interaction

An interaction is a message sent to RORI with its metadatas (type, author, etc).

Module

A module is designed to prepare actions to do for the endpoints and to modify RORI state.
A module has the following attributes:

• priority (0 = top priority, then 1, 2...)
• enabled (boolean)
• a type (text, music, etc.)
• a condition (for text module, the condition is a regex to match to activate the module)
• a path to the script to launch
• meta datas (like a description, image, dependencies)

Handling process:

Life cycle of an interaction

1. Alice send to RORI a new interaction.
2. manager.rs receives the interaction via handle_signals
3. server.rs gets the Interaction object and creates a new ModuleManager
4. The ModuleManager retrieves all modules triggered by the Interaction with the loop in the next section.
5. Modules are executed and can answer or modify RORI.

module activation loop

def process:
    priority = 0 # Start with modules with the top priority
    stop = false
    while not stop:
        # Launch all modules with the same priority
        for each module in modules (with priority == priority && enabled):
            if module.condition.fulfilled:
                result = module.exec()
                if not result: stop = true
         priority += 1

Rules for modules

A module is a python 3 script. which can interact with rori.db. A complete issue will be created later. A python 3 library will be created to interact with the database and dbus to interact with GNU RING.

A new table is available in the database:

id|name|priority|enabled|type|condition|path|metadatas (json)

NOTE: the database schema will be fixed at the next release, so after that we should use migrations.

Current status:

• Write meta ticket
• Create new table for modules
• modulemanager.rs
• Modify database.rs to get modules lists
• Execute some simple modules (history, hello_world)
• Remove TODOs from PR
• comments
• unit-testing
• wiki page
• Improve load_module.py
• Update README.md

When some clients will be released:

• Execute some modules which need multi devices (hello_world, launch_music)