reactive-graph/reactive-graph

Reactive Graph and Flow Control

Reactive Graph

What is this?

➔ The Reactive Graph Flow is a graph database

➔ The Reactive Graph Flow is a document store

➔ The Reactive Graph Flow is a flow control runtime

➔ The Reactive Graph Flow is a web server

➔ The Reactive Graph Flow is pluggable and extensible

➔ The Reactive Graph Flow is fast, secure and small

What is it for?

➔ Game Entity Component System (ECS) - especially for Inexor

➔ Smart Home and Internet of Things

➔ Data Conversion Tools

➔ Flow Control System for card size computers and embedded devices

➔ Desktop Automation

➔ Content Management System

➔ Knowledge Graphs and Knowledge Processing

Graph

A graph organizes highly interconnected data. The state of an Entity Component System can be ideally represented with the help of the graph. Inexor is the first game engine to introduce a graph as a basis.

The main benefits of a graph are:

• A universal data structure for everything
• Relations are first class citizens
• Benefit from types and instances which makes things intuitive
• Benefit from navigation which is fast and intuitive
• Benefit from the semantics of highly connected, intuitive data
• Properties can store not only certain primitive data but complete documents

Reactive

Now that we understand how data is stored, here's how data interacts. The approach is that the data itself is "alive". To do this, Inexor adopts a concept from reactive programming.

In computing, reactive programming is a declarative programming paradigm concerned with data streams and the propagation of change.

It is the ingenious combination of a graph with reactive programming. The property instances are not static and only contain data. Rather, they are streams of data. If you change the value of a property instance, you fill the data stream of this property instance. Data streams are linked together. For example, if the stream of one property instance is linked to the stream of another property instance and you change the value of the first property instance, the value of the second property instance will automatically change as well. Data is thus propagated from one station to the next, triggering a cascade of propagations.

In addition, Inexor remembers the last value in each property instance. This is done by subscribing to your own data stream and caching it. This allows subsequent querying of the value of a property instance.

Remember this basic concept:

• Every property is a stream not only data
• Property streams can be subscribed and published
• The streams of two properties can be connected and changes will be propagated (cascade)

Behaviour driven design

The data flow is therefore automatic. Building on this, Inexor applies the concept of behaviour-driven design. The goal is to use these data streams to simulate behaviour.

Behaviors can be implemented on components, entities and relations. To do this, one or more incoming data streams are combined, calculations are performed and written to one or more outgoing data streams.

For example, the entity type "AND gate" implements a behavior by subscribing to the two input properties, combining them into a combination data stream and performing an AND operation on the incoming pairs of values. The result of the AND operation is itself a data stream and this is linked to the output property.

This example shows how an entity type is wired internally. They are all data streams that are cleverly combined with one another and thus depict behavior.

It is interesting that this behavior also works for relations. For example, connectors are also implemented behaviors of streams. It is interesting that connectors connect the data stream from a property instance of the outgoing entity instance with the data stream from a property instance of the incoming entity instance.

For example the AND-Gate accepts inputs at the properties lhs and rhs. Both streams are subscribed and zipped. The zipped stream is calculated with a function - in this case the AND-Operator. This results in another (invisible) stream which is connected with the property result. The entity type AND-Gate defines that the properties lhs, rhs and result have to exist. Furthermore, the socket types are defined: lhs and rhs are Input-Sockets and result is a Output-Socket. The behaviour is like the internal wiring of entity instances and of relation instances. A behaviour can be added to entity instances and removed from entity/relation instances.

Flow

Control flows can be implemented based on the graph, the data streams and the behavior-driven design. It is important here that the available modules that implement the behavior are linked with connectors.

For example, a flow can consist of a logic that links several AND gates with each other using connectors. Both the AND gate and the connector are behaviors. But the arrangement of these behaviors within a flow makes them powerful.

Entire game modes can be implemented with the help of flows. Or just parts of it that are used in multiple game modes, such as a mechanism to pick up, drop, and score flags.

Flows are also useful for making maps more interactive. With the help of flows and behaviors, it can be ensured that a door opens in a map when you press switch 1 and switch 2. Or you determine the color of your own team's base based on the current score. Or you control particle emitters, depending on how many players are near the emitter. The possibilities for this are endless and want to be used!

Plugins

The Plugin System

List of Plugins

Plugins Repository

Development

Build Instructions

Data Model

GraphQL API

Dynamic Graph API

Configuration

Configuration

GraphQL Type System

Flow Editor

Coming soon.

reactive-graph's Issues

Event System for Type System and Instance System

Goal

Notify about created or deleted components, entity types, relation types, entity instances, relation instances and flows.

Use cases

• Get notified if an entity instance has been created or deleted in plugins and via GraphQL subscription in the user interface

Tasks

• Create Component event
  • Property event
  • This is a required component in the application core
• Create entity type system_event
  • This is a required entity type in the application core
• Component Events
  • Create entity instance /org/inexor/events/types/components/created
  • Create entity instance /org/inexor/events/types/components/deleted
  • Emit event on component creation
  • Emit event on component deletion
• Entity Type Events
  • Create entity instance /org/inexor/events/types/entities/created
  • Create entity instance /org/inexor/events/types/entities/deleted
  • Emit event on entity type creation
  • Emit event on entity type deletion
• Relation Type Events
  • Create entity instance /org/inexor/events/types/relations/created
  • Create entity instance /org/inexor/events/types/relations/deleted
  • Emit event on relation type creation
  • Emit event on relation type deletion
• Entity Instance Events
  • Create entity instance /org/inexor/events/instances/entities/created
  • Create entity instance /org/inexor/events/instances/entities/deleted
  • Emit event on entity instance creation
  • Emit event on entity instance deletion
• Relation Instance Events
  • Create entity instance /org/inexor/events/instances/relations/created
  • Create entity instance /org/inexor/events/instances/relations/deleted
  • Emit event on relation instance creation
  • Emit event on relation instance deletion
• Flow Events
  • Create entity instance "/org/inexor/events/flows/created"
  • Create entity instance "/org/inexor/events/flows/deleted"
  • Emit event on flow creation
  • Emit event on flow deletion
• GraphQL: Provide example subscriptions

Components

Name	Properties	Data Type	Socket Type
Event	event	any	output

Entity Types

Name	Component	Description
ComponentEvent	event	Name of the component
	label	/org/inexor/events/types/components/created /org/inexor/events/types/components/deleted
EntityTypeEvent	event	Name of the entity type
	label	/org/inexor/events/types/entities/created /org/inexor/events/types/entities/deleted
RelationTypeEvent	event	Name of the relation type
	label	/org/inexor/events/types/relations/created /org/inexor/events/types/relations/deleted
EntityInstanceEvent	event	UUID of the entity instance
	label	/org/inexor/events/instances/entities/created /org/inexor/events/instances/entities/deleted
RelationInstanceEvent	event	UUID of the relation instance
	label	/org/inexor/events/instances/relations/created /org/inexor/events/instances/relations/deleted
FlowEvent	event	UUID of the flow
	label	/org/inexor/events/flows/created /org/inexor/events/flows/deleted

Add System Event TypeSystemChanged

The system event TypeSystemChanged fires if the type system has changed - either a component, entity type or relation type has been created or deleted.

This functionality is needed for the Dynamic Graph feature. Each time when the type system has changed, it is necessary to regenerate the GraphQL schema of the Dynamic Graph #55

Simply development

In order to simplify the development, the most important topics are tracked in this issue:

• Reduce boilerplate code in plugins by using abstractions and macros in the core
  • #62
  • #60
• Simplify behaviour development in plugins by providing a framework
  • #61
• Create model crates on top of reactive instances for every component, entity type and relation type
• Make plugins hot deployable and calculate a plugin dependency tree
  • #60
  • #67
• Make the GraphQL endpoints usable using a GraphQL client and a GraphQL schema visualization
  • #29
  • #31
• Provide a GraphQL endpoint with a dynamically created schema which is more convenient and more natural
  • Tracked in #18
• Merge all plugin repositories into a single mono repository to reduce overhead, boilerplate and compilation times
  • #63
• Merge all core repositories and the application repository into a single mono repository to reduce overhead and make refactoring easier
  • #65
• #75

Plugin API: Extend API + Remove dependencies

Plugin API: Extend API

• ComponentManager
  • find
  • import
  • export
• EntityTypeManager
  • find
  • import
  • export
• RelationTypeManager
  • find
  • import
  • export

Plugin API: Remove dependencies

Make plugins slim by removing some dependencies used by inexor-rgf-core-plugins:

• actix-http
• actix-web
• async-std
• query_interface
• inexor-rgf-core-reactive

GraphQL: Query flows by label

Tasks

• Store label in LabelTree when registering a reactive flow
• Add GraphQL Filter

GraphQL: Dynamic Graph: Generate SDL from Type System

Goal

Generate a GraphQL schema using Apollo Encoder from the RGF type system dynamically. This is the first step for the Dynamic Graph feature.

TODOs

• Scalar UUID
• Scalar JSON
• Create types for Components
  • Create fields from the Component's properties
• Create types for Entity Types
  • Create subtypes from the Component's which are defined by the Entity Type
    • Create fields from the Component's properties
  • Create fields for the inbound and outbound Relation Types. The name of the field is the relation type name. If inbound and outbound relations of the same relation type exists, the field name is prefixed.
• Create types for Relation Types
  • Create subtypes from the Component's which are defined by the Relation Type
    • Create fields from the Component's properties
  • Create fields for the inbound and outbound Entity Types. The name of the field is the entity type name. If inbound and outbound types are the same, the field name is prefixed.

Inexor Widget Toolkit

Goal

Implement widget toolkit for React using mantine.

The widgets are entirely self-managed and automatically communicates with the GraphQL interface.

Use Cases

• Playground
• Flow Editor
• Asset Manager
• Game Server List
• Game Chat
• Home Automation UI
• Robotics UI

Usage Example

<ApolloProvider client={client}>

  ...

  <InexorButton id="4faf3dad-3be5-45c7-b73e-96e5855b4c7d" property="trigger">
    <StarIcon /> Action
  </InexorButton>

  ...

</ApolloProvider>

Widgets

Widget	Mantine Component	Component	Property
InexorButton	Button	action	trigger
InexorSwitch	Switch	state_boolean	state
InexorIconSwitch	Switch	state_boolean	state
InexorTextInput	TextInput	state_string	state
InexorNumberInput	NumberInput	state_number	state
InexorImage	Image	state_string (data_url)	state
InexorColorPicker	ColorPicker	state_string	state
InexorJsonInput	JsonInput	state_array / state_object	state
InexorTransferList	TransferList	state_array	state
InexorSlider	Slider	state_number	state

Provide required components and entity types in the application core

Problem

Components, entity types and relation types which are required in the application core must exist.

Tasks

• Components
  • labeled
  • event
• Entity Types
  • generic_flow
• Remove components and entity types from inexor-rgf-plugin-base

Plugin File: Reactive Component: fs_notify

Goal

The component fs_entity triggers if the file with the given filename has been changed on the filesystem. This enables
hot reloading of file resources, for example JSON files, TOML configuration files and binary data.

Component Flow

stateDiagram
  direction LR
  filename --> component_fs_notify
  state component_fs_notify {
    direction LR
    fn_watch
  }
  fn_watch --> trigger : externally changed

Entity Flow

stateDiagram
  direction LR
  [*] --> trigger: manual trigger
  [*] --> filename
  state entity_load_something {
    filename --> component_fs_notify
    state component_fs_notify {
      direction LR
      fn_watch --> trigger : externally changed
      trigger --> result
    }
    state component_load_something {
      trigger --> fn_load
      filename --> fn_load
      fn_load --> result
    }
  }
  result --> [*]

Use Cases

This component is indented to work together with several entity types, which loads data from disk.

• load_json+ fs_notify: (Re-) Load JSON files every time the file has been changed on the FS
• load_config+ fs_notify: (Re-) Load TOML configurations every time the file has been changed on the FS
• load_binary_data+ fs_notify: (Re-) Load binary data every time the file has been changed on the FS

Components

fs_notify relies on the components action (property trigger) and file (property filename).

Component	Property	Data Type	Socket Type
action	trigger	bool	input
file	filename	string	input
fs_notify

Tasks

• Setup Plugin
• Implement Component Behaviour
• Make it work with + Test
  • load_json
  • load_config
  • load_binary_data

Resources

• https://github.com/notify-rs/notify
• https://github.com/aschaeffer/inexor-rgf-plugin-file
• https://github.com/aschaeffer/inexor-rgf-plugin-json
• https://github.com/aschaeffer/inexor-rgf-plugin-config
• https://github.com/aschaeffer/inexor-rgf-plugin-binary-data

Flow Types

Goal

With a flow type, an abstract flow can be defined and instantiated multiple times. Similar flows can be instantiated using parametrization. Flow types can be nested, which allows reusage of mechanics.

Use Cases

Everything which can run multiple times at the same time:

• A game mode
• A mechanic of a game mode which is a building block for concrete game modes (for example: flag mechanics is used by multiple game modes)
• A weapon mechanics
• A mechanic of a weapon which is a building block for concrete weapons
• A player
• A team
• A hardware device which exists multiple times
• An identical or similar room which exists multiple times on a map
• An IoT device which exists multiple times
• ...

Specification

• Flow types can be represented as JSON and can be imported from a JSON file and exported to a JSON file
• Flow types contains:
  • Virtual entity instances
  • Virtual relation instances
  • Specific entity instances (with a specific UUID or path which is known at design time - usually well known singletons)
  • Named entity instances (the entity instance is not known at design time but provided at instantiation)
  • Another flow types (which allows building complex scenarios with abstract sub flows)
• Flows can be instantiated from a flow type multiple times, because the actual UUIDs of the resulting entity instances are generated at instantiation time
• Immediate UUIDs
  • Virtual entity instances have an immediate UUID
  • Virtual relation instances connects virtual entity instances (and uses the immediate UUIDs of the outbound + inbound virtual entity instances)
• Dynamic Instantiation
  • Flow type parameters
    • Flow types can define named parameters
    • The value of the properties of the virtual entity instances and virtual relation instances can use named parameters
    • At instantiation of a flow from a flow type, the value of the named parameters are replaced by the provided values
  • Flow type instances
    • Flow types can define named entity instances
    • The UUIDs of virtual entity instances can use an entity instance name
    • At instantiation of a flow from a flow type, the UUIDs of the named entity instances are replaced by the provided entity instances

Tasks

• Import a flow type from JSON
• Export a flow type to JSON
• Create a new flow type
• Delete a flow type
• Update a flow type
  • Add a virtual entity instance to a flow type
  • Remove a virtual entity instance to a flow type
  • Add a virtual relation instance to a flow type
  • Remove a virtual relation instance to a flow type
• Create a flow (instance) from a flow type
  • Replace the immediate UUIDs in virtual entity instances and virtual relation instances
  • Replace the named parameters
  • Replace the named entity instance by the provided concrete entity instance
  • Create the flows for the nested flow types
• Extend Plugin API
  • FlowTypeProvider
• ChangeLog
• Documentation
• Tests

Generate typescript types and query/mutation hooks

Tasks

• Generate typescript types using the introspected GraphQL schema
• Generate typescript query/mutation hooks using parametrized queries and mutations using .graphql files

Depends on

• #18

Resources

• https://www.graphql-code-generator.com/

Refactor dependencies injection

• Rename / Make fork independent
• Code Formating
• Licensing File
• Update Dependencies to current versions

Plugin JSON: Load from file / Save to file

Goal

1. Load data from a JSON-File

   stateDiagram
       direction LR
       [*] --> trigger
       trigger --> load_json
       filename --> load_json
       state load_json {
         direction LR
         file --> [*]
       }
       load_json --> result
       result --> [*]
   

2. Store into a JSON-File

   stateDiagram
       direction LR
       [*] --> trigger
       trigger --> save_json
       filename --> save_json
       payload --> save_json
       state save_json {
         direction LR
         [*] --> file
       }
       save_json --> [*]
   

Components

Component	Property	Data Type	Socket Type	Note
file	filename	string	input
fs_notify	fs_notify	boolean	input	Hot reload JSON object

Behaviours

Entity Type	Component	Property	Data Type	Socket Type
load_json	action	trigger	boolean	input
	file	filename	string	input
		result	any	output
save_json	action	trigger	boolean	input
	file	filename	string	input
		payload	any	input

GraphQL: Dynamic Graph: Implement Resolvers for Querying the Instance System

Goal

Implement resolvers for querying the RGF instance system (entity instances, relation instances). This is the third step for the Dynamic Graph feature.

Tasks

• Implement NameSpaceManager to get all namespaces used in components, entity types, relation types and flow types
• Implement Root Query
• Generate objects for all namespaces
• Generate interfaces for all components (prefixed by namespace, suffixed by Component)
• Generate interface Entity for entity objects
• Generate interface Relation for relation objects
• Generate objects for all entity types (prefixed by namespace, suffixed by Entity)
• Generate fields for an entity (entity type properties + component properties)
  • boolean
  • f64 number
  • i64 number
  • u64 number
  • string
  • array
  • object
  • Check for divergent data types (component vs. entity type)
• Generate fields for relations of an entity
• Generate objects for all relation types (prefixed by namespace, suffixed by Relation)
• Generate fields for a relation
• Generate fields for the outbound entity of a relation
• Generate fields for the inbound entity of a relation
• Generate unions AllEntities and AllRelations
• Generate unions for all entities of a namespace
• Generate unions for all relations of a namespace
• Improved relation naming
  • Make it possible to rename an outbound entity field of a relation
  • Make it possible to rename an inbound entity field of a relation
  • Make it possible to rename an outbound relation field of an entity
  • Make it possible to rename an inbound relation field of an entity
• Shortcut relation links
  • Make it possible to link directly from an entity to the entity of an outbound relation (skip the relation itself)
  • Make it possible to link directly from an entity to the entity of an inbound relation (skip the relation itself)
• Exclusions
  • Make it possible to exclude a component
  • Make it possible to exclude an entity type
  • Make it possible to exclude a relation type
  • Make it possible to exclude field of a component
  • Make it possible to exclude field of an entity
  • Make it possible to exclude field of a relation

Complex Connectors

Goal

Implement more complex connectors.

Connectors

Connector	Description
DebounceConnector	Propagate only, if incoming data has changed (the connector have to remember the last state interally)
ThreadedConnector	Run propagation in a new thread
DelayedConnector	Run propagation after a configurable delay, for example after one second
BufferedFifoConnector	Buffer incoming data in a FIFO of a configurable size, for example 10 entries
NumericInterpolationConnector	Similar to BufferedFifoConnector, but a numeric interpolation function is executed over the entries of the buffer, for example the result is calculated as the average of the last 10 entries (or min, max)

Implement Actions and Generators

Components

Name	Properties	DataType	SocketType	Description
action	trigger	bool	input	Triggers an action on true
	result	any	output	The datatype may be overridden by the concrete entity type
generator	trigger	bool	input	Produces triggers (signals boolean true)

Tasks

• Plugin Logical
  • Provide components
  • Apply new components on existing entity types which are either an action or a generator
  • Implement Behaviour Toggle (incoming trigger toggles outgoing boolean)
• Plugin Arithmetic
  • Implement Behaviour Counter (incoming trigger increases outgoing number)
• Plugin Random
  • Implement Behaviour PseudoRandomNumber (with seed in order to get reproducible results)
  • Implement Behaviour RandomBool
  • Implement Behaviour RandomString

Search for type names with wildcards

• component_manager.find(search_string: String) -> Vec<Component>
• entity_type_manager.find(search_string: String) -> Vec<EntityType>
• relation_type_manager.find(search_string: String) -> Vec<RelationType>

Dynamically add/remove components and behaviours to reactive instances

• Add components to ReactiveEntityInstance
• Add components to ReactiveRelationInstance
• Allow adding components to entity instances and relation instances after initial construction
  • Add properties (if not already present)
  • Add component behaviours
• Allow removing components from entity instances and relation instances
  • Remove component behaviours
  • Do not remove properties

Snap package

User Story

As a user or a tester I want to download a snap package which is installable on my Linux computer.

Definition of Done

• A snap package is available
• Additional plugins are contained in the snap package, preferable it's possible to configure which plugins shall be contained in the snap package
• The package can be downloaded from GitHub

Resources

A good walk through:
https://snapcraft.io/blog/lets-build-a-snap-together-a-complex-snapcraft-yaml-walkthrough

Improve Behaviours

Data Model

• Remove field behaviours from EntityTypes and RelationTypes
• Add field behaviours to EntityInstance and RelationInstance (the field reflects the effectively applied behaviours)
• Each behaviour should add itself to the field behaviours if applied on a EntityInstance or RelationInstance
• Each behaviour should remove itself from the field behaviours if applied on a EntityInstance or RelationInstance
• GraphQL: Query instances by behaviour

ComponentBehaviourManager

• add_behaviours_to_entity: Iterate through the components of an EntityInstance and call add_behaviours_to_entity with component
• add_behaviours_to_relation: Iterate through the components of an RelationInstance and call add_behaviours_to_relation with component
• remove_behaviours_from_entity: Iterate through the components of an EntityInstance and call remove_behaviours_from_entity with component
• remove_behaviours_from_relation: Iterate through the components of an RelationInstance and call remove_behaviours_from_relation with component

ComponentBehaviourProvider

• fn add_behaviours_to_entity(&self, entity_instance: Arc<ReactiveEntityInstance>, component: Component);
• fn add_behaviours_to_relation(&self, relation_instance: Arc<ReactiveRelationInstance>, component: Component);
• fn remove_behaviours_from_entity(&self, entity_instance: Arc<ReactiveEntityInstance>, component: Component);
• fn remove_behaviours_from_relation(&self, relation_instance: Arc<ReactiveRelationInstance>, component: Component);

Improve flexibility of behaviours

Problem

1. There are plugins which reactive instances interacts with the business layer (file::fs_notify, scheduler::scheduler)
2. There are plugins which reactive instances runs in threads / asynchronously
3. There are plugins which have to implement a different behaviour on disconnect

Tasks

• Make it possible to implement a different behaviour on disconnect
• Make it possible to store thread handles in a behaviour
• Make it possible to stop and clean up threads on disconnect
• Make it possible to construct a behaviour which having a tokio runtime

Plugin Value: States

Goal

Introduce new entity types which stores, synchronizes and debounces the state of an external thing/a digital twin, for example, a user interface button or an IRL-switch.

• Represents "something external" and stores it's state
• Debounces propagation in order to break feedback loops

Use Cases

• User Interface
  • Internal state of a user interface button. Synchronizes the button state and the internal state
• Home Automation
  • Switch. Synchronizes the switch state and the internal state

Behaviours

Property	Socket Type	Description
set_state	input	Sets the internal state and propagates value if and only if the internal state has changed.
state	none	The internal state.
value	output	The debounced state.

Components

Component	Properties	Data Type	Socket Type
value_boolean	value	boolean	output
value_number	value	number	output
value_string	value	string	output
value_arrray	value	array	output
value_object	value	object	output
state_boolean	state	boolean	none
	set_state	boolean	input
state_number	state	number	none
	set_state	number	input
state_string	state	string	none
	set_state	string	input
state_array	state	array	none
	set_state	array	input
state_object	state	object	none
	set_state	object	input

Entity Types

Entity Type	Components
value_boolean	value_boolean
value_number	value_number
value_string	value_string
value_arrray	value_arrray
value_object	value_object
state_boolean	state_boolean
	value_boolean
state_number	state_number
	state_number
state_string	state_string
	state_string
state_array	state_array
	state_array
state_object	state_object
	state_object

Plugin Base: Component Ordered

Goal

Relations between two entities can be ordered by a numeric value. The property order is placed on the relation.

flowchart
    Entity1 -- order: 1 --> Entity2
    Entity1 -- order: 2 --> Entity3
    Entity1 -- order: 3 --> Entity4
    Entity2 -- order: 1 --> Entity3
    Entity2 -- order: 2 --> Entity4

Use cases

Every relation which has a cardinality of 1:N

Components

Component	Property	Data Type	Socket Type
ordered	order	number	input

mdBook Documentation

Tool

https://rust-lang.github.io/mdBook/

Chapters

• About
  • Introduction
  • Design Goals
  • Roadmap
• Installation
  • Linux
  • Raspberry Pi
  • Windows
• Configure
  • Logging
  • HTTP/GraphQL Server
  • Plugins
• Model
  • Type System
    • Component
    • Entity Type
    • Relation Type
    • Property Type
  • Instance System
    • Entity Instance
    • Relation Instance
    • Property Instance
    • Flow
• GraphQL API
  • Type System
    • Component
    • Entity Type
    • Relation Type
  • Instance System
    • Entity Instance
    • Relation Instance
    • Flow
• Plugin system
  • Trait Plugin
  • Trait PluginContext
  • Lifecycle
  • Component Provider
  • Entity Type Provider
  • Relation Type Provider
  • Flow Provider
  • Component Behaviour Provider
  • Entity Behaviour Provider
  • Relation Behaviour Provider
  • Web Resource Provider
• Development
  • Supported Platforms
  • Build
    • Install build tools
    • Build main application
    • Build plugins
  • Coding Guidelines
  • Code Formatting
  • Binaries
  • Packaging
  • Documentation
  • License

Publishing

• https://rust-lang.github.io/mdBook/continuous-integration.html

Extend Model API 0.8.0

Model API 0.8

• TypeIds
  • Model
    • Namespaces
    • Types
      • ComponentTypeId
      • EntityTypeId
      • RelationTypeId
      • FlowTypeId
      • RelationInstanceTypeId
        • Is a RelationTypeId
        • Has a InstanceId
        • InstanceId is either
          • empty (only one relation instance can exist between two entity instances having the same relation type)
          • specified on creation (there may be multiple relation instances between two entity instances having the same relation type but only one with the same instance id)
          • random (there may be lots of relation instances and the instance id is generated randomly for each new relation)
      • ComponentOrEntityTypeId
        • A relation outbound or inbound can be either a EntityType or a Component
        • Variant
          • ComponentTypeId
          • EntityTypeId
      • ComponentBehaviourTypeId
      • EntityBehaviourTypeId
      • RelationBehaviourTypeId
      • EntityComponentTypeId
      • RelationComonentTypeId
    • Type Trait Implementations
      • Implementierung type_identifier()
      • Implementierung type_name()
      • Implementierung to_string()
      • Implementierung from(Identifier)
      • Implementierung from(String)
      • Unit Tests
  • Builder
    • ComponentTypeId
    • EntityTypeId
    • RelationTypeId
    • FlowTypeId
    • RelationInstanceTypeId
  • API + Plugin API
    • Method name consistency
      • xyz(XyzTypeId)
      • xyz_by_type(namespace, type_name)
    • Additional getter and query methods
      • get_by_namespace(namespace)
      • count()
      • count_by_namespace(namespace)
    • Apply the APIs on the complete code base
  • GraphQL
    • NamespacedTypeDefinition
      • From NamespacedTypeDefinition for ComponentTypeIdDefinition
      • From NamespacedTypeDefinition for EntityTypeIdDefinition
      • From NamespacedTypeDefinition for RelationTypeIdDefinition
      • From NamespacedTypeDefinition for RelationTypeIdDefinition
    • Flatten InputType
• Unified Display Trait
  • TypeId {namespace}__{type_name}
  • Reactive Instances
    • Entity Instance {type_id}__{id}
    • Relation Instance {outbound_id}--[{type_id}]-->{inbound_id}

GraphQL: Dynamic Graph

Problem

Currently, the GraphQL interface allows access to the type system and the instance system. This allows general access to all types and instances. But the access is done in a way that takes place on a technical level.

For example, if you are looking for a specific player, you are querying an entity instance - not the player.

query keys {
  instances {
    keys: entities(type: "or") {
      id
      label
      properties(
        names: [
          "lhs"
          "rhs"
          "result"
        ]
      ) {
        name
        value
      }
    }
  }
}

As a consumer of the GraphQL API, you have to handle the result. For example, you convert an EntityInstance to the actual type, which produces a lot of overhead. Or you introduce an abstraction layer for such conversion.

Goal

The aim is to also offer a GraphQL schema that is dynamically generated from the type system. Querying is shall be much more comfortable:

query GetAllAdds {
  or {
    logical_gate {
      ...LogicalGate
    }
  }
}

fragment LogicalGate on cLogicalGate {
  lhs
  rhs
  result
}

As you can see, you query for instances of the or type. Also, the or type has a field logical_gate of the type cLogicalGate which represents the component logical_gate.

Another advantage is that you can generate the types for TypeScript or your favourite language using the GraphQL schema of the Dynamic Graph.

Limitations

The disadvantage of this approach is, that only the components which are defined in the entity type are considered. Components which have been added to the entity instance at runtime cannot be resolved. It would be possible to make a hint possibleComponents

Roadmap

• #52
• #55
• #56
• #64
• Update the GraphQL schema visualization plugin
• Update the GraphQL client plugin
• Update the documentation

GraphQL: Search instances by property value

query keys {
  instances {
    keys: entities(
      type: "input_device_key",
      properties: [
        {
          "name": "key_code",
          "value": 74
        }
      ]
    ) {
      id
      label
      properties(
        names: [
          "name"
          "key"
          "key_code"
          "key_down"
        ]
      ) {
        name
        value
      }
    }
  }
}

Benchmark tests for the type system

User Story

As a developer, I want to benchmark the system.

Definition of Done

• I can run a benchmark which shows me how many components are created within a fixed time span.
• I can run a benchmark which shows me how many entity types are created within a fixed time span.
• I can run a benchmark which shows me how many relation types are created within a fixed time span.

Resources

• https://doc.rust-lang.org/1.16.0/book/benchmark-tests.html
• https://nnethercote.github.io/perf-book/title-page.html
• https://github.com/bheisler/criterion.rs

Request Logging

• Configuration graphql.toml: log_pattern with default fallback
• Middleware
• Documentation

HTTP Server: Default web resource

Goal

The root URL (/) can be routed to a web resource depending on the purpose.

For example, in the case of a game server, to the plugin that provides the front end of the game server, in the case of the game client, to the plugin that provides the HUD and the menu of the game client, or in the case of home automation, to the plugin that provides the flow Editor provides.

Tasks

• Add configuration
• Create actix-web route
• Route to default web resource

GraphQL: Dynamic Graph: Live-Introspection

Goal

The goal is that GraphQL clients can resolve the GraphQL schema via GraphQL Introspection. This is the second step for the Dynamic Graph feature.

Tasks

• Type System Changed Event
• Regenerate Dynamic Schema on Type System has changed
• Store generated Dynamic Schema
• Implement Getter for the generated Dynamic Schema
• Add actix-web Endpoint and convert query and result

Improve GraphQL API

• Rename mutation types (Remove prefix GraphQl)
• Make extensions a separate type (used in components, entity types and relation types)
• Navigate from components to entity types and to relation types
• Rename type_name to type
• Navigate from a property instance to the property type
• Shortcut labels
• Filter flows by flow type
• Stream property instances in subscriptions instead of handcrafted JSONs

Extend plugin API

• Provide default implementations for trait Plugin
  • Apply changes to all existing plugins
• Provide default implementations for trait Lifecycle
  • Apply changes to all existing services
• Provide macros to reduce boilerplate code in plugins
  • component_provider!
  • entity_type_provider!
  • relation_type_provider!
  • flow_type_provider!
  • flow_instance_provider!
  • web_resource_provider!
  • Apply changes to all existing plugins
• Return a Result in all methods of trait Plugin
  • Apply changes to all existing plugins
• Change group to namespace
  • Apply changes to all existing JSON type definitions
• Add FlowTypeProvider and change FlowProvider to FlowInstanceProvider
  • Apply changes to all existing plugins
• Add FlowTypeManager and change FlowManager to FlowInstanceManager
  • Apply changes to all existing plugins

Plugin System: Plugin Dependencies

Plugin API 0.8.0

• Plugin State and Lifecycle
  • https://eclipsesource.com/de/blogs/2013/01/23/how-to-track-lifecycle-changes-of-osgi-bundles/
  • Installed
    • The plugin is there
    • The plugin name is known
    • The plugin version is known
    • The plugin api version is known
    • The rustc version is known
    • The dependencies are known
  • Resolved
    • The rustc version is compatible
    • The plugin api version is compatible
    • None of the dependencies has state "Installed"
    • It is possible to start the plugin
  • Starting
    • The plugin is calling the "initialize" method
  • Active
    • The plugin is running
  • Stopping
    • The plugin is calling the "destroy" method
• PluginContainer Struct
  • Rustc Version
  • Plugin API Version
  • Name
  • Description
  • Version
  • Dependencies
    • Name
    • Version
  • Library
• Read Plugin Declaration
• Plugin State Transitions
  • Void → Installed
  • Installed → Resolved
  • Resolved → Starting
  • Starting → Active
  • Active → Stopping
  • Stopping → Resolved
  • Resolved → Void
• Start Plugin
• Prevent already started plugin to start twice
• Prevent starting a plugin which has unsatisfied dependencies
• Provide macros in -core-plugins
  • Macro for ComponentProvider
  • Macro for EntityTypeProvider
  • Macro for RelationTypeProvider
  • Macro for FlowTypeProvider
  • Macro for FlowInstanceProvider
• Provide macros for behaviours in -core-reactive
• Provide traits for behaviours in -core-reactive
• Shutdown plugin gracefully
  • State transition (Stopping)
  • Shutdown dependent plugins
  • Shutdown plugin (more states)
• Uninstall plugin gracefully
  • State transition (Uninstalling)
  • Unload DLL
  • Delete DLL
  • Remove plugin from plugin container manager
• Refresh plugin gracefully
  • State transition (Refreshing)
  • Shutdown dependent plugins
  • Stop plugin
  • Start plugin
  • Start dependent plugins
• Redeploy plugin gracefully
  • State transition (Refreshing)
  • Stop plugin
  • Uninstall old plugin
  • Deploy new plugin
  • Resolve plugin
  • Start plugin
  • Start dependent plugins
• Deploy new plugin
  • Deploy new plugin
  • Resolve plugin
  • Start plugin
  • Start dependent plugins
• Hot reloading plugin
  • Create Plugin Folder
  • Cargo Command for copying the produced library into the plugin folder
  • Watch fs events on deploy folder
  • On fs event (Refresh)
    • If Active → State transition (Restarting)
    • If Resolved → State transition (Reinstalling)
    • If Uninstalled → State transition (Installing)
  • State transition (Reinstall)
  • Shutdown dependent plugins
  • Uninstall current library
  • Install new library
  • Don't deactivate before dependent plugins are stopped
  • Start dependent plugins
• GraphQL API
  • Queries
    • List Plugins
      • Plugin State
      • stem
      • path
      • name
      • description
      • version
      • rustc_version
      • plugin_api_version
      • Dependencies
      • Dependents
      • Unsatisfied Dependencies
  • Mutations
    • Stop Plugin
    • Start Plugin
    • Restart Plugin
    • Uninstall Plugin
    • Redeploy Plugin

User Stories

• Plugin: Metadata: Each plugin defines dependencies
• Plugin-Manager: Build dependency tree
• Plugin-Manager: Ensure plugins are loaded in the correct order
• Plugin: Document all traits (Plugin)
• Write unit tests

Merge the single-crate core repositories into a mono repository

Merge the inexor-rgf-core-* repositories into a mono repository:

• inexor-rgf-core-di ⇒ crates/core/di
• inexor-rgf-core-di-codegen ⇒ crates/core/di-codegen
• inexor-rgf-core-frp ⇒ crates/core/frp
• inexor-rgf-core-model ⇒ crates/core/model
• inexor-rgf-core-reactive ⇒ crates/core/reactive
• inexor-rgf-core-builder ⇒ crates/core/builder
• inexor-rgf-core-plugins ⇒ crates/core/plugins
• inexor-rgf-application/src ⇒ crates/runtime

How to merge

Remodel Entity Behaviours to Component Behaviours (Generators / Actions / Auto-trigger)

Introduction

Lately, new components generator and action have been created to enable composable actions.

• A trigger is a signal with is a boolean true.
• A generator output a trigger.
• An action will be executed on trigger input.

Different types of generators are possible, for example: a button, a keystroke, a menu item, a scheduled job, an HTTP/MQTT message, ...
Different types of actions are possible, for example: a system command, an HTTP/MQTT message, ...

Goal

Use the power of components and remodel behaviours to be composable.

Steps for each Behaviour

• Create a component
• Make the entity type provide the newly created component
• Make the entity type provide component action
• Refactor the EntityBehaviour to a ComponentBehaviour
• Test if the newly created component behaviour is composable with a generator, for example tray_menu_item
• Implement auto trigger: trigger on trigger and a specific other input property

Behaviours

Plugin	EntityBehaviour	auto trigger Property	action/trigger	component
system-command	system_command	parameters	✓	✓
config	config_file	filename	✓	✓
binary	load_binary_data	filename	✓	✓
	save_binary_data	filename data_url	✓	✓
json	load_json	filename	✓	✓
	save_json	filename payload	✓	✓
http	http	payload	✓	✓
	jsonrpc	payload	✓	✓
mqtt	mqtt_publisher	payload
	mqtt_subscriber	payload

Improve Plugin Hot Reloading

Problems

1. Currently, no build is triggered for a plugin if ...
   • The type system definition has changed
   • The source code hasn't changed
2. While redeploying plugins, already registered type system definitions are ignored
3. Plugin hangs in refresh transition

Goal

1. Reduce application restarts
2. Add multi instance capability

Tasks

• Rebuild if types/**/.json have changed
  • build.rs cargo:rerun-if-changed
• Merge type system definitions (#73)
• Fix refresh transition
• Add configuration option for disable/enable hot reloading
• Add configuration options for plugin folders
• Remove duplicate plugin files in installation folder on startup
• Yield during plugin resolving to avoid blocking the async runtime

Inexor App Toolkit

Goal

Provides a simple dynamic declarative user interface.

Use Cases

• Dashboards
• Management UIs
• Debugging UIs
• Home Automation
• Desktop Automation

Declarative Definition of a User Interface

{
  "name": "Test",
  "pages": [
    {
      "title": "Page 1",
      "columns": [
        {
          "widget": "the-uuid",
        }
      ]
    }
  ]
}

Entity Types

Entity Type	Property	Data Type	Socket Type
app	layout	object	none
	css	string	none

Web Resource Provider

• URL /apps/{name}
  • Renders the declarative user interface
• URL /apps/{name}/styles.css
  • Renders the stylesheet

Plugin Management User Interface

• List of plugins and their state (active/inactive)
• Start Plugin
• Stop Plugin
• Restart Plugin

Merge the single-crate plugin repositories into a mono repository

Merge the inexor-rgf-plugin-* repositories into a mono repository:

• inexor-rgf-plugin-arithmetic
• inexor-rgf-plugin-base
• inexor-rgf-plugin-binary
• inexor-rgf-plugin-comparison
• inexor-rgf-plugin-config
• inexor-rgf-plugin-connector
• inexor-rgf-plugin-file
• inexor-rgf-plugin-graphql-client
• inexor-rgf-plugin-graphql-schema-visualization
• inexor-rgf-plugin-http
• inexor-rgf-plugin-json
• inexor-rgf-plugin-logical
• inexor-rgf-plugin-metadata
• inexor-rgf-plugin-numeric
• inexor-rgf-plugin-random
• inexor-rgf-plugin-string
• inexor-rgf-plugin-system-environment
• inexor-rgf-plugin-taxonomy
• inexor-rgf-plugin-value

(How to merge: https://mattmazzola.medium.com/creating-a-monorepo-from-separate-repos-merging-repositories-f7942885ace6)

GraphQL: Dynamic Graph: Implement Resolvers to Mutate the Instance System

Goal

Implement resolvers to mutate the RGF instance system (entity instances, relation instances). This is the fourth step of the Dynamic Graph feature.

Tasks

• Implement Root Mutation
• Generate objects for all namespaces
• Generate "Create" operation for all entity types of a namespace
  • Return the created entity instance
• Generate "Update" operation for all entity types of a namespace
  • Query entity instances by filters
    • By ID
    • By label
    • By property value(s)
    • By having a component
    • By having a behaviour
  • Update reactive stream of the entity instance property
  • Return the updated entity instance(s)
• Generate "Delete" operation for all entity types of a namespace
• Generate "Relate" operation for all entity types of a namespace
  • relateOutbound(possibleOutboundRelationTypeName(instance_id: "optional", property_name_1: "", ...) { instance_id property_name_1 property_name_2 })
• Generate "Create" operation for all relation types of a namespace
  • Return the created relation instance
• Generate "Update" operation for all relation types of a namespace
  • Query entity instances by filters
    • By ID
    • By label
    • By property value(s)
    • By having a component
    • By having a behaviour
  • Update reactive stream of the relation instance property
  • Return the updated relation instance(s)
• Generate "Delete" operation for all relation types of a namespace

Plugin: GraphQL Schema Visualization

Tasks

• Web Resource Provider
• GraphQL Voyager
• Configuration plugins.toml
• Documentation

Name	Repository
inexor-rgf-plugin-graphql-schema-visualization	https://github.com/aschaeffer/inexor-rgf-plugin-graphql-schema-visualization

Components can have extensions

Goal

Components should be able to provide extensions.

Tasks

• Add extensions to the model of Components
• Add builder for Components
• Add GraphQL resolvers

Support secure connections HTTPS/TLS/WSS

• REST
• GraphQL
• Websockets
• Configuration

  # Enable HTTPS and WSS
  secure = false

Improve Reactive Behaviours API

Reactive API 0.8

Disadvantages of Reactive API 0.7.0

• Registration of behaviours has a lot of boilerplate code in each plugin
• Management of behaviours is decentralized and repetitive
• In order to be able to restart plugins, behaviours have to shut down gracefully

Goals

1. Finite State Machine

• Reduce boilerplate code in plugins
• Streamline plugin implementations
• Abstract common code into base crates
• Implement a finite state machine for behaviours and state transitions
• Implement a framework which provides extension points for
  • initialization
  • validation
  • connecting (internal wiring)
  • disconnecting
  • reconnecting
  • query state

2. Validation

• Standardize validation of behaviours
• Provide macros for defining a validator easily

3. Connect & Disconnect

• Implement a framework which manages the observer handles
• Automatically drop stream observers when the behaviour drops
• Integrate managed observers with the finite state machine

4. Centralized Behaviour Store "Behaviours are First Class Citizens"

• Extent model to have behaviours as first class citizens
• Change service method signatures to use behaviour types
• Centralize behaviour storages with a two-level-hashmap containing the entity id/relation edge key and the behaviour type as hash keys.
• Provide a registration interface for plugins which registers and unregisters behaviours to the centralized behaviour storage

Tasks

• Behaviour Validation
• Behaviour Error Types
• Require Drop
• Behaviour Container
  • ty()
• Reactive Instance Container
  • get_reactive_instance()
• Behaviour Finite State Machine
  • get_state()
  • transition(state: BehaviourState)
    • Created --validate()--> Validated
      • validate()
    • Validated --init()--> Ready
      • init()
    • Ready --connect()--> Connected
      • connect()
      • Automatically add behaviour reactive_instance.add_behaviour()
    • Connected --disconnect()--> Ready
      • disconnect()
      • Automatically remove behaviour reactive_instance.remove_behaviour()
    • Ready --shutdown()--> Drop
      • shutdown()
  • enum BehaviourState
    • Created
    • Validated
    • Ready
    • Connected
• BehaviourStorage
  • ComponentBehaviourStorage
  • EntityBehaviourStorage
  • RelationBehaviourStorage
  • EntityComponentStorage
  • RelationComponentStorage
• Centralized Behaviour Management
  • Registry for EntityBehaviours
  • Registry for EntityComponentBehaviours
  • Registry for RelationBehaviours
  • Registry for RelationComponentBehaviours
  • Plugin Interface
  • Gracefully shutdown plugin
    • Shutdown all behaviours on plugin shutdown
• Macros
  • behaviour!
  • behaviour_factory!
  • behaviour_fsm!
  • behaviour_transitions!
  • behaviour_types!
• GraphQL Behaviours Query
  • Behaviour GraphQL Types
    • EntityBehaviour
      • EntityInstances
    • EntityComponentBehaviour
    • RelationBehaviour
      • RelationInstances
    • RelationComponentBehaviour
  • Components
    • List Component Behaviours
  • Entity Types
    • List Entity Behaviours
    • Count Entity Behaviours
    • Count Entity Behaviours Instances
    • Count Entity Component Behaviours
    • Count Entity Component Behaviours Instances
  • Relation Types
    • List Relation Behaviours
    • Count Relation Behaviours
    • Count Relation Behaviours Instances
    • Count Relation Component Behaviours
    • Count Relation Component Behaviours Instances
  • Entity Instance
    • EntityBehaviours
    • Behaviour State (created, ready, connected)
    • EntityComponentBehaviours
  • Relation Instance
    • RelationBehaviours
    • Behaviour State (created, ready, connected)
    • RelationComponentBehaviours
• GraphQL Behaviour Mutations
  • Entity Instance
    • Mutation: Reconnect Behaviour(behaviour_ty)
    • Mutation: Disconnect Behaviour
    • Mutation: Connect Behaviour
  • Relation Instance
    • Mutation: Reconnect Behaviour(behaviour_ty)
    • Mutation: Disconnect Behaviour
    • Mutation: Connect Behaviour

Experiment: Async Reactive Streams

Problem

• Reactive stream subscribers are executed serially
• If a subscriber is blocking, the following subscribers are not executed
• Behaviours cannot create green threads without a runtime

Idea

The GraphQL API is async already. If the reactive streams would be async, it would be possible to avoid blocking completely and further increase performance.

Research tasks

• Annotate the main function with tokio main
• Make reactive streams async
  • Make subscriber "callback" async FnOnce
  • Observer: Run all subscribers asynchronously (await all)
• Make property setter calls async
• Make business layer async
• Is it possible to simplify the GraphQL subscriptions implementation?

This may work or not - the goal is to get an idea if it would be possible.

Security / Authentication

Goal

Secure the GraphQL endpoint

Crates

• crate: casbin-rs
• crate: jwt
• crate: actix-web-grants

Tasks

• Extend Configuration config/graphql.toml

  [authentication]
  enabled = true
  keystore_path = ""
  secret = ""
  issuer = ""
  audience = ""
  # Expiration in seconds
  expiration = 3600

• Create endpoint for login

  Endpoint	Description
  /auth/login	Returns a newly created JWT
  /auth/renew	Renew JWT

• Middleware: Check JWT token
• Documentation

Plugin: GraphQL Client

Tasks

• Setup Repository
• Setup GitHub Actions
• Implement Web Resource Provider
• Integrate Altair GraphQL Client
• Configuration plugins.toml
• Documentation

Name	Repository
inexor-rgf-plugin-graphql-client	https://github.com/aschaeffer/inexor-rgf-plugin-graphql-client

Plugin Binary Data: Web Resource

Tasks

• Implement GET endpoint

Download binary resources via HTTP

HTTP Method	Resource Pattern	Description
GET	/entities/uuid/{uuid}/{property_name}	Converts the Data-URL into a binary data and returns it as web resource
GET	/entities/label/*label	Converts the Data-URL into a binary data and returns it as web resource

Examples

Request	Entity Instance	Property Name
GET /binary/entity/dce4bd25-7b25-4a6a-8567-5429a2b3a101/data_url	id=dce4bd25-7b25-4a6a-8567-5429a2b3a101	data_url
GET /binary/entity/label/org/inexor/logo/data_url	label=/org/inexor/logo/{:property}	data_url

Plugin Scheduler: inexor-rgf-plugin-scheduler

Scheduler and Timer

Name	Repository
inexor-rgf-plugin-scheduler	https://github.com/aschaeffer/inexor-rgf-plugin-scheduler

This plugin provides a scheduler which starts tasks on a regular basis.

Use cases

• Update the server list every X minutes
• Reindex the texture directory every day at 04:00 am
• Start a game server at 08:00 h and shutdown it at 20:00 h because of the opening hours of the internet

Entity Types

Name	Component	Property	Data Type	Socket Type	Description
scheduled_job		schedule	string	input	Cron Expression
	generator	trigger	bool	output
timer		duration	number	input	The duration to wait before the next
	generator	trigger	bool	output