cyberbit/telem

Trivial Extract and Load Engine for Minecraft

Telem - Trivial Extract and Load Engine for Minecraft

Tired of creating complex logic to monitor inventory or production? Want something more modern, modular, and scalable? Want something that can empower a dashboard like the screenshot below? You have come to the right place.

Requirements

• ComputerCraft (Tweaked and Restitched also supported)
• http access for installer and certain adapters

Note: These docs are being actively written so please excuse my dust! If you have questions, post them in this discussion.

Install

This runs the installer. You have a choice to install minified, packaged, and source versions.

wget run https://pinestore.cc/d/14

Usage

Please visit telem.cc for full documentation.

local telem = require 'telem'

local backplane = telem.backplane()                  -- setup the fluent interface
  :addInput('hello_in', telem.input.helloWorld(123)) -- add a named input
  :addOutput('hello_out', telem.output.helloWorld()) -- add a named output

-- call a function that reads all inputs and writes all outputs, then waits 3 seconds, repeating indefinitely
backplane:cycleEvery(3)()

-- alternative threadable option
parallel.waitForAny(
  backplane:cycleEvery(3),
  
  function ()
    while true do
      -- listen for events, control your reactor, etc.
      
      -- make sure to yield somewhere in your loop or the backplane will not cycle correctly
      sleep()
    end
  end)

Input Adapters

• Hello World (testing)
• Item Storage
• Fluid Storage
• Refined Storage (RS Bridge)
• ME Storage (ME Bridge)
• Mekanism Fission Reactor
• Mekanism Fusion Reactor
• Mekanism Induction Matrix
• Mekanism Industrial Turbine
• Custom Inputs
• Secure Modems ✨
• More to come!

Output Adapters

• Hello World (testing)
• Grafana* (Grafana Cloud or self-hosted)
• Basalt labels
• Custom Outputs
• Secure Modems ✨
• More to come!

*You will need to set up a free account on Grafana Cloud or provision a self-hosted Grafana + InfluxDB + Telegraf instance to use the Grafana output. A more detailed guide for Grafana is planned for the future.

telem's Issues

Improve build script with auto-discovery

Adding individual files to the build script is bad. Auto-discovering files is good!

Some ideas:

• Search source directories for Lua files that are well-defined (maybe with a top-of-file header?)

Actually that's my only idea.

Speaker output adapter

Ideas:

• Simple heartbeat: Speaker adapter at end of list indicates output has completed
• Simple indicator: Filter for metric, measuring against a minimum and maximum. Minimum = low tone, maximum = high tone. Trigger on output
• Panic indicator: Filter for metric, if value outside a given range (or missing), rapidly trigger a note sequence until recovered
• probably other annoying things :)

Publish to RequireDB

Cool repository, should add: Ale32bit/RequireDB

Add fusion reactor adapter

ChartLine: Resizing window does not update chart area

Reported by juh9870 in Discord.

When a window (in this case a monitor) is resized, the plot area does not automatically pick up the change and becomes cut off.

Original monitor configuration:

Removing one column of monitors from the left:

A possible fix would involve checking the size of the assigned window at the start of every write call and updating internal structures. In addition to this, a function should be provided for triggering a layout refresh on the adapter manually.

Filesystem output adapter

Dump point-in-time metrics to filesystem with configurable filename and format (CSV, JSON). For multiplayer worlds, this could be used as a way to enable out-of-game resource leaderboards with a script that scrapes all the computer filesystems on a schedule.

Add support for custom adapters

Concept:

backplane:addInput('custom_in_1', function ()
  -- do stuff

  -- can also return MetricCollection, this is sugar
  return {
    custom_metric_1 = 123,
    custom_metric_2 = 4.999127
  }
end)

backplane:addOutput('custom_out_1', function (metrics)
  -- metrics = {
  --   {
  --     name = 'metric',
  --     value = 123,
  --     unit = 'unit',
  --     adapter = 'adapter',
  --     source = 'source'
  --   },
  --   ...
  -- }

  -- do stuff
end)

Implement middleware

EDIT: Now that Fluent exists, a direct implementation of middleware may be possible.

Things middleware should be able to do:

• Renaming (prefix/suffix metric names and labels)
• Filtering (isolate metrics by property matching)
• Ordering (reorder metrics)

Two kinds of middleware:

• MetricMiddleware. These operate on Metric objects. Renaming and Filtering may be written at this scope. This design enables intelligent chaining to execute in O(n) time.
• CollectionMiddleware. These operate on MetricCollection objects. Ordering may be written at this scope.

Publish to unicornpkg

I'd love to get this on unicornpkg but the installer seems broken at this time :(

Creating adapters against non-existent peripherals is not recoverable

Need to make adapters resilient against peripherals not existing when booting up. A good fix may be to execute peripheral.wrap(id) before read() or write() calls if the stored peripheral is determined to be nil.

Add CloudSolutions output adapter

See CloudSolutions.

• KV output would be fast and secure storage, isolated per-gamer
• StatsD bridge would be a Grafana-alike with better efficiency

OpenComputers support

Need abstraction layer for:

• peripherals
• filesystem
• http
• wireless? (rednet or something else)
• monitors

I did some brief testing, it looks like it is possible in theory, need a lot of abstraction but managed to get a janky version of inventory reading working.

Using Ocelot Desktop for emulation.

Add secure modem input and output adapters

There needs to be some kind of in-game communication possible within Telem for coordinating systems across dimensions. Each computer attached to a peripheral can directly output its metrics to whatever local display it wants to, but having a home base that collects everything in one place is going to be important.

The input adapter would be a very specialized design as it needs to listen to events in order to collect them. The output would work like a normal output adapter, just screaming metrics into the void.

The input adapter sends requests to output adapters. The output adapters listen for requests.

Design notes:

• Should modem inputs overwrite adapter? Strictly speaking it should, but I think it makes more sense if it preserves the remote adapter and source names. Maybe prefix adapter? If remote input adapter is chest_1 and modem input adapter is storage_terminal, the final adapter is storage_terminal:chest_1 This has been implemented at the Backplane layer. For every input adapter that passes through cycle(), every metric will have <adapter>: prefixed to its adapter property.
• Secure Modem has been implemented with ECNet 1.x for the time being. I am planning on releasing a beta version of this adapter while I work up updating to ECNet 2.

Prometheus Pushgateway output adapter

This is technically for further supporting Grafana, but should support anything that can ingest Prometheus.

Add computer ID as a label

Precise storage input adapter

• gather high level of detail on each slot of an attached storage
• preserve slot IDs
• no deduplication

Gist output adapter

The API to make this work is a little annoying, but could be a fun way to export for systems that don't support webhooks but can read from a URL.

Basalt output adapters

Basalt added a Graph, now I want to use it.

• Label output adapter
• Graph output adapter

these should all require filtering at load time

Plotter output adapter

Now that Plotter is at least mostly stable, I want Telem to use it. Like this!

Metadata input adapter

Gather metrics about metrics!

Concepts:

• count of input adapters
• count of output adapters
• last count of output metrics
• count of cycles
• count of faults
• last input read time
• last output write time

SecureModem: serialized message is too large

When a secure modem output adapter tries to send a large (nearly 1,000 metrics) collection, the adapter crashes the backplane:

The solution may be to send multipart responses, where the requesting input adapter assembles the full collection when all parts are received. If the input adapter times out waiting for all parts, it can either stop listening to parts and assemble what it has received, or throw an error.

Alternatively, I can create a custom serde that is more efficient with keys and utilizes compression.

First-party HTTP output target

• Public ID + Secret Key authentication
• Limited rate per ID
• Limited storage size per ID
• Automatic bookmarkable live charts for provided ID
• History? (would eat storage probably)

Add Energy Detector input

See https://docs.advanced-peripherals.de/peripherals/energy_detector/.

Offers transfer rate and transfer rate limit.