ernstl/flowd

An inter-language runtime for flow-based programming (FBP)

flowd's Issues

launch/tcp-server: Add distinct connections handling, connection notifications

Add TCP server component

Allow creation of all kinds of TCP-based servers for FBP network input.

flowd: Remove args passing via component metadata

Reasons:

• is very limited with regards to allowed characters
• cannot even bring in two arguments
• comma is not possible to bring through
• no quoting allowed

Thus, remove in favor of IIPs (#43), which are now implemented.

flowd/launch: Add IIP support

Goal: No more argument passing to components necessary anymore.

• Argument passing via component metadata difficult using .fbp format, see compMeta specification in currently-used .fbp parser

Add TCP keepalive feature

Quickly detect connection having gone away.

• Useful: tcpkeepalive package
• Add to tcp-server component.
• Add to tcp-client component.

Add error reporting and debug reporting

Goal: To allow launch instances and their components to run without a terminal, ie. in the background.

Requires:

• DONE errors from component -> stderr capture
• WONT errors which launch encounters (network etc.)
  • also handle error that child has exited or panicked
• error output:
  • stderr (default and overrided by -debug)
  • syslog (priority, daemon name etc.)
  • send error frame with own output port -> centralized flowd error reporting
  • external program / script
• same for debug logging
• change meaning of flag -debug to require a debugging destination
• new flag -err

Add TCP endpoint support

flowd: Add way to give name of port on other side so that a frame's destination Port can be set

duplicate of #13

Add CI and badge

• Requires #5 (really?)
• Add link to godoc documentation for libflowd
• Add waffle stats

flowd: Add support for systemd socket activation

systemd socket activation is roughly the functionality of inetd. It passes in a file descriptor, which is the network connection.

Where could this be useful?

Useful resources:

• example in Go
• example in Ruby describing configuring systemd as well
• example in C

Implementation:

• May be interesting to start flowd on demand, but not really possible since flowd has no knowledge of network connections.
• A component like tcp-server would need the file descriptor.
• Would need to add functionality to pass that from flowd to a component using special marker in the .fbp network definition as an IIP.

Resolution:

• Not worth the effort ATM.

Add port EOF detection possibility for components

Send in a frame of type control.EOF or something and note name of port somewhere. Or, end of stream, rather.

What needs to be changed:

• DONE flowd has to send port-closed notification frame to downstream components if a component has exited (normally), eg. file has been read in full.
• DONE libflowd.ParseFrame() should forward io.EOF on first Read() so that ...
• DONE flowd knows this is a normal EOF and can react accordingly.
• DONE flowd should handle EOF from component differently than non-zero exit code.

Add sort component

• To know if all data has arrived, this requires #33
• First version using brackets can be done now

Add port unconnected detection possibility for components

• meaning: A component can detect if port X is connected or not.
• Since importing the UnixFBP concept for the alpha version, this is easily possible since:
  • the component has to open the ports = named pipes itself and
  • if there is no component on the other side of the named pipe, it will block on opening.

flowd: Add component health monitoring and crash recovery

• Detect if sending of IPs into component STDIN is blocked for longer than timeout.
• Handle exiting of child - restart or shut down network?

Question: What to do with any possibly missing IPs? - Solution: Once the IP is written to component Stdin, it is the problem of the component. Even if a component would acknowledge each IP, this could not prevent crashes.

Question: How to return the component to the same state as before? Certainly, some components are stateful. Which IPs to re-send? What if the component crashed in the middle of a chain of IPs, which depend on each other and only make sense as a whole? Maybe implement a safe-keeper component, which caches a chain of IPs until it gets acknowledgment of successful completion from downstream.

• Probably useful to do in conjunction with Persistence sub-protocol in #55.

Add port closed detection possibility for components

Graceful shutdown

• On component exit or if all is said and done, all data processed etc.
• Goes hand in hand with launch instrumentation, monitoring, health checks etc.
• flowd monitors and keeps connection with all its launch subprocesses
• If all upstream/input ports have been drained = end of stream or deliberately closed and component has exited, and all upstream processes have been closed down.
• How to send signals to components -- first send INT, then TERM, then KILL.

Changes necessary:

• TODO Close STDIN of components if all their input ports have been closed.
• TODO If component has exited abnormally, then send shutdown frame to all remaining components, timeout and then kill processes.

Add acknowledgment of receipt for frames

• Goal: No loss of information packets.
• If connection is lost, try to re-establish and re-send unacknowledged frames.

flowd, launch: Add destination port naming during routing

Add way to give name of port on other side so that a frame's destination Port can be set and verification be done according to port name.

Recognition of correctly addressed frames on input endpoint. How to do this? In launch in the URL specification.

Currently detected using the name in the Port frame header field. If keeping discrimination via some header field: Who should set the correct Port header field - sender or receiver? Does the sender have to know the name of the target port? How is this done in other protocols, for example SMTP with regards to its From and To header fields.

Add tls-server component

Add deploy helper

• Distributes binaries to destination machines (with correct architecture) including dependent files (TLS certificates, own configuration files) according to a network description file.
• Not really necessary any more since distribution of part of a network is done using SSH and therefore not part of the network definition any more.
• Has to be done using external script or system.
• WONT Create example of such an external script. Won't be universally useful; but can use output of flowd -deps to copy / scp list of these components over to target machine(s) and restart systemd service for the network via ssh.
• DONE Add example of systemd service definition for a flowd-based network.
• Compilation of components is outside the scope of this helper.
• Might be useful during development of tls-server #62 and tls-client #68
• Not really in scope for flowd, but list of components can be output.
• DONE Add -deps flag for flowd displaying the list of component dependencies for the given network.
• DONE Add dependency information as metadata into .fbp file; display that in -deps flag code.

Add multiple port support

Add TCP endpoint reconnection feature

Add README file

Add a descriptive project info file.

Add array port broadcast

• Sending a frame to output port out[*] should copy it to all defined out[0..n] output ports. The responsible launch instance should do this.
• Solution: Application can do this by itself by calling libunixfbp.ArrayPortMemberWriters("OUT") and then serializing into all writers.
• A function in libflowd could serialize once into a buffer then write the bytes into the given multiple writers.

Add centralized debug and error output collection

Probably best done as extension to orchestrator.

• Own connection/port from each component resp. its launch instance back to the orchestrator.

Add TCP output endpoint reconnection feature

Add health monitoring of components

Probably best done as extension to orchestrator.

• Own connection/port from each component resp. its launch instance back to the orchestrator.

Add instructions for running the examples

Add partial array broadcast

• After adding array broadcast in #14, enhance that feature to allow sending a frame to output ports out[2:max] and out[2:4].
• Result: Can easily be done on the application level, no runtime support necessary.

Add orchestrator (flowd)

Launches a network of components based network description file.

• Support network description in DrawFBP formats - it has an XML-based format (.drw) and a .fbp format
• Useful .fbp parser library
• The fbp parser from NoFlo
• (Abandoned) Network description file probably best made in .toml file format
• Name of binary = flowd
• Calls all preparatory helper programs (deploy, TLS certificates generator, etc.) or sub-routines

Add possibility to re-use non-FBP programs

Need possibility to pass command-line arguments and pipe something into it, then catching output. Recently removed support to pass args via metadata (#45), but this was limited anyway.

Implementation:

• switch in meta-data to pass the IIP as command-line argument instead? Like iip=args?
• launch: Read all output from command into one IP, send this on into FBP network for further processing.
• meta-data argument framing=off ... but how to handle multiple incoming IPs? Line-separate them? Zero-byte-separate them?
• Or handle this use-case as a regular FBP component, which launches the external program and handles everything?
• Each IP would have to start a new instance of the external program. Or maybe each bracket series of IPs..?

Notes toward a solution:

• needs parameters = IIP
• maybe the program creates only output, or maybe consumes only input (needs to be handled separately and configurable separately)
• streaming mode vs. 1 call per each input IP / frame / bracket (depends on when the program starts working - only after its STDIN hits EOF = closed?) -> need to call anew for each IP/frame/bracket
• maybe configurable parameters for each call (-> needs own port for this)
• too much configuration and if 1 subprocess start per IP is required, then not useful to do in launch, therefore handle this in a component extcmd or pipecmd or similar.

Further notes:

• Extend launch with flag -external=[stream/perip] to enable this feature.

flowd: Add support for XML DrawFBP .drw format

• DrawFBP
• XML schema
• Upstream problem (2.13.23): element <drawfbp_file> closed by </net>

libflowd/framing: Make frame.Body []byte instead of *[]byte

Many useless conversions and indirections and more complicated code because of it being a pointer of []byte. Sounded like a good idea in the beginning (moving data around as a pointer), but so far incurred many useless temporary variables and conversions etc.

May be reversed in the future, but so far this seems like a sensible choice to make it []byte.

Add Unix domain endpoint support

• (DONE) Including Linux's abstract ones with @ at the beginning
• (DONE) Make @/flowd/1234 work
• (DONE) Make @flowd/1234 work
• (DONE) Make regular path-based ones work
• (DONE) What to do with publishing the port? Avahi only supports TCP and UDP. Currently disabled for Unix domain endpoints.
• (DONE) Add unix2stdout

README: Update examples to reflect TCP and UNIX endpoints

Add instrumentation of FBP networks

Can inspect state of component, statistics, etc. and its launch instance during runtime.

• Probably best done as extension to orchestrator/flowd.
• Own connection/port from each component resp. its launch instance back to the orchestrator.

tls-client, tls-server: Add component certificate generator

Generates certificates for mutual authentication of component connections, one for each component, or even one for each port. So this tool must be flexible.

• To be based on openssl ca tool or tinyca.
• It gets network information from network description file.

Not required any more as described initially:

• flowd has no notion of LAN network connections any more.

Resolution:

• Might be useful during the development of tls-server (#62) and tls-client (#69).
• Might scan an .fbp file for tls-server and tls-client components.
• There are already some useful automation scripts for quickly setting up a CA and quickly signing some certificates. Closing an WONTFIX.

Add capability to receive from a specific named port

• Idea: Realize as component, which takes command via a port to release all its frames now until EOF, like a controllable flood gate. Call it gate.

README: Add information on how to write a component

Content ideas:

• Components should forward existing headers from the incoming frames/IPs, because downstream connections might lead to a loop back to the sender requiring a header field present for correlation, like for example a TCP connection ID.

Add multiple ports support

launch:

• requires port flag parsing
• requires new flag to turn framing/routing on or off
• requires internal frame routing to/from right ports

examples:

• update

Fix launch tests for disabled datagram protocols

Add ssh-client component

• Only client functionality.
• Using Golang SSH package.
• Maybe use sharkey tool in conjunction with SSH. Also possibility to forward ssh-agent connection to remote server (with cmd component).

NOTE: Already possible using cmd component using the security-audited OpenSSH.

Migration of components / spread of FBP network

Add compression and decompression components

Options:

• Using TLS compression (nope, see CRIME attack) - though would have been wonderful to offload compression to TLS and have it compress and encrypt all connections between the components.
• Header field Content-Coding, see the HTTP spec for a useful list.

In flowd, there is no negotiation like in HTTP. So it might be tricky to introduce new compression methods. OTOH, at least inside own organization / application, the launch program is always homogeneous.

For organization traversal, a special component would have to do cross-organization authentication anyway and could also do any necessary re-encoding.

Golang packages:

• compress stdlib packages = gzip, lzw, (de)flate, zlib
• kothar/brotli-go (need to update the vendored version to latest) - or use this version which is more up-to-date
• dsnet/compress
• JoshStrobl/xzed = xz/lzma2 as binding to liblzma

Ideas:

• Create two compress and decompress components, which do the most common and the most best-compressing algorithms.
• Don't do any negotiation - this has to be designed by the network programmer. Therefore no need to mark the compression algorithm in the frame header.
• Compress only body.
• Header cannot be compressed - only the extension headers maybe if these are removed from the frame and compressed together with the body.

Fix multiple frames per Read from component Stdout

Probably need a Scanner with a custom scan function.

Add nanomsg-client component

Since the ZeroMQ successor nanomsg is v1.0-production now, inclusion of its socket capabilities would be a good fit.

Nanomsg bindings for Go see this list.

flowd: Add full test coverage or rewrite using TDD

• Fix launch tests for disabled datagram protocols

Status:

• Part test coverage now from commit e320e91.

flowd: Show line-based output from launch instances with name prefix

Add http-server component

Goal: allow creation of web API.

Steps:

• DONE Depends on #36 (add TCP server component)
• Ideas from WebsocketReceive and WebsocketRespond
• Uses http-id or req-id header field, same idea as in tcp-server component.
• Data flow: tcp-server+conf -> HTTP+conf -> Router+conf -> Handlers -> HTTP in -> tcp-server in
• Ideas for router from Gorilla mux
• Sets header fields and basic connection info of HTTP request as frame headers, thus routing can then continue using generic router on header field.

Add message framing format