imgflo/imgflo

Node-based image processing with GEGL and Flowhub

imgflo

imgflo is an image-processing runtime built on top of GEGL which can be visually programmed using Flowhub.io.

It is used by imgflo-server, which adds a HTTP API for image processing.

imgflo is pronounced "Imageflo(w)".

Changelog

See ./CHANGES.md

License

MIT

Note: GEGL itself is under LGPLv3.

About

The imgflo runtime implements the [FBP runtime protocol] (http://noflojs.org/documentation/protocol) over WebSockets. It also provides an executable that can load and run a FBP graph defined as JSON.

The runtime uses GEGLs native graph structure, wrapped to be compatible with FBP conventions and protocols:

• All GEGL operations are automatically made available as imgflo components
• Each imgflo process is a GeglNode
• imgflo edges can pass data flowing between GeglPad

The edge limitation means that only ports with type GeglBuffer (image data) can be connected together. Other data-types are in GEGL exposed as a GProperty, and can currently only be set it to a FBP IIP literal. In the future, support for streaming property changes from outside is planned.

One exception is for the special Processor component, which is specific to imgflo. This component is attached to outputs which are to be computed interactively. Because GEGL does processing fully on-demand, something needs to pull at the edges where image data should be realized.

Deploying to Heroku

Configure. Flowhub user id is found in the Flowhub user interface (Settings or Register runtime)

heroku config:set HOSTNAME=YOURAPP.herokuapp.com
heroku config:set FLOWHUB_USER_ID=MYUSERID

Check the log that the initial registration was successful, and then save the runtime ID permanently

heroku logs
heroku config:set IMGFLO_RUNTIME_ID=MYRUNTIMEID

See "Run the runtime" for more detail

Developing and running locally

Note: imgflo has only been tested on GNU/Linux and Mac OSX systems. Root is not needed for any of the build.

Download it via git

git clone https://github.com/imgflo/imgflo.git
cd imgflo

Pre-requisites

imgflo requires git master of GEGL and BABL, as well as a custom version of libsoup. It is recommended to let make setup this for you, but you can use existing checkouts by customizing PREFIX.

Install node.js dependencies. Only needed for tests

npm install

For Mac OSX, you must install Homebrew

Building dependencies from source (recommended on Linux)

You only need to build the dependencies once, or when they have changed. See 'git log -- thirdparty'

git submodule update --init
make dependencies

If you are on an older distribution, you may also need a newer glib version

# make glib # only for older distros, where GEGL fails to build due to too old glib

Installing pre-built dependencies (recommended on OSX)

If you use this option, you must specify RELOCATE_DEPS=true in the commands below.

make travis-deps

Build

Now you can build & install imgflo itself

make install

To verify that things are working, run the test suite

make check

Run

To start the runtime

make run GRAPH=graphs/checker.json

This should open Flowhub automatically in your browser and connect to the runtime.

If the browser does not open, and you get "Operation not supported", add NOAUTOLAUNCH=1. Then you need to copy/paste the "Live URL:" into your browser manually to connect.

Registering runtime

imgflo can automatically ping Flowhub when it is available.

• Open Flowhub
• Login with your Github account
• Click "Register" under "runtimes" to find your user ID. Copy it and paste in command below

Set up registration

export FLOWHUB_USER_ID=MYUSERID

Finally, to run the Flowhub.io runtime use. You can customize the port used by setting PORT=3322

make run

If successful, you should see a message 'Registered runtime' with the new id. You should save this, and on subsequent runs on same machine use this id.

export IMGFLO_RUNTIME_ID=MYID

In Flowhub, refresh the runtimes and you should see your new "imgflo" instance. Note: sometimes a page refresh is needed.

You should now be able to create a new project in Flowhub of the "imgflo" type, select your local runtime and create image processing graphs!

imgflo's Issues

Missing documentation on how to open in Flowhub

On-canvas IIP editors and edge inspectors in UI

Currently the IIP editors and edge inspectors are cards in the left hand menu of noflo-ui. They however refer to items in the graph, and there is nothing establishing this relationship.
It would be better if these elements were directly on canvas.

Give information on accepted ranges

An example: color-temperature accepts values 1000-12000, but the UI doesn't know about this.

Remove letterboxing from video thumbnails

It would be great if a component could process this image and intelligently remove the black bars from the top and bottom.

Source: http://img.youtube.com/vi/Dg3lVNcQlhc/hqdefault.jpg

Show enums in UI

There are many ports/properties which takes an enumerated value in GEGL. These should be shown as a drop-down list in the user interface.

Support running as Flowhub runtime on Heroku

To avoid having to run imgflo locally for use with Flowhub, user should be able to host easily on Heroku. Should be able to just push an app to their own Heroku dyno, and then connect Flowhub

Should automatically register / ping with the runtime registry
Should respect the environment variables FLOWHUB_USER_ID / FLOWHUB_RUNTIME_ID

Runtime crash on client/UI disconnect

Testcase in branch runtime-crash-on-disconnect, commit 3153b49
Looks like it comes from libsoup...

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff70d5514 in ?? () from /usr/lib/libgio-2.0.so.0

Show preview automatically in noflo-ui on first preview message

Currently one has to click the eye icon manually to get the preview

Add deployment checks

Currently there are no automated tests that run in the actual deployment environment (just locally / on Travis).

Could for instance be a self-check which runs on start-up, and breaks Heroku deploy if failed.
Alternatively, or in addition, could be ran/started from the outside by a monitoring service.

Segfault on IIP removal

NoFlo integration: allow passing image buffers in/out

Currently the image buffers in imgflo can only come from other nodes in the imgflo graph. It is desirable to be able to pass in images into a imgflo graph from NoFlo (both browser and node.js), and to be able to retreive them.
As a first iteration it would be acceptable to only be able to do this on ports exposed on a subgraph.

Output could maybe be done usin g the same mechanism we use for the Flowhub interactive preview, on invalidation passing a HTTP URL which can be used to realize the image. An equivalent message could be sent to an inport. Then we'd have to implement image loading over HTTP in imgflo/GEGL.

Ideally the setup would be able to integrate with for instance noflo-canvas

Very closely related to #11

Allow to disable/enable operation(s) in graph

Should be possible to enable/disable one or more operations, using a boolean parameter for the graph.
Proposed implementation is to do a GEGL op with a boolean enable property. Should use no-op fast-path logic similar to https://git.gnome.org/browse/gimp/tree/app/operations/gimpoperationmaskcomponents.c

Support creating custom components in UI

GEGL supports writing operations in C, by implementing one of 4 base operation classes. It should be possible to create these directly in the UI and have the runtime use them right away. FBP protocol methods get:source and set:source
Might need to add a way to reload one (or all) operations in GEGL while the process is running. We should for safety probably stop/tear down graphs on such changes.

One could also create one or more component base classes which would allow writing components in JavaScript, for faster and more convenient prototyping. It would also not require a C compiler.

Implement queuing of requests in server

The design should be such that it is easy to replace an internal queue for one worker with a distributed queue like RabbitMQ.

NoFlo integration: allow driving property values using NoFlo components

Right now properties can only be set to literals (IIPs), not changed programatically by the graph. It would be desirable to be able establish connections to these ports and change them at runtime, especially from NoFlo (which has many general purpose components)

It would be acceptable as a first iteration to only allow this for ports exposed in a subgraph.

• Store the property edge information somehow. Can we use gegl meta op/node?
• Implement the inport/outport part of runtime protocol

Sliders for scalar values in UI

Currently the numeric entries in noflo-ui are just plain text fields. These should instead be sliders, allowing to easily see and change the value within the valid range (#20).

IIPs changes should only be set after a small idle period, say 100 ms.
It may be relevant to have logarithmic sliders in addition to linear. This will probably require more metadata in GEGL.

Allow to pick images directly from web service

In the general case (imgflo not running on same machine as UI), it would be nice to have image pickers that allows getting images by HTTP, or directly from Google image search, Picasa, Facebook gallery, Flicker, Creative Commons search.

This should probably be built with NoFlo, as there are also many non-imgflo uses for such widgets.

End-to-end tests for server, including performance

Currently we only run one image/graph in the server process test, and we do not validate the resulting image data.

The should be made into data-driven test, parametrized by the request url (API call). Each (example) graph should have a set of test images+properties, and a reference output for comparison.
We should also have instrumentation so that we can measure the time spent on each request.

GEGL itself has something similar under ./tests/compositions. It might be interesting to use their image comparison tool.

Support GIF files, at least static / single frame

Looks like it will require work in GEGL, as I do not see any file loaders/savers or it.
Animations will also require some special logic

Persist Flowhub information and ping

Currently imgflo registers a runtime with Flowhub, but the UUID is not stored anywhere. This makes it impossible to update the runtime when for example the user's IP or port number changes.

See how persistence is handled in noflo-nodejs https://github.com/noflo/noflo-nodejs/blob/master/index.js & https://github.com/noflo/noflo-nodejs/blob/master/bin/noflo-nodejs

Show file/image selector when running locally

When running imgflo on localhost, we should show a file picker in NoFlo UI for chosing paths to input files.

Interactive preview in Flowhub

Right now there is no way to get a preview/output of the pipeline in Flowhub, so one is "flying blind". Proposed is something like our "Processor" which can emit an URL with new output over Websocket on new data, or actually send image frames over WebSocket directly.

Add normalization component

Something similar to the normalize-contrast found in GIMP. Mostly for usage together with gradient maps

Color IIPs are interpreted as linear-light RGB instead of sRGB

Unexpected because the strings are CSS color strings, so colors come out lighter than expected. Also gives really poor dynamic range.

All parsing happens in GEGL, so needs to be fixed in GeglColor

Automate dependency building for Heroku deployment

Right now we use a "vendor lib" with our custom dependencies that was hand-assembled on Heroku based on:

• The Perl and XML::Parser steps from https://github.com/piotras/MDTs/blob/master/build-heroku-packs.sh#L19
• The "heroku-deps" and "dependencies" and targets in imgflo Makefile

Result was then uploaded to ftp://vps.jonnor.com/imgflo/

This needs to be automated so that we can easily update our dependencies, in particular the libraries which we use from git: libsoup, gegl, babl
It is also likely that we want to add more dependencies, particularly to enable more GEGL ops.

We should probably split this out into a separate git repo "imgflo-dependencies", so that we can have a separate Travis CI job which triggers the build on Heroku to produce the new vendor lib. This repo could then replace the git submodule we currently have under thirdparty/

Allow created graphs to be used as GEGL meta operations

GEGL has the concept of meta operations: operations implemented in terms of other operations. This maps very nicely to the subgraph concept in FBP/imgflo. It has always been the intention that GEGL could load a serialized description of, but the current GEGL XML format does not support that.

This will require:

• Upstreaming the .json FBP graph loading code to GEGL (currently in lib/graph.c)
• Implement construction of a GeglMetaOperation from this format
• Fix the GEGL module/operation loader to load the .json files found in GEGL_PATH as operations in library

Enable GEGL workshop operations

There are many GEGL ops only built when passing --enable-workshop

Scaling: allow to on sevice spread across many Heroku dynos

It could be that with #14 in place this would just work.

Runtime support for download/fetch images (JPG/PNG) from HTTP url

Should probably be done as ops in GEGL. Possibly using GIO, as the GIMP file loaders are now doing.

<pippin> having gegl:load accept URIs would be nice

Easy way to run locally on Linux/Windows/OSX

Currently one has to run imgflo as a server from commandline, register manually using imgflo-register, and manually open app.flowhub.io in browser. Which is not for everyone :)

Low priority for me, as delivering native solutions to multiple operating systems is tons of work. A proper solution would be to use Travis CI, or something that integrates with it, to continiously build and test a native app.

Public test server instance

Several people have asked for a server they can use for checking the project out without having to do install/setup themselves. Might also give us some nice robustness-testing...
Will require having sane rate (per host) and image size limiting functionality, with very strict values.

Investigate integration with client-side processing

To enable low-latency interactive image processing, it would be very beneficial if the image processing could be done client-side while changing parameters. When done changing parameters, convert it into a request URL for the server-side, which can be used to get the same image always - without requiring client-side processing.

Major challenge here is compatibility, as essentially two implementations are needed, giving same results.

Show default values in noflo-ui

Need to send the data over the protocol, and make sure the UI displays it

Implicitly add source-over OP when connecting multiple inputs

Right now, latest connection wins

IIP support for GeglColor

Histogram UI for output/edge preview

In addition to the preview of the image data it should be possible to see the histogram of the image. This is very useful when evaluating/developing signal processing algorithms, and sometimes for artistic use.

Allow imgflo to be used interactively in GIMP

After #9 is done, a natural extension is to see how one could integrate imgflo into GIMP, with Flowhub acting as a node-editor extension to it.
Would need to

• Host the imgflo runtime in GIMP
• Pass buffers/node referencs from GIMP to imgflo, and back

This may not be possible to achieve with the GIMP plugin API (as it is tailored for one-shot, non-interactive use), and would then require hooking into GIMP as a module/tool.
It may also require infrastructure work in GIMP to allow to get access to the GeglBuffer/GeglNodes for layers,masks etc.

Scaling: allow to store images in S3

Currently images (inputs and processed) are only stored on local disk. To be able to scale horizonally, need to be able to store to a location shared between all nodes like Amazon S3

Show units for IIP values

Right now user does not know values should be pixels, meter, points or anything else (though most things are pixels)

Requires some work in NoFlo UI, and maybe in protocol. It may also be that units are not provided by the GEGL ops, and that we need to add it or apply heuristics.

Out-of-box deploy support for Flowhub runtime on Heroku

Like with the sever, it should be possible to just push imgflo to Heroku and access it as a runtime in Flowhub.
The runtime is ideally automatically registed with Flowhub when one does this.

Should respect Heroku environment variables for at the user ID.

Not using a crop element can cause out-of-memory

The Processor node used interactively should probably have max height+width properties, and the HTTP server as well.

Allow hsl() values for colors

Currently one can use HTML #rrggbbaa colors, but not hsl which designers generally prefer.
http://dev.w3.org/csswg/css-color/#the-hsl-notation

Should be implemented in GeglColor directly.

Preview buffer at edges

It would be super-useful to get a small preview image of how the image looks like at each edge, maybe integrated with the edge inspector (and sending thumbnails as data URLs).

FilterForge is doing something like this:

Allow to use graphs saved in Flowhub project directly in server

Right now the sever makes available the graphs found in "examples" folder. Flowhub however saves to its own git repository when using the git integration for a project.

It should be possible to tell the server to use this project directory to look up assets. The project could also contain settings for the server. The server should be able to pick up changed/new graphs at runtime.

Full, tested, implementation of graph/network protocol

Need to handle

• remove edge, remove iip
• stop network
• background processing of network when started
• multiple graphs

Error conditions also need to be throw over to UI side

Support multiple graphs/networks

Currently can only handle one graph, though NoFlo UI protocol specifies that there may be multiple.

Invalidation mechanism for cached processed images where the URL data have changed

Currenltly will not re-fetch the resource

Can't install imgflo

Hi,

I'd really like to try your prototype, but can't make it work until now.

Here are the steps I followed so far to install it :
1/ Manjaro (Arch based) Linux system with nodejs 0.10 and GEGL 0.2
2/ git clone https://github.com/jonnor/imgflo.git
3/ go to cloned dir
4/ npm install websocket serialport commander pkginfo coffee-script node-static chai mocha
5/ start server with "node index.js" ?
6/ localhost:8080 in Firefox : console log : "NetworkError: 500 Internal Server Error - http://localhost:8080/"

Errors with serialport:
Failed at the [email protected] install script

I tried "./configure", but no such file, and "make" too

make: /home/nylnook/Crea/dev/imgflo/install/env.sh: commande introuvable
/home/nylnook/Crea/dev/imgflo/install/env.sh gcc -o ./bin/imgflo bin/imgflo.c -I. -Wall -Werror -std=c99 -g 
make: /home/nylnook/Crea/dev/imgflo/install/env.sh: commande introuvable
Makefile:22: recipe for target 'imgflo' failed
make: *** [imgflo] Error 127

I suppose the "install" or ".install" directory is missing ;p

Please give me some details, or complete the README ;p

Map port datatypes to recognized types

Currently the MicroFlo runtime gives "raw" GObject types for its ports, which are not recognized by the Flowhub UI. These should be mapped to the Flowhub-compatible equivalents.

List here:

https://github.com/noflo/noflo/blob/95a833e09a1e006ab28959888d3a29a8ccd3299a/src/lib/BasePort.coffee#L11

Support subgraphs

Should ideally be done using #9, but could be we can implement this quickly without it