inakianduaga/kafka-image-processor

Process client images using Kafka & reactive architecture

kafka-image-processor's Issues

Fix CycleJS websocket driver

CycleJS can't seem to push to the websocket, always throwing

WebSocket is already in CLOSING or CLOSED state.next @ websocket.ts:13Stream._n @ core.js:909Stream.shamefullySendNext @ core.js:1688next @ index.js:25next @ index.js:50Stream._n @ core.js:913MapOperator._n @ core.js:729Stream._n @ core.js:909MapFlattenInner._n @ core.js:635Stream._n @ core.js:909intervalHandler @ core.js:260 150:1 GET https://unsplash.it/150/150?image=462 404 (Not Found)

next: outgoing => ws.send(outgoing), // debug error

However a simple test shows the websocket connection should be working from scratch:

var ws = new WebSocket('ws://localhost:9000');
ws.send(JSON.stringify({"test": 123}));

Setup Avro schemas

We want

Frontend:

• Has toggle to distinguish between V1 or V2. When V2 is chosen, it will forward the filter type to be used to process the image

UI-Backend:

• This will produce using either V1 of the Avro schema (where the filter type is not specified) or V2 of the schema, where the filter is specified

Image Processor:

This will always use V2 of the schema to read the data, with Avro schema evolution filling the gap for V1 produced data.

So what we will test initially is the case of "old producers", which are pushing data into the stream with an older version of the schema, and consumers still can read data properly.

TODO

• Avro schemas for the UI-backend for V1 & V2, and copy V2 over to the processor
• Install Avro Hugger on UI-backend / processor to be able to hydrate from the schemas

Add schema registry UI visualization

Update docker compose and hook this up to the schema registry
https://github.com/Landoop/schema-registry-ui
Then the schemas can be visualized

UI: V1 vs V2 selector

Related to #15

We want to introduce a toggle to select the filter to be applied. If it's not enabled, we will
Has toggle to distinguish between V1 or V2. When V2 is chosen, it will forward the filter type to be used to process the image

Setup image processor

• Setup SBT to run the processor hooks (maybe instantiate class manually instead of using dependency injection on the main method)
• Add binding to kafka topic, print to console that it's receiving them
• Add WS play to fetch images from the url, republish to kafka topic
• Add 2nd processor to listen to downloaded images, run processing to apply filter to images
• Push image binary to kafka so it can be downloaded by the frontend

Improve websocket-Akka integration

Play 2.5 changed a few things,

• Use dependency injection of the actor system

https://www.playframework.com/documentation/2.5.x/ScalaWebSockets

• Remove random websocket id / actor mapping, we don't need it, we'll use the actor's own path as the unique identifier to be able to recognize it.

• Add custom JSON data types for the InEvent / OutEvents

Switch to Akka streams kafka to do the processing

Regular bare API & kafka streams are not working too nicely, let's give Akka streams a shot

Add kafka connectors visualization

http://kafka-connect-ui.landoop.com/

Seems to display what things are hooked to Kafka, which is nice to understand the topology

Add Kafka topics visualization

http://kafka-topics-ui.landoop.com/

To visualize the topics coming/going to kafka (also allows to see the system topics)

Avro doc generation

Check out
https://github.com/ept/avrodoc
http://avrodoc.herokuapp.com/#/schema/com.example.avro.EmailAddress

Enable debugging for docker container

Seems it's straightforward, just need to run sbt with the option and open the port for external connections

http://daodecode.com/2013/04/19/debug-an-sbt-plugin-in-intellij-idea/
https://github.com/Tapad/sbt-docker-compose/blob/master/examples/basic-with-tests/docker/docker-compose.yml

Add frontend schema selector

Frontend should

• Have the option to choose between specifying the color or leaving it undefined (with a toggle).
• Backend should apply a monochrome filter for when there is no information passed, or a color filter when it is passed
• Backend image processor should save a schema with V1 if the filter information is not there, or V2 if the filter info is there

Add Kafka bindings to the client backend

We need to hook listening events to Kafka topics so we can stream the data to the corresponding websocket (such as when the image processing is done, processing stats, etc)

This needs to be done on app startup, Play mentions this can be the constructor of any dependency injected class

https://www.playframework.com/documentation/2.5.x/GlobalSettings

Add Avro integration

Registry:

• We need to setup an Avro schema registry
• Save schema V1 with only image URL
• Save schema V2 with image Url + Filter Type
• Maybe a 3rd schema that adds some property automatically using schema evolution

Setup schema registry

Example of confluent platform
https://github.com/confluentinc/cp-docker-images/blob/master/examples/cp-all-in-one/docker-compose.yml

Setup Docker compose

Setup docker compose to run

• Kafka container
• SBT container w/ play app
• CycleJS UI running on webpack, mapped to port 80.
• Image processor sbt app as well

After docker compose is done

• Run the client to see if we can get a valid websocket connection to the server
• Push data from client to server and check server logs to make sure we can read the data.

Improve documentation

Avro:

• Add section about Avro
• How choosing the filter on the frontend switches the avro schema used in the backend to test schema evolution

Investigate AvroHugger

Right now we do the whole deserializing / serializing from an Avro GenericRecord by hand. It would be nice to be able to hydrate directly into a generated Avro case class from AvroHugger. We have to see how this can be done

Switch to Kafka stream for processing instead of using manual Consumer/Producer API

http://docs.confluent.io/3.1.1/streams/developer-guide.html#writing-a-kafka-streams-application