thecodeteam/mosaicme

Micro service application that retrieves images from Twitter feeds, stores and process them to build a mosaic based on a selected picture.

mosaicme's Issues

Check if needed binaries are available

For example, the engine service should check if the convert and metapixel binaries are available in the system.

ECS Cluster Deployment + add to github

Engine and Sync components log to stdout

Engine and Sync components should send the logs to stdout, instead of redirecting them to a log file.

More info: 12-factor app.

Could not get container for s3

Hi, I tried to deploy the app and use Amazon S3 buckets to play with it. It says that it can't find a container for S3... Do I need ECS service also with Amazon S3? Or am I missing something? It's my first time trying a Docker app.

Remove Puppet bits

Since we are not using Puppet anymore to provision hosts, let's remove the existing Puppet bits from the repository.

Set up a CI/CD pipeline

Set up a CI/CD pipeline to automate testing and artifact delivery.

Some CI system that we can consider using: Travis, Circle, Semaphore

The artifacts (Docker images) will be pushed to Docker Hub.

Enhance the documentation

Better documentation of the project is needed. It would also be very useful to have some sequence and data flow diagrams.

Vagrant up not finishing

Hi,

just tried to vagrant up this app but it did not work unfortunately. He had issues doing the configurations with puppet. Any ideas on how to try and use the app?

br,

Daniel

Avoid using supervisord as the main container process

Currently, some components are using supervisord to run more than one process inside the container. Instead of that, each process should be started in a different container. Even though they could share the same Docker image.

Update the listener/collector component

Update the Listener component to the new business logic.

The Listener should do the following:

1. Listen to the Twitter Streaming API for tweets containing the specified hashtag (e.g. #mosaicme)
2. When there is a new tweet, extract the tweet data (i.e. author's name, handler, picture URL). If the tweet has no picture attached, discard it.
3. Send a message to a specific queue with the extracted information.

Example:

{
    "twitter_handler": "johndoe",
    "name": "John Doe",
    "img_url": "http://example.com/394756138745.jpg"
}

Alternative source to download raw images

Right now the Engine relies on the raw bucket and it needs a minimum amount of images to start building mosaics. If there are not enough images in the raw bucket, the Engine will fall into an active loop to check and download images until the threshold is met.

It should have a fallback mechanism when there are not enough images and download them from a known source. For example, it could download images from the following website: http://www.vision.caltech.edu/Image_Datasets/Caltech256/

Standardize component configuration

At this time, MosaicMe components read the configuration in different ways. Which makes it confusing when we are developing and using them.

Let's standardize the way components read their configuration. According to the 12-factor app principles, the configuration should be obtained from environment variables.

Therefore, let's use the existing example.env that is used by the collector and cacher components and extend it with any needed env variable so that it is used by all the components.

Create independent dependency manifests

Create an independent dependency manifest (e.g. requirements.txt for Python) for each component.

Dockerize the sync component

Create a Dockerfile to generate an image of the sync component. Also provide documentation about how to build and use the image.

Create a unified Docker Compose file

Create a Docker Compose file to deploy all MosaicMe components in one shot.

Create tests for all components

Create test to increase the code coverage and robustness of all the components.

Show the timestamp in logs for all components

Some of the components are not showing the timestamp in the logs. Make sure they do.

Dockerize the engine component

Create a Dockerfile to generate an image of the engine component. Also provide documentation about how to build and use the image.