kuriringohankamehameha/mini-poller

mini-poller

A simple demonstration of how polling can be done asyncronously as a service. This uses async methods from Python, publishes messages into a Message Queue (RabbitMQ is used here) and inserts persistent content into MongoDB.

---

Instructions

Before running this program, you need to have Docker installed on your machine.

Start all containers for this service using:

cd mini-poller
docker-compose up -d

All necessary setup will be done automatically by Docker.

A sample web view is provided using Flask, to simply keep the main container alive. Going to localhost will simply give you an index page.

Now open 2 different terminals, for spawning both the producer and the consumer

You need to connect to the app docker container from both the terminals:

docker exec -it mini-poller_app_1 /bin/bash # On both terminal 1 and terminal 2

On the first terminal, start the consumer:

cd /usr/src/app
python poll_consumer.py

On the second terminal, start the core service:

cd /usr/src/app
python poll_service.py

Notice that I start the consumer first, so that it is ready for any published messages from the poll service.

Immediately, you'll start observing the polling changes in action!

---

Methodology

• There are two programs:

1. poll_service.py: Consists of the main service, which does the actual polling and publishes messages
2. poll_consumer.py: A listener which retrieves incoming messages and stores them into DB.

The poll service basically keeps requesting a list of API endpoints and publish messages into RabbitMQ.

The poll consumer will keep listening to incoming messages on that channel and insert any records into MongoDB directly.

---