A simple demonstration of how polling can be done asyncronously as a service.
This uses async
methods from Python, publishes messages into a Message Queue (RabbitMQ is used here) and inserts persistent content into MongoDB.
Before running this program, you need to have Docker installed on your machine.
Start all containers for this service using:
cd mini-poller
docker-compose up -d
All necessary setup will be done automatically by Docker.
A sample web view is provided using Flask, to simply keep the main container alive. Going to localhost
will simply give you an index page.
Now open 2 different terminals, for spawning both the producer and the consumer
You need to connect to the app
docker container from both the terminals:
docker exec -it mini-poller_app_1 /bin/bash # On both terminal 1 and terminal 2
On the first terminal, start the consumer:
cd /usr/src/app
python poll_consumer.py
On the second terminal, start the core service:
cd /usr/src/app
python poll_service.py
Notice that I start the consumer first, so that it is ready for any published messages from the poll service.
Immediately, you'll start observing the polling changes in action!
- There are two programs:
poll_service.py
: Consists of the main service, which does the actual polling and publishes messagespoll_consumer.py
: A listener which retrieves incoming messages and stores them into DB.
The poll service basically keeps requesting a list of API endpoints and publish messages into RabbitMQ.
The poll consumer will keep listening to incoming messages on that channel and insert any records into MongoDB directly.