peterwade153/house-bob

A django application for scraping properties with scrapy.

house-bob

Description

A Django, Scrapy application which pulls properties data .i.e houses and offices available for rent in Kampala and neighbouring districts. The Scraped data containing property price, location, district, bedrooms, bathrooms, status whether its available for rent or sale and agent details is then saved in a Postgres database.

Its uses Django 3.0 and Scrapy 2.0

Installation

Create a virtual env and activate it.

Clone the project

git clone https://github.com/peterwade153/house-bob.git

Create a Postgres Database.

Create a .env file from the .env.example and replace with the actual values.

Install Dependencies.

pip install requirements.txt

Run migrations with the command below

make migrate

Create a superuser to access to the admin page, run the command to register.

make superuser

To crawl properties with the command below

make crawl

To view the crawled properties, run the server

make start

Visit 127.0.0.1/admin and login with the superuser account credentials, created earlier. Properties can also be added / removed from here

To view dashboard Visit 127.0.0.1 after starting the server.

To run tests

make test

The Scraping is automated to run, every sunday midnight. This requires installation of redis as a broker. start the redis server

redis-server

In another terminal, start the worker.

make celery-worker

In another terminal, start the beat.

make celery-beat

Developer :- Peter, reach him at [email protected].