Scan (and monitor) Craigslist for new apartments within the defined parameters.
- sends notifications (after first run) for any new apartment
- extract features like gym, pool, availability and apartment size from the description
- calculate distance from a location (i.e.: your work place)
- stores everything conveniently in a sqlite database
- Python 3.10+
- create a folder for the app and change your workdir:
mkdir craigscraper; cd craigscraper
- create a notification file using the instructions below
vi notifications.yaml
- prepare a
.env
file using the.env_template
as model. Remember to uncomment the last two lines to allow data persistance - run docker:
docker run --env-file ./craigscraper/.env -v ./craigscraper:/persist ghcr.io/porelli/craigscraper:main
docker run -it --rm -p 8080:8080 -v ./craigscraper:/data -e SQLITE_DATABASE=rents.db coleifer/sqlite-web
- clone the package
- copy
.env_template
to.env
- replace values in
.env
- create venv:
python3 -m venv .venv
- activate venv:
source .venv/bin/activate
- install dependencies:
pip3 install -r requirements.txt
- profit:
scrapy crawl rent
- suppress the notification test in the .env file using
SUPPRESS_TEST_NOTIFICATION='True'
- specify the scan interval in the .env file using
MINUTES_INTERVAL
- run:
python3 main.py
- currently it works only for Vancouver, BC
- code is not very organized and does not follow all the scrapy best practices
By default, the script will try to send a desktop notification. A first notification is sent at the beginning as a test. Subsequent notifications are sent when new apartments appear in the search on there is a price change.
You can override the default behavior specifying your own notification provider(s) with an apprise compatible configuration file and using the -a notifications_file=<NOTIFICATION_FILE>
CLI option or via ENV NOTIFICATION_FILE
.
Please note, that for notifications to work on Mac or Windows, you may need to install additional packages. If you experience any errors with this, please refer to the apprise wiki.
brew install terminal-notifier
pip install pywin32