erisyang2025/scrapy-project-template

Scrapy project template. Use it to quickly spin up a new web scraping project

Project Name: Scrapy Project Template

Description:

this is an example Scrapy project template that can be used to quickly spin up a new web scraping project with the most necessary features.

Features:

• scrapy-poet integration - a better way to organize Scrapy projects
• proxy middleware - a simple way to start using proxies with your project
• database integration with peewee orm - a simple way to store web scraping output in a database
• ruff linter and code formatter
• scrapy-redis integration - a simple way to allow distributed spider runs
• example spiders: google search and amazon.com

Usage

To use this example Scrapy project template, simply fork the repository and start writing your own code. Customize the spiders, pipelines, and items to suit your specific web scraping project needs.

License: MIT