hund030/acrawler

Acrawler

Installation for linux

1. install Miniconda
2. install g++
3. create python3 enviroment
4. install scrapy and scrapy-splash and faker and maybe selenium
5. install docker and run splash

sudo apt install g++
conda create -n py3 python
conda activate py3
pip install Scrapy
pip install scrapy-splash
pip install faker
pip install selenium
sudo apt install docker.io
tmux
sudo docker run -p 8050:8050 scrapinghub/splash

Usage

1. clone this repo
2. change directory to this repo root
3. git branch switch to splash
4. ensure your docker image running and python enviorment is alright
5. start your crawl task(recommend using tmux)

git clone [email protected]:hund030/Acrawler.git
cd Acrawler
git checkout splash
tmux
scrapy runspider douban/spiders/replies_spider.py -o reply.json -s JOBDIR=~/logs

acrawler's Issues

Re-sort the original data to mapping input and output

load balance for different layers