spider-press's Introduction

Spider Press Man

高可用代理IP池高并发爬虫不均匀的压力分发系统

Highly Available Proxy IP Pool, Highly Concurrent Spider, Uneven Pressure Distribution System

Key

Highly Available Proxy IP Pool
- By obtaining data from Gatherproxy, Goubanjia, xici etc. Free Proxy WebSite
- Analysis the Goubanjia port data
- Quickly verify IP availability
- Cooperate with Requests to automatically assign proxy Ip, with Retry mechanism, fail to write DB mechanism
Netease
- classify -> playlist id -> song_detail
- V1 Write file, One run version, no proxy, no record progress mechanism
- V1.5 Small amount of proxy IP
- V2 Proxy IP pool, Record progress, Write to MySQL
  - Optimize the write to DB Load data/ Replace INTO
Press Test
- By highly available proxy IP pool to pretend user.
- Give some web service uneven pressure
- To do: press uniform

Development

$ git clone https://github.com/iofu728/spider.git
$ cd spider
$ ipython
# netease spider
$ import netease.netease_music_db
$ xxx = netease.netease_music_db.Get_playlist_song()
# press
$ import press.press
$ xxx = press.press.Press_test()
$ xxx.one_press_attack(url, host, qps, types, total)

Structure

.
├── LICENSE                        // LICENSE
├── README.md                      // README
├── log                           // failured log
├── netease
│   ├── netease_music_base.py      // v1 spider
│   ├── netease_music_db.py        // v2 spider
│   ├── result.txt                 // result
│   └── table.sql                  // netease sql
├── press
│   └── press.py                   // press
├── proxy
│   ├── gatherproxy                // gatherproxy data
│   ├── getproxy.py                // proxy pool
│   └── table.sql                  // proxy sql
├── song_detail
└── utils
    ├── db.py                      // db operation
    └── utils.py                   // requests operation