dqsdatalabs Goto Github PK

internet-worm

网络爬虫基础

libcloud

Apache Libcloud is a Python library which hides differences between different cloud provider APIs and allows you to manage different cloud resources through a unified and easy to use API

newspaper

News, full-text, and article metadata extraction in Python 3. Advanced docs:

newsscraper

Messing around with scrapy. Scrapes news websites.

openapi-generator

OpenAPI Generator allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec (v2, v3)

parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

pdfminer

Python PDF Parser (Not actively maintained). Check out pdfminer.six.

personalized-sector-with-knowledge-graph

python3webspider

Python3 网络爬虫开发实战

pythonspider

JS逆向系列教程，模拟登录，AES、RSA、DES加密等，持续更新，欢迎 star！

requests-ip-rotator

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.

rotating-proxies-with-python

Learn about how to rotate proxies by using Python.

scrapinghub-stack-scrapy

Software stack with latest Scrapy and updated deps

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy-boilerplate

Scrapy project boilerplate done right

scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

scrapy-distributed

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

scrapy-exercise-solution

scrapy-gui

A simple, Qt-Webengine powered web browser with built in functionality for basic scrapy webscraping support.

scrapy-poet

Page Object pattern for Scrapy

scrapy-rotating-proxies

use multiple proxies with Scrapy

scrapy-training

Scrapy Training companion code

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

scrapy_crawler_template

scrapy_demo

all kinds of scrapy demo

scrapy_utils

scrapy_utils is configuration template project, Contains the extraction of scrapy configuration

scrapydouban

豆瓣电影/豆瓣读书 Scarpy 爬虫

scrapyrt

HTTP API for Scrapy spiders

shud

sparkpipelineframework

Framework for simpler Spark Pipelines