Topic: crawler Goto Github
Some thing interesting about crawler
👇 Here are 6666 public repositories matching this topic...
-
adbar / trafilatura
crawler,Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
User: adbar
Home Page: https://trafilatura.readthedocs.io
article-extractor corpus corpus-builder corpus-tools crawler html-to-markdown html2text news news-aggregator news-crawler nlp readability rss-feed scraping tei text-cleaning text-extraction text-mining text-preprocessing web-scraping -
alirezamika / autoscraper
crawler,A Smart, Automatic, Fast and Lightweight Web Scraper for Python
User: alirezamika
scraping scraper scrape webscraping crawler web-scraping ai artificial-intelligence python webautomation -
apify / crawlee
crawler,Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev
web-scraping web-crawling npm headless-chrome puppeteer automation apify scraping crawling crawler -
arachni / arachni
crawler,Web Application Security Scanner Framework
Organization: arachni
Home Page: http://www.arachni-scanner.com
arachni dom ruby audit detection security-audit analysis modular javascript scanners -
bda-research / node-crawler
crawler,Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Organization: bda-research
cheerio crawler extract-data javascript jquery nodejs spider -
-
boris-code / feapder
crawler,🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
User: boris-code
Home Page: http://feapder.com
scrapy feapder spider crawler python feaplat -
brucedone / awesome-crawler
crawler,A collection of awesome web crawler,spider in different languages
User: brucedone
web-crawler crawler web-scraper spider node-crawler scraper awesome -
charlespikachu / decryptlogin
crawler,DecryptLogin: APIs for loginning some websites by using requests.
User: charlespikachu
Home Page: https://httpsgithubcomcharlespikachudecryptlogin.readthedocs.io/zh/latest/
requests login python3 spider crawler zhihu bilibili weibo taobao jingdong -
-
code4craft / webmagic
crawler,A scalable web crawler framework for Java.
User: code4craft
Home Page: http://webmagic.io/
crawler java scraping framework -
codelucas / newspaper
crawler,newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
User: codelucas
Home Page: https://goo.gl/VX41yK
python news crawler crawling scraper news-aggregator -
constverum / proxybroker
crawler,Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
User: constverum
Home Page: http://proxybroker.readthedocs.io
proxy anonymity privacy socks http-proxy crawler proxy-server anonymous proxy-checker proxy-list proxypool proxies -
crawlab-team / crawlab
crawler,Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Organization: crawlab-team
Home Page: https://www.crawlab.cn
webcrawler scrapy crawlab spiders-management go scrapyd-ui spider crawler webspider web-crawler -
dataabc / weibo-crawler
crawler,新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频
User: dataabc
crawler weibo weibo-spider -
dedsecinside / torbot
crawler,Dark Web OSINT Tool
Organization: dedsecinside
tor tor-network deepweb dark-web psnappz crawler spider dedsec-inside python-web-crawler security-tools -
dotnetcore / dotnetspider
crawler,DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Organization: dotnetcore
crawler cross-platform csharp distributed dotnetcore -
dropsdevopsorg / ecommercecrawlers
crawler,实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:
Organization: dropsdevopsorg
Home Page: http://wechat.doonsec.com/
python3 crawler baidu-tieba taobao-spider dazhong-spider douban-movie douban-music alitask baotu quanjing -
elliotgao2 / toapi
crawler,Every web site provides APIs.
User: elliotgao2
Home Page: https://gaojiuli.github.io/toapi/
html json api python web spider crawler flask toapi -
evil0ctal / douyin_tiktok_download_api
crawler,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
User: evil0ctal
Home Page: https://douyin.wtf
python pywebio tiktok douyin api scraper fastapi no-watermark online-parsing async -
-
guyueyingmu / avbook
crawler,AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
User: guyueyingmu
javbus avmoo javlibrary spider crawler laravel scraper adult magnet-link magnet -
hardkoded / puppeteer-sharp
crawler,Headless Chrome .NET API
Organization: hardkoded
Home Page: https://www.puppeteersharp.com
puppeteer chrome chromium automation crawler crawling csharp e2e e2e-testing webautomation -
hiddenstrawberry / crawler_illegal_cases_in_china
crawler,Collection of China illegal cases about web crawler 本项目用来整理所有**大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在**大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户
User: hiddenstrawberry
Home Page: http://kgkg.kg
china law crawler -
-
-
-
-
-
jhao104 / proxy_pool
crawler,Python ProxyPool for web spider
User: jhao104
Home Page: https://jhao104.github.io/proxy_pool/
crawler http proxy redis spider -
jumper2014 / lianjia-beike-spider
crawler,链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个**主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。
User: jumper2014
lianjia spider crawler beike house -
kanasimi / work_crawler
crawler,Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
User: kanasimi
cejs comic-downloader comics crawler download-comic downloader ebook epub manga manga-downloader narou novel-downloader novels webcomics -
lorien / awesome-web-scraping
crawler,List of libraries, tools and APIs for web scraping and data processing.
User: lorien
web-scraping captcha-bypass captcha-recaptcha crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python -
madawei2699 / mygptreader
crawler,A community-driven way to read and chat with AI bots - powered by chatGPT.
User: madawei2699
Home Page: https://www.i365.tech/
chatgpt slack-bot prompt crawler embedding gpt-35-turbo scraper ai reader hot-news -
mendableai / firecrawl
crawler,🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Organization: mendableai
Home Page: https://firecrawl.dev
ai crawler data markdown scraper html-to-markdown llm rag scraping web-crawler -
montferret / ferret
crawler,Declarative web scraping
Organization: montferret
Home Page: https://www.montferret.dev/
golang query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler go chrome cli tool library hacktoberfest -
naibowang / easyspider
crawler,A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
User: naibowang
Home Page: https://www.easyspider.net
code-free crawler gui layman spider parameters web www input-parameters frontend -
niespodd / browser-fingerprinting
crawler,Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
User: niespodd
Home Page: https://niespodd.github.io/browser-fingerprinting/
bot detection chromium stealth puppeteer scraper webscraping web automation chromium-browser bot-detection chromedriver fingerprinting crawler recaptcha spider browser-fingerprinting -
projectdiscovery / katana
crawler,A next-generation crawling and spidering framework.
Organization: projectdiscovery
cli crawler gocrawler headless spider-framework web-spider -
qianlitp / crawlergo
crawler,A powerful browser crawler for web vulnerability scanners
User: qianlitp
headless-chrome crawler chrome-devtools golang vulnerability-scanner headless web-vulnerability-scanners chromedp crawlergo blackhat -
rmax / scrapy-redis
crawler,Redis-based components for Scrapy.
User: rmax
Home Page: http://scrapy-redis.readthedocs.io
scrapy crawler distributed redis -
-
scrapy / scrapy
crawler,Scrapy, a fast high-level web crawling & scraping framework for Python.
Organization: scrapy
Home Page: https://scrapy.org
crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python -
shengqiangzhang / examples-of-web-crawlers
crawler,一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
User: shengqiangzhang
crawler spider taobao tmall example python selenium pyquery stock fund -
spiderclub / haipproxy
crawler,:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Organization: spiderclub
Home Page: https://spiderclub.github.io/haipproxy/
high-availability scrapy ipproxy distributed redis crawler scheduler spider -
ssssssss-team / spider-flow
crawler,新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Organization: ssssssss-team
Home Page: https://www.spiderflow.org
crawler jsoup spider spider-flow web-crawler web-spider webcrawler webspider xpath -
tuhinshubhra / red_hawk
crawler,All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
User: tuhinshubhra
scanner crawler information-gathering admin-scanner backups-finder sql-vulnerability-scannig cms-detector cloudflare-detection geo-ip subdomain-scanner -
-
-
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.