Topic: scraper Goto Github

👇 Here are 8493 public repositories matching this topic...

aapatre / automatic-udemy-course-enroller-get-paid-udemy-courses-for-free
scraper,Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
User: aapatre
python scraping selenium python3 scraper

adyzng / jd-autobuy
scraper,Python爬虫，京东自动登录，在线抢购商品
User: adyzng
python crawler jingdong scraper

ahmadibrahiim / website-downloader
scraper,💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
User: ahmadibrahiim
Home Page: https://website-downloader.onrender.com
assets downloader offline-web-pages scraper

aliparlakci / bulk-downloader-for-reddit
scraper,Downloads and archives content from reddit
User: aliparlakci
Home Page: https://pypi.org/project/bdfr
reddit imgur gfycat scraper python downloader archive

alirezamika / autoscraper
scraper,A Smart, Automatic, Fast and Lightweight Web Scraper for Python
User: alirezamika
scraping scraper scrape webscraping crawler web-scraping ai artificial-intelligence python webautomation

apify / crawlee
scraper,Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev
web-scraping web-crawling npm headless-chrome puppeteer automation apify scraping crawling crawler

apify / crawlee-python
scraper,Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev/python/
apify automation beautifulsoup crawler crawling headless headless-chrome pip playwright python

avnsx / fansly-downloader
scraper,Easy to use fansly.com content downloading tool. Written in python, but ships as a standalone Executable App for Windows too. Enjoy your Fansly content offline anytime, anywhere in the highest possible content resolution! Fully customizable to download in bulk or single: photos, videos & audio from timeline, messages, collection & specific posts 👍
User: avnsx
Home Page: https://fansly.com/
fansly datascraping python fansly-scraper fansly-downloader fansly-download windows macos linux image-download

brucedone / awesome-crawler
scraper,A collection of awesome web crawler,spider in different languages
User: brucedone
web-crawler crawler web-scraper spider node-crawler scraper awesome

cheeriojs / cheerio
scraper,The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Organization: cheeriojs
Home Page: https://cheerio.js.org
cheerio dom hacktoberfest html htmlparser htmlparser2 jquery parser scraper selector

cinemagoer / cinemagoer
scraper,Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people, characters and companies
Organization: cinemagoer
Home Page: https://cinemagoer.github.io/
imdb movies actors cinema movie-database python database sql cast internet-movie-database

codelucas / newspaper
scraper,newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
User: codelucas
Home Page: https://goo.gl/VX41yK
python news crawler crawling scraper news-aggregator

edoardottt / cariddi
scraper,Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
User: edoardottt
Home Page: https://edoardoottavianelli.it
bugbounty crawler crawling endpoint-discovery endpoints go golang hacktoberfest infosec osint penetration-testing pentesting recon reconnaissance redteam scraper secret-keys secrets-detection security security-tools

evil0ctal / douyin_tiktok_download_api
scraper,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
User: evil0ctal
Home Page: https://douyin.wtf
python pywebio tiktok douyin api scraper fastapi no-watermark online-parsing async

extractus / article-extractor
scraper,To extract main article from given URL with Node.js
Organization: extractus
Home Page: https://extractor-demos.pages.dev/article-extractor
nodejs article-parser readability article article-extractor crawler extract scraper

facundoolano / google-play-scraper
scraper,Node.js scraper to get data from Google Play
User: facundoolano
google-play scraper crawler api nodejs

felipecsl / wombat
scraper,Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
User: felipecsl
Home Page: https://felipecsl.github.io/wombat/
ruby scraper crawler dsl

fent / node-ytdl-core
scraper,YouTube video downloader in javascript.
User: fent
node scraper video-downloader youtube youtube-downloader

geziyor / geziyor
scraper,Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Organization: geziyor
crawler go scraper scraping spider

go-rod / rod
scraper,A Chrome DevTools Protocol driver for web automation and scraping.
Organization: go-rod
Home Page: https://go-rod.github.io
cdp chrome-headless chrome-devtools chrome-devtools-protocol headless web-scraping automation scraper devtools devtools-protocol

gocolly / colly
scraper,Elegant Scraper and Crawler Framework for Golang
Organization: gocolly
Home Page: https://go-colly.org/
crawler crawling framework go golang scraper scraping spider

guyueyingmu / avbook
scraper,AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
User: guyueyingmu
javbus avmoo javlibrary spider crawler laravel scraper adult magnet-link magnet

huginn / huginn
scraper,Create agents that monitor and act on your behalf. Your agents are standing by!
Organization: huginn
automation notifications scraper webscraping feedgenerator rss agent monitoring feed twitter-streaming

iawia002 / lux
scraper,👾 Fast and simple video download library and CLI tool written in Go
User: iawia002
bilibili crawler download downloader go golang iqiyi qq scraper tumblr video youku youtube

ionicabizau / scrape-it
scraper,🔮 A Node.js scraper for humans.
User: ionicabizau
Home Page: http://ionicabizau.net/blog/30-how-to-write-a-web-scraper-in-node-js
scraper node-scraper hacktoberfest

jae-jae / querylist
scraper,:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
User: jae-jae
Home Page: https://querylist.cc
querylist crawler spider scraper

javscraper / emby.plugins.javscraper
scraper,Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。
User: javscraper
Home Page: https://javscraper.com
emby jav-scraper jav plugin scraper fanart-poster synology japanese adult metadata

joeyism / linkedin_scraper
scraper,A library that scrapes Linkedin for user data
User: joeyism
linkedin linkedin-scraper scraper profile driver linkedin-profile linkedin-url scrapes-linkedin chrome firefox

justanotherarchivist / snscrape
scraper,A social networking service scraper in Python
User: justanotherarchivist
python scraper social-media social-network

justfoolingaround / animdl
scraper,A highly efficient, fast, powerful and light-weight anime downloader and streamer for your favorite anime.
User: justfoolingaround
anime download python scraper

lorey / mlscraper
scraper,🤖 Scrape data from HTML websites automatically by just providing examples
User: lorey
Home Page: https://pypi.org/project/mlscraper/
scraping crawling html machine-learning extraction-engine scraper crawler crawler-python

lucasjinreal / weibo_terminater
scraper,Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
User: lucasjinreal
scraper weibo sina corpus chinese chatbot

madawei2699 / mygptreader
scraper,A community-driven way to read and chat with AI bots - powered by chatGPT.
User: madawei2699
Home Page: https://www.i365.tech/
chatgpt slack-bot prompt crawler embedding gpt-35-turbo scraper ai reader hot-news

meetdeveloper / freedictionaryapi
scraper,There was no free Dictionary API on the web when I wanted one for my friend, so I created one.
User: meetdeveloper
Home Page: https://dictionaryapi.dev/
dictonary google api scraper google-dictionary dictionary-api free-api

mendableai / firecrawl
scraper,🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Organization: mendableai
Home Page: https://firecrawl.dev
ai crawler data markdown scraper html-to-markdown llm rag scraping web-crawler

metafates / mangal
scraper,📖 The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!
User: metafates
anime cli go manga manga-downloader scraper tui manga-reader golang anilist terminal command-line lua comic-downloader pdf mangadex linux macos windows mangal

mishushakov / llm-scraper
scraper,Turn any webpage into structured data using LLMs
User: mishushakov
ai browser gpt langchain llm openai scraper browser-automation playwright puppeteer

montferret / ferret
scraper,Declarative web scraping
Organization: montferret
Home Page: https://www.montferret.dev/
golang query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler go chrome cli tool library hacktoberfest

naibowang / easyspider
scraper,A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
User: naibowang
Home Page: https://www.easyspider.net
code-free crawler gui layman spider parameters web www input-parameters frontend

niespodd / browser-fingerprinting
scraper,Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
User: niespodd
Home Page: https://niespodd.github.io/browser-fingerprinting/
bot detection chromium stealth puppeteer scraper webscraping web automation chromium-browser bot-detection chromedriver fingerprinting crawler recaptcha spider browser-fingerprinting

paulmcinnis / jobfunnel
scraper,Scrape job websites into a single spreadsheet with no duplicates.
User: paulmcinnis
jobs search indeed monster glassdoor scraper job automated csv beautifulsoup beautifulsoup4 python tfidf yaml waterloo international

paulpierre / informer
scraper,A Telegram Mass Surveillance Bot in Python
User: paulpierre
Home Page: https://www.twitter.com/paulpierre
bot cryptocurrency listen listener monitor python scraper spy surveillance telegram telegram-api telegram-bot telegram-bot-api telegrambot telethon telethon-userbot

pwxcoo / chinese-xinhua
scraper,:orange_book: 中华新华字典数据库。包括歇后语，成语，词语，汉字。
User: pwxcoo
data scraper chinese-traditional python3 chinese chinese-characters chinese-nlp chinese-language chinese-simplified json-dataset

sqzw-x / mdcx
scraper,Movie metadata scraper
User: sqzw-x
crawler emby jav-scraper jellyfin metadata movie-crawler movie-metadata movie-scrapper movies python

teamnewpipe / newpipeextractor
scraper,NewPipe's core library for extracting data from streaming sites
Organization: teamnewpipe
newpipe youtube crawler extractor soundcloud peertube mediaccc bandcamp scraper

th3unkn0n / osi.ig
scraper,Information Gathering Instagram.
User: th3unkn0n
information-gathering information-retrieval instagram instagram-scraper linux osint python python3 scraper termux termux-tool

ultimahoarder / ultimascraper
scraper,Scrape all the media from an OnlyFans account - Updated regularly
User: ultimahoarder
archive datascraping onlyfans scraper

website-scraper / node-website-scraper
scraper,Download website to local directory (including all css, images, js, etc.)
Organization: website-scraper
Home Page: https://www.npmjs.org/package/website-scraper
hacktoberfest javascript nodejs scraper website-scraper

yujiosaka / headless-chrome-crawler
scraper,Distributed crawler powered by Headless Chrome
User: yujiosaka
chrome chromium crawler crawling headless-chrome jquery promise puppeteer scraper scraping

zerodytrash / tiktok-live-connector
scraper,Node.js library to receive live stream events (comments, gifts, etc.) in realtime from TikTok LIVE.
User: zerodytrash
Home Page: https://discord.gg/2c6kX6g3Pa
api api-wrapper bot broadcast chat chat-reader connector hacktoberfest javascript live livestream nodejs package scraper stream tiktok tiktok-api tiktok-live webcast websocket