paulpierre/markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

🏗️ building evm & llm 🏴‍☠️ warez

█▀▀ █▀█ █▄░█ █▀▄ █░█ █▀▀ ▀█▀ █ █░█ █▀▀
█▄▄ █▄█ █░▀█ █▄▀ █▄█ █▄▄ ░█░ █ ▀▄▀ ██▄
https://conductive.ai  // @conductiveai

⭐️ repos	💬 RasaGPT - Headless chat platform built on top of Rasa and Langchain 2k+ ⭐️'s #3 trending on HN 🐙 Hydralisk - scale and fund millions of EVM-chain wallets via CLI 🕵️ Informer - Telegram Mass Surveillance w/ 1.3k+ ⭐️'s and 170+ ⑂ featured on the frontpage of HN 🐦 twig.py - a twitter web3 influencer truffle pig used for finding engaged users
🧠 tech	evm, solidity, erigon, geth, gpt, llms, agents, langchain, llamaindex, stable diffusion, lora
💬 lang	python, golang, js, kotlin, swift, vue, nodejs
💽 data	clickhouse, postgres, timescaledb, ksqldb
🚇 pipelines	kafka, redpanda, celery, rabbitmq, redis, debezium, airflow
🧑‍🍳 orchest	k8s, hashicorp, docker, nomad l00n1x
👷‍♂️ work	build0r of web3 user acquistion tech and conversational AI
🌱 xp	adtech, data engineering, mobile games, machine learning
❤️ ❤️	🐍🎷, graph theory, sci-fi, games, 🤖 openly a balaji's network state npc
🎙️ podcasts	all-in-podcast, lex fridman, yc, network state, a16z, MoZ, MFM, proof, jre, flagrant2

gm

• follow: @paulpierre
• email: hi🍜paulpierre.com

markdown-crawler's Issues

Not all links are found when custom `target_content` is set.

I wanted to try out this package for an internal RAG presentation based on some of our company website data. Our website has the typical structure: header, main content and footer. Most of the links are in the header and footer part, but if I set target_content to only the main container, no links are getting found.

Is there a way to have all URLs collected outside of the container i ultimately want to parse into markdown? When i don't specify the target i am getting quite a mess of navigation and footer link data that repeats on every page (which would have to be cleaned up for every single page)..

Thanks in advance!

---

Edit: I think it was actually a combination of the domain, base path flags and the target content that returns zero results. once I removed all the extra flags, the target content worked as advertised.

Continue where you left off feature is not working robustly

I believe there's a problem when crawling massive documentation portals in a scenario where the crawler cannot handle a certain condition and exists suddenly, leading to the continuation task to start from the beginning but not intelligently, therefore leading to all the previous pages being checked before moving forward with the new download. This is my constant experience with stripe's documentation portal where every time after around an hour the crawler fails to move forward and there are no option to move forward exactly where you left off.
my suggestion is to switch the default behavior to continue where you left off without previous check, since I assume what you're doing with the current implantation is to also detect if there any changes and apply them as well on the existing files instead of literally sticking to the current state of the progress and attempting to finish the crawl job which is taking massive times.