https://github.com/aimlpm/markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

ai-agents anthropic-claude data-extraction gemini ingestion-pipeline llm markdown-extraction openai pgvector python rag sitemap-crawler structured-data supabase vector-database webcrawler

Last synced: 23 Apr 2026