https://github.com/aaronlifton/fastcrawl

an agentic atomics-based low-heap allocation web crawler written in Rust, that can crawl Wikipedia at a rate of ~75 pages/sec.

agentic-ai agentic-framework agentic-web-crawler ai-agents atomics rust stack-allocation streaming-parser web-crawler wikipedia

Last synced: 24 Dec 2025