Projects in Awesome Lists tagged with distributed-crawler
A curated list of projects in awesome lists tagged with distributed-crawler .
https://github.com/SpiderClub/weibospider
:zap: A distributed crawler for weibo, building with celery and requests.
data-analysis distributed-crawler python3 sina weibo weibospider
Last synced: 26 Mar 2025
https://github.com/spiderclub/weibospider
:zap: A distributed crawler for weibo, building with celery and requests.
data-analysis distributed-crawler python3 sina weibo weibospider
Last synced: 12 Mar 2026
https://github.com/brendonboshell/supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
crawler distributed-crawler robot sitemap web-crawler
Last synced: 12 Jan 2026
