https://github.com/shavit/crawlero

Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.
https://github.com/shavit/crawlero

crawler marketing-automation marketing-tools pbn proxy rabbitmq tor

Last synced: about 1 month ago
JSON representation

Distributed web crawlers. Fault tolerance, user-agent randomizer, RabbitMQ, Tor, PostgreSQL.

• Host: GitHub
• URL: https://github.com/shavit/crawlero
• Owner: shavit
• Created: 2017-11-30T13:44:13.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2018-01-17T22:30:36.000Z (almost 7 years ago)
• Last Synced: 2023-03-06T08:31:23.973Z (almost 2 years ago)
• Topics: crawler, marketing-automation, marketing-tools, pbn, proxy, rabbitmq, tor
• Language: Go
• Homepage:
• Size: 7.81 KB
• Stars: 14
• Watchers: 4
• Forks: 0
• Open Issues: 0

Awesome Lists containing this project