https://github.com/prorobot-ai/worker
A concurrent web worker written in Go (Golang) designed to crawl websites efficiently while respecting basic crawling policies. The worker stops automatically after crawling a specified number of links (default: 64).
https://github.com/prorobot-ai/worker
crawler golang grpc-server scraper
Last synced: 11 months ago
JSON representation
A concurrent web worker written in Go (Golang) designed to crawl websites efficiently while respecting basic crawling policies. The worker stops automatically after crawling a specified number of links (default: 64).
- Host: GitHub
- URL: https://github.com/prorobot-ai/worker
- Owner: prorobot-ai
- Created: 2025-02-19T23:03:23.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-05T22:09:26.000Z (over 1 year ago)
- Last Synced: 2025-03-24T17:07:01.914Z (about 1 year ago)
- Topics: crawler, golang, grpc-server, scraper
- Language: Go
- Homepage:
- Size: 40 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0