https://github.com/zdunecki/crawlerd

crawlerd is a crawler daemon for small crawling jobs.
https://github.com/zdunecki/crawlerd

Last synced: 14 days ago
JSON representation

crawlerd is a crawler daemon for small crawling jobs.

• Host: GitHub
• URL: https://github.com/zdunecki/crawlerd
• Owner: zdunecki
• Created: 2021-04-06T00:13:09.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2022-11-24T01:14:45.000Z (about 2 years ago)
• Last Synced: 2024-06-20T09:15:17.156Z (7 months ago)
• Language: Go
• Homepage:
• Size: 6.86 MB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# crawlerd

**crawlerd** is a crawler daemon for small crawling jobs. 

## commands

* `make grpc-gen` - generate grpc code

* `make compose-up` - run all crawlerd services in docker

* `make run-e2e-tests` - run e2e tests inside docker

* `make run-worker-integration-tests` - run integration tests inside docker

## TODO

**crawlerd** is still not production ready, below is the list needed for first production release:

- [ ] Rotating IP

- [ ] HTTP/Socks Proxy

- [ ] Fix bugs