https://github.com/zdunecki/crawlerd
crawlerd is a crawler daemon for small crawling jobs.
https://github.com/zdunecki/crawlerd
Last synced: 11 months ago
JSON representation
crawlerd is a crawler daemon for small crawling jobs.
- Host: GitHub
- URL: https://github.com/zdunecki/crawlerd
- Owner: zdunecki
- Created: 2021-04-06T00:13:09.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2022-11-24T01:14:45.000Z (over 3 years ago)
- Last Synced: 2025-02-12T14:30:55.504Z (over 1 year ago)
- Language: Go
- Homepage:
- Size: 6.86 MB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# crawlerd
**crawlerd** is a crawler daemon for small crawling jobs.
## commands
* `make grpc-gen` - generate grpc code
* `make compose-up` - run all crawlerd services in docker
* `make run-e2e-tests` - run e2e tests inside docker
* `make run-worker-integration-tests` - run integration tests inside docker
## TODO
**crawlerd** is still not production ready, below is the list needed for first production release:
- [ ] Rotating IP
- [ ] HTTP/Socks Proxy
- [ ] Fix bugs