https://github.com/dan-leech/go-link-crawler
Simple web crawler
https://github.com/dan-leech/go-link-crawler
Last synced: 12 months ago
JSON representation
Simple web crawler
- Host: GitHub
- URL: https://github.com/dan-leech/go-link-crawler
- Owner: dan-leech
- License: mit
- Created: 2019-12-26T08:57:54.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-01-07T11:19:15.000Z (over 6 years ago)
- Last Synced: 2025-02-24T16:42:40.390Z (over 1 year ago)
- Language: Go
- Size: 22.5 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Go Link Crawler
Crawler for links and titles. It benchmarks requests per second.
Copy `./config/.go-link-crawler.example.yaml` to `./config/.go-link-crawler.yaml`
## Build
`make help`
### Docker
`make run-docker`
if you need use other file than `list.txt` run docker container with
```
docker run -v $PATH_TO_YOUR_FILE:/app/crawl_list.txt -it go-link-crawler ./main ./crawl_list.txt
```
# License
MIT