https://github.com/crackcomm/crawl-links

https://github.com/crackcomm/crawl-links

Last synced: 4 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/crackcomm/crawl-links
• Owner: crackcomm
• License: apache-2.0
• Created: 2017-03-21T22:21:53.000Z (about 9 years ago)
• Default Branch: master
• Last Pushed: 2017-03-24T22:52:19.000Z (about 9 years ago)
• Last Synced: 2024-06-20T05:04:08.563Z (almost 2 years ago)
• Language: Go
• Size: 6.84 KB
• Stars: 2
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# crawl-links

[![Circle CI](https://img.shields.io/circleci/project/crackcomm/crawl-links.svg)](https://circleci.com/gh/crackcomm/crawl-links)

Links crawler.

## Usage

Example usage from command line:

```sh

# Install command line application for crawl scheduling

$ go install github.com/crackcomm/crawl/nsq/crawl-schedule

# Schedule crawl of google search results

$ crawl-schedule \

      --nsq-topic crawl_links \

      --callback github.com/crackcomm/crawl-links/spider.Links \

      "https://www.google.com/search?q=Github"

```

## License

                                 Apache License

                           Version 2.0, January 2004

                        http://www.apache.org/licenses/

## Authors

* [Łukasz Kurowski](https://github.com/crackcomm)