https://github.com/crackcomm/crawl

Lightweight library for scalable crawlers in Go.
https://github.com/crackcomm/crawl

crawl crawler go

Last synced: 4 months ago
JSON representation

Lightweight library for scalable crawlers in Go.

• Host: GitHub
• URL: https://github.com/crackcomm/crawl
• Owner: crackcomm
• License: apache-2.0
• Created: 2015-03-14T04:29:11.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2017-07-24T07:04:27.000Z (almost 9 years ago)
• Last Synced: 2024-06-19T01:54:33.019Z (almost 2 years ago)
• Topics: crawl, crawler, go
• Language: Go
• Homepage:
• Size: 103 KB
• Stars: 19
• Watchers: 4
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# crawl

[![GoDoc](https://godoc.org/github.com/crackcomm/crawl?status.svg)](https://godoc.org/github.com/crackcomm/crawl) [![Circle CI](https://img.shields.io/circleci/project/crackcomm/crawl.svg)](https://circleci.com/gh/crackcomm/crawl)

Lightweight library for crawlers in Go.

HTML parsing and extracting is done thanks to [goquery](https://godoc.org/github.com/PuerkitoBio/goquery/).

## Usage

You can take a look at [example](https://github.com/crackcomm/crawl/blob/master/examples/imdb/main.go) crawler code.

## License

                                 Apache License

                           Version 2.0, January 2004

                        http://www.apache.org/licenses/