https://github.com/aimuz/colly

Elegant Scraper and Crawler Framework for Golang
https://github.com/aimuz/colly

Last synced: 6 months ago
JSON representation

Elegant Scraper and Crawler Framework for Golang

• Host: GitHub
• URL: https://github.com/aimuz/colly
• Owner: aimuz
• License: apache-2.0
• Fork: true (gocolly/colly)
• Created: 2018-05-14T11:27:19.000Z (about 8 years ago)
• Default Branch: master
• Last Pushed: 2018-05-14T10:15:37.000Z (about 8 years ago)
• Last Synced: 2024-11-24T17:56:16.588Z (over 1 year ago)
• Language: Go
• Homepage: http://go-colly.org/
• Size: 4.03 MB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

          
# Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

[![GoDoc](https://godoc.org/github.com/gocolly/colly?status.svg)](https://godoc.org/github.com/gocolly/colly)

[![Backers on Open Collective](https://opencollective.com/colly/backers/badge.svg)](#backers) [![Sponsors on Open Collective](https://opencollective.com/colly/sponsors/badge.svg)](#sponsors) [![build status](https://img.shields.io/travis/gocolly/colly/master.svg?style=flat-square)](https://travis-ci.org/gocolly/colly)

[![report card](https://img.shields.io/badge/report%20card-a%2B-ff3333.svg?style=flat-square)](http://goreportcard.com/report/gocolly/colly)

[![view examples](https://img.shields.io/badge/learn%20by-examples-0077b3.svg?style=flat-square)](https://github.com/gocolly/colly/tree/master/_examples)

[![Code Coverage](https://img.shields.io/codecov/c/github/gocolly/colly/master.svg)](https://codecov.io/github/gocolly/colly?branch=master)

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_shield)

[![Twitter URL](https://img.shields.io/badge/twitter-follow-green.svg)](https://twitter.com/gocolly)

## Features

 * Clean API

 * Fast (>1k request/sec on a single core)

 * Manages request delays and maximum concurrency per domain

 * Automatic cookie and session handling

 * Sync/async/parallel scraping

 * Caching

 * Automatic encoding of non-unicode responses

 * Robots.txt support

 * Distributed scraping

 * Configuration via environment variables

 * Extensions

## Example

```go

func main() {

	c := colly.NewCollector()

	// Find and visit all links

	c.OnHTML("a[href]", func(e *colly.HTMLElement) {

		e.Request.Visit(e.Attr("href"))

	})

	c.OnRequest(func(r *colly.Request) {

		fmt.Println("Visiting", r.URL)

	})

	c.Visit("http://go-colly.org/")

}

```

See [examples folder](https://github.com/gocolly/colly/tree/master/_examples) for more detailed examples.

## Installation

```

go get -u github.com/gocolly/colly/...

```

## Bugs

Bugs or suggestions? Visit the [issue tracker](https://github.com/gocolly/colly/issues) or join `#colly` on freenode

## Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

 * [greenpeace/check-my-pages](https://github.com/greenpeace/check-my-pages) Scraping script to test the Spanish Greenpeace web archive

 * [altsab/gowap](https://github.com/altsab/gowap) Wappalyzer implementation in Go

If you are using Colly in a project please send a pull request to add it to the list.

## Contributors

This project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md).



## Backers

Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/colly#backer)]



## Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/colly#sponsor)]





















## License

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_large)