https://github.com/ericz99/go-crawler

Simple lightweight crawler, that will find all endpoints on any website.
https://github.com/ericz99/go-crawler

crawler golang

Last synced: about 1 month ago
JSON representation

Simple lightweight crawler, that will find all endpoints on any website.

• Host: GitHub
• URL: https://github.com/ericz99/go-crawler
• Owner: ericz99
• License: mit
• Created: 2020-03-25T14:52:51.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2020-03-25T15:28:46.000Z (almost 5 years ago)
• Last Synced: 2024-06-20T06:46:48.458Z (7 months ago)
• Topics: crawler, golang
• Language: Go
• Size: 4.88 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## go-crawler

Will crawl through the website, and scrape all endpoints, paths, hashtags, etc.

## Installation

Installation is done using `go get`.

```

go get -u github.com/ericz99/go-crawler

```

## Example

```golang

package example

import (

	crawler "github.com/ericz99/go-crawler"

)

func main() {

	// # create a crawler instance

	spider := crawler.Crawler{}

	// # crawl the page

	result, domain := spider.Crawl("https://kith.com/")

	// # download result

	spider.Download(result, domain)

}

```

## Todo

- [ ] Find all links base on regex, instead of relying on goquery

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details