https://github.com/nakabonne/netsurfer

netsurfer is a very lightweight scraping framework
https://github.com/nakabonne/netsurfer

crawler go library scraping

Last synced: 10 days ago
JSON representation

netsurfer is a very lightweight scraping framework

• Host: GitHub
• URL: https://github.com/nakabonne/netsurfer
• Owner: nakabonne
• License: mit
• Created: 2017-11-04T07:23:08.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2018-02-21T14:03:01.000Z (over 6 years ago)
• Last Synced: 2024-06-20T13:38:22.778Z (5 months ago)
• Topics: crawler, go, library, scraping
• Language: Go
• Homepage:
• Size: 11.7 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# netsurfer

netsurfer is a very lightweight scraping framework.  

This has only the minimum function necessary for scraping.

# What can be done

- You can know the ranking of the web site.

- You can know the URL displayed when searching with the google engine.

- You can get HTML.

# Installation

```

$ go get -u github.com/nakabonne/netsurfer

```

# Usage

If you want to know more, please read [examples](https://github.com/ryonakao/netsurfer/tree/master/examples).

```go

import (

	"fmt"

	"net/url"

	"github.com/ryonakao/netsurfer"

)

// If you want to know the rank of a page, please write the following code.

u, _ := url.Parse("https://qiita.com/ryonakao")

rank, _ := netsurfer.GetRank(u, "ryonakao", 2)

fmt.Println("Rank is ", rank)

// If you want to know the search result URL, please write following

urls, _ := netsurfer.OrganicSearch("ruby", 3)

for _, url := range urls {

	// If you want to know the title, please write the following

	title, _ := netsurfer.GetTitle(url.String())

	fmt.Println("Title is ", title)

}

```

# dependencies

- [goquery](https://github.com/PuerkitoBio/goquery)

# License

`netsurfer` source code is available under the MIT [License](https://github.com/ryonakao/netsurfer/blob/master/LICENSE).