https://github.com/medson/ocrawl
A simple crawler to map sites relations
https://github.com/medson/ocrawl
charts golang goquery webcrawling
Last synced: 4 months ago
JSON representation
A simple crawler to map sites relations
- Host: GitHub
- URL: https://github.com/medson/ocrawl
- Owner: medson
- License: mit
- Created: 2020-03-17T13:51:54.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-07-26T10:33:43.000Z (almost 6 years ago)
- Last Synced: 2025-08-15T12:18:03.012Z (11 months ago)
- Topics: charts, golang, goquery, webcrawling
- Language: Go
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# OCrawl - A simple crawler to map sites relations
## Installation
Install your terminal type:
`$ go get github.com/lmedson/ocrawl`
## Setup a site to crawl
First, you need to create a go file and set a url to be crawled. To visualize the relations graphically, it is possible to plot a graph through an html file, just use the plot function of the graph, passing the result of what was crawled and a desired name, to the html file.
## Plot example:
```go
func main(){
crawledData := crawler.Crawl("https://clojure.org/")
crawler.Plot(crawledData, "index") // to plot a index.html file with the graph
}
```
## Crawling assets example:
```go
func main(){
crawler.CrawlAssets("https://clojure.org/") // you can to plot or parse to json the outputed data
}
```
## Formating output data example:
```go
func main(){
crawledData := crawler.Crawl("https://clojure.org/")
crawler.JsonParse(crawledData, "data")
}
```
## Runinng
Make sure you have installed all the dependencies. Run your created file, with above code:
`$ go run .go`.
## Result
After run the crawl and plot a .html file with the graph, you can open it in your favorite browser and see the relations infos passing the mouse cursor by the nodes and through line connections.
The package structure folder
```bash
.
├── .gitignore # File with ignored files
├── LICENSE # Our kind of license
├── ocrawl.go # The main method to crawl
├── README.md # Readme with how to use the crawler
├── types.go # Types used at crawler
├── Gopkg.lock # It locks the version of packages
├── Gopkg.toml # Controls the import instructions, used by the lock file
├── .html # After graph plot, you will have this output file
├── .json # After json parse crawled data, you will have this output file
└── utils.py # File with some helpers
```