https://github.com/konsalex/colly-neo4j-storage

Neo4j Storage implementation for Colly
https://github.com/konsalex/colly-neo4j-storage

Last synced: 24 days ago
JSON representation

Neo4j Storage implementation for Colly

• Host: GitHub
• URL: https://github.com/konsalex/colly-neo4j-storage
• Owner: konsalex
• License: mit
• Created: 2021-07-20T16:51:27.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2021-07-20T18:26:18.000Z (over 3 years ago)
• Last Synced: 2024-10-30T09:28:12.522Z (2 months ago)
• Language: Go
• Size: 8.79 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# colly-neo4j-storage

[![GitHub Action - Test Go Package](https://github.com/konsalex/colly-neo4j-storage/actions/workflows/workflow.yml/badge.svg)](https://github.com/konsalex/colly-neo4j-storage/actions/workflows/workflow.yml)

![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/konsalex/colly-neo4j-storage)

[![Go Report Card](https://goreportcard.com/badge/github.com/konsalex/colly-neo4j-storage)](https://goreportcard.com/report/github.com/konsalex/colly-neo4j-storage)

A Neo4j storage back end for the Colly web crawling/scraping framework https://go-colly.org

Example Usage:

```go

package main

import (

	"fmt"

	"github.com/gocolly/colly"

	"github.com/konsalex/colly-neo4j-storage/colly"

)

func main() {

	c := colly.NewCollector()

	storage := &neo4j.Storage{

        URI:      "bolt://localhost:7687",

        Username: "neo4j",

        Password: "password",

        Database: "colly" , // Override default database "neo4j" (optional)

	}

	if err := c.SetStorage(storage); err != nil {

		panic(err)

	}

	// Find and visit all links

	c.OnHTML("a[href]", func(e *colly.HTMLElement) {

		e.Request.Visit(e.Attr("href"))

	})

	c.OnRequest(func(r *colly.Request) {

		fmt.Println("Visiting", r.URL)

	})

	c.Visit("http://go-colly.org/")

}

```