https://github.com/rayspock/go-web-crawler

A web crawler to fetch all the links from a given website via go routines.
https://github.com/rayspock/go-web-crawler

concurrency crawler golang goroutine

Last synced: 3 days ago
JSON representation

A web crawler to fetch all the links from a given website via go routines.

• Host: GitHub
• URL: https://github.com/rayspock/go-web-crawler
• Owner: rayspock
• Created: 2021-01-19T23:36:35.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2023-08-17T14:27:07.000Z (over 1 year ago)
• Last Synced: 2024-06-21T03:25:53.558Z (5 months ago)
• Topics: concurrency, crawler, golang, goroutine
• Language: Go
• Homepage:
• Size: 10.7 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Web Crawler

A web crawler to fetch all the links from a given website via go routines. 

## Getting Started

To get started with the repository, simply clone or download the code examples and open them in your favorite text

editor or IDE.

Before running the examples, make sure you have Go installed on your machine. You can download and install the latest

version of Go from the official website at https://golang.org.

To run the code, simply navigate to the directory containing the code example and run the following command:

```bash

# Run the code

$ go run cmd/main.go -u https://github.com -d 2 -o output.txt

```

```bash

# Parameters

  -d, --depth int       Maximum of depth to crawl (default 1)

  -o, --output string   Output file name

  -u, --url string      Website URL (default "https://github.com")

```

## Development 

```bash

# Run the tests

$ make test

# Generate mock

$ make generate

```