https://github.com/x-way/crawlerdetect

Golang module to detect bots and crawlers via the user agent
https://github.com/x-way/crawlerdetect

bot-detection crawler crawler-detection detect go spider user-agent

Last synced: 3 months ago
JSON representation

Golang module to detect bots and crawlers via the user agent

• Host: GitHub
• URL: https://github.com/x-way/crawlerdetect
• Owner: x-way
• License: mit
• Created: 2020-08-09T14:54:07.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2025-03-08T21:23:39.000Z (4 months ago)
• Last Synced: 2025-04-01T16:21:27.670Z (3 months ago)
• Topics: bot-detection, crawler, crawler-detection, detect, go, spider, user-agent
• Language: Go
• Homepage:
• Size: 3.63 MB
• Stars: 61
• Watchers: 3
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# crawlerdetect

[![CircleCI](https://circleci.com/gh/x-way/crawlerdetect.svg?style=svg)](https://circleci.com/gh/x-way/crawlerdetect)

[![Go Report Card](https://goreportcard.com/badge/github.com/x-way/crawlerdetect)](https://goreportcard.com/report/github.com/x-way/crawlerdetect)

[![PkgGoDev](https://pkg.go.dev/badge/github.com/x-way/crawlerdetect)](https://pkg.go.dev/github.com/x-way/crawlerdetect)

## About

**crawlerdetect** is a Go version of PHP class @[CrawlerDetect](https://github.com/JayBizzle/Crawler-Detect). 

It helps to detect bots/crawlers/spiders via the user agent and other HTTP-headers. Currently able to detect 1,000's of bots/spiders/crawlers.

## Installation

`go get github.com/x-way/crawlerdetect`

## Basic Usage

```

import "fmt"

import "github.com/x-way/crawlerdetect"

func main() {

    uastring := "curl/7.54.0"

    if crawlerdetect.IsCrawler(uastring) {

        fmt.Println("Found a crawler")

    }

}

```

## Contributing

The patterns and testcases are synced from the PHP repo.

If you find a bot/spider/crawler user agent that crawlerdetect fails to detect, please submit a pull request with the regex pattern and a testcase to the [upstream PHP repo](https://github.com/JayBizzle/Crawler-Detect).