https://github.com/arp242/isbot

Go library to detect HTTP bots.
https://github.com/arp242/isbot

bot

Last synced: 2 months ago
JSON representation

Go library to detect HTTP bots.

• Host: GitHub
• URL: https://github.com/arp242/isbot
• Owner: arp242
• License: mit
• Created: 2020-03-27T01:08:07.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2022-04-17T03:08:24.000Z (over 2 years ago)
• Last Synced: 2024-10-14T20:47:43.839Z (3 months ago)
• Topics: bot
• Language: Go
• Size: 129 KB
• Stars: 19
• Watchers: 3
• Forks: 5
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
Go library to detect bots based on the HTTP request. A "bot" is defined as any

request that isn't a regular browser request initiated by the user. This

includes things like web crawlers, but also stuff like "preview" renderers and

the like.

`Bot()` accepts a `http.Request` since it looks at *all* information, not just

the `User-Agent`. You can use `UserAgent()` if you just have a `User-Agent`, but

it's highly recommended to use `Bot()`.

Import as `zgo.at/isbot`; API docs: https://godocs.io/zgo.at/isbot

There is a command-line tool in `cmd/isbot` to check if User-Agents are bots:

    $ isbot 'Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0' 'Wget/1.13.4 (linux-gnu)'

    false (1: NoBotNoMatch) ← Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0

    true  (4: BotClientLibrary) ← Wget/1.13.4 (linux-gnu)

It's not 100% reliable, and there are some known cases where it gets things

wrong. See [`isbot_test.go`](/isbot_test.go) for a list of test cases.

The performance is pretty good; turns out that running a few `string.Contains()`

is loads faster than a `(bot|crawler|search|...)` regexp.