https://github.com/arcjet/well-known-bots
List of well-known bots and user-agent patterns to detect them
https://github.com/arcjet/well-known-bots
Last synced: about 1 year ago
JSON representation
List of well-known bots and user-agent patterns to detect them
- Host: GitHub
- URL: https://github.com/arcjet/well-known-bots
- Owner: arcjet
- License: mit
- Created: 2024-09-06T13:46:25.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-19T17:38:38.000Z (over 1 year ago)
- Last Synced: 2025-05-08T02:18:46.636Z (about 1 year ago)
- Language: JavaScript
- Homepage:
- Size: 752 KB
- Stars: 40
- Watchers: 4
- Forks: 1
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Well Known Bots
This repository contains a list of Well Known Bots, including robots, crawlers,
validators, monitors, and spiders, in a single JSON file. Each bot is identified
and provided a RegExp `pattern` to match against an HTTP `User-Agent` header.
Additional metadata is available on each item.
## Install
### Direct download
Download the [`well-known-bots.json` file][raw-json-url] directly.
## Realities
It's impossible to create a system that can detect all bots. Well-behaving bots
identify themselves in a consistent manner, usually via the User-Agent patterns
this project provides. It is straightforward to identify these well-behaving
bots, but misbehaving bots pretend to be real clients and use various mechanisms
to evade detection.
For more details, see [Non-Technical Notes in the
browser-fingerprinting][non-tech-notes-url] project.
## Structure
Each entry in the JSON represents a specific bot or crawler and includes the following fields:
- id: A unique identifier for the bot
- categories: An array of categories the bot belongs to (e.g., "search-engine", "advertising")
- pattern: A regular expression pattern used to identify the bot in user agent strings
- url: (optional) A URL with more information about the bot
- verification: A list of supported methods for verifying the bot's identity (if the bot is not verifiable it should be empty).
- instances: An array of example user agent strings for the bot
- aliases: Extra unique identifiers for the bot that can be used to identify it across other data sources
### Verification
Each verification entry contains the following fields:
- type: The method of verification (`dns` and `cidr` are supported)
If you specify `dns` verification then these fields are expected:
- masks: An array of mask patterns used for verification
If you specify `cidr` verification then these fields are expected:
- sources: An array of sources to pull cidr range data from (at least one is required)
### Verification mask patterns
The mask patterns use the following special characters:
- *: Represents 0 or 1 of any character
- @: Acts as a wildcard, matching any number of characters
All other characters in the mask require an exact match.
### Cidr verification sources
Each cidr source requires the following fields:
- type: The type of source (Currently only `http-json`) is supported
- url: The url that hosts the ip ranges
- selector: A JsonPath selector that selects all of the IP ranges in the source
## License
The project is a hard-fork of [crawler-user-agents][forked-repo-url] at commit
`46831767324e10c69c9ac6e538c9847853a0feb9`, which is distributed under the [MIT
License][mit-license].
[raw-json-url]: https://raw.githubusercontent.com/arcjet/well-known-bots/main/well-known-bots.json
[forked-repo-url]: https://github.com/monperrus/crawler-user-agents/commit/46831767324e10c69c9ac6e538c9847853a0feb9
[non-tech-notes-url]: https://github.com/niespodd/browser-fingerprinting/blob/baecc60821cefd06eb89a54d18be39d87dd16f2e/README.md#non-technical-notes
[mit-license]: https://opensource.org/licenses/MIT