https://github.com/microlinkhq/cloudflare-bot-directory
CloudFlare Radar verified bots directory – 500+ web crawlers and user agents as JSON.
https://github.com/microlinkhq/cloudflare-bot-directory
bot-detection bots cloudflare cloudflare-radar crawler crawlers dataset datasets googlebot user-agent user-agents user-agents- verified-bots web-crawler web-scraping
Last synced: about 2 months ago
JSON representation
CloudFlare Radar verified bots directory – 500+ web crawlers and user agents as JSON.
- Host: GitHub
- URL: https://github.com/microlinkhq/cloudflare-bot-directory
- Owner: microlinkhq
- License: mit
- Created: 2026-01-29T09:26:02.000Z (4 months ago)
- Default Branch: master
- Last Pushed: 2026-01-29T20:45:06.000Z (4 months ago)
- Last Synced: 2026-01-30T04:42:48.434Z (4 months ago)
- Topics: bot-detection, bots, cloudflare, cloudflare-radar, crawler, crawlers, dataset, datasets, googlebot, user-agent, user-agents, user-agents-, verified-bots, web-crawler, web-scraping
- Language: JavaScript
- Homepage:
- Size: 184 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
# CloudFlare Bot Directory
[](https://www.npmjs.org/package/cloudflare-bot-directory)
> A comprehensive list of **500+ verified bots** and web crawlers from [CloudFlare Radar](https://radar.cloudflare.com/traffic/verified-bots), available as a JSON dataset for bot detection, user agent analysis, and web scraping identification.
## Why
Identifying legitimate bots from malicious scrapers is essential for web security and analytics. This package provides the official CloudFlare Radar verified bots directory, helping you:
- **Detect verified bots** – Identify legitimate crawlers like Googlebot, Bingbot, and more
- **Filter analytics** – Exclude known bots from your traffic reports
- **Allow-list crawlers** – Permit verified bots while blocking suspicious traffic
- **User agent lookup** – Match user agent strings against known bot patterns
## Data Structure
Each bot entry includes:
| Field | Description |
| ------------------- | ---------------------------------------------- |
| `slug` | URL-friendly unique identifier |
| `name` | Human-readable bot name |
| `kind` | Bot kind classification |
| `operator` | Company or organization operating the bot |
| `operatorUrl` | URL to operator's documentation |
| `category` | Bot category (Search Engine, Monitoring, etc.) |
| `description` | What the bot does |
| `followsRobotsTxt` | Whether the bot respects robots.txt |
| `userAgentPatterns` | User agent pattern(s) for matching |
| `userAgents` | Known user agent string(s) |
## Install
```bash
npm install cloudflare-bot-directory
```
## Usage
```js
const bots = require('cloudflare-bot-directory')
// Get all bot slugs
console.log(bots.map(bot => bot.slug))
// ['2checkout', '360monitoring', 'googlebot', 'bingbot', ...]
// Find a specific bot
const googlebot = bots.find(bot => bot.slug === 'googlebot')
// Filter by category
const searchBots = bots.filter(bot => bot.category === 'Search Engine Crawler')
// Check if a user agent is a known bot
const isKnownBot = (userAgent) =>
bots.some(bot => userAgent.includes(bot.name))
```
## Related
- [top-user-agents](https://github.com/microlinkhq/top-user-agents) – An always up-to-date list of the top 100 HTTP user-agents most used over the Internet.
## License
**cloudflare-bot-directory** © [Microlink](https://microlink.io), released under the [MIT](https://github.com/microlinkhq/cloudflare-bot-directory/blob/master/LICENSE.md) License.
Authored and maintained by [Kiko Beats](https://kikobeats.com) with help from [contributors](https://github.com/microlinkhq/cloudflare-bot-directory/contributors).
> [microlink.io](https://microlink.io) · GitHub [microlinkhq](https://github.com/microlinkhq) · X [@microlinkhq](https://x.com/microlinkhq)