{"id":41276713,"url":"https://github.com/cnlangzi/botrate","last_synced_at":"2026-01-23T02:35:34.077Z","repository":{"id":331716567,"uuid":"1119611071","full_name":"cnlangzi/botrate","owner":"cnlangzi","description":"🛡️ An GEO/SEO-friendly rate limiter for Go. Protects your server from abuse/DDoS while safely whitelisting verified Search Engines \u0026 AI Bots (Google, Bing, GPTBot, etc.) via IP/RDNS verification.","archived":false,"fork":false,"pushed_at":"2026-01-11T07:05:50.000Z","size":28,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-11T07:12:07.077Z","etag":null,"topics":["ai","ai-robots","geo-robots","googlebot","gptbot","ratelimit","ratelimiter","seo-robots","seo-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cnlangzi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-19T14:52:33.000Z","updated_at":"2026-01-11T05:05:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cnlangzi/botrate","commit_stats":null,"previous_names":["cnlangzi/botrate"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/cnlangzi/botrate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnlangzi%2Fbotrate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnlangzi%2Fbotrate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnlangzi%2Fbotrate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnlangzi%2Fbotrate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cnlangzi","download_url":"https://codeload.github.com/cnlangzi/botrate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnlangzi%2Fbotrate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28679003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T01:00:35.747Z","status":"online","status_checked_at":"2026-01-23T02:00:08.296Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-robots","geo-robots","googlebot","gptbot","ratelimit","ratelimiter","seo-robots","seo-tools"],"created_at":"2026-01-23T02:35:33.993Z","updated_at":"2026-01-23T02:35:34.064Z","avatar_url":"https://github.com/cnlangzi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# botrate\n\n\u003cdiv align=\"center\"\u003e\n\n**High-performance, SEO-friendly rate limiter for Go applications**\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/cnlangzi/botrate.svg)](https://pkg.go.dev/github.com/cnlangzi/botrate)\n[![Go Report Card](https://goreportcard.com/badge/github.com/cnlangzi/botrate)](https://goreportcard.com/report/github.com/cnlangzi/botrate)\n[![codecov](https://codecov.io/gh/cnlangzi/botrate/branch/main/graph/badge.svg)](https://codecov.io/gh/cnlangzi/botrate)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\u003c/div\u003e\n\n---\n\n## Overview\n\nBotRate is a high-performance rate limiter designed specifically for modern web applications. Unlike traditional rate limiters that blindly block high-frequency IPs, **botrate intelligently distinguishes between malicious scrapers and verified bots** (Search Engines, AI Crawlers, etc.).\n\nThis ensures your site remains protected from abuse **without sacrificing your SEO rankings** or AI knowledge base presence.\n\n## Features\n\n- 🛡️ **Smart Bot Detection** - Uses `knownbots` library for verified bot identification (Googlebot, Bingbot, GPTBot, ClaudeBot, etc.)\n- 🔒 **Behavior Analysis** - Asynchronous IP+URL pattern detection to identify scrapers\n- ⚡ **High Performance** - \u003c2μs hot path latency, only rate limits blacklisted IPs\n- 💾 **Memory Efficient** - Only creates token buckets for blacklisted IPs\n- 🎯 **Flexible** - HTTP callback handling is left to caller for maximum compatibility\n- 🔄 **Graceful Shutdown** - Proper resource cleanup with `Close()` method\n\n## Installation\n\n```bash\ngo get github.com/cnlangzi/botrate\n```\n\n## Quick Start\n\n### Basic Usage\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/cnlangzi/botrate\"\n\t\"golang.org/x/time/rate\"\n)\n\nfunc main() {\n\tlimiter, err := botrate.New(\n\t\t// Rate limiting for blacklisted IPs only\n\t\tbotrate.WithLimit(rate.Every(10*time.Minute)),\n\n\t\t// Behavior analysis\n\t\tbotrate.WithAnalyzerWindow(time.Minute),\n\t\tbotrate.WithAnalyzerPageThreshold(50),\n\t\tbotrate.WithAnalyzerQueueCap(10000),\n\t)\n\tif err != nil {\n\t\tlog.Fatalf(\"Failed to create limiter: %v\", err)\n\t}\n\tdefer limiter.Close()\n\n\thandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tua := r.UserAgent()\n\t\tip := extractIP(r)\n\n\t\tif !limiter.Allow(ua, ip) {\n\t\t\thttp.Error(w, \"Too Many Requests\", http.StatusTooManyRequests)\n\t\t\treturn\n\t\t}\n\n\t\tw.Write([]byte(\"Hello, World!\"))\n\t})\n\n\tfmt.Println(\"Server started on :8080\")\n\thttp.Handle(\"/\", handler)\n\thttp.ListenAndServe(\":8080\", nil)\n}\n\n// extractIP extracts the real client IP from the request.\nfunc extractIP(r *http.Request) string {\n\tif xff := r.Header.Get(\"X-Forwarded-For\"); xff != \"\" {\n\t\treturn strings.TrimSpace(strings.Split(xff, \",\")[0])\n\t}\n\tif xri := r.Header.Get(\"X-Real-IP\"); xri != \"\" {\n\t\treturn xri\n\t}\n\treturn r.RemoteAddr\n}\n```\n\n### Using Wait Method (Blocking)\n\nFor scenarios where you want to wait instead of immediately rejecting:\n\n```go\nhandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\tif err := limiter.Wait(r.Context(), ua, ip); err != nil {\n\t\thttp.Error(w, err.Error(), http.StatusTooManyRequests)\n\t\treturn\n\t}\n\tw.Write([]byte(\"Hello!\"))\n})\n```\n\n### Middleware Pattern\n\n```go\nfunc BotRateMiddleware(l *botrate.Limiter) func(http.Handler) http.Handler {\n\treturn func(next http.Handler) http.Handler {\n\t\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\t\tif !l.Allow(r.UserAgent(), extractIP(r)) {\n\t\t\t\thttp.Error(w, \"Rate limit exceeded\", http.StatusTooManyRequests)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tnext.ServeHTTP(w, r)\n\t\t})\n\t}\n}\n\n// Usage\nhttp.Handle(\"/\", BotRateMiddleware(limiter)(myHandler))\n```\n\n## API Reference\n\n### Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `WithLimit(rate.Limit)` | Requests per second for blocked IPs | `rate.Every(10*time.Minute)` |\n| `WithAnalyzerWindow(time.Duration)` | Analysis window duration | `5*time.Minute` |\n| `WithAnalyzerPageThreshold(int)` | Max distinct pages threshold | `50` |\n| `WithAnalyzerQueueCap(int)` | Event queue capacity | `10000` |\n| `WithKnownbots(*knownbots.Validator)` | Custom knownbots validator | `nil` (use default) |\n\n### Methods\n\n#### `Allow(ua, ip string) bool`\n\nNon-blocking check if the request should proceed. Returns `true` if allowed, `false` if blocked.\n\n**Bot Detection Logic:**\n- **Verified bot** (StatusVerified): ✅ Allow immediately\n- **RDNS lookup failed** (StatusPending): ✅ Allow, retry verification next time\n- **Fake bot** (StatusFailed): ❌ Block immediately\n- **Normal user**: Continue to analyzer and blocklist check\n\n```go\nallowed := limiter.Allow(ua, ip)\nif !allowed {\n    // Request was blocked (fake bot or blacklisted IP)\n}\n```\nresult := limiter.Allow(ua, ip)\nif !result.Allowed {\n\t// Handle denial\n}\n```\n\n#### `Wait(ctx context.Context, ua, ip string) error`\n\nBlocks until the request is allowed or the context ends. Returns `nil` if allowed, `ErrLimit` if blocked.\n\n**Bot Detection Logic:**\n- **Verified bot** (StatusVerified): ✅ Allow immediately\n- **RDNS lookup failed** (StatusPending): ✅ Allow, retry verification next time\n- **Fake bot** (StatusFailed): ❌ Block immediately\n- **Normal user**: Continue to analyzer and blocklist check\n\n```go\nerr := limiter.Wait(ctx, ua, ip)\nif err != nil {\n    // Handle denial (ErrLimit) or context cancellation\n}\n```\n\n#### `Close()`\n\nGracefully shuts down the limiter and releases resources. **Always call this when the limiter is no longer needed.**\n\n```go\nlimiter, err := botrate.New(...)\nif err != nil {\n    log.Fatalf(\"Failed to create limiter: %v\", err)\n}\ndefer limiter.Close()\n```\n\n## How It Works\n\n```\nRequest\n  │\n  ▼\n┌─────────────────────────────────────┐\n│ 1. KnownBots Verification           │  Hot path: \u003c1μs\n│    - Check if UA matches known bot  │\n│    - Verified → Allow immediately   │\n│    - RDNS failed → Allow, retry     │\n│    - Fake bot → Block immediately   │\n└─────────────────────────────────────┘\n  │\n  ▼ (only for normal users)\n┌─────────────────────────────────────┐\n│ 2. Blocklist Check                  │  Atomic read: \u003c100ns\n│    - Check if IP is blacklisted     │\n│    - Not blocked → Record + Allow   │\n│    - Blocked → Rate limit           │\n└─────────────────────────────────────┘\n  │\n  ▼ (async)\n┌─────────────────────────────────────┐\n│ 3. Behavior Analysis                │  Background worker\n│    - Record IP+URL combination      │\n│    - Bloom filter deduplication     │\n│    - Visit counter increment        │\n│    - Threshold exceeded → Block     │\n└─────────────────────────────────────┘\n```\n\n### Key Design Decisions\n\n1. **Fake bots blocked immediately** - Known bot UAs with failed verification are blocked without rate limiting\n2. **RDNS lookup failures are tolerated** - Failed DNS lookups allow the request (will retry next time)\n3. **Verified bots bypass everything** - Googlebot, Bingbot, etc. are allowed without rate limiting\n4. **Normal users go through analyzer** - Behavior analysis only applies to regular users\n5. **Async behavior analysis** - Request processing is never blocked by analysis\n\n## Performance\n\n| Scenario | Latency | Memory |\n|----------|---------|--------|\n| **Normal user** | \u003c1.5μs | 0 bytes |\n| **Verified bot** | \u003c1μs | 0 bytes |\n| **Blacklisted IP** | \u003c2μs | ~200 bytes/IP |\n| **Fake bot** | \u003c1μs | 0 bytes |\n\n**Total memory budget**: \u003c5MB (Bloom: 1MB + Counter: 1MB + Blacklisted IPs: variable)\n\n### Benchmark Results\n\n```bash\n$ go test -run=^$ -bench=. -benchmem -cpu=1,4,8\n```\n\nKey metrics to monitor:\n- **ns/op** - Nanoseconds per operation (lower is better)\n- **B/op** - Bytes allocated per operation (should be 0 for hot path)\n- **allocs/op** - Allocations per operation (should be 0 for hot path)\n\n## Error Handling\n\n### Errors\n\n```go\nvar ErrLimit = context.DeadlineExceeded\n```\n\nCheck errors with:\n\n```go\nif errors.Is(err, botrate.ErrLimit) {\n    // Request was denied (fake bot or blacklisted IP)\n}\n```\n\n### Denial Reasons\n\n`Allow()` returns `false` when:\n\n1. **Fake bot** - Known bot UA (e.g., \"GPTBot\") but IP verification failed\n2. **Blacklisted IP** - IP was flagged by behavior analysis\n\n`Wait()` returns `ErrLimit` when:\n\n1. **Fake bot** - Blocked immediately\n2. **Rate limited** - Normal user on blocklist hitting rate limit\n\n```go\nallowed := limiter.Allow(ua, ip)\n\nif !allowed {\n    // Request was denied\n    // - Fake bot: blocked immediately\n    // - Blacklisted IP: rate limited\n}\n```\n\n## Configuration Examples\n\n### Strict Rate Limiting\n\n```go\nlimiter, err := botrate.New(\n\tbotrate.WithLimit(rate.Every(10*time.Minute)),\n)\nif err != nil {\n    log.Fatalf(\"Failed to create limiter: %v\", err)\n}\n```\n\n### Aggressive Bot Detection\n\n```go\nlimiter, err := botrate.New(\n\tbotrate.WithAnalyzerWindow(30*time.Second),\n\tbotrate.WithAnalyzerPageThreshold(20),\n)\nif err != nil {\n    log.Fatalf(\"Failed to create limiter: %v\", err)\n}\n```\n\n### High-Throughput Configuration\n\n```go\nlimiter, err := botrate.New(\n\tbotrate.WithAnalyzerWindow(10*time.Minute),\n\tbotrate.WithAnalyzerPageThreshold(100),\n\tbotrate.WithAnalyzerQueueCap(50000),\n)\nif err != nil {\n    log.Fatalf(\"Failed to create limiter: %v\", err)\n}\n```\n\n### Custom KnownBots Validator\n\n```go\n// Create custom validator with specific configuration\ncustomKB, err := knownbots.New(\n\tknownbots.WithRoot(\"./custom-bots\"),\n\tknownbots.WithSchedulerInterval(12*time.Hour),\n)\nif err != nil {\n    log.Fatalf(\"Failed to create validator: %v\", err)\n}\n\n// Use custom validator\nlimiter, err := botrate.New(\n\tbotrate.WithKnownbots(customKB),\n\tbotrate.WithLimit(rate.Every(5*time.Minute)),\n)\nif err != nil {\n    log.Fatalf(\"Failed to create limiter: %v\", err)\n}\n```\n\n## Architecture\n\n```\nbotrate/\n├── limiter.go          # Main Limiter type and API\n├── botrate.go          # Error definitions\n├── config.go           # Configuration struct\n├── options.go          # Functional options\n├── analyzer/           # Behavior analysis engine\n│   ├── analyzer.go    # Core analyzer with worker\n│   ├── bloom.go       # Double-buffered Bloom filter\n│   └── counter.go     # LRU visit counter (O(1))\n└── example/\n    └── main.go        # Working example\n```\n\n## Development\n\n### Makefile Commands\n\nA Makefile is provided for common development tasks:\n\n```bash\nmake help          # Show available commands\nmake test          # Run all tests (short + race)\nmake test-short    # Run short tests (fast)\nmake test-race     # Run tests with race detector\nmake test-coverage # Run tests with coverage report\nmake bench         # Run benchmarks (1 and 4 CPUs)\nmake bench-all     # Run all benchmarks (1, 4, 8 CPUs)\nmake build         # Build the project\nmake clean         # Clean build artifacts\n```\n\n### Examples\n\n```bash\n# Run all tests\nmake test\n\n# Run benchmarks\nmake bench\n\n# Generate coverage report\nmake test-coverage\n\n# Run benchmarks with detailed output\nmake bench-all\n```\n\n## Contributing\n\nContributions are welcome! Please read our contributing guidelines before submitting PRs.\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for your changes\n4. Ensure all tests pass: `make test`\n5. Run benchmarks: `make bench`\n6. Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Acknowledgments\n\n- [knownbots](https://github.com/cnlangzi/knownbots) - Bot detection library\n- [bloom](https://github.com/bits-and-blooms/bloom/v3) - Bloom filter implementation\n- [golang/x/time](https://pkg.go.dev/golang.org/x/time/rate) - Token bucket rate limiter\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnlangzi%2Fbotrate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcnlangzi%2Fbotrate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnlangzi%2Fbotrate/lists"}