{"id":40340986,"url":"https://github.com/aniketchaudhari3/botmd","last_synced_at":"2026-01-20T09:02:56.751Z","repository":{"id":325519513,"uuid":"1069906779","full_name":"aniketchaudhari3/botmd","owner":"aniketchaudhari3","description":null,"archived":false,"fork":false,"pushed_at":"2025-11-21T19:31:15.000Z","size":191,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-11-21T21:11:37.273Z","etag":null,"topics":["agent","ai","bot","llm","markdown","middleware","nextjs"],"latest_commit_sha":null,"homepage":"https://botmd-docs.vercel.app","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aniketchaudhari3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-04T21:19:43.000Z","updated_at":"2025-11-21T19:31:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/aniketchaudhari3/botmd","commit_stats":null,"previous_names":["aniketchaudhari3/botmd"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/aniketchaudhari3/botmd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketchaudhari3%2Fbotmd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketchaudhari3%2Fbotmd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketchaudhari3%2Fbotmd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketchaudhari3%2Fbotmd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aniketchaudhari3","download_url":"https://codeload.github.com/aniketchaudhari3/botmd/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aniketchaudhari3%2Fbotmd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28599840,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T08:51:33.170Z","status":"ssl_error","status_checked_at":"2026-01-20T08:51:10.855Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","ai","bot","llm","markdown","middleware","nextjs"],"created_at":"2026-01-20T09:02:56.680Z","updated_at":"2026-01-20T09:02:56.738Z","avatar_url":"https://github.com/aniketchaudhari3.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# botmd\n\n\u003e Universal AI bot markdown middleware for any JavaScript framework.\n\nConvert your HTML pages to clean, structured markdown automatically when AI bots visit your site. Reduce token usage, improve AI comprehension, and make your content more accessible to AI models.\n\n[![npm version](https://img.shields.io/npm/v/botmd.svg)](https://www.npmjs.com/package/botmd)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**📚 [View Full Documentation →](https://botmd-docs.vercel.app)**\n\n## ✨ Features\n\n- **🤖 Smart Bot Detection** - Automatically detects 50+ AI bots and crawlers\n- **📝 HTML to Markdown** - Clean, structured markdown with absolute URLs\n- **⚡ High Performance** - Built-in LRU cache with TTL\n- **🎯 Path Control** - Fine-grained control over which paths get converted\n- **🌐 Framework Agnostic** - Works with Next.js, Express, Hono, Bun, NestJS, and more\n- **🚀 Edge Ready** - Runs in Node.js and Edge runtimes\n- **📦 Zero Config** - Works out of the box with sensible defaults\n- **🔒 SSRF Protection** - Built-in security against server-side request forgery\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\nnpm install botmd\n# or\npnpm add botmd\n# or\nyarn add botmd\n# or\nbun add botmd\n```\n\n### Next.js Example\n\n```typescript\n// middleware.ts\nimport { Botmd } from 'botmd';\nimport { NextRequest, NextResponse } from 'next/server';\n\nconst botmd = new Botmd({\n  paths: {\n    allowed: ['/docs/**', '/blog/**'],\n    disallowed: ['/api/**', '/admin/**']\n  },\n  logRequests: true\n});\n\nexport async function middleware(request: NextRequest) {\n  // Skip internal requests\n  if (Botmd.shouldSkip(request)) {\n    return NextResponse.next();\n  }\n\n  const result = await botmd.createResponse(request);\n  \n  if (!result.shouldConvert) {\n    return NextResponse.next();\n  }\n  \n  return new NextResponse(result.content, { \n    headers: result.headers \n  });\n}\n\nexport const config = {\n  matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)']\n};\n```\n\n### Express Example\n\n```typescript\nimport express from 'express';\nimport { Botmd } from 'botmd';\n\nconst app = express();\nconst botmd = new Botmd({\n  paths: { disallowed: ['/api/**'] }\n});\n\napp.use(async (req, res, next) =\u003e {\n  const result = await botmd.createResponse(req);\n  \n  if (!result.shouldConvert) {\n    return next();\n  }\n  \n  res.set(result.headers);\n  res.send(result.content);\n});\n\napp.listen(3000);\n```\n\n## 🤖 Detected Bots\n\nBotmd automatically detects 50+ AI bots including:\n\n### AI Assistants \u0026 Search\n- **OpenAI**: GPTBot, ChatGPT-User, OAI-SearchBot\n- **Anthropic**: ClaudeBot, Claude-Web, anthropic-ai\n- **Perplexity**: PerplexityBot, Perplexity-User\n- **Google**: Google-Extended, Googlebot\n- **Meta**: meta-externalfetcher\n- **Microsoft**: bingbot\n\n### Coding Assistants\n- **GitHub Copilot**: GitHubCopilot, CopilotBot\n- **Cursor**: Cursor, CursorAgent, CursorBot\n- **Codeium**: Windsurf, CodeiumAgent\n- **Tabnine**: TabnineAgent\n- **Replit**: ReplitAgent, ReplitAI\n\n### Crawlers \u0026 Tools\n- **Firecrawl**: FirecrawlAgent\n- **Jina**: JinaBot, JinaReader\n- **Tavily**: TavilyBot, TavilySearchBot\n- **Exa**: ExaBot\n- **Amazon**: Amazonbot\n- **Apple**: Applebot, iTMS\n- **Others**: CCBot, Diffbot, DuckAssistBot, Bytespider, TikTokSpider\n\n[See full list](./packages/botmd/src/ai-bots.ts)\n\n## ⚙️ Configuration\n\n```typescript\ninterface BotmdConfig {\n  // Enable/disable the middleware\n  enabled?: boolean; // default: true\n  \n  // Path filtering\n  paths?: {\n    allowed?: (string | RegExp)[]; // e.g., ['/docs/**', '/blog/*']\n    disallowed?: (string | RegExp)[]; // e.g., ['/api/**', '/admin/**']\n  };\n  \n  // User agent filtering\n  userAgents?: {\n    allowed?: (string | RegExp)[]; // Custom bots to allow\n    disallowed?: (string | RegExp)[]; // Bots to block\n  };\n  \n  // Caching\n  cache?: {\n    enabled?: boolean; // default: true\n    ttl?: number; // default: 86400000 (1 day in ms)\n    maxSize?: number; // default: 1000 entries\n  };\n  \n  // Logging\n  logRequests?: boolean; // default: false\n  debug?: boolean; // default: false\n}\n```\n\n### Path Patterns\n\n```typescript\n'/docs'        // Exact match\n'/docs/*'      // Single level: /docs/intro ✓, /docs/guide/setup ✗\n'/docs/**'     // Multi level: /docs/intro ✓, /docs/guide/setup ✓\n/^\\/api\\/.*/   // RegExp patterns\n```\n\n### Common Configurations\n\n```typescript\n// Allow all paths (default)\nconst botmd = new Botmd();\n\n// Only specific paths\nconst botmd = new Botmd({\n  paths: { allowed: ['/docs/**', '/blog/**'] }\n});\n\n// Exclude sensitive paths\nconst botmd = new Botmd({\n  paths: { disallowed: ['/api/**', '/admin/**'] }\n});\n```\n\n## 📚 API\n\n```typescript\nnew Botmd(config?)                    // Create instance\nawait botmd.createResponse(request)   // Process request → BotmdResponse\nbotmd.clearCache()                    // Clear cache\nBotmd.shouldSkip(request)             // Check if internal request\n```\n\n[Complete API documentation →](https://botmd-docs.vercel.app/docs/api-reference)\n\n## 🧪 Testing\n\nTest with curl:\n\n```bash\n# Regular request (gets HTML)\ncurl http://localhost:3000/docs\n\n# Bot request (gets Markdown)\ncurl -H \"User-Agent: GPTBot\" http://localhost:3000/docs\ncurl -H \"User-Agent: Claude-Web\" http://localhost:3000/docs\n\n# Explicit markdown request\ncurl -H \"Accept: text/markdown\" http://localhost:3000/docs\n```\n\n## 🎯 How It Works\n\n1. **Request Normalization** - Extract URL and headers from any request format\n2. **Configuration Check** - Verify botmd is enabled and path is allowed\n3. **Bot Detection** - Check `Accept: text/markdown` header or user-agent patterns\n4. **Cache Check** - Return cached markdown if available (with TTL)\n5. **HTML Fetch** - Internally fetch HTML with loop prevention\n6. **Conversion** - Transform HTML to clean markdown with absolute URLs\n7. **Cache Store** - Store result for future requests\n8. **Response** - Return markdown with appropriate headers\n\n\n## 🚀 Performance\n\n- **Zero dependencies** for HTML conversion (regex-based)\n- **LRU cache** with TTL prevents redundant conversions\n- **Edge compatible** - no Node.js-specific APIs required\n- **Fast path matching** - optimized for common patterns\n- **~14KB minified** - minimal bundle impact\n\n## 📖 Documentation\n\n**[Read the full docs at botmd-docs.vercel.app →](https://botmd-docs.vercel.app)**\n\n## 📝 License\n\nMIT License - see [LICENSE](./packages/botmd/LICENSE) file for details.\n\n---\n\n**Made with ❤️ for developers building AI-accessible applications**\n\nFor questions, issues, or feature requests, please [open an issue](https://github.com/aniketchaudhari3/botmd/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniketchaudhari3%2Fbotmd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faniketchaudhari3%2Fbotmd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faniketchaudhari3%2Fbotmd/lists"}