{"id":30650548,"url":"https://github.com/devhims/weblinq","last_synced_at":"2025-08-31T06:10:08.213Z","repository":{"id":308471547,"uuid":"993347036","full_name":"devhims/weblinq","owner":"devhims","description":"High-performance web scraping and browser automation platform built on Cloudflare","archived":false,"fork":false,"pushed_at":"2025-08-29T12:26:55.000Z","size":2692,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-29T14:51:17.722Z","etag":null,"topics":["better-auth","d1","durableobject","honojs","nextjs","r2","workers","workers-ai"],"latest_commit_sha":null,"homepage":"https://weblinq.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devhims.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"docs/support/contact.mdx","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-30T16:23:52.000Z","updated_at":"2025-08-29T12:26:58.000Z","dependencies_parsed_at":"2025-08-06T06:10:09.115Z","dependency_job_id":"7d350d55-9a19-483a-b2c2-079c04ef3799","html_url":"https://github.com/devhims/weblinq","commit_stats":null,"previous_names":["devhims/weblinq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/devhims/weblinq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devhims%2Fweblinq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devhims%2Fweblinq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devhims%2Fweblinq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devhims%2Fweblinq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devhims","download_url":"https://codeload.github.com/devhims/weblinq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devhims%2Fweblinq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272944304,"owners_count":25019514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-31T02:00:09.071Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["better-auth","d1","durableobject","honojs","nextjs","r2","workers","workers-ai"],"created_at":"2025-08-31T06:01:58.810Z","updated_at":"2025-08-31T06:10:08.199Z","avatar_url":"https://github.com/devhims.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WebLinq\n\n\u003e High-performance web scraping and browser automation platform\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.8-blue)](https://www.typescriptlang.org/)\n[![Cloudflare Workers](https://img.shields.io/badge/Cloudflare-Workers-orange)](https://workers.cloudflare.com/)\n[![Node.js](https://img.shields.io/badge/Node.js-18+-green)](https://nodejs.org/)\n[![Hono](https://img.shields.io/badge/Hono-4.7-orange)](https://hono.dev/)\n\n## 🚀 Overview\n\nWebLinq is a modern web scraping and browser automation platform that revolutionizes performance through **intelligent browser session reuse**. Built on Cloudflare's edge infrastructure, it provides lightning-fast web operations while maintaining reliability and scalability.\n\n**🎯 Perfect for:** Realtime web access in chat apps, Browser automation, Data aggregation, Competitor analysis, and Market research.\n\n### Key Features\n\n- **🔄 Browser Session Reuse**: Intelligent architecture that reduces operation latency from ~2-3s to ~200-500ms\n- **⚡ High Performance**: Built on Cloudflare Workers for global edge deployment\n- **🎯 Comprehensive API**: Search, Screenshot capture, Markdown / HTML extraction, PDF generation, AI data extraction\n- **🔧 MCP Integration**: Model Context Protocol server for AI assistant integration\n- **🛡️ Enterprise Ready**: Authentication, rate limiting, and secure API key management\n- **📱 Modern Dashboard**: Full-featured web interface for API management\n\n## 🌐 Live Demo\n\nTry WebLinq instantly with our interactive API:\n\n```bash\n# Extract markdown from any webpage\ncurl -X POST \"https://api.weblinq.dev/v1/web/markdown\" \\\n  -H \"Authorization: Bearer YOUR_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"https://example.com\"}'\n\n# Take a screenshot\ncurl -X POST \"https://api.weblinq.dev/v1/web/screenshot\" \\\n  -H \"Authorization: Bearer YOUR_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"https://example.com\"}'\n```\n\n**🔗 [Get your free API key](https://weblinq.dev/dashboard/api-keys)** • **📖 [View live documentation](https://docs.weblinq.dev)**\n\n## 🏗️ Directory Structure\n\n```\nweblinq/\n├── backend/                    # Core API server (Cloudflare Worker)\n│   ├── src/\n│   │   ├── durable-objects/   # Browser session management\n│   │   ├── routes/            # API endpoints\n│   │   ├── lib/               # Core utilities and operations\n│   │   └── middlewares/       # Authentication and CORS\n│   └── scripts/               # Build and deployment scripts\n├── frontend/                   # Next.js 15 dashboard application\n│   ├── src/\n│   │   ├── app/               # App router pages\n│   │   ├── components/        # Reusable UI components\n│   │   └── lib/               # Client utilities\n├── weblinq-mcp/              # Model Context Protocol server\n│   └── src/                   # MCP implementation\n├── docs/                      # Mintlify documentation site\n│   ├── api-reference/         # API documentation\n│   └── guides/                # User guides and examples\n└── tests/                     # Integration testing suite\n```\n\n## 🔄 Browser Session Reuse Innovation\n\nWebLinq's core innovation lies in its **intelligent browser session reuse architecture** powered by Cloudflare Durable Objects:\n\n### Architecture Overview\n\n- **BrowserManagerDO**: Orchestrates up to 10 concurrent browser sessions\n- **BrowserDO**: Manages individual Playwright/Puppeteer browser instances\n- **Session Pooling**: Maintains warm sessions across requests\n- **Blue-Green Refresh**: Zero-downtime session rotation every 8.5 minutes\n\n### Performance Benefits\n\n| Metric                    | Traditional                   | WebLinq                    |\n| ------------------------- | ----------------------------- | -------------------------- |\n| **Cold Start Latency**    | 2-3 seconds                   | 200-500ms                  |\n| **Resource Efficiency**   | ❌ New browser per request    | ✅ Persistent sessions     |\n| **Concurrent Operations** | Limited by startup time       | Up to 10 parallel sessions |\n| **Cost Optimization**     | High browser startup overhead | Reduced slot usage         |\n\n### How It Works\n\n1. **Session Management**: Durable Objects maintain persistent browser sessions\n2. **Intelligent Allocation**: Available sessions are reused; new ones created on-demand\n3. **Proactive Refresh**: Sessions are refreshed before Cloudflare's 10-minute limit\n4. **Fault Tolerance**: Automatic recovery from crashes and network issues\n\n## 🛠️ Core Technologies\n\n### Backend Stack\n\nBuilt with modern, high-performance technologies:\n\n- **[Hono.js](https://hono.dev/)** `^4.7.10` - Ultra-fast web framework\n- **[Drizzle ORM](https://orm.drizzle.team/)** `^0.43.1` - Type-safe database operations\n- **[Zod](https://zod.dev/)** `^3.25.28` - Runtime type validation\n- **[Better Auth](https://better-auth.com/)** `^1.2.8` - Modern authentication\n- **[@cloudflare/puppeteer](https://github.com/cloudflare/puppeteer)** - Browser automation\n\n## 🔌 MCP Server Integration\n\nThe **`weblinq-mcp/`** directory contains a complete [Model Context Protocol](https://modelcontextprotocol.io/) server implementation, enabling seamless integration with AI assistants like Claude Desktop and other MCP-compatible clients.\n\n### Features\n\n- **🔗 Direct API Integration**: Connect AI assistants to WebLinq's full API\n- **🔄 Real-time Operations**: Screenshot capture, data extraction, web search\n- **🛡️ Secure Authentication**: API key-based access control\n- **📊 Structured Responses**: Type-safe data exchange with AI models\n\n### Usage\n\n```bash\ncd weblinq-mcp\nnpm install\nnpm run dev  # Development server\nnpm run deploy  # Deploy to Cloudflare Workers\n```\n\nThe MCP server provides AI assistants with tools for web scraping, screenshot capture, and data extraction, making WebLinq's capabilities directly accessible within AI workflows.\n\n## 🚀 Quick Start\n\n### For Developers\n\n```bash\n# Clone and setup\ngit clone https://github.com/devhims/weblinq.git\ncd weblinq\n\n# Install dependencies\ncd backend \u0026\u0026 pnpm install\ncd ../frontend \u0026\u0026 pnpm install\n\n# Setup environment\ncp backend/.env.example backend/.env\ncp frontend/.env.example frontend/.env.local\n\n# Start development\ncd backend \u0026\u0026 pnpm dev    # Backend: http://localhost:8787\ncd frontend \u0026\u0026 pnpm dev   # Frontend: http://localhost:3000\n```\n\n**Requirements:** Node.js 18+, Cloudflare account with Workers/D1/Durable Objects enabled\n\n📖 **[Full setup guide in CONTRIBUTING.md](CONTRIBUTING.md)**\n\n## 📚 Documentation\n\n- **📖 [API Documentation](./docs/)** - Complete API reference and guides\n- **🚀 [Quick Start Guide](./docs/getting-started/quickstart.mdx)** - Get started in 5 minutes\n- **🔧 [Developer Guide](./docs/guides/examples.mdx)** - Integration examples\n- **🔒 [Authentication](./docs/getting-started/authentication.mdx)** - API key setup\n- **🛡️ [Security Policy](./SECURITY.md)** - Vulnerability reporting and best practices\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for detailed information on:\n\n- 🛠️ Development setup and workflow\n- 📋 Code style and standards\n- 🧪 Testing requirements\n- 🐛 Bug reporting process\n- 💡 Feature request guidelines\n\n## 📄 License\n\nThis project is licensed under the **MIT License** - see the [LICENSE](./LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- **Cloudflare** - For Workers, Durable Objects, and Browser Rendering API\n- **Hono.js** - For the clean, lightning-fast web framework\n- **Better Auth** - The most complete authentication framework\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**[Documentation](./docs/) • [API Reference](./docs/api-reference/) • [Examples](./docs/guides/examples.mdx) • [Contributing](CONTRIBUTING.md)**\n\nMade with ❤️ by the WebLinq team\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevhims%2Fweblinq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevhims%2Fweblinq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevhims%2Fweblinq/lists"}