{"id":30106114,"url":"https://github.com/blocklet/snap-kit","last_synced_at":"2026-06-19T01:31:40.656Z","repository":{"id":294559290,"uuid":"986805915","full_name":"blocklet/snap-kit","owner":"blocklet","description":"Snap Kit is a powerful, Puppeteer-based service designed for seamless web automation. It enables you to effortlessly capture high-fidelity web page screenshots and efficiently scrape web content for precise data extraction.","archived":false,"fork":false,"pushed_at":"2025-12-26T11:10:10.000Z","size":6878,"stargazers_count":3,"open_issues_count":4,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-12-27T23:05:12.647Z","etag":null,"topics":["browser-automation","puppeteer","self-hosting","web-crawling","web-screenshot"],"latest_commit_sha":null,"homepage":"https://snap.createblocklet.dev/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blocklet.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-20T06:39:47.000Z","updated_at":"2025-12-17T04:10:58.000Z","dependencies_parsed_at":"2025-05-21T02:26:39.784Z","dependency_job_id":"d95655ca-90e7-4b04-8cdb-b2d92d7fc8e6","html_url":"https://github.com/blocklet/snap-kit","commit_stats":null,"previous_names":["blocklet/snap-kit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/blocklet/snap-kit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blocklet%2Fsnap-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blocklet%2Fsnap-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blocklet%2Fsnap-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blocklet%2Fsnap-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blocklet","download_url":"https://codeload.github.com/blocklet/snap-kit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blocklet%2Fsnap-kit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34514282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser-automation","puppeteer","self-hosting","web-crawling","web-screenshot"],"created_at":"2025-08-10T01:01:47.013Z","updated_at":"2026-06-19T01:31:40.628Z","avatar_url":"https://github.com/blocklet.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📸 Snap Kit\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"blocklets/snap-kit/logo.png\" alt=\"Snap Kit Logo\" width=\"128\" height=\"128\"\u003e\n\n  **Enterprise-grade web automation platform powered by Puppeteer**\n\n  ![License](https://img.shields.io/badge/license-MIT-blue.svg)\n  ![Version](https://img.shields.io/badge/version-1.3.0-green.svg)\n  ![Node.js](https://img.shields.io/badge/Node.js-18%2B-brightgreen.svg)\n  ![TypeScript](https://img.shields.io/badge/TypeScript-5.7%2B-blue.svg)\n\n  [Features](#features) • [Quick Start](#quick-start) • [Architecture](#architecture) • [API Reference](#api-reference) • [Examples](#examples)\n\u003c/div\u003e\n\n---\n\n## 🚀 Why Snap Kit?\n\n**Snap Kit** is a production-ready web automation platform that transforms how you handle web scraping, screenshot generation, and SEO optimization. Built on the powerful Blocklet ecosystem, it delivers enterprise-grade reliability with developer-friendly APIs.\n\n### 💡 Key Benefits\n\n- **Zero Configuration**: Deploy instantly with Docker or Blocklet Server\n- **Production Scale**: Handle thousands of concurrent requests with built-in queuing\n- **SEO Powerhouse**: Pre-render SPAs for perfect search engine indexing\n- **Developer Experience**: Modern TypeScript APIs with comprehensive documentation\n- **Cost Effective**: Self-hosted solution with no per-request fees\n\n## ✨ Features\n\n### 🎯 Core Capabilities\n\n- **High-Fidelity Screenshots**: Capture pixel-perfect web page snapshots\n- **Smart Content Extraction**: Extract structured data with advanced parsing\n- **Batch Processing**: Process multiple URLs efficiently with queue management\n- **Sitemap Crawling**: Automatically discover and crawl entire websites\n- **SEO Pre-rendering**: Generate search-engine-friendly HTML for SPAs\n- **Custom Headers \u0026 Cookies**: Full control over request customization\n\n### 🔧 Technical Highlights\n\n- **Puppeteer Integration**: Latest Chrome automation capabilities\n- **SQLite Database**: Efficient data storage with Sequelize ORM\n- **React 19.1 UI**: Modern, responsive web interface\n- **Express 4.21 API**: RESTful endpoints with TypeScript\n- **Blocklet Platform**: One-click deployment and scaling\n\n## 📦 What's Included\n\nThis monorepo contains three production-ready modules:\n\n### 🏗️ Snap Kit Blocklet\n\n**Location**: `blocklets/snap-kit`\n\nThe main application featuring:\n\n- **React Frontend**: Modern UI for managing crawling tasks\n- **Express API**: RESTful endpoints for automation\n- **DID Authentication**: Secure access control\n- **Real-time Dashboard**: Monitor crawling progress\n\n### 🕷️ Crawler Engine\n\n**Location**: `packages/crawler`\n\nCore automation engine with:\n\n- **Puppeteer Integration**: Latest Chrome automation\n- **Database Management**: SQLite with migrations\n- **Queue System**: Efficient batch processing\n- **Scheduled Tasks**: Automated crawling workflows\n\n### 🌐 SEO Middleware\n\n**Location**: `packages/middleware`\n\nExpress middleware for:\n\n- **Pre-rendering**: Generate static HTML for SPAs\n- **Cache Management**: Intelligent caching strategies\n- **Search Engine Optimization**: Perfect SEO for dynamic content\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Node.js 18+\n- pnpm (recommended) or npm\n- Docker (optional)\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/blocklet/snap-kit.git\ncd snap-kit\n\n# Install dependencies\npnpm install\n\n# Start development environment\npnpm dev\n```\n\n### Docker Deployment\n\n```bash\n# Build and run with Docker\ndocker run -p 3000:3000 arcblock/snap-kit\n```\n\n### Blocklet Server\n\n```bash\n# Deploy to Blocklet Server\nnpm run deploy\n```\n\n## 🏗️ Architecture\n\n```\n┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐\n│                     │    │                     │    │                     │\n│   React Frontend    │───▶│   Express API       │───▶│   Crawler Engine    │\n│                     │    │                     │    │                     │\n│  • Modern UI        │    │  • RESTful API      │    │  • Puppeteer        │\n│  • Real-time        │    │  • Authentication   │    │  • Queue System     │\n│  • Dashboard        │    │  • Rate Limiting    │    │  • SQLite Storage   │\n│                     │    │                     │    │                     │\n└─────────────────────┘    └─────────────────────┘    └─────────────────────┘\n```\n\n### Technology Stack\n\n- **Frontend**: React 19.1, TypeScript, Vite 7.0\n- **Backend**: Express 4.21, TypeScript, DID Auth\n- **Database**: SQLite with Sequelize ORM\n- **Automation**: Puppeteer, @blocklet/puppeteer\n- **Deployment**: Blocklet Platform, Docker\n\n## 📚 API Reference\n\n### Screenshot Generation\n\n```typescript\nPOST /api/screenshot\n{\n  \"url\": \"https://example.com\",\n  \"options\": {\n    \"width\": 1920,\n    \"height\": 1080,\n    \"fullPage\": true\n  }\n}\n```\n\n### Content Extraction\n\n```typescript\nPOST /api/extract\n{\n  \"url\": \"https://example.com\",\n  \"selectors\": {\n    \"title\": \"h1\",\n    \"description\": \"meta[name='description']\"\n  }\n}\n```\n\n### Batch Processing\n\n```typescript\nPOST /api/crawl/batch\n{\n  \"urls\": [\"https://site1.com\", \"https://site2.com\"],\n  \"options\": {\n    \"priority\": \"high\",\n    \"schedule\": \"immediate\"\n  }\n}\n```\n\n## 🎯 Use Cases\n\n### 📊 Website Monitoring\n\nMonitor competitor websites and track changes automatically.\n\n### 🔍 SEO Optimization\n\nPre-render SPAs for perfect search engine indexing.\n\n### 📈 Data Analytics\n\nExtract structured data from websites for business intelligence.\n\n### 🖼️ Visual Testing\n\nGenerate screenshots for visual regression testing.\n\n### 📱 Social Media\n\nCreate automated social media preview generation.\n\n## 🛠️ Development\n\n### Commands\n\n```bash\n# Development\npnpm dev                    # Start all services\npnpm build:packages        # Build all packages\npnpm lint                  # Lint all packages\npnpm lint:fix              # Fix lint issues\n\n# Snap Kit Blocklet\ncd blocklets/snap-kit\nnpm run dev                # Development server\nnpm run bundle             # Production build\nnpm run deploy             # Deploy to Blocklet Server\n```\n\n### Project Structure\n\n```\nsnap-kit/\n├── blocklets/snap-kit/     # Main Blocklet application\n│   ├── src/               # React frontend\n│   ├── api/               # Express API\n│   └── public/            # Static assets\n├── packages/\n│   ├── crawler/           # Core crawler engine\n│   └── middleware/        # SEO middleware\n└── scripts/               # Build and utility scripts\n```\n\n## 🔒 Security\n\n- **DID Authentication**: Secure decentralized identity\n- **Rate Limiting**: Prevent abuse and ensure fair usage\n- **Input Validation**: Comprehensive request sanitization\n- **CORS Configuration**: Secure cross-origin requests\n\n## 📖 Documentation\n\n- [Snap Kit Blocklet](https://github.com/blocklet/snap-kit/blob/master/blocklets/snap-kit/blocklet.md)\n- [Crawler Engine](https://github.com/blocklet/snap-kit/blob/master/packages/crawler/README.md)\n- [SEO Middleware](https://github.com/blocklet/snap-kit/blob/master/packages/middleware/README.md)\n\n## 🤝 Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## 🌟 Support\n\n- 📧 Email: blocklet@arcblock.io\n- 🐛 Issues: [GitHub Issues](https://github.com/blocklet/snap-kit/issues)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cstrong\u003eBuilt with ❤️ by the ArcBlock team\u003c/strong\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://www.arcblock.io\"\u003eArcBlock\u003c/a\u003e •\n  \u003ca href=\"https://www.blocklet.io\"\u003eBlocklet\u003c/a\u003e •\n  \u003ca href=\"https://github.com/blocklet\"\u003eGitHub\u003c/a\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblocklet%2Fsnap-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblocklet%2Fsnap-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblocklet%2Fsnap-kit/lists"}