{"id":34948302,"url":"https://github.com/crawlee-cloud/crawlee-cloud","last_synced_at":"2026-05-03T22:04:55.773Z","repository":{"id":330499307,"uuid":"1122895972","full_name":"crawlee-cloud/crawlee-cloud","owner":"crawlee-cloud","description":"Self-hosted, open-source platform for running Apify Actors. Drop-in compatible with the Apify SDK.","archived":false,"fork":false,"pushed_at":"2026-04-28T11:58:30.000Z","size":1211,"stargazers_count":18,"open_issues_count":4,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-28T13:36:35.144Z","etag":null,"topics":["apify","apify-actor","apify-platform","apify-sdk","automation","crawlee","crawlee-framework","crawlee-project","docker","open-source","scraper","scraping","self-hosted","web-scraping"],"latest_commit_sha":null,"homepage":"https://crawlee.cloud","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crawlee-cloud.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["aminembarki"],"ko_fi":"crawleecloud"}},"created_at":"2025-12-25T19:01:16.000Z","updated_at":"2026-04-28T11:56:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/crawlee-cloud/crawlee-cloud","commit_stats":null,"previous_names":["crawlee-cloud/crawlee-cloud"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/crawlee-cloud/crawlee-cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crawlee-cloud%2Fcrawlee-cloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crawlee-cloud%2Fcrawlee-cloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crawlee-cloud%2Fcrawlee-cloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crawlee-cloud%2Fcrawlee-cloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crawlee-cloud","download_url":"https://codeload.github.com/crawlee-cloud/crawlee-cloud/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crawlee-cloud%2Fcrawlee-cloud/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32399010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apify","apify-actor","apify-platform","apify-sdk","automation","crawlee","crawlee-framework","crawlee-project","docker","open-source","scraper","scraping","self-hosted","web-scraping"],"created_at":"2025-12-26T20:55:05.843Z","updated_at":"2026-05-03T22:04:55.766Z","avatar_url":"https://github.com/crawlee-cloud.png","language":"TypeScript","funding_links":["https://github.com/sponsors/aminembarki","https://ko-fi.com/crawleecloud"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"./logo-dark.svg\"\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"./logo-light.svg\"\u003e\n    \u003cimg src=\"./logo-dark.svg\" width=\"450\" alt=\"Crawlee Cloud Logo\" /\u003e\n  \u003c/picture\u003e\n\n**Self-hosted, open-source platform for running Apify Actors on your own infrastructure.**\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.0-blue)](https://www.typescriptlang.org/)\n[![Node.js](https://img.shields.io/badge/Node.js-18+-green)](https://nodejs.org/)\n\n\u003ca href=\"https://www.producthunt.com/products/crawlee-cloud?utm_source=badge-featured\u0026utm_medium=badge\u0026utm_campaign=badge-crawlee-cloud\" target=\"_blank\"\u003e\u003cimg src=\"https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=1054817\u0026theme=light\" alt=\"Crawlee Cloud - Self-hosted platform for running web scrapers | Product Hunt\" width=\"250\" height=\"54\" /\u003e\u003c/a\u003e\n\n[Dashboard](#dashboard) · [Quick Start](#quick-start) · [Documentation](#documentation) · [Contributing](#contributing)\n\n\u003c/div\u003e\n\n---\n\n## Dashboard\n\nA purpose-built operator console — monitor runs in real time, debug webhook deliveries, browse datasets, and triage failures without leaving the page.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./docs/screenshots/02-dashboard-home.png\" width=\"900\" alt=\"Operator dashboard — KPIs, throughput, recent activity\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eOperator dashboard — actor count, runs, success rate, hourly throughput, recent activity feed\u003c/em\u003e\n\u003c/p\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/05-webhooks.png\" width=\"100%\" alt=\"Webhooks — test fire, deliveries drawer, last-seen indicator\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eWebhooks — fire test events per subscription, inline delivery log with HTTP code + body, last-seen status\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/06-run-detail.png\" width=\"100%\" alt=\"Run detail — logs, runtime, storage IDs\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eRun detail — live logs, container runtime, dataset / KV / queue IDs, status timeline\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/04-settings.png\" width=\"100%\" alt=\"Settings — version, scaler state, storage health\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eSettings — live version, scaler state, storage health probes (PG / Redis / S3) with latency, API keys\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/08-kv-inline-preview.png\" width=\"100%\" alt=\"KV store inline preview\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eKV stores — click any record to pretty-print its JSON inline, no new tab needed\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/03-runs.png\" width=\"100%\" alt=\"Runs — execution history\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eRuns — full execution history with status filters, durations, dataset links\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cimg src=\"./docs/screenshots/07-actors.png\" width=\"100%\" alt=\"Actors — deploy and manage\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eActors — your deployed scrapers with version + last-modified at a glance\u003c/em\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n## Why Crawlee Cloud?\n\nLove the Crawlee/Apify ecosystem but want the freedom to run things your way? Crawlee Cloud brings the same great developer experience to your own infrastructure. Keep using the tools you love — just host them wherever you want.\n\n### Key Benefits\n\n- **� Your infrastructure** — Deploy on your own servers, cloud, or anywhere you like\n- **🔒 Complete privacy** — Your data stays exactly where you want it\n- **⚡ SDK compatible** — Works seamlessly with the Apify SDK you already know\n- **🐳 Container-based** — Each Actor runs in an isolated Docker container\n- **📊 Beautiful dashboard** — Monitor runs, explore datasets, manage everything visually\n\n---\n\n## How It Works\n\n```bash\n# Instead of pointing to Apify's servers...\nexport APIFY_API_BASE_URL=https://api.apify.com/v2\n\n# Point to your own Crawlee Cloud instance\nexport APIFY_API_BASE_URL=https://your-server.com/v2\nexport APIFY_TOKEN=your-token\n```\n\nYour existing Actor code works without any modifications:\n\n```typescript\nimport { Actor } from 'apify';\n\nawait Actor.init();\nawait Actor.pushData({ title: 'Scraped data' });\nawait Actor.exit();\n```\n\n---\n\n## Quick Start\n\n### Prerequisites\n\n- Node.js 18+\n- Docker \u0026 Docker Compose\n- PostgreSQL, Redis, and S3-compatible storage (or use our Docker setup)\n\n### 1. Clone \u0026 Install\n\n```bash\ngit clone https://github.com/crawlee-cloud/crawlee-cloud.git\ncd crawlee-cloud\nnpm install\n```\n\n### 2. Start Infrastructure\n\n```bash\n# Starts PostgreSQL, Redis, and MinIO\nnpm run docker:dev\n```\n\n### 3. Configure Environment\n\n```bash\ncp .env.example .env\n# Edit .env with your settings\n```\n\n### 4. Build \u0026 Run\n\n```bash\nnpm run build\nnpm run db:migrate\nnpm run dev\n```\n\nThe API server starts at `http://localhost:3000`.\n\n---\n\n## Deploy\n\nDeploy your own instance in minutes:\n\n| Method                                                                                                                                                                               | Status         | Description                                                            |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------- | ---------------------------------------------------------------------- |\n| [![Deploy to DO](https://www.deploytodo.com/do-btn-blue.svg)](https://cloud.digitalocean.com/apps/new?repo=https://github.com/crawlee-cloud/crawlee-cloud/tree/main\u0026refcode=crawlee) | ✅ Supported   | Automated full stack — App Platform, managed PG/Redis, Runner Droplet  |\n| [VPS Deploy Script](deploy/vps/)                                                                                                                                                     | ✅ Supported   | Full stack on any Ubuntu VPS with auto-HTTPS via Caddy                 |\n| Railway                                                                                                                                                                              | 🚧 Coming soon | One-click PaaS deploy (template scaffolding present, not yet verified) |\n| Render                                                                                                                                                                               | 🚧 Coming soon | One-click PaaS deploy (blueprint present, not yet verified)            |\n\nSee [deploy/](deploy/) for detailed instructions.\n\n---\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                        Your Actors                              │\n│            (using official Apify SDK, no changes)               │\n└────────────────────────────┬────────────────────────────────────┘\n                             │\n                             ▼\n┌─────────────────────────────────────────────────────────────────┐\n│                     Crawlee Cloud API                           │\n│              (Apify-compatible REST endpoints)                  │\n└─────────────────────────────────────────────────────────────────┘\n        │                    │                    │\n        ▼                    ▼                    ▼\n   ┌──────────┐         ┌─────────┐         ┌─────────┐\n   │PostgreSQL│         │  Redis  │         │ S3/MinIO│\n   │ metadata │         │ queues  │         │  blobs  │\n   └──────────┘         └─────────┘         └─────────┘\n```\n\n### Components\n\n| Component      | Description                                                 |\n| -------------- | ----------------------------------------------------------- |\n| **API Server** | Fastify-based REST API compatible with Apify's v2 endpoints |\n| **Runner**     | Polls job queue and executes Actors in Docker containers    |\n| **Dashboard**  | Next.js web UI for monitoring and management                |\n| **CLI**        | Command-line tool for pushing and running Actors            |\n\n---\n\n## Documentation\n\n| Guide                                                                 | Description                  |\n| --------------------------------------------------------------------- | ---------------------------- |\n| [API Reference](https://crawlee.cloud/docs/api)                       | REST API endpoints and usage |\n| [CLI Guide](https://crawlee.cloud/docs/cli)                           | Command-line interface       |\n| [Dashboard](https://crawlee.cloud/docs/dashboard)                     | Web interface overview       |\n| [Deployment](https://crawlee.cloud/docs/deployment)                   | Production deployment guide  |\n| [Runner](https://crawlee.cloud/docs/runner)                           | Actor execution engine       |\n| [SDK Compatibility](https://crawlee.cloud/docs/apify-sdk-environment) | Apify SDK integration        |\n\n---\n\n## Supported Apify SDK Features\n\n| Feature                                      | Status       |\n| -------------------------------------------- | ------------ |\n| Datasets (`Actor.pushData`)                  | ✅ Supported |\n| Key-Value Stores (`Actor.getValue/setValue`) | ✅ Supported |\n| Request Queues                               | ✅ Supported |\n| Request deduplication                        | ✅ Supported |\n| Distributed locking                          | ✅ Supported |\n| Builds \u0026 versioning                          | ✅ Supported |\n| Webhooks                                     | ✅ Supported |\n| Schedules                                    | ✅ Supported |\n| Auto-scaling runners (local Docker, GHCR)    | ✅ Supported |\n\n---\n\n## What's New\n\n**v0.8.0** — auto-scaling with image registry support, build versioning, scale-aware run pagination, CLI profiles + `crc info`, dashboard rewrite (Builds, Schedules, Webhooks, Request Queues, Runners, KV browser). See the [full changelog](CHANGELOG.md).\n\n\u003e ⚠️ **Upgrading from v0.7?** `GET /metrics` and `GET /v2/scaler/status` are now admin-only. If you scrape `/metrics` with Prometheus, send an admin token in the `Authorization` header, or set `METRICS_PUBLIC=true`. See [CHANGELOG → 0.8.0 Breaking](CHANGELOG.md#080---2026-05-01).\n\n---\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n```bash\n# Run tests\nnpm test\n\n# Type checking\nnpm run typecheck\n\n# Linting\nnpm run lint\n```\n\n---\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**Built with ❤️ for the web scraping community**\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrawlee-cloud%2Fcrawlee-cloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrawlee-cloud%2Fcrawlee-cloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrawlee-cloud%2Fcrawlee-cloud/lists"}