{"id":20044128,"url":"https://github.com/jurooravec/crawlee-one","last_synced_at":"2026-02-09T23:07:33.568Z","repository":{"id":193949608,"uuid":"630626572","full_name":"JuroOravec/crawlee-one","owner":"JuroOravec","description":"Professional scrapers that provide full control to the users. Crawlee One builds on top of Crawlee and Apify and extends them with features for robust and highly configurable web scrapers.","archived":false,"fork":false,"pushed_at":"2024-05-29T11:35:54.000Z","size":2050,"stargazers_count":30,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-16T20:15:54.905Z","etag":null,"topics":["actor","apify","crawlee","crawler","framework","scraper","scraping","web"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JuroOravec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-20T19:40:12.000Z","updated_at":"2025-03-15T11:37:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"68a710be-ffe5-41e3-93fb-39ee39a35c0e","html_url":"https://github.com/JuroOravec/crawlee-one","commit_stats":null,"previous_names":["jurooravec/crawlee-one","jurooravec/apify-actor-utils"],"tags_count":59,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuroOravec%2Fcrawlee-one","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuroOravec%2Fcrawlee-one/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuroOravec%2Fcrawlee-one/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuroOravec%2Fcrawlee-one/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JuroOravec","download_url":"https://codeload.github.com/JuroOravec/crawlee-one/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252466948,"owners_count":21752467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actor","apify","crawlee","crawler","framework","scraper","scraping","web"],"created_at":"2024-11-13T10:59:56.037Z","updated_at":"2026-02-09T23:07:33.563Z","avatar_url":"https://github.com/JuroOravec.png","language":"TypeScript","funding_links":["https://www.buymeacoffee.com/jurooravec"],"categories":[],"sub_categories":[],"readme":"# CrawleeOne\n\n[![npm version](https://img.shields.io/npm/v/crawlee-one)](https://www.npmjs.com/package/crawlee-one)\n[![npm downloads](https://img.shields.io/npm/dm/crawlee-one)](https://www.npmjs.com/package/crawlee-one)\n[![license](https://img.shields.io/npm/l/crawlee-one)](https://github.com/jurooravec/crawlee-one/blob/main/LICENSE)\n[![TypeScript](https://img.shields.io/badge/TypeScript-5.9-blue)](https://www.typescriptlang.org/)\n[![node](https://img.shields.io/node/v/crawlee-one)](https://nodejs.org/)\n[![GitHub stars](https://img.shields.io/github/stars/jurooravec/crawlee-one)](https://github.com/jurooravec/crawlee-one)\n\n**Production-ready web scraping. Out of the box.**\n\nCrawleeOne wraps [Crawlee](https://crawlee.dev/) with everything production scrapers need -- data transforms, privacy compliance, error tracking, caching, and more -- in a single function call. Write the extraction logic. CrawleeOne handles the rest.\n\nWorks seamlessly with [Apify](https://apify.com/), but the storage backend is pluggable -- you're not locked in.\n\n```sh\nnpm install crawlee-one\n```\n\n## Quick start\n\n```ts\nimport { crawleeOne } from 'crawlee-one';\n\nawait crawleeOne({\n  type: 'cheerio',\n  routes: {\n    mainPage: {\n      match: /example\\.com\\/home/i,\n      handler: async (ctx) =\u003e {\n        const { $, pushData, pushRequests } = ctx;\n        await pushData([{ title: $('h1').text() }], {\n          privacyMask: { author: true },\n        });\n        await pushRequests([{ url: 'https://example.com/page/2' }]);\n      },\n    },\n    otherPage: {\n      match: (url, ctx) =\u003e url.startsWith('/') \u0026\u0026 ctx.$('.author').length \u003e 0,\n      handler: async (ctx) =\u003e {\n        /* ... */\n      },\n    },\n  },\n});\n```\n\nThat's it. No `Actor.main()` boilerplate, no manual router setup, no input wiring. CrawleeOne handles initialization, routing, input resolution, error handling, and teardown.\n\n## Why CrawleeOne?\n\n### One function. Full crawler.\n\nReplace 100+ lines of Actor + Router + input boilerplate with a single `crawleeOne()` call.\n\n### Switch strategies, not code.\n\nGo from `cheerio` to `playwright` by changing one prop. Your route handlers stay the same.\n\n### Reshape output without touching scraper code.\n\nUsers filter, transform, rename, and limit results via input config -- no code changes needed.\n\n```json\n{\n  \"outputPickFields\": [\"name\", \"email\"],\n  \"outputRenameFields\": { \"photo\": \"media.photos[0].url\" },\n  \"outputMaxEntries\": 500,\n  \"outputFilter\": \"(entry) =\u003e entry.rating \u003e 4.0\"\n}\n```\n\n### Fully typed out of the box.\n\nRoute handlers and context objects are typed based on your crawler type. TypeScript knows whether you have `ctx.page` or `ctx.$` -- no extra setup.\n\n### Privacy compliance, built in.\n\nMark fields as personal data. CrawleeOne redacts them automatically when `includePersonalData` is off.\n\n### Incremental scraping.\n\nOnly process entries you haven't seen before. Built-in cache with KeyValueStore tracks what's been scraped across runs.\n\n### Errors captured, not lost.\n\nFailed requests are saved to a dataset automatically. Plug in Sentry with one line, or implement your own telemetry.\n\n### Match routes by URL or content.\n\nRegex, functions, or both. CrawleeOne auto-routes unlabeled requests to the right handler.\n\n[See all features](./packages/crawlee-one/docs/features.md)\n\n## Before and after\n\n\u003cdetails\u003e\n\u003csummary\u003eWhat CrawleeOne replaces (click to expand)\u003c/summary\u003e\n\n**With CrawleeOne:**\n\n```ts\nawait crawleeOne({\n  type: 'cheerio',\n  routes: {\n    mainPage: {\n      match: /example\\.com\\/home/i,\n      handler: async (ctx) =\u003e {\n        const data = [\n          /* ... */\n        ];\n        await ctx.pushData(data, { privacyMask: { author: true } });\n        await ctx.pushRequests([{ url: 'https://...' }]);\n      },\n    },\n  },\n});\n```\n\n**Without CrawleeOne (vanilla Crawlee + Apify):**\n\n```ts\nimport { Actor } from 'apify';\nimport { CheerioCrawler, createCheerioRouter } from 'crawlee';\n\nawait Actor.main(async () =\u003e {\n  const rawInput = await Actor.getInput();\n  const input = {\n    ...rawInput,\n    ...(await fetchInput(rawInput.inputFromUrl)),\n    ...(await runFunc(rawInput.inputFromFunc)),\n  };\n\n  const router = createCheerioRouter();\n\n  router.addHandler('mainPage', async (ctx) =\u003e {\n    await onBeforeHandler(ctx);\n    const data = [\n      /* ... */\n    ];\n    const finalData = await transformAndFilterData(data, ctx, input);\n    const dataset = await Actor.openDataset(input.datasetId);\n    await dataset.pushData(data);\n    const reqs = ['https://...'].map((url) =\u003e ({ url }));\n    const finalReqs = await transformAndFilterReqs(reqs, ctx, input);\n    const queue = await Actor.openRequestQueue(input.requestQueueId);\n    await queue.addRequests(finalReqs);\n    await onAfterHandler(ctx);\n  });\n\n  router.addDefaultHandler(async (ctx) =\u003e {\n    await onBeforeHandler(ctx);\n    const url = ctx.request.loadedUrl || ctx.request.url;\n    if (url.match(/example\\.com\\/home/i)) {\n      const req = { url, userData: { label: 'mainPage' } };\n      const finalReqs = await transformAndFilterReqs([req], ctx, input);\n      const queue = await Actor.openRequestQueue(input.requestQueueId);\n      await queue.addRequests(finalReqs);\n    }\n    await onAfterHandler(ctx);\n  });\n\n  const crawler = new CheerioCrawler({ ...input, requestHandler: router });\n  crawler.run(['https://...']);\n});\n```\n\nAnd that's far from everything -- the vanilla version still doesn't include data transforms, privacy masking, error tracking, caching, or input validation.\n\n\u003c/details\u003e\n\n## Common use cases\n\nCrawleeOne scrapers support these out of the box, all configurable via input:\n\n| Use case                                                                                       | What it does                                                    |\n| ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |\n| **[Import URLs](./packages/crawlee-one/docs/playbook-01-import-urls.md)**                      | Load URLs from databases, datasets, or custom functions.        |\n| **[Data transforms](./packages/crawlee-one/docs/playbook-03-results-mapping-simple.md)**       | Rename, select, limit, and reshape output without code changes. |\n| **[Request filtering](./packages/crawlee-one/docs/playbook-06-requests-mapping-filtering.md)** | Control what gets scraped to save time and money.               |\n| **[Caching](./packages/crawlee-one/docs/playbook-07-caching.md)**                              | Incremental scraping -- only process new entries.               |\n| **[Privacy compliance](./packages/crawlee-one/docs/playbook-10-privacy-compliance.md)**        | Redact personal data with a single toggle.                      |\n| **[Error capture](./packages/crawlee-one/docs/playbook-11-errors.md)**                         | Centralized error tracking across scrapers.                     |\n\n[See all 12 use cases](./packages/crawlee-one/docs/use-cases.md)\n\n## Getting started\n\n### Installation\n\n```sh\nnpm install crawlee-one\n```\n\n### For scraper developers\n\n1. Read the [getting started guide](./packages/crawlee-one/docs/getting-started.md) for a full walkthrough of `crawleeOne()` and its options.\n2. See [example projects](#example-projects) for real-world usage.\n3. Managing multiple crawlers in one project? Use [codegen](./packages/crawlee-one/docs/codegen.md) to generate typed helper functions from a config file.\n\n### For end users\n\nScrapers built with CrawleeOne are configurable by the end users (via Apify platform). Transform, filter, limit, and reshape scraped data and requests -- all through input fields, no code changes needed.\n\n[User guide](./packages/crawlee-one/docs/user-guide.md)\n\n![Apify actor input page](./packages/crawlee-one/docs/user-guide-input-ui-open.png)\n\n## Documentation\n\n| Document                                                                             | Description                                                 |\n| ------------------------------------------------------------------------------------ | ----------------------------------------------------------- |\n| [Getting started](./packages/crawlee-one/docs/getting-started.md)                    | Developer guide with full `crawleeOne()` options reference. |\n| [Features](./packages/crawlee-one/docs/features.md)                                  | Complete feature catalog with code examples.                |\n| [Use cases](./packages/crawlee-one/docs/use-cases.md)                                | All 12 use cases with links to detailed guides.             |\n| [Input reference](./packages/crawlee-one/docs/reference-input.md)                    | All available input fields.                                 |\n| [Deploying to Apify](./packages/crawlee-one/docs/deploying-to-apify.md)              | Step-by-step Apify deployment guide.                        |\n| [Codegen](./packages/crawlee-one/docs/codegen.md)                                    | Generate typed crawler definitions from config.             |\n| [Integrations](./packages/crawlee-one/docs/integrations.md)                          | Custom telemetry and storage backends.                      |\n| [User guide](./packages/crawlee-one/docs/user-guide.md)                              | Guide for end users of CrawleeOne scrapers.                 |\n| [API reference](./packages/crawlee-one/docs/typedoc/globals.md)                      | Auto-generated TypeScript API docs.                         |\n| [Crawlee \u0026 Apify overview](./packages/crawlee-one/docs/scraping-workflow-summary.md) | Background on how Crawlee and Apify work.                   |\n\n## Example projects\n\n- [SKCRIS Scraper](https://github.com/JuroOravec/apify-actor-skcris) -- Slovak research database scraper.\n- [Profesia.sk Scraper](https://github.com/JuroOravec/apify-actor-profesia-sk) -- Slovak job board scraper.\n\n## Contributing\n\nFound a bug or have a feature request? Please [open an issue](https://github.com/jurooravec/crawlee-one/issues).\n\nWhen contributing code, please fork the repo and submit a pull request. See [CONTRIBUTING.md](./CONTRIBUTING.md) for dev setup and guidelines.\n\n## Development\n\nWant to build, test, or hack on CrawleeOne? The [development guide](./packages/crawlee-one/docs/development/README.md) covers prerequisites, all npm scripts, project structure, architecture, and testing strategy.\n\n## Supporting CrawleeOne\n\nCrawleeOne is a labour of love. If you find it useful, you can support the project on [Buy Me a Coffee](https://www.buymeacoffee.com/jurooravec).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjurooravec%2Fcrawlee-one","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjurooravec%2Fcrawlee-one","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjurooravec%2Fcrawlee-one/lists"}