{"id":52092102,"url":"https://github.com/mrkamel/kraps-ts","last_synced_at":"2026-08-04T06:30:40.998Z","repository":{"id":362553245,"uuid":"1254642389","full_name":"mrkamel/kraps-ts","owner":"mrkamel","description":"Kraps allows to process and perform calculations on very large datasets in parallel using nodejs","archived":false,"fork":false,"pushed_at":"2026-06-04T20:54:21.000Z","size":109,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-04T21:33:16.208Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrkamel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-30T20:36:49.000Z","updated_at":"2026-06-04T20:54:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mrkamel/kraps-ts","commit_stats":null,"previous_names":["mrkamel/kraps-ts"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mrkamel/kraps-ts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fkraps-ts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fkraps-ts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fkraps-ts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fkraps-ts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrkamel","download_url":"https://codeload.github.com/mrkamel/kraps-ts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fkraps-ts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36265484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-04T02:00:06.901Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-08-04T06:30:40.237Z","updated_at":"2026-08-04T06:30:40.989Z","avatar_url":"https://github.com/mrkamel.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kraps\n\n**Easily process big data in TypeScript/Node**\n\nKraps allows you to process and perform calculations on very large datasets in\nparallel using a map/reduce framework similar to [Spark](https://spark.apache.org/),\nbut runs on a background job framework you already have. You just need some\nspace on your filesystem, S3 as a storage layer (with a temporary lifecycle\npolicy enabled), a background job framework, and Redis to track progress.\n\n## Install\n\n```bash\nnpm install kraps ioredis\n# optional: only if you use the S3 driver\nnpm install @aws-sdk/client-s3\n```\n\n## Configure\n\n```ts\nimport { Redis } from 'ioredis';\nimport { S3Client } from '@aws-sdk/client-s3';\nimport { configure, S3Driver } from 'kraps';\n\nconfigure({\n  driver: new S3Driver({\n    client: new S3Client({ region: 'eu-central-1' }),\n    bucket: 'some-bucket',\n    prefix: 'temp/kraps/',\n  }),\n  redis: new Redis(),\n  namespace: 'my-application',  // optional, used as a redis key prefix\n  jobTtl: 7 * 24 * 60 * 60,     // optional, default 4 days (seconds)\n  showProgress: true,            // optional, default true; prints a TTY progress bar per step\n  enqueuer: async (json) =\u003e {\n    // hand off the job to your background queue\n    await myQueue.add('KrapsWorker', { json });\n  },\n  jobs: { SearchLogCounter },  // see \"Define a job\" below\n});\n```\n\n## Define a job\n\nPipelines are built by chaining steps on a `Job`. Each step's block returns an\n`Iterable` (or `AsyncIterable`) of emitted items — sync arrays for small,\neager outputs and generators (`function*` / `async function*`) for lazy\nproduction.\n\n```ts\nimport { Job } from 'kraps';\n\nclass SearchLogCounter {\n  constructor(private readonly startDate: string, private readonly endDate: string) {}\n\n  run() {\n    const startDate = this.startDate;\n    const endDate = this.endDate;\n\n    return new Job()\n      .parallelize(function* () {\n        for (let date = new Date(startDate); date \u003c= new Date(endDate); date.setDate(date.getDate() + 1)) {\n          yield date.toISOString().slice(0, 10);\n        }\n      }, { partitions: 128 })\n      .map(async function* (date) {\n        const lines = await fetchLogFile(date);\n\n        for (const line of lines) {\n          const parsed = JSON.parse(line);\n          yield [parsed.q, 1] as [string, number];\n        }\n      })\n      .reduce((_key, leftCount, rightCount) =\u003e leftCount + rightCount)\n      .eachPartition(async (partition, pairs) =\u003e {\n        const lines: string[] = [];\n\n        for await (const [query, count] of pairs) {\n          lines.push(JSON.stringify({ q: query, count }));\n        }\n\n        await uploadToS3(`results/${partition}.jsonl`, lines.join('\\n'));\n      });\n  }\n}\n```\n\nType inference flows through the chain — `parallelize` produces a\n`Job\u003cstring, null\u003e`, `map` produces `Job\u003cstring, number\u003e`, and `eachPartition`\nsees `pairs: AsyncIterable\u003c[string, number]\u003e`. No `as` casts needed on the\nkeys/values.\n\n**`this` and generators:** `function*` / `async function*` create their own\n`this` binding, so capture instance state in locals first (as `startDate` /\n`endDate` above) — this is a JavaScript language limitation, not a kraps one.\n\n**Pipeline registration:** the worker process rebuilds the job graph from the\npayload's name, so every class you run must appear in the `jobs` dict passed\nto `configure()`. The dict key is the identifier sent over the wire —\n`jobs: { SearchLogCounter }` uses shorthand to bind the key `'SearchLogCounter'`\nto the class. The key is a string literal in source code, so it survives\nbundler minification. (Note: the Runner resolves a class back to its name via\nan identity map built at `configure()` time. If HMR swaps a class for a fresh\nidentity, re-run `configure()` so the map sees the new constructor.) The\nclass only needs an instance `run()` method.\n\n## Worker\n\nThe `enqueuer` you configure receives a JSON payload and is responsible for\nhanding it off to a background queue. In the worker process, instantiate\n`Worker` to handle that payload:\n\n```ts\nimport { Worker } from 'kraps';\n\nasync function handleKrapsJob(json: string) {\n  const worker = new Worker(json, {\n    memoryLimit: 16 * 1024 * 1024,  // bytes\n    chunkLimit: 64,\n    concurrency: 8,\n  });\n\n  await worker.run({ retries: 3 });\n}\n```\n\n* `memoryLimit` — how large a single in-memory chunk may grow before it spills\n  to a temp file (gzipped, line-delimited JSON).\n* `chunkLimit` — caps the number of files open during k-way merges.\n* `concurrency` — parallelism for storage I/O (uploads/downloads).\n\n## Run\n\n```ts\nimport { Runner } from 'kraps';\n\nawait new Runner(SearchLogCounter).run('2018-01-01', '2022-01-01');\n```\n\n## Job API\n\nEvery step method takes the **block first, options second**. Options are\noptional where every field has a default.\n\n| Method | Block signature | Block returns |\n| --- | --- | --- |\n| `parallelize(block, { partitions, partitioner?, enqueuer?, before? })` | `() =\u003e …` | `Iterable\u003cNewKey\u003e \\| AsyncIterable\u003cNewKey\u003e` |\n| `map(block, { partitions?, partitioner?, jobs?, enqueuer?, before? }?)` | `(key, value) =\u003e …` | `Iterable\u003c[NewKey, NewValue]\u003e \\| AsyncIterable\u003c…\u003e` |\n| `mapPartitions(block, { … }?)` | `(partition, pairs) =\u003e …` (pairs is `AsyncIterable\u003c[Key, Value]\u003e`, sorted) | same as `map` |\n| `reduce(block, { jobs?, enqueuer?, before? }?)` | `(key, leftValue, rightValue) =\u003e Value \\| Promise\u003cValue\u003e` | a single merged value |\n| `combine(otherJob, block, { jobs?, enqueuer?, before? }?)` | `(key, leftValue, rightValue \\| null) =\u003e …` (right is `null` when no match) | `Iterable\u003c[Key, ResultValue]\u003e \\| AsyncIterable\u003c…\u003e` |\n| `append(otherJob, { jobs?, enqueuer?, before? }?)` | — (no block) | — |\n| `eachPartition(block, { jobs?, enqueuer?, before? }?)` | `(partition, pairs) =\u003e …` | `void \\| Promise\u003cvoid\u003e` (side effects only) |\n| `repartition({ partitions, partitioner?, jobs?, enqueuer?, before? })` | — | — |\n| `dump({ prefix, enqueuer? })` | — | per-partition file written under `prefix/\u003cn\u003e/chunk.json` |\n| `load({ prefix, partitions, partitioner, concurrency, enqueuer? })` | — | seeds a fresh job from previously dumped data |\n\n`partitioner` defaults to `hashPartitioner`. `jobs` caps the number of\nwake-ups the runner pushes for that step (one wake-up triggers one\n`Worker.run`; if your workers are long-lived and drain the queue per run,\nset `jobs` close to your worker concurrency to avoid no-op wake-ups).\n\n`combine` combines the results of two jobs by joining every key available in\nthe current job with the corresponding key from `otherJob`. When `otherJob`\ndoes not have a corresponding key, `null` is passed to the block. **Keys which\nare only available in `otherJob` are completely omitted** (left-outer join,\nnot full-outer). The keys, partitioners and number of partitions must match\nbetween the two jobs, and `otherJob` must be reduced (every key unique).\n`otherJob` does not need to be listed in the array returned from `run()` —\nkraps detects the dependency.\n\n`append` requires the partitioners and number of partitions to match between\nthe two jobs.\n\n## Type safety\n\n`Job\u003cKey, Value\u003e` is generic over the current step's key/value type, and the\nmethods narrow these as you chain. `KrapsKey` (sortable JSON-safe values:\n`string | number | boolean | null | KrapsKey[]`) is the constraint on keys;\n`JsonValue` is the constraint on values.\n\nIf TypeScript can't infer the new types from the block (commonly when the\nblock yields no concrete data, or yields literals that need widening), pass\nthe type arguments explicitly:\n\n```ts\n.parallelize\u003cstring\u003e(/* block */, { partitions: 8 })\n.map\u003cstring, number\u003e(/* block */)\n```\n\n## Datatypes\n\nAll keys and values round-trip through JSON. Keys must be sortable — strings,\nnumbers, booleans, `null`, and arrays of those work; objects do not have a\nstable comparison order and are intentionally excluded from `KrapsKey`. The\ndefault `hashPartitioner` takes the first 5 hex digits of\n`SHA1(JSON.stringify(key))` and returns the result modulo `numPartitions`.\n\n## Storage\n\nKraps stores temporary results in the configured driver. The S3 driver\nexpects you to set up a lifecycle policy on the bucket (or a prefix) to delete\nstale objects, since kraps itself does not clean up — leaving rubbish behind\nis safer than risking premature deletion on error.\n\n```ts\nnew S3Driver({\n  client: new S3Client({ /* ... */ }),\n  bucket: 'some-bucket',\n  prefix: 'temp/kraps/',\n});\n```\n\nFor tests, use `FakeDriver`:\n\n```ts\nimport { FakeDriver } from 'kraps';\n\nconfigure({ driver: new FakeDriver({ bucket: 'test' }), redis: new Redis({ db: 15 }) });\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fkraps-ts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrkamel%2Fkraps-ts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fkraps-ts/lists"}