{"id":22088155,"url":"https://github.com/lazycuh/lazy-pipeline","last_synced_at":"2025-03-23T22:41:25.680Z","repository":{"id":59641181,"uuid":"537542556","full_name":"lazycuh/lazy-pipeline","owner":"lazycuh","description":"A super light-weight, tree-shakeable, reusable, lazy pipeline TypeScript library with functional APIs and no 3rd-party dependencies.","archived":false,"fork":false,"pushed_at":"2022-09-25T20:19:14.000Z","size":479,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T16:52:23.401Z","etag":null,"topics":["functional-pipeline","lazy","lazy-evaluation","lazy-pipeline"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lazycuh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-16T16:46:13.000Z","updated_at":"2022-09-19T08:46:47.000Z","dependencies_parsed_at":"2023-01-18T22:01:25.528Z","dependency_job_id":null,"html_url":"https://github.com/lazycuh/lazy-pipeline","commit_stats":null,"previous_names":["tinkeringprogrammer/lazy-pipeline","nhuyvan/lazy-pipeline","babybeet/lazy-pipeline","lazycuh/lazy-pipeline"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lazycuh%2Flazy-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lazycuh%2Flazy-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lazycuh%2Flazy-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lazycuh%2Flazy-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lazycuh","download_url":"https://codeload.github.com/lazycuh/lazy-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245181527,"owners_count":20573717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functional-pipeline","lazy","lazy-evaluation","lazy-pipeline"],"created_at":"2024-12-01T02:07:40.883Z","updated_at":"2025-03-23T22:41:25.646Z","avatar_url":"https://github.com/lazycuh.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `lazy-pipeline` [![](https://circleci.com/gh/nhuyvan/lazy-pipeline.svg?style=svg\u0026logo=appveyor)](https://app.circleci.com/pipelines/github/nhuyvan/lazy-pipeline?branch=main)\n\nA super light-weight, tree-shakeable, reusable, lazy pipeline TypeScript library with functional APIs and no 3rd-party\ndependencies.\n\n`lazy-pipeline` package is [already tiny](https://bundlephobia.com/package/lazy-pipeline), but with its tree-shakeable\ndesign, the final package size will be reduced much more by the build tool.\n\n## Installation\n\n```shell\nnpm i -S lazy-pipeline\n```\n\n## Table of contents\n\n- [What is a reusable, lazy pipeline?](#what-is-a-reusable-lazy-pipeline)\n- [Terminologies](#terminologies)\n- [Advantages of tree-shakeable, functional APIs](#advantages-of-tree-shakeable-functional-apis)\n- [How to add a new operator](#how-to-add-a-new-operator)\n- [How to add a new collector](#how-to-add-a-new-collector)\n- [Built-in collectors](./lib/src/collectors/)\n- [Built-in operators](./lib/src/operators/)\n\n## What is a reusable, lazy pipeline?\n\nConsider the following common pattern with arrays\n\n```typescript\nconst source = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]; // 10 elements\n\nsource\n  .map(value =\u003e value + 1) // Iterate 10 times and add 1 to each number each time\n  .filter(value =\u003e value % 2 === 0) // Iterate 10 times and filter out any odd numbers\n  .reduce((left, right) =\u003e left + right); // Iterate 5 times and sum up the numbers\n```\n\nThe combination of `map`, `filter`, and `reduce` operations above constitutes a pipeline, each operation executes\n_eagerly_ as soon as it is added, but most notably, each operation runs as many times as there are elements.\n\nThe example above would look similar to the following using `for-of` loops:\n\n```typescript\nconst source = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];\n\n// map(value =\u003e value + 1)\nconst mappedValues = [];\nfor (const value of source) {\n  mappedValues.push(value + 1);\n}\n\n// filter(value =\u003e value % 2 === 0)\nconst filteredValues = [];\nfor (const value of mappedValues) {\n  if (value % 2 === 0) {\n    filteredValues.push(value);\n  }\n}\n\n// reduce((left, right) =\u003e left + right)\nlet sum = 0;\nfor (const value of filteredValues) {\n  sum += value;\n}\n```\n\nAs you can see, it needs to iterate through the entire array every time an operation (`map`, `filter`, and `reduce`) is\nadded which is not something that we want.\n\n\u003cbr/\u003e\n\nNow consider a similar pipeline using `lazy-pipeline` APIs.\n\n```typescript\nimport { LazyPipeline } from 'lazy-pipeline';\nimport { filter, map, peek } from 'lazy-pipeline/operators';\nimport { sum } from 'lazy-pipeline/collectors';\n\nconst source = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];\n\nconst result = LazyPipeline.from(source) // Creating a new pipeline object\n  // Begin adding operations to this pipeline\n  .add(\n    /*\n     * `peek()` allows to perform some action on each pipeline element with no transformations applied to it,\n     * here we're logging each element to verify that we're only iterating over `source` 10 times\n     */\n    peek(value =\u003e console.log(value)),\n\n    // Adding a mapping stage, incrementing each number by 1\n    map(value =\u003e value + 1),\n\n    // Adding a filtering stage, discarding any odd numbers\n    filter(value =\u003e value % 2 === 0)\n  )\n  // Telling the pipeline to start executing all of the added stages (operations)\n  .collect(\n    // Adding a terminal stage that sums all elements in the pipeline and return the result to the caller\n    sum()\n  );\n\nconsole.log(result); // Outputs 30\n```\n\nThis second example is no different than the first one in terms of the final result, however, there are a few things\nthat distinguish it from the first example. First of all, you can add as many intermediate stages (`map`, and `filter`\nor [any intermediate stages](./lib/src/operators)) to the pipeline, those stages won't be executed until `collect()` is\ncalled, that's the reason for its _lazy_ evaluation nature, `collect()` expects a terminal stage which is the final\nstage to produce the result, these terminal stages' sole responsibility is to collect the remaining pipeline elements\ninto some final result container. Second of all, all stages (`map`, `filter` and `sum` in this example) run in the same\niteration which is much more efficient as compared to the first example, the second example would look similar to the\nfollowing with a regular `for-of` loop:\n\n```typescript\nconst source = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];\n\nlet sum = 0;\nfor (const value of source) {\n  // peek(...)\n  console.log(value);\n\n  // map(...)\n  const mappedValue = value + 1;\n\n  // filter(...)\n  if (mappedValue % 2 === 0) {\n    // sum()\n    sum += mappedValue;\n  }\n}\n\nconsole.log(sum);\n```\n\nAs you can see, we only need to iterate once per element to evaluate all of our operations which is obviously much more\nefficient.\n\nThe most interesting bit of `lazy-pipeline` module is its reusability feature. After constructing your pipeline, and\nadding any number of intermediate stages to it, you can pass this pipeline object around and reuse it on as many\niterable data sources as desired, but keep it mind that after each call to `collect()` to get the result, the pipeline\nwill be frozen to prevent any new stages from being added to it, to unfreeze it, simply call `unfreeze()` on the same\npipeline instance, for example:\n\n```typescript\nimport { LazyPipeline } from 'lazy-pipeline';\nimport { filter, map, skip } from 'lazy-pipeline/operators';\nimport { findFirst, findLast, max, min } from 'lazy-pipeline/collectors';\n\nconst source1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];\n\nconst pipeline = LazyPipeline.from(source1).add(\n  map(value =\u003e value + 1), // (1)\n  filter(value =\u003e value % 2 === 0) // (2)\n);\n\nconst maxNumber = pipeline.collect(\n  /*\n   * max() collector returns a terminal stage that finds the max number by default, if elements in the pipeline\n   * are not numbers, then you must provide a comparator to `max()` to return the max element.\n   */\n  max()\n);\n\n// Calling resume() is required before reusing the same pipeline instance\npipeline.resume();\n\n// Calling unfreeze() so that we can add new stages to this pipeline, otherwise, it will throw an error\npipeline.unfreeze();\n\nconst minNumber = pipeline\n  .add(\n    // map(...) and filter(...) from (1) and (2) will execute before this new limit(5) stage\n    limit(5) // (3)\n  )\n  .collect(\n    /*\n     * min() collector returns a terminal stage that finds the min number by default, if elements in the pipeline\n     * are not numbers, then you must provide a comparator to `min()` to return the min element.\n     */\n    min()\n  );\n\n// Calling resume() again so that we can calculate a new result\npipeline.resume();\n\n// Have to unfreeze so that we can add a new `skip()` stage to it\npipeline.unfreeze();\n\n// The pipeline instance will execute the same source `source1` passed to `LazyPipeline.from(...)` above\nconst firstElement = pipeline\n  .add(\n    // map(...), filter(...) and limit(...) from (1), (2), and (3) will execute before skip(2) does\n    skip(2) // (4)\n  )\n  .collect(\n    /*\n     * findFirst() returns the very first element in the pipeline, or `undefined` when pipeline has no elements\n     */\n    findFirst() // (5)\n  );\n\n// We have to call `resume()` anytime we need to calculate a new result\npipeline.resume();\n\nconst source2 = [1, 10, 100, 1000, 10000];\n/*\n * Because we want to use `source2` instead of `source1` that was passed to `LazyPipeline.from()` above,\n * we're calling `readFrom()` on `pipeline` to switch to a different data source. From this point on,\n * `pipeline` will use `source2` as the data source to read elements from.\n */\npipeline.readFrom(source2);\n\n// Because we don't need to add new stages to this pipeline, we didn't call `unfreeze()` on it\n\nconst lastElement = pipeline.collect(\n  /**\n   * `findLast()` collector finds and retrieves the very last element in the pipeline, if the pipeline has no\n   * elements, `undefined` is returned.\n   */\n  findLast()\n);\n```\n\nOnce again, all stages `(1)`, `(2)`, `(3)`, `(4)` and `(5)` from the example above will _all run in the same iteration_.\n\n## Terminologies\n\n- [Pipeline](./lib/src/ReusablePipeline.ts): A series of stages that execute in order and in the same iteration, and end\n  with a terminal stage that returns a result. Once the pipeline has been consumed by triggering a terminal stage with\n  `collect()`, it cannot be consumed again until `resume()` is called, otherwise, the pipeline will throw an error to\n  indicate as such.\n- [Stage](./lib/src/stages/Stage.ts): A particular operation that executes on each element in the pipeline.\n- [Intermediate stage](./lib/src/stages/IntermediateStage.ts): A stage that performs some transformation on each element\n  in the pipeline and forwards the new element to the next stage in line downstream. The built-in intermediate stages'\n  creators are located inside [operators module](./lib/src/operators).\n- [Terminal stage](./lib/src/stages/TerminalStage.ts): The final stage that collects all remaining elements in the\n  pipeline to produce a result. Each pipeline doesn't execute until a terminal stage is triggered by calling `collect()`\n  on the pipeline instance which expects a `TerminateStage` object. Terminal stages' creators are located inside\n  [collectors module](./lib/src/collectors/).\n- [Collector](./lib/src/collectors): A function that returns a new object instance of type\n  [`TerminalStage`](./lib/src/stages/TerminalStage.ts).\n- [Operator](./lib/src/operators/): A function that returns a new object instance of type\n  [`IntermediateStage`](./lib/src/stages/IntermediateStage.ts).\n\n## Advantages of tree-shakeable, functional APIs\n\nAll of the stages (intermediate and terminal) are created using standalone functions, this not only makes the APIs\neasier to use, it also allows bundling tools to effortlessly tree-shake those functions that are not imported into your\napplication. Additionally, with a functional programming model, it's straightforward to extend this library with your\nown operators and collectors.\n\n## How to add a new operator\n\nTo add a new operator, create a function that returns a new object instance of type\n[`IntermediateStage`](./lib/src/stages/IntermediateStage.ts) each time it's called.\n\nAs an example, let's create an operator that increments each number in the pipeline by some provided value:\n\n```typescript\nimport { IntermediateStage } from 'lazy-pipeline/stages';\n\nclass IncrementByStage\u003cIN extends number\u003e extends IntermediateStage\u003cIN, number\u003e {\n  constructor(private readonly _valueToIncrementBy: number) {\n    super();\n  }\n\n  override consume(element: IN, hasMoreDataUpstream: boolean): void {\n    this._throwErrorIfNotNumber(element);\n    this._downstream.consume(element + this._valueToIncrementBy, hasMoreDataUpstream);\n  }\n\n  private _throwErrorIfNotNumber(element: IN) {\n    if (typeof element !== 'number') {\n      throw new Error(\n        `[incrementBy() operator] Numbers expected, erroneous pipeline element received was ${JSON.stringify(element)}`\n      );\n    }\n  }\n}\n\n/**\n * Return an intermediate stage that increments each pipeline numeric element by `value`,\n * if at least one element is not numeric, then an error will occur.\n *\n * @param value The value to increment each element by\n * @returns\n */\nexport function incrementBy\u003cIN extends number\u003e(value: number): IntermediateStage\u003cIN, number\u003e {\n  if (typeof value !== 'number') {\n    throw new Error(\n      `[incrementBy() operator] The provided value to increment by is not a number, it was ${JSON.stringify(value)}`\n    );\n  }\n  return new IncrementByStage\u003cIN\u003e(value);\n}\n```\n\nTo use the new operator, simply add it to the pipeline, for example:\n\n```typescript\nimport { LazyPipeline } from 'lazy-pipeline';\nimport { sum } from 'lazy-pipeline/collectors';\n\nimport { incrementBy } from './incrementBy';\n\nconst result = LazyPipeline.from([1, 2, 3, 4, 5]).add(incrementBy(10)).collect(sum());\n```\n\nTo view all built-in operators, please see the [operators module](./lib/src/operators/)\n\n## How to add a new collector\n\nTo add a new collector, create a function that returns a new object instance of type\n[`TerminalStage`](./lib/src/stages/TerminalStage.ts) each time it's called. Each terminal stage must additionally\nimplement `get()` method to return the final result when the pipeline terminates.\n\nAs an example, let's create a collector that multiplies all numbers in the pipeline:\n\n```typescript\nimport { TerminalStage } from 'lazy-pipeline/stages';\n\nclass MultiplyStage\u003cIN extends number\u003e extends TerminalStage\u003cIN, number\u003e {\n  private _product = 1;\n\n  override consume(element: IN, hasMoreDataUpstream: boolean): void {\n    this._throwErrorIfNotNumber(element);\n    this._product *= element;\n  }\n\n  private _throwErrorIfNotNumber(element: IN) {\n    if (typeof element !== 'number') {\n      throw new Error(\n        `[multiply() collector] Numbers expected, erroneous pipeline element received was ${JSON.stringify(element)}`\n      );\n    }\n  }\n\n  override get(): number {\n    return this._product;\n  }\n}\n\n/**\n * Return a terminal stage that multiplies all pipeline elements together, if some element\n * is not numeric, then an error will occur.\n *\n * @param value The value to increment each element by\n * @returns\n */\nexport function multiply\u003cIN extends number\u003e(): IntermediateStage\u003cIN, number\u003e {\n  return new MultiplyStage\u003cIN\u003e(value);\n}\n```\n\nTo use the new collector, simply provide it to `collect()` as followed:\n\n```typescript\nimport { LazyPipeline } from 'lazy-pipeline';\nimport { filter, map } from 'lazy-pipeline/operators';\n\nimport { multiply } from './multiply';\n\nconst result = LazyPipeline.from([1, 2, 3, 4, 5])\n  .add(\n    filter(e =\u003e e % 2 === 0),\n    map(e =\u003e e * e)\n  )\n  .collect(multiply());\n```\n\nTo view all built-in collectors, please see the [collectors module](./lib/src/collectors/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazycuh%2Flazy-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flazycuh%2Flazy-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazycuh%2Flazy-pipeline/lists"}