{"id":13807578,"url":"https://github.com/scramjetorg/framework-js","last_synced_at":"2025-07-07T19:41:27.450Z","repository":{"id":41589902,"uuid":"358021349","full_name":"scramjetorg/framework-js","owner":"scramjetorg","description":"Simple yet powerful live data computation framework.","archived":false,"fork":false,"pushed_at":"2022-04-29T14:49:47.000Z","size":5855,"stargazers_count":38,"open_issues_count":10,"forks_count":0,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-05-05T19:53:47.961Z","etag":null,"topics":["data-stream","es6","event-stream","javascript","nodejs","promise","react","reactive-programming","spark-streaming","stream","stream2","transformations","typescript"],"latest_commit_sha":null,"homepage":"https://www.scramjet.org","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scramjetorg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":"FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"scramjetorg"}},"created_at":"2021-04-14T19:35:14.000Z","updated_at":"2024-08-26T18:18:38.000Z","dependencies_parsed_at":"2022-08-27T06:12:41.552Z","dependency_job_id":null,"html_url":"https://github.com/scramjetorg/framework-js","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/scramjetorg/framework-js","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scramjetorg%2Fframework-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scramjetorg%2Fframework-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scramjetorg%2Fframework-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scramjetorg%2Fframework-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scramjetorg","download_url":"https://codeload.github.com/scramjetorg/framework-js/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scramjetorg%2Fframework-js/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264139573,"owners_count":23563251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-stream","es6","event-stream","javascript","nodejs","promise","react","reactive-programming","spark-streaming","stream","stream2","transformations","typescript"],"created_at":"2024-08-04T01:01:27.151Z","updated_at":"2025-07-07T19:41:27.429Z","avatar_url":"https://github.com/scramjetorg.png","language":"TypeScript","readme":"Scramjet Framework TypeScript\n==================\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://badge.fury.io/js/@scramjet%2Fframework\"\u003e\n        \u003cimg src=\"https://badge.fury.io/js/@scramjet%2Fframework.svg\" alt=\"npm version\" height=\"18\"\u003e\n    \u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"https://github.com/scramjetorg/framework-js/actions/workflows/test.yml/badge.svg?branch=main\" alt=\"Tests\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://snyk.io/test/github/scramjetorg/framework-js\"\u003e\n        \u003cimg src=\"https://snyk.io/test/github/scramjetorg/framework-js/badge.svg\" alt=\"Known Vulnerabilities\" /\u003e\n    \u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"https://img.shields.io/github/license/scramjetorg/framework-js?color=green\u0026style=plastic\" alt=\"License\" /\u003e\u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"https://img.shields.io/github/v/tag/scramjetorg/framework-js?label=version\u0026color=blue\u0026style=plastic\" alt=\"Version\" /\u003e\u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"https://img.shields.io/github/stars/scramjetorg/framework-js?color=pink\u0026style=plastic\" alt=\"GitHub stars\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=7F7V65C43EBMW\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/Donate-PayPal-green.svg?color=yellow\u0026style=plastic\" alt=\"Donate\" /\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e⭐ Star us on GitHub — it motivates us a lot! 🚀 \u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://assets.scramjet.org/images/framework-logo-256.svg\" width=\"420\" alt=\"Scramjet Framework\"\u003e\n\u003c/p\u003e\n\nScramjet is a simple reactive stream programming framework. The code is written by chaining functions that transform the streamed data, including well known map, filter and reduce.\n\nThe main advantage of Scramjet is running asynchronous operations on your data streams concurrently. It allows you to perform the transformations both synchronously and asynchronously by using the same API - so now you can \"map\" your stream from whatever source and call any number of API's consecutively.\n\nThis is a pre-release of the next major version (v5) of [JavaScript Scramjet Framework](https://www.npmjs.com/package/scramjet).\n\n**We are open to your feedback!** We encourage you to report issues with any ideas, suggestions and features you would like to see in this version. You can also upvote (`+1`) existing ones to show us the direction we should take in developing Scramjet Framework.\n\n**Not interested in JavaScript/TypeScript version?** Check out [Scramjet Framework in Python](https://github.com/scramjetorg/framework-python)!\n\n## Table of contents\n\n- [Installation](#installation)\n- [Usage](#usage)\n- [Requesting features](#requesting-features)\n- [Reporting bugs](#reporting-bugs)\n- [Contributing](#contributing)\n- [Development Setup](#development-setup)\n\n## Installation\n\nSimply run:\n\n```bash\nnpm i @scramjet/framework\n```\n\nAnd then you can require it in the JS/TS code like:\n\n_sample-file.ts_\n```ts\nimport { DataStream } from \"@scramjet/framework\";\n```\n\n\u003c!-- DEV_ONLY_START --\u003e\n\nYou can also use nightly build as npm dependency by referring to `nightly` branch (which is the latest build) from this repository:\n\n_package.json_\n```json\n{\n    \"dependencies\": {\n        \"scramjet\": \"scramjetorg/framework-js#nightly\"\n    }\n}\n```\n\nAfter adding Scramjet Framework as dependency it needs to be installed via `npm` (or similar):\n\n```\nnpm i\n```\n\nYou can also build Scramjet Framework yourself. Please refer to [Development Setup](#development-setup) section for more details.\n\n\u003c!-- DEV_ONLY_END --\u003e\n\n## Usage\n\nScramjet streams are similar and behave similar to native nodejs streams and to streams in any programing language in general. They allow operating on streams of data (were each separate data part is called a `chunk`) and process it in any way through transforms like mapping or filtering.\n\nLet's take a look on how to create and operate on Scramjet streams.\n\n_If you would like to dive deeper, please refer to [streams source files](https://github.com/scramjetorg/framework-js/tree/main/src/streams)_.\n\n### Creating Scramjet streams\n\nThe basic method for creating Scramjet streams is `from()` static method. It accepts iterables (both sync and async) and native nodejs streams. As for iterables it can be a simple array, generator or anything iterable:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream = DataStream.from([\"foo\", \"bar\", \"baz\"]);\n```\n\nScramjet streams are asynchronous iterables itself, which means one stream can be created from another:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream1 = DataStream.from([\"foo\", \"bar\", \"baz\"]);\nconst stream2 = DataStream.from(stream1);\n```\n\nThey can be also created from native nodejs `Readable`s:\n\n```ts\nimport { createReadStream } from \"fs\";\nimport { DataStream } from \"scramjet\";\n\nconst stream = DataStream.from(createReadStream(\"path/to/file\"));\n```\n\nThe more \"manual\" approach is creating streams using constructor:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream = new DataStream();\n```\n\nSuch approach is useful when one needs to manually write data to a stream or use it as a pipe destination:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream = new DataStream();\nstream.write(\"foo\");\n\nconst stream2 = new DataStream();\nstream.pipe(stream2);\n```\n\n### Getting data from Scramjet streams\n\nSimilar as to creating Scramjet streams, there are specific methods which allow getting data out of them. Those are sometimes called `sink` methods as they allow data to flow through and out of the stream. As those methods needs to wait for the stream end, they return a `Promise` which needs to be awaited and is resolved when all data from source is processed.\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream1 = DataStream.from([\"foo\", \"bar\", \"baz\"]);\nawait stream1.toArray(); // [\"foo\", \"bar\", \"baz\"]\n\nconst stream2 = DataStream.from([\"foo\", \"bar\", \"baz\"]);\nawait stream2.toFile(\"path/to/file\"); // Writes to a file, resolves when done.\n\nconst stream3 = DataStream.from([\"foo\", \"bar\", \"baz\"]);\nawait stream3.reduce(\n    (prev, curr) =\u003e `${ prev }-${ curr }`,\n    \"\"\n); // \"foo-bar-baz\"\n```\n\nAs Scramjet streams are asynchronous iterables they can be iterated too:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream = DataStream.from([\"foo\", \"bar\", \"baz\"]);\n\nfor await (const chunk of stream) {\n    console.log(chunk);\n}\n// Logs:\n// \"foo\"\n// \"bar\"\n// \"baz\"\n```\n\nSimilar to writing, there is also more \"manual\" way of reading from streams using `.read()` method:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream = DataStream.from([\"foo\", \"bar\", \"baz\"]);\n\nawait stream.read(); // \"foo\"\nawait stream.read(); // \"bar\"\n```\n\nRead returns a `Promise` which waits until there is something ready to be read from a stream.\n\n### Basic operations\n\nThe whole idea of stream processing is an ability to quickly and efficiently transform data which flows through the stream. Let's take a look at basic operations (called `transforms`) and what they do:\n\n#### Mapping\n\nMapping stream data is basically the same as mapping an array. It allows to map a `chunk` to a new value:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([\"foo\", \"bar\", \"baz\"])\n    .map(chunk =\u003e chunk.repeat(2))\n    .toArray(); // [\"foofoo\", \"barbar\", \"bazbaz\"]\n```\n\nThe result of the map transform could be of different type than initial chunks:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([\"foo\", \"bar\", \"baz\"])\n    .map(chunk =\u003e chunk.charCodeAt(0))\n    .toArray(); // [102, 98, 98]\n\nDataStream\n    .from([\"foo\", \"bar\", \"baz\"])\n    .map(chunk =\u003e chunk.split(\"\"))\n    .toArray(); // [[\"f\", \"o\", \"o\"], [\"b\", \"a\", \"r\"], [\"b\", \"a\", \"z\"]]\n```\n\n#### Filtering\n\nFiltering allows to filter out any unnecessary chunks:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([1, 2, 3, 4, 5, 6])\n    .filter(chunk =\u003e chunk % 2 === 0)\n    .toArray(); // [2, 4, 6]\n```\n\n#### Grouping\n\nBatching allows to group chunks into arrays, effectively changing chunks number flowing though the stream:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([1, 2, 3, 4, 5, 6, 7, 8])\n    .batch(chunk =\u003e chunk % 2 === 0)\n    .toArray(); // [[1, 2], [3, 4], [5, 6], [7, 8]]\n```\n\nWhenever callback function passed to `.batch()` call returns `true`, new group is emitted.\n\n#### Flattening\n\nOperation opposite to batching is flattening. At the moment, Scramjet streams provides `.flatMap()` method which allows first to map chunks and then flatten the resulting arrays:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([\"foo\", \"bar\", \"baz\"])\n    .flatMap(chunk =\u003e chunk.split(\"\"))\n    .toArray(); // [\"f\", \"o\", \"o\", \"b\", \"a\", \"r\", \"b\", \"a\", \"z\"]\n```\n\nBut it can be also used to only flatten the stream by providing a callback which only passes values through:\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nDataStream\n    .from([1, 2, 3, 4, 5, 6, 7, 8])\n    .batch(chunk =\u003e chunk % 2 === 0)\n    .flatMap(chunk =\u003e chunk)\n    .toArray(); // [1, 2, 3, 4, 5, 6, 7, 8]\n```\n\n#### Piping\n\nPiping is essential for operating on streams. Scramjet streams can be both used as pipe source and destination. They can be also combined with native nodejs streams having native streams as pipe source or destination.\n\n```ts\nimport { DataStream } from \"scramjet\";\n\nconst stream1 = DataStream.from([1, 2, 3, 4, 5, 6, 7, 8]);\nconst stream2 = new DataStream();\n\nstream1.pipe(stream2); // All data flowing through \"stream1\" will be passed to \"stream2\".\n```\n\n```ts\nimport { createReadStream } from \"fs\";\nimport { DataStream } from \"scramjet\";\n\nconst readStream = createReadStream(\"path/to/file\"));\nconst scramjetStream = new DataStream();\n\nreadStream.pipe(scramjetStream); // All file contents read by native nodejs stream will be passed to \"scramjetStream\".\n```\n\n```ts\nimport { createWriteStream } from \"fs\";\nimport { DataStream } from \"scramjet\";\n\nconst scramjetStream = DataStream.from([1, 2, 3, 4, 5, 6, 7, 8]);\n\nscramjetStream.pipe(createWriteStream(\"path/to/file\")); // All data flowing through \"scramjetStream\" will be written to a file via native nodejs stream.\n```\n\n## Requesting Features\n\nAnything missing? Or maybe there is something which would make using Scramjet Framework much easier or efficient? Don't hesitate to fill up a [new feature request](https://github.com/scramjetorg/framework-js/issues/new?assignees=\u0026labels=\u0026template=feature_request.md\u0026title=)! We really appreciate all feedback.\n\n## Reporting Bugs\n\nIf you have found a bug, inconsistent or confusing behavior please fill up a [new bug report](https://github.com/scramjetorg/framework-js/issues/new?assignees=\u0026labels=\u0026template=bug_report.md\u0026title=).\n\n## Contributing\n\nYou can contribute to this project by giving us feedback ([reporting bugs](#reporting-bugs) and [requesting features](#reporting-features)) and also by writing code yourself! We have some introductory issues labeled with `good first issue` which should be a perfect starter.\n\nThe easiest way is to [create a fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of this repository and then [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) with all your changes. In most cases, you should branch from and target `main` branch.\n\n\u003c!-- DEV_ONLY_START --\u003e\n\nPlease refer to [Development Setup](#development-setup) section on how to setup this project.\n\n## Development Setup\n\n### Project setup\n\n1. Install nodejs (`14.x`).\n\nRefer to [official docs](https://nodejs.org/en/download/). Alternatively you may use Node version manager like [nvm](https://github.com/nvm-sh/nvm).\n\n2. Clone this repository:\n\n```bash\ngit clone git@github.com:scramjetorg/framework-js.git\n```\n\n3. Install project dependencies:\n\n```bash\nnpm i\n```\n\n### Commands\n\nThere are multiple npm commands available which helps run tests, build the project and help during development.\n\n#### Running tests\n\n```bash\nnpm run test\n```\n\nRuns all tests from `test` directory. It runs `build` internally so it doesn't have to be run manually.\n\n```bash\nnpm run test:unit[:w]\n```\n\nRuns all unit tests (`test/unit` directory). It runs `build` internally so it doesn't have to be run manually. When run with `:w` it will watch for changes, rebuild and rerun test automatically. To run unit tests without rebuilding the project use `npm run test:run:unit`.\n\n```bash\nnpm run test:unit:d -- build/test/.../test.js [--host ...] [--port ...]\n```\n\nRuns specified test file in a debug mode. It runs `build` internally so it doesn't have to be run manually. This is the same as running\n`npm run build \u0026\u0026 npx ava debug --break build/test/.../test.js [--host ...] [--port ...]`. Then it can be inspected e.g. via Chrome inspector\nby going to `chrome://inspect`.\n\n```bash\nnpm run test:bdd\n```\n\nRuns all BDD tests (`test/bdd` directory). It runs `build` internally so it doesn't have to be run manually. To run BDD tests without rebuilding the project use `npm run test:run:bdd`.\n\n**Running single test file or specific tests**\n\nSingle test file can be run by passing its path to `test` command:\n\n```bash\nnpm run test:unit -- build/test/ifca/common.spec.js\n```\n\nWhile specific test cases can be run using `-m` (match) option:\n\n```bash\nnpm run test:unit -- -m \"*default*\"\n```\n\nBoth can be mixed to run specific tests from a given file or folder:\n\n```bash\nnpm run test:unit -- build/test/ifca/common.spec.js -m \"*default*\"\n```\n\n#### Building the project\n\n```bash\nnpm run build[:w]\n```\n\nTranspiles `.ts` sources and tests (`src` and `test` directories) and outputs JS files to `build` directory. When run with `:w` it will watch for changes and rebuild automatically.\n\n```bash\nnpm run dist\n```\n\nBuilds dist files - similar to `build` but skips `test` directory and additionally generates source maps.\n\n#### Miscellaneous\n\n```bash\nnpm run lint\n```\n\nLints `src` and `test` directories. Used as a `pre-commit` hook.\n\n```bash\nnpm run lint:f\n```\n\nFixes lint warnings/errors in `src` and `test` files.\n\n```bash\nnpm run coverage\n```\n\nChecks code coverage, generates HTML report and serves it on 8080 port.\n\n```bash\nnpm run coverage:check\n```\n\nChecks code coverage. Will fail if it is below a threshold defined in `package.json`. Useful as a CI job.\n\n```bash\nnpm run coverage:generate\n```\n\n\u003c!-- DEV_ONLY_END --\u003e\n","funding_links":["https://github.com/sponsors/scramjetorg","https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=7F7V65C43EBMW"],"categories":["Table of Contents"],"sub_categories":["Streaming Library"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscramjetorg%2Fframework-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscramjetorg%2Fframework-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscramjetorg%2Fframework-js/lists"}