{"id":16399462,"url":"https://github.com/mtrunkat/flowage","last_synced_at":"2025-08-06T15:31:54.292Z","repository":{"id":57238836,"uuid":"168944010","full_name":"mtrunkat/flowage","owner":"mtrunkat","description":"Easy transformations and filtering for NodeJS object streams. ","archived":false,"fork":false,"pushed_at":"2019-02-12T08:12:01.000Z","size":36,"stargazers_count":83,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-02T14:38:51.363Z","etag":null,"topics":["filtering","nodejs","streams","transformations"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mtrunkat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-03T12:41:36.000Z","updated_at":"2023-08-20T13:07:51.000Z","dependencies_parsed_at":"2022-09-05T07:51:11.878Z","dependency_job_id":null,"html_url":"https://github.com/mtrunkat/flowage","commit_stats":null,"previous_names":["mtrunkat/js-flowage"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtrunkat%2Fflowage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtrunkat%2Fflowage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtrunkat%2Fflowage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtrunkat%2Fflowage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mtrunkat","download_url":"https://codeload.github.com/mtrunkat/flowage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228920546,"owners_count":17992005,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filtering","nodejs","streams","transformations"],"created_at":"2024-10-11T05:24:23.559Z","updated_at":"2024-12-09T16:12:34.321Z","avatar_url":"https://github.com/mtrunkat.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Flowage\n\n[![npm version](https://badge.fury.io/js/flowage.svg)](https://www.npmjs.com/package/flowage)\n\n## Contents\n\n\u003c!-- toc --\u003e\n\n- [Motivation](#motivation)\n- [Basic usage](#basic-usage)\n- [Reference](#reference)\n  * [merge `stream1.merge(stream2)`](#merge-stream1mergestream2)\n  * [collect `stream.collect()`](#collect-streamcollect)\n  * [filter `stream.filter(function)`](#filter-streamfilterfunction)\n  * [chunk `stream.chunk(length)`](#chunk-streamchunklength)\n  * [map `stream.map(function)`](#map-streammapfunction)\n  * [omit `stream.omit(field1, field2, ...)`](#omit-streamomitfield1-field2-)\n  * [pick `stream.pick(field1, field2, ...)`](#pick-streampickfield1-field2-)\n  * [pluck `stream.pluck(field);`](#pluck-streampluckfield)\n  * [uniq `stream.uniq(field)`](#uniq-streamuniqfield)\n  * [weakSort `stream.weakSort(sortFunction, [bufferMinSize=75], [bufferMaxSize=100])`](#weaksort-streamweaksortsortfunction-bufferminsize75-buffermaxsize100)\n  * [onSeries `stream.onSeries(async function, [concurrency=1])`](#onseries-streamonseriesasync-function-concurrency1)\n\n\u003c!-- tocstop --\u003e\n\n## Motivation\n\nThis package simplifies transformations and filtering of NodeJS object streams. Think about it as [Underscore.js](http://underscorejs.org)\nfor streams.\n\nThe basic use case I faced many times was a transformation of a large number of JSON objects that are finally stored in some database.\nTransformation is the quick part but then you have to then chunk data in size allowed by your database to limit the number of queries\nand control the flow of the whole stream based on how fast you are able to save the transformed data.\n\n## Basic usage\n\n```javascript\nconst { Readable } = require('stream');\nconst Flowage =  require('flowage');\n\n// Let's have some stream that will output a series of objects { n: 0 }, { n: 1 }, { n: 2 }, { n: 3 }, ...\nconst readable = new Readable({ objectMode: true });\nlet n = 0;\nsetInterval(() =\u003e readable.push({ n: n++ }), 1000);\n\n// Pipe it thru Flowage() to get stream extended by helper methods.\nconst flowage = readable.pipe(new Flowage());\n\n// Split the stream into a stream of odd objects and even objects and extend them with some field is='odd' or is='even'.\nconst oddStream = flowage\n    .filter(obj =\u003e obj.n % 2)\n    .map(obj =\u003e Object.assign({}, obj, { is: 'odd' }));\n\nconst evenStream = flowage\n    .filter(obj =\u003e obj.n % 2 === 0)\n    .map(obj =\u003e Object.assign({}, obj, { is: 'even' }));\n\n// Then merge them back.\nconst mergedStream = oddStream.merge(evenStream);\n\n// Chunk them by 100 records.\nconst chunkedStream = mergedStream.chunk(100);\n\n// Save them to MongoDB in batches of 100 items with concurrency 2.\n// This also corks the stream everytime the period when max concurrency is reached.\nchunkedStream.onSeries(async (arrayOf100Items) =\u003e {\n    await datase.collection('test').insert(arrayOf100Items);\n}, { concurrency: 2 });\n\n```\n\n## Reference\n\n### merge `stream1.merge(stream2)`\n\nReturns stream containing values merged from 2 given streams. Merged stream ends when both streams ends.\n\n```javascript\nconst mergedStream = stream1.merge(stream2);\n```\n\n### collect `stream.collect()`\n\nReturns Promise that gets resolved when stream ends to an array of all the values.\n\n```javascript\nconst data = await stream.collect();\n```\n\n### filter `stream.filter(function)`\n\nReturns stream containing filtered values.\n\n```javascript\n// Filter out even items from stream.\nconst filteredStream = stream.filter(val =\u003e val.index % 2 === 0);\n```\n\n### chunk `stream.chunk(length)`\n\nReturns stream where each item is an array given number of items from original stream.\n\n```javascript\n// Chunk values into arrays of 10 items.\nconst chunkedStream = stream.chunk(10);\n```\n\n### map `stream.map(function)`\n\nReturns stream where original items are transformed using given function.\n\n```javascript\n// Extend each object in the stream with `.foo = 'bar'` field.\nconst mappedStream = stream.map(val =\u003e Object.assign({}, val, { foo: 'bar' }));\n```\n\n### omit `stream.omit(field1, field2, ...)`\n\nReturns stream where given fields where omitted.\n\n```javascript\n// Omit field1 and field2 from stream objects.\nconst resultingStream = stream.omit('field1', 'field2');\n```\n\n### pick `stream.pick(field1, field2, ...)`\n\nReturns stream where each item contains only the given fields.\n\n```javascript\n// Pick only field1 and field2 from stream objects.\nconst resultingStream = stream.pick('field1', 'field2');\n```\n\n### pluck `stream.pluck(field);`\n\nReturns stream with given field picked from each item.\n\n```javascript\n// Pick only field1 and field2 from stream objects.\nconst resultingStream = stream.pluck('field1');\n```\n\n### uniq `stream.uniq(field)`\n\nReturns stream containing only unique items based on given field.\nYou need enough memory to keep a set of all unique values hashed using sha256.\n\n```javascript\n// Filter unique items based on id field.\nconst uniquesStream = stream.uniq('id');\n```\n\n### weakSort `stream.weakSort(sortFunction, [bufferMinSize=75], [bufferMaxSize=100])`\n\nReturns stream containing values sorted using given function and floating buffer of a given size.\n\nThis method is helpful when only a few neighboring items may have the wrong order. This may happen\nfor example when a client is pushing data into the storage via API with concurrency higher than 1 and the\nquests reach the server in the wrong order. Or the API has multiple redundant instances that may process\nthe incoming requests with different speed.\n\nThis method uses a buffer for streamed items. Every time the buffer reaches `bufferMaxSize` gets\nsorted and `bufferMaxSize - bufferMinSize` items are outputted to the stream.\n\n```javascript\nconst sortFunction = (a, b) =\u003e a.index \u003c b.index ? -1 : 1;\nconst sortedStream = stream.sort(sortFunction, 75, 100);\n```\n\n### onSeries `stream.onSeries(async function, [concurrency=1])`\n\nReturns a promise that gets resolved when given function gets finished for the last item of the stream.\n\nEverytime the given concurrency is reached it pauses the stream.\n\n```javascript\n// Store items in MongoDB with concurrency 10.\nawait stream.onSeries(async (item) =\u003e {\n    await database.collection('items').insert(item);\n}, 10);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtrunkat%2Fflowage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtrunkat%2Fflowage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtrunkat%2Fflowage/lists"}