{"id":13510299,"url":"https://github.com/ominibyte/richflow","last_synced_at":"2026-02-19T22:25:25.247Z","repository":{"id":79111449,"uuid":"98733218","full_name":"ominibyte/richflow","owner":"ominibyte","description":"A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable \u0026 Transformable Pipeline data processing.","archived":false,"fork":false,"pushed_at":"2017-08-08T02:37:18.000Z","size":125,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-18T01:02:03.913Z","etag":null,"topics":["data-flow","data-pipeline","data-processor","data-stream","data-transformation","flow","javascript","nodejs","pipe-data","pipeline-framework","streaming-data","synchronous"],"latest_commit_sha":null,"homepage":"http://richflow.richboy.me/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ominibyte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-07-29T13:11:45.000Z","updated_at":"2024-10-17T00:49:40.000Z","dependencies_parsed_at":"2024-01-13T19:26:12.955Z","dependency_job_id":"22626319-d088-4cc6-88d9-2eba4d3b5861","html_url":"https://github.com/ominibyte/richflow","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"9660ff0a510a1caf26deca05c5a9ad0124d2b40f"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ominibyte/richflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ominibyte%2Frichflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ominibyte%2Frichflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ominibyte%2Frichflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ominibyte%2Frichflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ominibyte","download_url":"https://codeload.github.com/ominibyte/richflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ominibyte%2Frichflow/sbom","scorecard":{"id":706245,"data":{"date":"2025-08-11","repo":{"name":"github.com/ominibyte/richflow","commit":"9660ff0a510a1caf26deca05c5a9ad0124d2b40f"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.7,"checks":[{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Code-Review","score":0,"reason":"Found 0/11 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact 0.0.7 not signed: https://api.github.com/repos/ominibyte/richflow/releases/7312683","Warn: release artifact 0.0.7 does not have provenance: https://api.github.com/repos/ominibyte/richflow/releases/7312683"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-22T06:40:59.729Z","repository_id":79111449,"created_at":"2025-08-22T06:40:59.729Z","updated_at":"2025-08-22T06:40:59.729Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29635300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T18:02:07.722Z","status":"ssl_error","status_checked_at":"2026-02-19T18:01:46.144Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-flow","data-pipeline","data-processor","data-stream","data-transformation","flow","javascript","nodejs","pipe-data","pipeline-framework","streaming-data","synchronous"],"created_at":"2024-08-01T02:01:32.621Z","updated_at":"2026-02-19T22:25:25.205Z","avatar_url":"https://github.com/ominibyte.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","nodejs"],"sub_categories":[],"readme":"RichFlow: Data processing for JavaScript\n========================================\nA framework for javascript data pipeline processing, data sharing and stream processing. Actionable \u0026 Transformable Pipeline data processing.\n\nRichFlow is an extract of Flow, a node.js library built for data processing in the [JAMScript Framework](https://github.com/anrl/JAMScript-beta)\n\nInstallation\n------------\n\n`npm install richflow`\n\nFor use in the browser, download richflow.js from [github.com/ominibyte/richflow](https://github.com/ominibyte/richflow)\n\nOnline in-browser testing of RichFlow is available at [richflow.richboy.me](http://richflow.richboy.me)\n\nUsage\n-----\nThe RichFlow library comes with several classes which can be used for different purposes. RichFlow is mostly based on JavaScript ES6.\nFor the basic Flow, 'require' it as follows:\n\n```javascript\n//in node.js\nvar {Flow} = require('richflow');\n\n//in your browser\n\u003cscript src=\"richflow.js\"\u003e\u003c/script\u003e\n\u003cscript\u003e\n        var Flow = RichFlow.Flow;\n\u003c/script\u003e        \n```\n\n\nFlow can be used to operate on several data types including Arrays, Sets, Maps, FileSystem, Objects, Generators.\nIn addition, RichFlow comes with a Streamer object that allows data stream processing and allows Data sharing opportunities.\n\n```javascript\nvar array = [1, 2, 3, 4, 5];\n\n//For a very simple example. Let us count the number of even numbers in the array\nvar count = Flow.from(array).where(elem =\u003e elem % 2 == 0).count();\n\n//create a data window and return a new array\nvar range = Flow.from(array).skip(1).limit(3).collect();\n//The above line is equivalent to\nvar range = Flow.from(array).range(1, 4).collect();\n\n//a few more possibilities\nvar anotherArray = [6, 7, 8, 9];\nvar average = Flow.from(array).merge(anotherArray).select(elem =\u003e elem * 5).average();\n\n//check if all students passed\nvar studentScores = [71, 90, 55, 50, 88, 67];\nvar allPassed = Flow.from(studentScores).allMatch(score =\u003e score \u003e= 50);\n\n//an example of selectExpand: prints [\"my\",\"name\",\"is\",\"richboy\"]\nconsole.log(Flow.from(\"my name is richboy\").selectExpand(input =\u003e input.split(\" \")).collect());\n\n//an example of selectFlatten: prints [1,2,3,4,5,6,7,8,9]\nconsole.log(Flow.from([[1,2,3],[4,5,6],[7,8,9]]).selectFlatten().collect());\n```\n\nUnderstanding RichFlow\n----------------------\nA Flow is a data abstraction encapsulated within a JS ES6 class object that allows several operations on several data structures. Large collections of data can be processed efficiently. Flow allows programmers operate on data in somewhat similar way to SQL operations and it uses relatively similar query words.\nFlow operations can either be methods/transformations (operations that yield other Flows) or actions (operations that yield a result).\n\n\nFlow Creation\n-------------\nA Flow can be created from several Javascript data structures including: Array, Set, Map, Object, FileSystem, Generator, and Streamer (an in-built bare-bones class for supporting data streaming). The last two could potentially produce an infinite stream of data.\n\nHere is an example of how a Flow can be created from a simple array:\n\n```javascript\nvar array = [1, 0, 5, 13, -1];\nvar flow = Flow.from(array);\n```\n\nThe above example creates an Iterator from the array from which data is pipelined.\nFlow can also be created from a number range using:\n\n```javascript\nvar flow = Flow.fromRange(3, 8);\t//creates a Flow with [3,4,5,6,7,8]\n```\n\nFlow can also be created from several arguments using:\n\n```javascript\nvar flow = Flow.of(1, 3, 4, 7);\t//creates a Flow with [1,3,4,7]\n```\n\nThe Flow.of(…) also allows creating Flow with empty array elements which could be operated on later. Flow.of(…) default to Flow.from(…) when the argument to the method is not a number and is a single argument. An example is shown below:\n\n```javascript\nvar flow = Flow.of(3);\t\t//creates a Flow with [[],[],[]]\n```\n\nLet us show a very simple use case for Flow.of(…) that is actually used within the Flow implementation:\n\n```javascript\n//A lazy way to create 5 queues.\nvar flow = Flow.of(5).map(array =\u003e new Queue());\n```\n\n\nFlow Methods\n------------\nFlow methods are data transformations that yield other Flows. Each Flow maintains a link to the Flow operation before it.\nFlow methods are lazily computed, nothing happens to the underlying data until an action is called.\nWhen an action is called on a Flow, data is continually streamed/piped down to the next Flow level for further processing as they are produced.\nThis can reduce the execution time because some operations can be handled together. The currently supported methods are listed below:\n\nFor most of the examples, we will be using the following extracted sample dataset of nobel prize winners for physics in 2016. The complete dataset is available at: [http://api.nobelprize.org/v1/prize.json](http://api.nobelprize.org/v1/prize.json)\n\n```javascript\nvar winners = [\n    {\n      \"id\": \"928\",\n      \"firstname\": \"David J.\",\n      \"surname\": \"Thouless\",\n      \"motivation\": \"\\\"for theoretical discoveries of topological phase transitions and topological phases of matter\\\"\",\n      \"share\": \"2\"\n    },\n    {\n      \"id\": \"929\",\n      \"firstname\": \"F. Duncan M.\",\n      \"surname\": \"Haldane\",\n      \"motivation\": \"\\\"for theoretical discoveries of topological phase transitions and topological phases of matter\\\"\",\n      \"share\": \"4\"\n    },\n    {\n      \"id\": \"930\",\n      \"firstname\": \"J. Michael\",\n      \"surname\": \"Kosterlitz\",\n      \"motivation\": \"\\\"for theoretical discoveries of topological phase transitions and topological phases of matter\\\"\",\n      \"share\": \"4\"\n    }\n];\n```\n\n#### select(function | String) \\[alias: map\\]\nThis is similar to map in mad-reduce operations. This selects one or more parts of a data from a given dataset. As an example\n\n```javascript\n//we wish to get the surnames of all the winners\nvar selectFlow = Flow.from(winners).select(winner =\u003e winner.surname);   //returns a Flow object\n\n//For objects as with the working example, we can also do:\nvar selectFlow = Flow.from(winners).select(\"surname\");  //returns a Flow object\n```\n\n#### limit(Number)\nTo limit the number of results obtained after the previous operation.\n\n```javascript\n//let us say we want to restrict the result to the first two winners\nvar limitFlow = Flow.from(winners).limit(2);   //returns a Flow object\n\n//get the ids of the first two winners\nvar ids = Flow.from(winners).limit(2).select(\"id\").collect();   //returns an array\n```\n\n#### skip(Number)\nTo ignore the first given number of results found after a previous operation.\n\n```javascript\n//skip the first result\nvar skipFlow = Flow.from(winners).skip(1);   //returns a Flow object\n```\n\n#### range(startIndex: Number, endIndex: Number)\nThis method combines the implementations of limit and skip. It creates a bound for the data to be used for further processing.\nstartIndex is inclusive while endIndex is exclusive.\n\n```javascript\n//so if we want to get only the second person:\nvar rangeFlow = Flow.from(winners).range(1,2);   //returns a Flow object\n```\n\n#### skipUntil(function)\nSkip until the condition in the function argument returns true. The function will receive each piped input and should return a boolean\n\n```javascript\nvar remaining = Flow.fromRange(1,10).skipUntil(num =\u003e num \u003e 6).collect();\n//returns [7, 8, 9, 10]\n```\n\n#### skipWhile(function)\nSkip while the condition in the function argument returns true. The function will receive each piped input and should return a boolean\n\n```javascript\nvar remaining = Flow.fromRange(1,10).skipWhile(num =\u003e num != 6).collect();\n//returns [6, 7, 8, 9, 10]\n```\n\n#### takeUntil(function)\nKeep accepting the piped data until the condition in the function argument returns true. This method also takes the data that meets the condition but skips after. The function will receive each piped input and should return a boolean\n\n```javascript\nvar taken = Flow.fromRange(1,10).takeUntil(num =\u003e num == 4).collect();\n//returns [1, 2, 3, 4]\n```\n\n#### takeWhile(function)\nKeep accepting the piped data while the condition in the function argument returns true. The function will receive each piped input and should return a boolean\n\n```javascript\nvar taken = Flow.fromRange(1,10).takeWhile(num =\u003e num * 8 \u003c= 50).collect();\n//returns [1, 2, 3, 4, 5, 6]\n```\n\n\n#### selectExpand(function)\nThis maps one input to many outputs as generated by the function. The collection generated by function must be supported by Flow.from(…).\n\n```javascript\nvar sentence = \"my name is richboy\";\nvar parts = Flow.from(sentence).selectExpand(input =\u003e input.split(\" \")).collect();\n//returns  [\"my\",\"name\",\"is\",\"richboy\"]\n\n//Another Example: rewrite the sentence with only words that are above 2 chars long\nsentence = Flow.from(sentence).selectExpand(input =\u003e input.split(\" \")).where(word =\u003e word.length \u003e 2).join(\" \");\n//returns \"name richboy\"\n```\n\n#### selectFlatten()\nThis is similar to selectExpand, except that this doesn't take a function. Select flatten assumes that the input from the pipe is a collection that is supported by Flow.from(…).\n\n```javascript\nvar flattened = Flow.from([[1,2,3],[4,5,6],[7,8,9]]).selectFlatten().collect();\n//returns  [1,2,3,4,5,6,7,8,9]\n```\n\n#### where(function) \\[alias: filter\\]\nThis method performs a filtering operation on the data to match a constraint.\n\n```javascript\n//get all the even numbers from the array\nvar whereFlow = Flow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0); //returns a Flow object\n```\n\n#### orderBy(function | Flow.ASC | Flow.DESC | Flow.NUM_ASC | Flow.NUM_DESC)\nThis performs a sorting operation on the data based on a given function. Flow has internal operations to sort based on descending and ascending order.\nYou can provide your own sorting implementation which will normally be submitted to Array.prototype.sort() function.\n*Flow.ASC* and *Flow.DESC* will sort according to each character's Unicode code point value, according to the string conversion of each element with the only difference being that *Flow.ASC* will sort in ascending order and *Flow.DESC* in descending order.\n*Flow.NUM_ASC* and *Flow.NUM_DESC* with sort the elements as numbers.\n\n```javascript\n//sort the winners based on their surname\nvar orderedFlow = Flow.from(winners).select(\"surname\").orderBy(Flow.ASC);   //returns a Flow object\n```\n\n#### partitionBy(function | String)\nThis performs data grouping on the elements of the data, determined by the function. This is similar to the Flow action - groupBy, but this returns a Flow for further pipelining.\nThe argument can either be a function (that receives an item each time to generate the group/partition that items belongs to) or a key (from which the group/partition will be determined using JS object syntax like `input[key]`).\nAfter partitioning, data is emitted one partition at a time in the format:\n`{key: \"partition key\", value:[...array of elements in that partition]}`\n\nAs an example:\n\n```javascript\nvar array = [\n    {entity: \"book\", bookID: 12},\n    {entity: \"student\", studentID: 23434},\n    {entity: \"student\", studentID: 12233},\n    {entity: \"book\", bookID: 998}\n];\n\n//we want to partition by entity so all entries with same entity value would be grouped together\nlet partitions = Flow.from(array).partitionBy(\"entity\").collect();\n//partitions will contain:\n/*\n[\n\t{key: \"book\", value: [{entity: \"book\", bookID: 12}, {entity: \"book\", bookID: 998}]},\n\t{key: \"student\", value: [{entity: \"student\", studentID: 23434}, {entity: \"student\", studentID: 12233}]}\n]\n*/\n```\n\n#### merge(data)\nThis method is only available to an object of IteratorFlow and is used to merge a supported data structure as with Flow.from(data). Merging creates an Iterator and adds it to the current Iterator or Iterators.\nThis function also returns an IteratorFlow so one can do multiple merging on the return value.\n\n```javascript\n//let us merge the data for those who won the nobel prize for chemistry in 2016\nvar chemistryWinners = [\n   {\n     \"id\": \"931\",\n     \"firstname\": \"Jean-Pierre\",\n     \"surname\": \"Sauvage\",\n     \"motivation\": \"\\\"for the design and synthesis of molecular machines\\\"\",\n     \"share\": \"3\"\n   },\n   {\n     \"id\": \"932\",\n     \"firstname\": \"Sir J. Fraser\",\n     \"surname\": \"Stoddart\",\n     \"motivation\": \"\\\"for the design and synthesis of molecular machines\\\"\",\n     \"share\": \"3\"\n   },\n   {\n     \"id\": \"933\",\n     \"firstname\": \"Bernard L.\",\n     \"surname\": \"Feringa\",\n     \"motivation\": \"\\\"for the design and synthesis of molecular machines\\\"\",\n     \"share\": \"3\"\n   }\n];\n\nvar iteratorFlow = Flow.from(winners);  //IteratorFlow is the first flow in the chain\n//merge both datasets and return the full names of all the winners\nvar allWinners = iteratorFlow.merge(chemistryWinners).select(winner =\u003e winner.firstname + \" \" + winner.surname).collect();\n//returns [\"David J. Thouless\", \"F. Duncan M. Haldane\", \"J. Michael Kosterlitz\", \"Jean-Pierre Sauvage\", \"Sir J. Fraser Stoddart\", \"Bernard L. Feringa\"]\n```\n\n#### discretize(span, spanLength\\[, spawnFlows\\])\nThis method is best understood in the context of data streams. It allows processing data in windows.\n*span* is the number of data streams to focus on in a window.\n*spanLength* can either be a Number or a function that tells when we get to the end of a window.\n*spawnFlows* is an optional boolean value that states if the output should be objects of DiscretizedFlow or simple arrays. spawnFlows defaults to true.\n\nThis method is available to all Flow objects but the implementation differs as with the IteratorFlow. See the advanced section for Usage.\n\n\nFlow Actions\n------------\nFlow actions are operations that yield results that are not themselves Flows. When an action is called on a Flow, the Flow engine begins operating on the data and pipes each produces data to the next layer until the condition for the action is met. The currently supported actions are listed below:\n\n#### count()\nReturns the total number of datasets left after the last Flow method.\n\n```javascript\n//get the count of all the even numbers from the array\nvar count = Flow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0).count();\n//returns 4\n```\n\n#### findFirst()\nReturns the first data available in a Flow.\n\n```javascript\n//get the first even number\nvar first = Flow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0).findFirst();\n//returns 2\n```\n\n#### findLast()\nReturns the last data available in a Flow.\n\n```javascript\n//get the last even number\nvar last = Flow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0).findLast();\n//returns 8\n```\n\n#### findAny()\nThis returns any data from the Flow. This currently does the same as findFirst(). This methods is expected to work best in a parallel computing sense with ParallelFlow.\n\n```javascript\n//get the count of all the even numbers from the array\nvar any = Flow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0).findAny();\n//returns 2\n```\n\n#### groupBy(function | String)\nThis returns the data as a JS object partitioned into array of groups, determined by the function.\nThe argument can either be a function (that receives an item each time to generate the group that items belongs to) or a key (from which the group will be determined using JS object syntax like `input[key]`).\n\n```javascript\nvar array = [\n    {entity: \"book\", bookID: 12},\n    {entity: \"student\", studentID: 23434},\n    {entity: \"student\", studentID: 12233},\n    {entity: \"book\", bookID: 998}\n];\n\n//we want to group by entity so all entries with same entity value would be grouped together\nlet groups = Flow.from(array).groupBy(\"entity\");\n//groups will contain:\n/*\n{\n\t\"book\": [{entity: \"book\", bookID: 12}, {entity: \"book\", bookID: 998}],\n\t\"student\": [{entity: \"student\", studentID: 23434}, {entity: \"student\", studentID: 12233}]\n}\n*/\n```\n\n#### collect(\\[function\\])\nThis returns the data as either an Array, Set or Map. The function argument is optional and default to returning an array.\nThe function argument is a Flow internal function which can either be `Flow.toSet()`, `Flow.toArray()` or `Flow.toMap(keyFunc)`. It is also possible to ignore the parenthesis for the array and set as `Flow.toArray` and `Flow.toSet` respectively.\n`collect`ing with `Flow.toSet` returns a distinct dataset, `collect`ing with `Flow.toArray` returns all the data left after the last Flow method as an array while `collect`ing with `Flow.toMap(keyFunc)` returns a JS ES6 Map. The keyFunc in `Flow.toMap()` is same as the function supplied to groupBy. The only difference between calling collect with toMap(keyFunc) and calling groupBy(keyFunc) is that toMap returns an ES6 Map object while groupBy returns a plain JS object.\n\n```javascript\nvar array = [\n    {entity: \"book\", bookID: 12},\n    {entity: \"student\", studentID: 23434},\n    {entity: \"student\", studentID: 12233},\n    {entity: \"book\", bookID: 998}\n];\n\n//collecting to Array. Note that this exactly same without Flow.toArray\nvar entities = Flow.from(array).select(\"entity\").collect(Flow.toArray);\n//returns [\"book\", \"student\", \"student\", \"book\"]\n\n//collecting to Set\nvar entitySet = Flow.from(array).select(\"entity\").collect(Flow.toSet);\n//returns Set(2) {\"book\", \"student\"}\n\n//collecting all to Map\nvar map = Flow.from(array).collect(Flow.toMap(\"entity\"));\n/*\n    returns:\n    Map(2) {\n        \"book\" =\u003e (2) [{entity: \"book\", bookID: 12}, {entity: \"book\", bookID: 998}],\n        \"student\" =\u003e (2) [{entity: \"student\", studentID: 23434}, {entity: \"student\", studentID: 12233}]\n    }\n*/\n```\n\n#### join([delimiter: String])\nThis function joins the outputs by a delimiter which is optional. The delimiter argument defaults to \",\".\n\n```javascript\nvar joined = Flow.from([1,2,3,4,5]).map(num =\u003e num * 5).limit(3).join(\" | \");\n//returns 5 | 10 | 15\n```\n\n#### forEach(function) \\[alias: foreach\\]\nThis sends the remaining data from the last Flow in the chain to the custom function provided. The user may wish to operate on each data outside the context of Flow.\n\n```javascript\n//print all even numbers to the console\nFlow.from([1,2,3,4,5,6,7,8,9]).where(num =\u003e num % 2 == 0).foreach(console.log);\n```\n\n#### anyMatch(function)\nThis returns a boolean to check if the remaining data matches the definition in the user defined function.\n\n```javascript\n//check if there is any number in the array that if we multiply with 5 yields 35\nvar match = Flow.from([1,2,3,4,5,6,7,8,9]).anyMatch(num =\u003e num * 5 == 35);\n//returns true\n```\n\n#### allMatch(function)\nSimilar to anyMatch, this checks that all the remaining data matches the condition defined in the function.\n\n```javascript\n//check if multiplying 5 with all numbers in the array yields 35\nvar match = Flow.from([1,2,3,4,5,6,7,8,9]).allMatch(num =\u003e num * 5 == 35);\n//returns false\n```\n\n#### noneMatch(function)\nThis may look like the inverse of allMatch but it is more closely related to anyMatch. This basically checks that no item matches the condition defined in the function argument.\n\n```javascript\n//check that multiplying 5 with any numbers in the array DOES NOT yield 35\nvar match = Flow.from([1,2,3,4,5,6,7,8,9]).noneMatch(num =\u003e num * 5 == 35);\n//returns false\n```\n\n#### reduce(initial, function)\nThis allows a Flow to be reduced to a single value. It takes the initial value for the reduce operation and the function that defines how the reduce would be carried out.\nThe function parameter takes two arguments (in the order: currentValue and newValue) and is expected to return a value which is further fed in as the currentValue for the next iteration. The function is called until all values are piped out of the Flow chain.\n\n```javascript\n//let us implement getting the sum of numbers\nvar sum = Flow.from([1,2,3,4,5]).reduce(0, (cv, nv) =\u003e cv + nv);\n//returns 15\n```\n\n#### sum()\nThis is a reduce operation that returns the sum. It is expected that the values a the last Flow item in the chain return Number types.\n\n```javascript\nvar sum = Flow.from([1,2,3,4,5]).sum();\n//returns 15\n```\n\n#### average()\nThis is also a reduce operation that returns the average. It is expected that the values a the last Flow item in the chain return Number types.\n\n```javascript\nvar avg = Flow.from([1,2,3,4,5]).average();\n//returns 3\n```\n\n#### max()\nThis returns the maximum number. It is expected that the values a the last Flow item in the chain return Number types.\n\n```javascript\nvar max = Flow.from([1,2,3,4,5]).max();\n//returns 5\n```\n\n#### min()\nThis returns the minimum number. It is expected that the values a the last Flow item in the chain return Number types.\n\n```javascript\nvar min = Flow.from([1,2,3,4,5]).min();\n//returns 1\n```\n\n\nFlow from FileSystem (for node.js)\n----------------------------------\nFlow does not current work with the browser FileReader due to the way the FileReader is designed, which differs from the synchronous design of Flow.\nFor working with files in node, The Flow.from() method accepts a string path to the file. However, the path needs to be prepended with \"fs://\". This is used to distinguish working with files from strings.\nFiles are processed by line. As an example:\n\n```javascript\n//we have a file called names.txt in the same directory\nFlow.from(\"fs://./names.txt\").range(0, 11).foreach(line =\u003e console.log(line));\n```\n\n\nAdvanced + Design Info\n----------------------\n### Flow Groups\nThere are 5 Flow groups namely: IteratorFlow, OutFlow, InFlow, DiscretizedFlow and Flow (the default Flow). They are grouped based on the type of operations that can be performed on them.\n\ni. IteratorFlow: This is mostly the first Flow in a Flow chain. When the Flow.from(…) method is called, an IteratorFlow is created. This flow extends the default Flow and provides a few more operations.\n\nii. OutFlow: This Flow is responsible for processing and sending data across applications. More information on this later.\n\niii. InFlow: This Flow is responsible for receiving data from another application. Also, more information on this later.\n\niv. DiscretizedFlow: This Flow splits data streams into chunks/windows to allow for Flow methods that require finite data operations. Discretized Flows are discussed much later.\n\nv. Flow: This is the default Flow that has all the basic operations for data processing.\n\n### Flow Chain Pipelining\nA Flow chain is a linked data structure of different Flow objects. Every Flow is aware of the previous Flow and the next Flow in the chain. A Flow chain is created when a Flow method is called on a Flow object.\nAs an example:\n\n```javascript\nvar flow = Flow.from(array).skip(2).where((num) =\u003e num % 2 == 0);\n```\n\nFrom the example above, there are three Flow objects in the Flow chain. When an action is called on the final flow object, data is piped through the Flow chain till it gets to the last Flow in the chain, from which the action is computed.\n\n### Flow Push \u0026 Pull Models\nFlow provides two modes of data pipelining: push and pull. The pull model is used to request that data be piped from the IteratorFlow (discussed later) through the chain. The data is generated from the Iterator when requested and sent through the chain. This mode is used by Flow actions to do a final computation on the dataset. For the push model, data is automatically piped through the Flow chain. The push model is used in Flow Streaming.\n\n### Flow Streaming \u0026 The Streamer Class\nFor continuous streams of data, Flow provides a data push model that can continuously pipe data through the Flow chain. This can be especially useful if computed data needs to be sent to another application for further processing. Each Flow pushes processed data to the next Flow in the chain or to a customizable terminal function (If the Flow is the last in the chain). The terminal function for a Flow can be set using the setTerminalFunction method. Flow streaming can be achieved when the Flow is created from either a Streamer or a function that generates continuous data like a JS Generator. An example of working with Streamer is shown below:\n\n```javascript\n//import Flow and Streamer\nvar Flow = RichFlow.Flow;\nvar Streamer = RichFlow.Streamer;\n//create a new streamer\nvar streamer = new Streamer();  \n\n//create a Flow from the streamer. Several streamers can be added via the merge method\nvar flow = Flow.from(streamer).filter(num =\u003e num % 2 != 0);  //filter for odd numbers\n//set the terminal function which will receive the data from the last Flow in the chain\nflow.setTerminalFunction(console.log); //print to the console\n//Inform the IteratorFlow to start listening for data from the streamer\nflow.startPush(); //This can be called from any Flow in the chain.\n\nsetInterval(() =\u003e {\n    streamer.send(parseInt(Math.random() * 10)); //send data to all listeners\n}, 500);\n```\n\n**NOTE**: If the `startPush` method is called after the Streamer starts generating data, some data may be lost at the initial stage.\n\nThe Streamer class is bare-bones and does minimal work. It can be extended to do much more like working as a finite dataset. Data could be received from the OutFlow and cached or data it generates could be cached and reused as a finite dataset using the Flow pull mode. If you wish to use the Streamer in Flow pull mode, you will need to extend the class and provide implementation for the `size` and `get` methods.\n\nThe Streamer class can act as a stream provider and a stream receiver as well. A function can be supplied to the constructor of the Streamer to receive stream data. More on this on the InFlow and OutFlow sections.\n\n### IteratorFlow\nThe IteratorFlow is a Flow that creates a unified means of retrieving data from different data structures. The IteratorFlow turns the data passed to Flow.from(…) into a Javascript Iterable by wrapping the data with an iterator implementation that makes retrieving data as easy as calling a next() method on the iterator handle. More Iterators can be added via the merge method on an object of IteratorFlow. The merge method takes the same type of parameter as the Flow.from(…) method.\n\nThis Flow is the Root Flow of the Flow chain and can be accessed from any Flow in the chain via the property rootFlow. As an example:\n\n```javascript\nvar flow = Flow.from(array).skip(2).where((num) =\u003e num % 2 == 0);\nvar iteratorFlow = flow.rootFlow;\t//get access to the IteratorFlow\n```\n\nFor data streaming in Flow, the IteratorFlow needs to listen for changes on the Streamer object(s) and retrieves new data when data is sent via the Streamer.send() method. The retrieved data is pushed through the Flow chain till it gets to an OutFlow or the terminal function of the last Flow object in the chain. To start data streaming in Flow, the startPush() method needs to be called on an object of the IteratorFlow. To stop the streaming at anytime, the stopPush() method can be called on the IteratorFlow object. When the stopPush() method is called, the IteratorFlow disconnects from the Streamers and stops listening for incoming data on the connected streams.\n\n\n### DiscretizerFlow \u0026 DiscretizedFlow\nDiscretizerFlow partitions streams of data flowing through the Flow chain into windows and each data window could be emitted as a DiscretizedFlow or an array. Actually, discretization can also occur for static/finite datasets like arrays or generators. DiscretizedFlows are IteratorFlows and could themselves be discretized and Flow actions can be called on them. Any Flow can be discretized (with an exception to OutFlow). However, the discretization implementation in IteratorFlow differs from the implementation on others Flow.\n\nIteratorFlow handles the discretization process internally, while the DiscretizerFlow handles discretization for all other Flows. For IteratorFlow discretization, the data window can be created from a single iterator or multiple iterators (this could be a single datastream or multiple datastream) while the discretization for other Flow groups are done on the input data. The discretize method takes three arguments namely - the window span, the span length and a boolean value indicating if data should be spawned as discretized flows or as arrays. The third argument is optional and defaults to true.\n\nFor IteratorFlow discretization, the window span talks about how many iterators should be included in creating the window. Recall that an Iterator can be added via the IteratorFlow.merge(…) method. A block of data is a data structure that has one item from each iterator from the window. The span length is the number of data blocks that should constitute a discrete block. Span length can be a number, a function or an object having an ‘isDataEnd’ function. The function receives two arguments - the last data added and the current length of the window span and should evaluate to a boolean.\n\nFor other Flow groups, discretization is on the input. It is the responsibility of the programmer to ensure that the data received as input to the DiscretizerFlow is fit for discretization and it is assumed that each data piped can be broken down is the way needed by the programmer. When DiscretizerFlow determines that it is not possible to discretize ‘perfectly’, the implementation respects the programmers wish and fills the remaining slots  in the data block with null values. The discretize method take in the same arguments and the span length follows the same as that of IteratorFlow. The window span here talks about how many parts each input piped to the DiscretizerFlow can be broken down. It is assumed that when each input is passed to Flow.from(…), it should be able to create an Iterator that will generate the amount of data required by the programmer.\n\n```javascript\n//lazily create 4 streamers\nvar streamers = Flow.of(4).map(a =\u003e new Streamer()).collect();\n//we need to merge all the streams so we start by adding one\nvar flow = Flow.from(streamers[0]);\nfor( i = 1; i \u003c streamers.length; i++ )\n    flow = flow.merge(streamers[i]);  //merge the remainder\n//discretize with a span covering all streams and data length of 1\nvar discretizerFlow = flow.discretize(streamers.length, 1);\n//set the terminal function\ndiscretizerFlow.setTerminalFunction(discretizedFlow =\u003e console.log(discretizedFlow.selectFlatten().collect()));\ndiscretizerFlow.startPush();  //start listening for data on the Streamers\n\nsetInterval(() =\u003e {\n    streamers.forEach(streamer =\u003e streamer.send(parseInt(Math.random() * 10)));\n}, 500);\n```\n\n### OutFlow\nIn [JAMScript](https://github.com/anrl/JAMScript-beta), OutFlow was built as a specialized Flow for the purpose of sending processed data to external applications. Here, the OutFlow has been stripped of that functionality. Though the concept is still part of it but that is now the responsibility of Streamer. When an OutFlow is created, a Flow object is supplied as a argument which the OutFlow links to, in order to receive pushed data. A Streamer object is also provided as the second argument in the constructor to which the OutFlow is expected to push the data and an identifier key which can optionally be supplied as a third argument (If none is supplied, one is auto generated). A Streamer could for instance generate stream data from sensors and write to a datastore such as Redis or send the computed data elsewhere. Let us see an example with Redis:\n\n```javascript\n//require OutFlow, Flow and Streamer\nconst {Flow, Streamer, OutFlow} = require('richflow');  //in node.js (See top for browser)\nvar Redis = require('redis-fast-driver'); //require Redis\nvar redis = new Redis({host: '127.0.0.1', port: 6379}); //establish connection\n\n//create 4 streamers. Could listen to sensors and obtain data\nvar streamers = Flow.of(4).map(a =\u003e new Streamer()).collect();\n\n//we need to merge all the streams so we start by adding one\nvar flow = Flow.from(streamers[0]);\nfor( i = 1; i \u003c streamers.length; i++ )\n    flow = flow.merge(streamers[i]);  //merge the remainder\n\nvar outFlow = new OutFlow(flow.discretize(streamers.length, 1),\n                            new Streamer((dFlow, key) =\u003e {\n                              let avg = dFlow.selectFlatten().average();\n                              let timestamp = new Date().getTime();\n                              redis.rawCall(['ZADD', key, timestamp, avg + '']);\n                            }), \"App1.Key\");\noutFlow.start();  //inform the IteratorFlow to begin pushing data\n\n//simulate sensor data\nsetInterval(() =\u003e {\n    streamers.forEach(streamer =\u003e streamer.send(parseInt(Math.random() * 10)));\n}, 500);\n```\n\nThe start() method in OutFlow calls the startPush() method in the IteratorFlow (the first flow in the chain) and informs the IteratorFlow to start listening for push data from the data source. This data is continually pushed and may or may not get to the OutFlow based on the constraints within each Flow object in the Flow chain. As data arrives at the OutFlow, it is sent to the Streamer which further sends it to Redis.\nTo stop listening to data streams,  the programmer can call the OutFlow stop method on the object handle which will in turn call the stopPush() method on the IteratorFlow. For example:\n\n```javascript\noutFlow.stop();\n```\n\n### InFlow\nThis is also another specialized Flow. In [JAMScript](https://github.com/anrl/JAMScript-beta) it is solely responsible for retrieving data from an external application. The retrieved data can be taken through further processing along a Flow chain before being used. Here, the InFlow can listens for new data from the Streamer and push them onwards to any connected Flow. An example with Redis following from the OutFlow:\n\n```javascript\n//require InFlow, Flow and Streamer\nconst {Flow, Streamer, InFlow} = require('richflow');  //in node.js (See top for browser)\nvar Redis = require('redis-fast-driver'); //require Redis\nvar redis = new Redis({host: '127.0.0.1', port: 6379}); //establish connection\n\nclass MyStreamer extends Streamer{\n  constructor(){\n    super();\n    this.lastIndex = 0;\n    //listen for new data on Redis\n    redis.rawCall(['config', 'set', 'notify-keyspace-events', 'Ez']);\n    redis.rawCall(['psubscribe', '__keyevent*'], this.notify);\n  }\n  notify(e, data){\n    if(data[0] == \"pmessage\" \u0026\u0026 data[3]){ //check if a message has arrived\n      //get data from Redis\n      var self = this;\n      redis.rawCall(['ZRANGE', \"App1.Key\", this.lastIndex + 1, -1], function(err, resp){\n          if( err )\n              throw new Error(err);\n\n          for (var i = 0; i \u003c resp.length; i++) {\n              self.lastIndex++;\n              self.send(resp[i]); //send data to all listeners like InFlow\n          }\n      });\n    }\n  }\n}\n\nvar inflow = new InFlow(new MyStreamer());\nvar flow = inflow.where(avg =\u003e avg \u003e 5);  //filter for averages above 5\nflow.setTerminalFunction(console.log);  //print to the console\n```\n\n### Flow Caching\n\nThis is an internal process that aims to speed up Flow reuse and works with static/finite data sets (does not work with Flow streaming). Flow attempts to get data from the Iterators each time an action is called on the Flow. However, for static/finite datasets, the iterators will produce same data each time leading to a time wastage when piping through the Flow chain each time. By caching processed data, when ever an action is called on a Flow (a second time), because it has already processed the data during the first round, it serves the processed data, saving processing time. Caching IteratorFlow data is trivial so they are never cached. However, this caching is on memory. Currently, the cache stays on for as long as the Flow has not be garbage collected.\n\n\nCommon Pitfalls\n---------------\nThe `null` value has a special meaning within Flow so your data should not contain it. This could cause Flow to give a fake report.\n\nBy default, Flow caches outputs for faster reuse. However, this can cause certain issues if the underlying data source changes. With caching, the changes will not be reflected when the constructed Flow is being reused. Let us see a simple example with Arrays:\n\n```javascript\n//with caching\nvar arr = [1,2,3,4,5,6,7,8,9];\nvar flow = Flow.from(arr).where(num =\u003e num % 2 == 0);\nconsole.log(flow.count());  //prints 4\narr.push(0);\nconsole.log(flow.count());  //prints 4\n\n//without caching\nvar arr = [1,2,3,4,5,6,7,8,9];\nvar flow = Flow.from(arr).where(num =\u003e num % 2 == 0);\nflow.rootFlow.shouldCache = false;\nconsole.log(flow.count());  //prints 4\narr.push(0);\nconsole.log(flow.count());  //prints 5\n```\n\nTo disable caching, after creating the Flow, on the IteratorFlow do the following:\n\n```javascript\nvar iteratorFlow = Flow.from(…);\niteratorFlow.shouldCache = false; //needs to be done before any action is called\n```\n\nAnother common pitfall you may have is in reusing Flows. Each Iterator in the IteratorFlow maintains a cursor on where the next data should be obtained from. Now, because the pipeline process ensures that the minimum amount of work is done to produce the desired result, it will sometimes be the case that an iterator may not get to the end and thus reusing will resume the cursor of the iterator from the last placed it stopped and will yield unexpected results. Do not reuse Flows if you do not understand this concept. As an example:\n\n```javascript\nvar flow = Flow.fromRange(1, 10); //creates a Flow with numbers from 1 to 10\nconsole.log(flow.limit(5).collect()); //prints [1,2,3,4,5]\nflow.forEach(console.log);  //prints 7 8 9 10\n```\n\nFrom the above code, you can notice that the call to forEach prints what is left as opposed to all the content from 1 to 10. This type of error can be fixed in most cases by flushing the contents of the flow before reusing. Sometimes, it can only be fixed with a combination of turning off caching and flushing and other times it may take more than that. **Flow reuse should be done with caution**. To fix the above error, we can do the following:\n\n```javascript\nvar flow = Flow.fromRange(1, 10); //creates a Flow with numbers from 1 to 10\nconsole.log(flow.limit(5).collect()); //prints [1,2,3,4,5]\n//flush the remaining contents. The iterators automatically reset for reuse when they get to the end\nflow.count();\nflow.forEach(console.log);  //prints 1 2 3 4 5 6 7 8 9 10\n```\n\nRoadmap\n-------\ni. ParallelFlow: A truly parallel pipeline data processing library.\n\nii. Flow Caching Offloading: An investigation needs to be made on when and which Flows to release memory, especially when the system is running low on RAM storage. There could be a listener that listens out for memory changes and probably informs Flows to either save processed data to disk or release the data. Based on the size of data held by the Flows in the middle of the chain, the runtime could decide which will be faster, saving to the disk and reloading from disk when needed or recomputing from the previous Flow in the chain.\n\nContact\n-------\nFor questions or suggestions please send a message to david.echomgbe \\[@\\] gmail.com. Please prefix your email subject with \"RichFlow -\".\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fominibyte%2Frichflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fominibyte%2Frichflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fominibyte%2Frichflow/lists"}