{"id":19506907,"url":"https://github.com/afsalthaj/fp-generator","last_synced_at":"2026-06-16T15:31:16.546Z","repository":{"id":80596542,"uuid":"134493135","full_name":"afsalthaj/fp-generator","owner":"afsalthaj","description":"The library that solves all your data generation problem !","archived":false,"fork":false,"pushed_at":"2019-02-28T23:40:32.000Z","size":225,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-25T22:25:45.289Z","etag":null,"topics":["fs2","functional-programming","generator","scala","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/afsalthaj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-23T01:01:47.000Z","updated_at":"2019-02-28T23:40:34.000Z","dependencies_parsed_at":"2023-06-26T22:32:24.085Z","dependency_job_id":null,"html_url":"https://github.com/afsalthaj/fp-generator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/afsalthaj/fp-generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsalthaj%2Ffp-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsalthaj%2Ffp-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsalthaj%2Ffp-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsalthaj%2Ffp-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/afsalthaj","download_url":"https://codeload.github.com/afsalthaj/fp-generator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsalthaj%2Ffp-generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34412785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fs2","functional-programming","generator","scala","stream-processing"],"created_at":"2024-11-10T22:38:52.715Z","updated_at":"2026-06-16T15:31:16.528Z","avatar_url":"https://github.com/afsalthaj.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fp-generator\n\n[![Build Status](https://travis-ci.com/afsalthaj/fp-generator.svg?branch=master)](https://travis-ci.com/afsalthaj/fp-generator)\n\nThis library is for data generation - solving almost all usecases of data generations!\n\nIn-fact, it is simple light weight FP abstraction that makes use of fs2 for data generation and processing with granular control of its behavior, managing state, batching, time delays, concurrency and backpressure.\n\nIn short, this abstraction allows you to forget about the mechanical coding that is required for any generation and processing application.\n\nWhy this abstraction? Get more information [here](docs/datagen_why.md)\n\nTo see the working usages, please refer to [examples](src/main/scala/com/thaj/generator/examples).\n\n## Examples:\n\n### Simple Generator\n\nSpecify a `rule for data generation` as a simple function and a `processing function` (println for demo purpose) and then call run!\n\n```scala\n\n  val generator = GeneratorLogic.create {\n    s =\u003e \n      if (s \u003c 10)\n        Some (s + 1, s + 1)\n      else \n        None\n  }\n  \n  Generator.run[IO, Int, Int](generator.withZero(0)){\n    a =\u003e IO { println (a) }\n  }.unsafeRunSync()\n  \n  // output\n  1\n  2\n  3\n  ...\n\n```\n\n### Multiple Generators\n\nSpecify multiple `rule for data generation`, the `processing function`, and then call run. This should generate all the instances of data each having its own termination condition or logic of generation interleaved with each other.\n\n```scala\n\n    val generator1 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 100).option {\n          val ss = s + 1\n          (ss, ss)\n        }\n      }\n    }.withZero(0)\n\n    val generator2 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 10000).option {\n          val ss = s + 100\n          (ss, ss)\n        }\n      }\n    }.withZero(2000)\n\n    val generator3 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 200000).option {\n          val ss = s + 10000\n          (ss, ss)\n        }\n      }\n    }.withZero(100000)\n\n    Generator.run[IO, Int, Int](generator1, generator2, generator3)(a =\u003e IO { println(a) }).unsafeRunSync()\n    \n    // output\n    1\n    110000\n    2100\n    120000\n    2\n    ...\n    6\n    2600\n    7\n    2700\n```\n\n### Batching\n\nSpecify a `rule of data generation`, specify a `batch size`, a `processing function` that operates on a batch, and then call run. Here, batching along with state management is handled with in the library. \n\n```scala\n\n    val generator = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 100).option {\n          val ss = s + 2\n          (ss, ss)\n        }\n      }\n    }\n\n\n    Generator.runBatch[IO, Int, Int](10, generator.withZero(0))(list =\u003e IO { println(list) }).unsafeRunSync()\n    \n    // output\n    List(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)\n    List(22, 24, 26, 28, 30, 32, 34, 36, 38, 40)\n    ...\n    List(82, 84, 86, 88, 90, 92, 94, 96, 98, 100)\n    \n   \n```\nPS: Note that when we using `runBatch` instead of `run` the processing function is `List[A] =\u003e F[Unit]` instead of `A =\u003e F[Unit]`. Ex: You will be using `sendEvents` function of eventhub instead of `sendEvent` in Azure SDK.\n\n\n### Batching with multiple generators\nSpecify multiple `rules of generation`, a `batch size`, a `processing function` that operates on a batch, and then call run. Here, batching along with state management is handled with in the library. \n\n```scala\n    val generator1 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 100).option {\n          val ss = s + 1\n          (ss, ss)\n        }\n      }\n    }.withZero(0)\n\n    val generator2 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 10000).option {\n          val ss = s + 100\n          (ss, ss)\n        }\n      }\n    }.withZero(2000)\n\n    val generator3 = GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 200000).option {\n          val ss = s + 10000\n          (ss, ss)\n        }\n      }\n    }.withZero(100000)\n\n    Generator.runBatch[IO, Int, Int](10, generator1, generator2, generator3)(list =\u003e IO { println(list) }).unsafeRunSync()\n    \n    \n    // output\n    List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)\n    List(110000, 120000, 130000, 140000, 150000, 160000, 170000, 180000, 190000, 200000)\n    List(2100, 2200, 2300, 2400, 2500, 2600, 2700, 2800, 2900, 3000)\n    ...\n    List(4100, 4200, 4300, 4400, 4500, 4600, 4700, 4800, 4900, 5000)\n    ...\n    List(71, 72, 73, 74, 75, 76, 77, 78, 79, 80)\n    List(9100, 9200, 9300, 9400, 9500, 9600, 9700, 9800, 9900, 10000)\n    ...\n    List(91, 92, 93, 94, 95, 96, 97, 98, 99, 100)\n    ...\n\n```\nNote that, the batch doesn't mix in values from different generators (unlike the method of using grouped in scala stream or fs2.groupAdjacent). The batch makes sure that it batches values in the same context. \n### Composability\nExample: generate x's account transactions along with y's  account transactions, such that x's account balance is always\nhigher than that of y's.\n\nRefer to `GeneratorComposition` in examples folder to get insights on compositionality, and how it can be used in your usecase.\n\n\n```scala\n\n    val complexGen =\n      for {\n        t \u003c- generator1\n        y \u003c- generator2.map(_ + t)\n      } yield y\n      \n    Generator.run(generator1, generator2, complexGen)..\n```\n\n### Concurrency \u0026 Time delays\nEx: The number of account transactions per day for person x is less than person y's, such that y shouldn't wait during the delays in x.\n\n```scala\n\n    val generator1: Generator[Int, Int] =  GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 1000).option {\n          val ss = s + 100\n          (ss, ss)\n        }\n      }\n    }.withZero(100)\n\n\n    // Incorporating time delay in generator2\n    val generator2: Generator[Int, Int] =  GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 10).option {\n          val ss = s + 1\n          (ss, ss)\n        }\n      }\n    }.withZero(1).withDelay(1000)\n\n    val generator3: Generator[Int, Int] =  GeneratorLogic.create {\n      s =\u003e {\n        (s \u003c 4000).option {\n          val ss = s + 10\n          (ss, ss)\n        }\n      }\n    }.withZero(2000)\n\n    // Thread.sleep(100) for generator2 results in the some of the results of generator2 to be the last ones to stdout\n    Generator.run[IO, Int, Int](generator1, generator2, generator3)(a =\u003e IO { println(a) }).unsafeRunSync()\n\n// Output\n// The output of generator2 is slow, however you can see that it isn't blocking other generators+process in execution.\n\nscala-execution-context-global-12 2010\nscala-execution-context-global-13 2020\n.....\nscala-execution-context-global-16 1000  \n....\nscala-execution-context-global-17 2\n...\nscala-execution-context-global-17 3290\n...  \n...\nscala-execution-context-global-11 4000\nscala-execution-context-global-16 5\n...\nscala-execution-context-global-18 9\nscala-execution-context-global-16 10\n\n```\n\nThe time delays and concurrency works fine when you use `runBatch`. In batch runs with `delay = t`, there will be a delay\nof `t seconds` between every batch generation/processing.\n\n### Back pressure\nThe processing of data involves sending it to external system such as eventhubs, kafka or other streaming services. \nHence with a fire and forget mechanism, we might be sending either too much or too less data depending on the load of target system, leading to intermittent errors and data loss. \n\nInternally, the backpressure is handled using async queue such that data generation depends on how fast the data is being processed. Again, we reuse fs2's capability as much as we can and refrain from low level concurrency primitives. \n\n## More\nTake a look at the source code which is a few lines of code, to find more functionalities that you may find useful.\n\n## Potential questions\n[Appendix](docs/appendix.md)\n\n## TODO\nMore tests and laws !\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafsalthaj%2Ffp-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fafsalthaj%2Ffp-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafsalthaj%2Ffp-generator/lists"}