{"id":20057649,"url":"https://github.com/todesking/scala-async_task_pipeline","last_synced_at":"2025-07-01T05:38:15.090Z","repository":{"id":17638128,"uuid":"20442501","full_name":"todesking/scala-async_task_pipeline","owner":"todesking","description":null,"archived":false,"fork":false,"pushed_at":"2016-01-08T10:12:13.000Z","size":33,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-12T21:46:17.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/todesking.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-03T12:54:08.000Z","updated_at":"2015-08-04T13:53:33.000Z","dependencies_parsed_at":"2022-09-23T23:10:51.827Z","dependency_job_id":null,"html_url":"https://github.com/todesking/scala-async_task_pipeline","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/todesking%2Fscala-async_task_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/todesking%2Fscala-async_task_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/todesking%2Fscala-async_task_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/todesking%2Fscala-async_task_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/todesking","download_url":"https://codeload.github.com/todesking/scala-async_task_pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241488152,"owners_count":19970826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T12:59:47.820Z","updated_at":"2025-03-02T09:41:40.496Z","avatar_url":"https://github.com/todesking.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AsyncTaskPipeline\n\n## Description\n\nAsyncTaskPipeline is framework for building in-process parallel task processing pipeline.\n\n```scala\nresolvers += \"com.todesking\" at \"http://todesking.github.io/mvn/\"\n\nlibraryDependencies += \"com.todesking\" %% \"async_task_pipeline\" % \"0.0.9\"\n```\n\n## Usage\n\n1. Build pipes/sink with `Dataflow.build*`\n2. Join pipes.\n3. `Dataflow.runPipe/runSink` and get Execution object.\n4. `feed()` values to execution context.\n\n```scala\nimport com.todesking.{async_task_pipeline =\u003e ap}\n\nimport ap.Dataflow.{buildPipe, buildSinkToGrowable}\nimport ap.Parallelism\n\nval par = Parallelism.Const(100) // concurrency = 100\n\nval output = new scala.collection.mutable.ArrayBuffer[Int]\n\nval context = ap.Dataflow.runSink(\n  buildPipe(par) {i: Int =\u003e if(i % 2 == 0) Seq(i) else Seq.empty }\n  \u003e\u003e\u003e\n  buildPipe(par) {i: Int =\u003e Seq(i * 2)}\n  \u003e\u003e~\n  buildSinkToGrowable(output)\n)\n\ncontext.feed(1)\ncontext.feed(2)\ncontext.feed(3)\ncontext.feed(4)\n\n// Wait until all process completed.\ncontext.await()\n\noutput.sorted == Seq(4, 8)\n```\n\n## Architecture\n```scala\nobject Dataflow {\n  def buildPipe[A, B](par: Parallelism)(f: A =\u003e Seq[B]): Pipe[A, B]\n  def buildSink[A, B](par: Parallelism)(f: A =\u003e Unit): Sink[A]\n  def buildSinkToGrowable[A](g: Growable): Sink[A]\n\n  def runPipe[A, B](pipe: Pipe[A, B]): PipeExecution[A, B]\n  def runSink[A](sink: Sink[A]): Sink[A]\n}\n\ntrait Dataflow\ntrait Sink[-A] extends Dataflow\ntrait Pipe[-A, +B] extends Dataflow with Sink[A] {\n  def \u003e\u003e\u003e[C](rhs: Pipe[B, C]): Pipe[A, C]\n  def \u003e\u003e~(rhs: Sink[B]): Sink[A]\n  def \u003c=\u003e(rhs: Pipe[A, B]): Pipe[A, B]\n}\n\ntrait DataflowExecution {\n  def await(): Unit\n}\ntrait SinkExecution[-A] extends DataflowExecution {\n  def feed(value: A): Unit\n}\ntrait PipeExecution[-A, +B] extends DataflowExecution with SinkExecution[A] {\n  def feedPipe(value: A)(callback: Seq[B] =\u003e Unit): Unit\n  def feedPipe1(value: A)(callback: B =\u003e Unit): Unit\n}\n```\n\n## TODO\n\n* Error handling\n* Througput in statusString\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftodesking%2Fscala-async_task_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftodesking%2Fscala-async_task_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftodesking%2Fscala-async_task_pipeline/lists"}