{"id":24735575,"url":"https://github.com/atanzu/conveyorpp","last_synced_at":"2025-03-22T17:15:59.124Z","repository":{"id":268388595,"uuid":"254813518","full_name":"atanzu/conveyorpp","owner":"atanzu","description":"A simple С++ library for building conveyor-like pipelines","archived":false,"fork":false,"pushed_at":"2020-04-13T07:05:20.000Z","size":40,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-27T20:47:55.104Z","etag":null,"topics":["parallel-algorithm","parallel-processing","sequencing","thread-library"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atanzu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-11T07:13:26.000Z","updated_at":"2024-06-01T02:37:57.000Z","dependencies_parsed_at":"2024-12-16T13:28:35.132Z","dependency_job_id":"a3b28c65-4a13-4d12-bef5-ff6230cc9ff0","html_url":"https://github.com/atanzu/conveyorpp","commit_stats":null,"previous_names":["atanzu/conveyorpp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atanzu%2Fconveyorpp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atanzu%2Fconveyorpp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atanzu%2Fconveyorpp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atanzu%2Fconveyorpp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atanzu","download_url":"https://codeload.github.com/atanzu/conveyorpp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244991177,"owners_count":20543627,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel-algorithm","parallel-processing","sequencing","thread-library"],"created_at":"2025-01-27T20:45:20.986Z","updated_at":"2025-03-22T17:15:58.887Z","avatar_url":"https://github.com/atanzu.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# conveyorpp\nA simple С++ library for building conveyor-like pipelines\n\n[![Build Status](https://travis-ci.org/atanzu/conveyorpp.svg?branch=master)](https://travis-ci.org/atanzu/conveyorpp)\n\n## Description\nThis library allows users to easily create linear parallel pipelines.\nFor instance. imagine you have three functions `f1`, `f2` and `f3`, and your result is calculated like\n```\noutput = f3(f2(f1(input)));\n```\nSo, if you continuously feed this pipeline with data, your timing looks like this:\n\n```\nf1: |XXXXXX|..............|XXXXXX|..............|XXXXXX|..............\nf2: .......|XXXXXXXXXX|..........|XXXXXXXXXX|..........|XXXXXXXXXX|...\nf3: ..................|XX|..................|XX|..................|XX|\n```\n\nAn easy way o speed up the process is to execute the function chain in parallel, i.e. create several threads for each `f3(f2(f1(...)))` call. However, sometimes it is not possible, e.g. your functions depend on some hardware calculation modules with exclusive access or aren't reentrant.\n\nConveyourpp lib solves this problems by simplifying building of a parallel worker chain. For instance, the example above with conveyorpp (with zero-queue settings) looks like this:\n\n```\nf1: |XXXXXX||XXXXXX|...|XXXXXX|....|XXXXXX|....|XXXXXX|....\nf2: .......|XXXXXXXXXX||XXXXXXXXXX||XXXXXXXXXX||XXXXXXXXXX|\nf3: ..................|XX|........|XX|........|XX|.........\n```\n\nSo if your calculation can be divided in several stages, you can utilize all the computational power you've got.\n\n## Installation\nTo install conveyourpp, clone this repository and perform the following actions:\n```\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake\n```\nOptionally you can install it via\n```\nsudo make install\n```\n\n## Usage\n### Usual functions\nImagine you have three functions:\n```\nint f1(float);\ndouble f2(int);\nbool f3(double);\n```\nYou can parallelize their execution by:\n```\nauto chainer = cnvpp::MakeChainer(f1, f2, f3);\n```\nThis line of code creates execution chain f1-\u003ef2-\u003ef3, which is equivalent to `f3(f2(f1(...)))`\nTo feed this conveyor with input data, simply call:\n```\nint x = 42;\nchainer.Call(x);\n```\nTo obtain result of the computation, call\n```\nauto result = chainer.GetResult();\n```\nMethod `GetResult` blocks until result is available. If the last function's return type is `void`,method `GetResult` is excluded from compilation.\n\n### Lambdas\nLambda-functions need to be processed with some special care. In particular, user has to manually describe their input and output types, e.g.\n```\ndouble multiplier = 2.5;\nauto l1 = [=] (int x) { return (float)x * multiplier; };\nauto l2 = [\u0026] (float y) { return (size_t)(y / multiplier + 1); };\n\nauto chainer = cnvpp::MakeChainer(\n  cnvpp::LambdaWrapper\u003cfloat, int\u003e(l1),\n  cnvpp::LambdaWrapper\u003csize_t, float\u003e(l2),\n};\n```\n\n## Roadmap\n* Add more test and examples\n* Add more documentation and diagrams\n* Find a way to get rid of LambdaWrapper class and handle functions, class methods and lambdas in a unified way\n* Make internal queuing mechanism more customizable (at the current moment each conveyor stage has internal queue with size 5)\n* Add mechanisms for auto-balancing the load (e.g. user adds a sequence of actions to perform, and conveyorpp's runtime determines optimal amount of threads to use, not just execute each stage in a separate thread)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatanzu%2Fconveyorpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatanzu%2Fconveyorpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatanzu%2Fconveyorpp/lists"}