{"id":17739019,"url":"https://github.com/pierrekieffer/multiprocessing-template","last_synced_at":"2025-03-31T19:44:52.163Z","repository":{"id":120128924,"uuid":"313341210","full_name":"PierreKieffer/multiprocessing-template","owner":"PierreKieffer","description":" Provides parallel and async execution of multiple methods, and processing of results in real time.","archived":false,"fork":false,"pushed_at":"2021-03-18T11:40:23.000Z","size":76,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-03-07T01:31:41.786Z","etag":null,"topics":["async","asynchronous","asynchronous-programming","multiprocessing","parallel-programming","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PierreKieffer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-16T15:13:15.000Z","updated_at":"2022-07-13T21:15:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"12db3871-f7fe-4cde-863f-852806a1937c","html_url":"https://github.com/PierreKieffer/multiprocessing-template","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PierreKieffer%2Fmultiprocessing-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PierreKieffer%2Fmultiprocessing-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PierreKieffer%2Fmultiprocessing-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PierreKieffer%2Fmultiprocessing-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PierreKieffer","download_url":"https://codeload.github.com/PierreKieffer/multiprocessing-template/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246531986,"owners_count":20792735,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asynchronous","asynchronous-programming","multiprocessing","parallel-programming","stream-processing"],"created_at":"2024-10-26T02:06:58.631Z","updated_at":"2025-03-31T19:44:52.142Z","avatar_url":"https://github.com/PierreKieffer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# multiprocessing-template \n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"logo.png\"\u003e\n\u003c/p\u003e\n\n\nProvides parallel and async execution of multiple methods, and processing of results in real time.\n\n`AsyncWorker` provides two types of runners : \nBoth runners init and run a pool of async tasks (same / different ones) in parallel. \n\n- stream runner : \nProvides a way to consume results in real time, asynchronously. \n\n- standard runner : \nReturns the result when all task are finished.\n\n\n* [Install](#install)\n* [Usage](#usage)\n\t* [Required](#required)\n\t* [AsyncWorker](#asyncworker)\n\t\t* [Stream runner](#stream-runner)\n\t\t* [Standard runner](#standard-runner)\n\n\n## Install \n```bash\npip install .\n```\n\n## Usage \n\n### Required \nThe executed methods must respect the following format:\n\n```python\ndef method(**kwargs): \n\t'''\n\tdo something ... \n\t'''\n```\n### AsyncWorker\n\n#### Stream runner \n\nInits and runs a pool of async tasks (same / different ones) in parallel. \nProvides a way to consume results in real time, asynchronously. \n\n`stream_run()` takes a method as param, to consume results in real time. \n\nAs soon as the result of a task is available, the method of consumption of the result, passed in parameter of `stream_run`, is executed and the result is also sent to the `output_buffer`.\n\n- Import \n```python\nfrom multiprocessing_template.template import AsyncWorker\n```\n\n- Methods example \n```python \ndef f_1(arg1 = \"default_value\"): \n    time.sleep(5)\n    return arg1\n\ndef f_2(arg1 = \"default_value\", arg2 = \"default_value\"):\n    time.sleep(1)\n    return arg1 + arg2\n```\n\n- Custom results consumer\n```python\ndef consumer(data): \n    print(\"consumed data : {}\".format(data))\n\n    # Example : Write to file concurrently\n    with open(\"data.txt\", \"a\") as f : \n        f.write(\"\\n\")\n        f.write(data)```\n\n```python \nif __name__==\"__main__\":\n\n    '''\n    Init\n    '''\n    async_worker = AsyncWorker(2) # number of workers as param, (max = os.cpu_count())\n\n    '''\n    Add methods and params to worker\n    '''\n    async_worker.add(f_1, {\"arg1\" : \"foo\"})\n    async_worker.add(f_2, {\"arg1\" : \"foo\", \"arg2\" : \"bar\"})\n\n    async_worker.stream_run(consumer)\n\n    print(async_worker.output_buffer)\n```\n\n- Output \n```bash \nconsumed data : foobar\nconsumed data : foo\n['foobar', 'foo']\n```\n\n#### Standard runner\nInits and runs a pool of async tasks in parallel.\nReturns the result when all task are finished.\n\n- Import \n```python\nfrom multiprocessing_template.template import AsyncWorker\n```\n- Methods example \n```python\ndef f_1(arg1 = \"default_value\"): \n    print(arg1)\n\ndef f_2(arg1 = \"default_value\", arg2 = \"default_value\"): \n    print(arg1)\n    print(arg2)\n\ndef f_3(**kwargs): \n    print(kwargs.get(\"arg\"))\n```\n\n```python\n`if __name__==\"__main__\":\n\n    '''\n    Init worker\n    '''\n    async_worker = AsyncWorker(4) # number of workers as param, (max = os.cpu_count())\n\n    '''\n    Add methods and params to worker\n    '''\n    async_worker.add(f_1, {\"arg1\" : \"foo\"})\n    async_worker.add(f_1, {\"arg1\" : \"bar\"})\n    async_worker.add(f_2, {\"arg1\" : \"foo\", \"arg2\" : \"bar\"})\n    async_worker.add(f_3, {\"arg\" : \"foobar\"})\n\n    output = async_worker.run()\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpierrekieffer%2Fmultiprocessing-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpierrekieffer%2Fmultiprocessing-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpierrekieffer%2Fmultiprocessing-template/lists"}