{"id":17984118,"url":"https://github.com/shadowmoose/pygroupedpool","last_synced_at":"2025-04-04T02:14:28.478Z","repository":{"id":95786486,"uuid":"206470663","full_name":"shadowmoose/PyGroupedPool","owner":"shadowmoose","description":"Python multiprocessing Pool with reserved, group-based concurrency.","archived":false,"fork":false,"pushed_at":"2019-11-06T01:17:55.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T13:41:52.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shadowmoose.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-05T03:56:41.000Z","updated_at":"2019-11-06T01:17:57.000Z","dependencies_parsed_at":"2023-04-23T20:27:19.999Z","dependency_job_id":null,"html_url":"https://github.com/shadowmoose/PyGroupedPool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadowmoose%2FPyGroupedPool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadowmoose%2FPyGroupedPool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadowmoose%2FPyGroupedPool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shadowmoose%2FPyGroupedPool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shadowmoose","download_url":"https://codeload.github.com/shadowmoose/PyGroupedPool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247107827,"owners_count":20884797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T18:19:32.985Z","updated_at":"2025-04-04T02:14:28.461Z","avatar_url":"https://github.com/shadowmoose.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Grouped Pool \n[![badge](https://github.com/shadowmoose/PyGroupedPool/workflows/Pytest/badge.svg)](https://github.com/shadowmoose/PyGroupedPool/actions)\n[![codecov](https://codecov.io/gh/shadowmoose/PyGroupedPool/branch/master/graph/badge.svg)](https://codecov.io/gh/shadowmoose/PyGroupedPool)\n![lines of code](https://my-api.link/gh/lines?repo=PyGroupedPool\u0026exts=py)\n\n*This is currently a proof-of-concept. No release currently exists.*\n\nThis project is a wrapper around the Python \"Pool\" implementation for multiprocessing.\n\nIt extends the basic functionality to add a few notable changes:\n\n+ The Pool object supports group-based tags, each of which guarantees their own amount of dedicated process 'slots'.\n+ A generic tag ('None') can also be provided, to allow all tagged groups to burst above their base limits as-needed.\n+ Group sizes can be adjusted live while running, to allow your program to reallocate subprocesses priority in realtime.\n+ The logic to launch infinite sub-processes has been streamlined to prevent excessive memory usage.\n+ The Pool supports data \u0026 error callbacks, or a simple generator to iterate the results as they're returned.\n\n---\n\n## Example:\n\nThis demo creates a Pool object with three groups. Each group can be named whatever you want. \nIn this example, we emulate a potential webserver + background-worker setup.\n\n+ All processes submitted to group `webserver` are guaranteed at least one dedicated slot.\n+ The `processors` group, in this example, is guaranteed 5 slots.\n+ The `None` group is the \"general purpose\" group, and all tagged groups may use these if they are out of dedicated slots.\n```python\n# Create an example Pool, using callbacks to print returned data \u0026 errors.\npool = PyPool(tags={\n    'webserver': 1,\n    'processors': 5,\n    None: 10\n}, callback=lambda r: print('Returned:', r, ', Pending:', pool.pending), on_error=print)\n```\n\nOnce the Pool has been created, it's easy to submit new tasks. The simplest way is to use the (asynchronous) `ingest` method.\n\nThis method accepts any iterable collection of values, and submits them all to the given group.\nTo avoid excessive memory consumption, it will wait to create new subprocesses until there is an open slot for them.\n\n```python\npool.ingest([1, 2, 4, 5, 'etc...'], 'tag_group', function_to_run, ['extra_func_arguments'])\n```\n\nIf you don't need to bulk-add, or you need more control over the functionality, you can also add sub-processes individually in a block fashion:\n\n```python\n# You can also pass custom result/error callbacks to override the defaults:\nresult = pool.put('tag_name', function_name, ('arg1', 'arg2', 'etc...'), callback=print, error=print)  \n# A result object is always returned, just in case you want to handle it manually.\n```\n\n\n## Resizing Pool Group Capacity\nIf needed, each group can be resized at-will, to allow for dynamic shifting of priorities.\nResizing will play nicely with running threads, only removing and adding capacity as it becomes available.\n\nBy default, resizing creates or destroys new subprocess slots. \nIf you specify the 'use_general_slots' flag, the change in slots will instead be added or removed from the general purpose ('None tag') group.\n```python\npool = PyPool(tags={\n    'test': 1,\n    'ignored_pool': 5\n}, on_error=print)\n\n\npool.adjust('test', 10)  # The group 'test' now supports up to 10 concurrent processes. 9 slots have been created.\n\n# You may also move slots to/from the general pool.\n# After this call, 'ignored_pool' group will only have 1 process slot available -\n#   but the generic group (None tag) will gain the removed 4 process slots:\npool.adjust('ignored_pool', 1, True)\n\npool.adjust('test', 14, True)  # The 'test' group now has 14 reserved slots for processes, and the general pool has 0.\n```\n\n## Iterator Example:\nIf you'd prefer to use an iterator instead of an async callback, simply create a Pool without a data callback:\n```python\npool = PyPool(tags={\n    'test': 1\n}, iteration=True)\n\npool.ingest([5, 5, 5, 5, 5], 'test', time.sleep, [])  # Asynchronously run a subprocess for each value, using util function.\n\nfor v in pool:\n    print(v)  # Prints all data as it becomes available. Exits once no further data is incoming.\n```\n\n## More documentation:\nThere's more functionality, such as `join` or `stop`, and the easiest way to learn about them is to [read the docs](./pool.py) for each method.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshadowmoose%2Fpygroupedpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshadowmoose%2Fpygroupedpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshadowmoose%2Fpygroupedpool/lists"}