{"id":17151229,"url":"https://github.com/soda480/mppbar","last_synced_at":"2026-02-27T12:06:11.112Z","repository":{"id":37794714,"uuid":"503517621","full_name":"soda480/mppbar","owner":"soda480","description":"A multi-processing enabled progress bar","archived":false,"fork":false,"pushed_at":"2024-09-20T19:17:53.000Z","size":3702,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-05T17:24:09.948Z","etag":null,"topics":["dockerfile","message-queue","multiprocessing","pool","progressbar","pybuilder","python","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soda480.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-14T20:54:35.000Z","updated_at":"2024-09-20T19:17:56.000Z","dependencies_parsed_at":"2022-06-23T17:58:07.862Z","dependency_job_id":null,"html_url":"https://github.com/soda480/mppbar","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmppbar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmppbar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmppbar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fmppbar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soda480","download_url":"https://codeload.github.com/soda480/mppbar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240072098,"owners_count":19743527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dockerfile","message-queue","multiprocessing","pool","progressbar","pybuilder","python","python3"],"created_at":"2024-10-14T21:37:33.348Z","updated_at":"2026-02-27T12:06:06.086Z","avatar_url":"https://github.com/soda480.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mppbar\n[![build](https://github.com/soda480/mppbar/actions/workflows/main.yml/badge.svg)](https://github.com/soda480/mppbar/actions/workflows/main.yml)\n[![complexity](https://img.shields.io/badge/complexity-A-brightgreen)](https://radon.readthedocs.io/en/latest/api.html#module-radon.complexity)\n[![vulnerabilities](https://img.shields.io/badge/vulnerabilities-None-brightgreen)](https://pypi.org/project/bandit/)\n[![PyPI version](https://badge.fury.io/py/mppbar.svg)](https://badge.fury.io/py/mppbar)\n[![python](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-teal)](https://www.python.org/downloads/)\n\nThe mppbar module provides a convenient way to scale execution of a function across multiple input values by distributing the input across a number of background processes, it displays the execution status of each background process using a [progress bar](https://pypi.org/project/progress1bar/).\n\nThe MPpbar class is a subclass of [MPmq](https://pypi.org/project/mpmq/) and the primary benefit of using `mppbar` is that the function being scaled requires minimal modification (if at all) since the multiprocessing and progress bar are completely abstracted. The progress bar is initialized and updated through the interception and processing of the messages logged within the function. The log messages in the function must use f-string style formatting.\n\n### Installation\n```bash\npip install mppbar\n```\n\n### `MPpbar class`\n```\nMPpbar(function, process_data=None, shared_data=None, processes_to_start=None, regex=None, **kwargs)\n```\n\n\u003cdetails\u003e\u003csummary\u003eDocumentation\u003c/summary\u003e\n\n\u003e `function` - The function to execute. It should accept two positional arguments; the first argument is the dictionary containing the input data for the respective process see `process_data` below, the second argument is the shared dictionary sent to all proceses see `shared_data` below.\n\n\u003e `process_data` - A list of dictionaries where each dictionary describes the input data that will be sent to the respective process executing the function; the length of the list dictates the total number of processes that will be executed.\n\n\u003e `shared_data` - A dictionary containing arbitrary data that will be sent to all processes.\n\n\u003e `processes_to_start` - The number of processes to initially start; this represents the number of concurrent processes that will be running. If the total number of processes is greater than this number then the remaining processes will be queued and executed to ensure this concurrency is maintained. Defaults to the length of the `process_data` lsit.\n\n\u003e `regex` - A dictionary whose key values are regular expressions for `total`, `count` and `alias`. The regular expressions will be checked against the log messages intercepted from the executing function, if matched the value will be used to assign the attribute for the respective progress bar. The `total` and `count` key values are required, the `alias` key value is optional. The default `regex` is: \n```\n{\n    'total': r'^processing total of (?P\u003cvalue\u003e\\d+) items$',\n    'count': r'^processed .*$',\n    'alias': r'^worker is (?P\u003cvalue\u003e.*)$'\n}\n```\n\n\u003e `**kwargs` - Pass through any other supported ProgressBar keyword argument, refer to [progress bar](https://pypi.org/project/progress1bar/) for supported arguments.\n\n\u003e **execute(raise_if_error=False)**\n\u003e\u003e Start execution the process’s activity. If `raise_if_error` is set to True, an exception will be raised if the function encounters an error during execution.\n\n\u003c/details\u003e\n\n### Examples\n\n#### [example1](https://github.com/soda480/mppbar/blob/main/examples/example1.py)\n\nDistribute work across multiple processes with all executing concurrently, each displays a progress bar showing its execution status. Uses default regex for assigning progress bar attributes.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport time, random, logging\nimport names\nfrom mppbar import MPpbar\nlogger = logging.getLogger(__name__)\n\ndef do_work(data, *args):\n    # log our intentions - messages will be intercepted as designated by MPpbar regex\n    logger.debug(f'worker is {names.get_last_name()}')\n    total = data['total']\n    logger.debug(f'processing total of {total} items')\n    for index in range(total):\n        # simulate work by sleeping\n        time.sleep(random.choice([.1, .2, .4]))\n        logger.debug(f'processed item {index}')\n    return total\n\ndef main():\n    # designate 6 processes total - each getting a different total\n    process_data = [{'total': random.randint(8, 16)} for item in range(6)]\n    print(f'\u003e\u003e Processing items using {len(process_data)} workers ...')\n    pbars =  MPpbar(function=do_work, process_data=process_data, timeout=1)\n    results = pbars.execute()\n    # add up totals from all processes\n    print(f\"\u003e\u003e {len(process_data)} workers processed a total of {sum(result for result in results)} items\")\n\nif __name__ == '__main__':\n    main()\n```\n\n\u003c/details\u003e\n\n![example1](https://raw.githubusercontent.com/soda480/mppbar/main/docs/images/example1.gif)\n\n#### [example2](https://github.com/soda480/mppbar/blob/main/examples/example2.py)\n\nDistribute work across multiple processes but only a subset are executing concurrently, each displays a progress bar showing its execution status. Useful if you can only afford to run a few processes concurrently. Uses custom regex for assigning progress bar attributes.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport time, random, logging\nimport names\nfrom mppbar import MPpbar\nlogger = logging.getLogger(__name__)\n\ndef do_work(data, *args):\n    # log our intentions - messages will be intercepted as designated by MPpbar regex\n    logger.debug(f'processor is {names.get_last_name()}')\n    total = data['total']\n    logger.debug(f'processing total of {total}')\n    for index in range(total):\n        # simulate work by sleeping\n        time.sleep(random.choice([.1, .2, .4]))\n        logger.debug(f'processed item {index}')\n    return total\n\ndef main():\n    # designate 6 processes total - each getting a different total\n    process_data = [{'total': random.randint(8, 16)} for item in range(6)]\n    # supply custom regex to intercept and set values for total count and alias\n    regex = {\n        'total': r'^processing total of (?P\u003cvalue\u003e\\d+)$',\n        'count': r'^processed item \\d+$',\n        'alias': r'^processor is (?P\u003cvalue\u003e.*)$',\n    }\n    # designate fill factor for total - to make progress bar look nicer\n    fill = {\n        'max_total': 100\n    }\n    print(f'\u003e\u003e Processing items using {len(process_data)} workers ...')\n    # set concurrency to 3 - max of 3 processes will be running at any given time\n    pbars =  MPpbar(function=do_work, process_data=process_data, regex=regex, fill=fill, processes_to_start=3, timeout=1, show_fraction=False)\n    results = pbars.execute()\n    # add up totals from all processes\n    print(f\"\u003e\u003e {len(process_data)} workers processed a total of {sum(result for result in results)} items\")\n\nif __name__ == '__main__':\n    main()\n```\n\n\u003c/details\u003e\n\n![example2](https://raw.githubusercontent.com/soda480/mppbar/main/docs/images/example2.gif)\n\n#### [example3](https://github.com/soda480/mppbar/blob/main/examples/example3.py)\n\nDistribute alot of work across a small set of processes using a thread-safe queue, each process gets work off the queue until there is no more work, all processes reuse a progress bar to show its execution status. Useful if you have alot of data to distribute across a small set of processes.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport time, random, logging\nfrom multiprocessing import Queue\nfrom queue import Empty\nimport names\nfrom mppbar import MPpbar\nlogger = logging.getLogger(__name__)\n\ndef do_work(total):\n    # log our intentions - messages will be intercepted as designated by MPpbar regex\n    logger.debug(f'worker is {names.get_last_name()}')\n    logger.debug(f'processing total of {total} items')\n    for index in range(total):\n        # simulate work by sleeping\n        time.sleep(random.choice([.001, .003, .005]))\n        logger.debug(f'processed item {index}')\n    return total\n\ndef prepare_queue():\n    # create queue to add all the work that needs to be done\n    queue = Queue()\n    for _ in range(75):\n        queue.put({'total': random.randint(100, 150)})\n    return queue\n\ndef run_q(data, *args):\n    queue = data['queue']\n    result = 0\n    while True:\n        try:\n            # get work from queue\n            total = queue.get(timeout=1)['total']\n            # process the work\n            result += do_work(total)\n            # this allows us to reset progress bar\n            logger.debug('reset-mppbar')\n        except Empty:\n            logger.debug('reset-mppbar-complete')\n            break\n    return result\n\ndef main():\n    queue = prepare_queue()\n    # designate 3 processes total - each getting reference to the queue\n    process_data = [{'queue': queue} for item in range(3)]\n    print(f'\u003e\u003e Processing {queue.qsize()} totals using {len(process_data)} workers ...')\n    pbars =  MPpbar(function=run_q, process_data=process_data, timeout=1, show_prefix=False, show_percentage=False)\n    results = pbars.execute()\n    # add up results from all workers\n    print(f\"\u003e\u003e {len(process_data)} workers processed a total of {sum(result for result in results)} items\")\n\nif __name__ == '__main__':\n    main()\n```\n\n\u003c/details\u003e\n\n![example3](https://raw.githubusercontent.com/soda480/mppbar/main/docs/images/example3.gif)\n\n\n### Development\n\nClone the repository and ensure the latest version of Docker is installed on your development server.\n\nBuild the Docker image:\n```sh\ndocker image build \\\n-t mppbar:latest .\n```\n\nRun the Docker container:\n```sh\ndocker container run \\\n--rm \\\n-it \\\n-v $PWD:/code \\\nmppbar:latest \\\nbash\n```\n\nExecute the build:\n```sh\npyb -X\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fmppbar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoda480%2Fmppbar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fmppbar/lists"}