{"id":17151239,"url":"https://github.com/soda480/pypbars","last_synced_at":"2025-08-18T17:09:22.006Z","repository":{"id":40591777,"uuid":"507673990","full_name":"soda480/pypbars","owner":"soda480","description":"Provides a convenient way to display progress bars for concurrent asyncio or multiprocessing Pool processes to the terminal.","archived":false,"fork":false,"pushed_at":"2024-09-20T20:03:01.000Z","size":2342,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-10T16:31:15.684Z","etag":null,"topics":["asynchronous","asyncio","docker","multiprocessing","progress-bar","pybuilder-example","python","terminal"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soda480.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-06-26T20:08:57.000Z","updated_at":"2024-09-20T20:03:04.000Z","dependencies_parsed_at":"2024-10-14T21:47:39.804Z","dependency_job_id":null,"html_url":"https://github.com/soda480/pypbars","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/soda480/pypbars","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fpypbars","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fpypbars/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fpypbars/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fpypbars/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soda480","download_url":"https://codeload.github.com/soda480/pypbars/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soda480%2Fpypbars/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271027687,"owners_count":24687082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","asyncio","docker","multiprocessing","progress-bar","pybuilder-example","python","terminal"],"created_at":"2024-10-14T21:37:34.808Z","updated_at":"2025-08-18T17:09:21.969Z","avatar_url":"https://github.com/soda480.png","language":"Python","readme":"# pypbars\n[![build](https://github.com/soda480/pypbars/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/soda480/pypbars/actions/workflows/main.yml)\n[![coverage](https://img.shields.io/badge/coverage-95%25-brightgreen)](https://pybuilder.io/)\n[![vulnerabilities](https://img.shields.io/badge/vulnerabilities-None-brightgreen)](https://pypi.org/project/bandit/)\n[![PyPI version](https://badge.fury.io/py/pypbars.svg)](https://badge.fury.io/py/pypbars)\n[![python](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-teal)](https://www.python.org/downloads/)\n\nThe `pypbars` module provides a convenient way to display progress bars for concurrent [asyncio](https://docs.python.org/3/library/asyncio.html) or [multiprocessing Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) processes. The `pypbars` class is a subclass of [list2term](https://pypi.org/project/list2term/) that displays a list to the terminal, and uses [progress1bar](https://pypi.org/project/progress1bar/) to render the progress bar.\n\n### Installation\n```bash\npip install pypbars\n```\n\n#### [example1 - ProgressBars with asyncio](https://github.com/soda480/pypbars/blob/main/examples/example1.py)\n\nCreate `ProgressBars` using a lookup list containing unique values, these identifiers will be used to get the index of the appropriate `ProgressBar` to be updated. The convention is for the function to include `logger.write` calls containing the identifier and a message for when and how the respective progress bar should be updated. In this example the default `regex` dict is used but the caller can specify their own, so long as it contains regular expressions for how to detect when `total`, `count` and optional `alias` are set.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport asyncio\nimport random\nfrom faker import Faker\nfrom pypbars import ProgressBars\n\nasync def do_work(worker, logger=None):\n    logger.write(f'{worker}-\u003eworker is {worker}')\n    total = random.randint(10, 65)\n    logger.write(f'{worker}-\u003eprocessing total of {total} items')\n    for count in range(total):\n        # mimic an IO-bound process\n        await asyncio.sleep(.1)\n        logger.write(f'{worker}-\u003eprocessed {count}')\n    return total\n\nasync def run(workers):\n    with ProgressBars(lookup=workers, show_prefix=False, show_fraction=False) as logger:\n        doers = (do_work(worker, logger=logger) for worker in workers)\n        return await asyncio.gather(*doers)\n\ndef main():\n    workers = [Faker().user_name() for _ in range(10)]\n    print(f'Total of {len(workers)} workers working concurrently')\n    results = asyncio.run(run(workers))\n    print(f'The {len(workers)} workers processed a total of {sum(results)} items')\n\nif __name__ == '__main__':\n    main()\n```\n\n\u003c/details\u003e\n\n![example1](https://raw.githubusercontent.com/soda480/pypbars/main/docs/images/example1.gif)\n\n#### [example2 - ProgressBars with multiprocessing Pool](https://github.com/soda480/pypbars/blob/main/examples/example2.py)\n\nThis example demonstrates how `pypbars` can be used to display progress bars from processes executing in a [multiprocessing Pool](https://docs.python.org/3/library/multiprocessing.html#using-a-pool-of-workers). The `list2term.multiprocessing` module contains a `pool_map` method that fully abstracts the required multiprocessing constructs, you simply pass it the function to execute, an iterable containing the arguments to pass each process, and an instance of `ProgressBars`. The method will execute the functions asynchronously, update the progress bars accordingly and return a multiprocessing.pool.AsyncResult object. Each progress bar in the terminal represents a background worker process.\n\nIf you do not wish to use the abstraction, the `list2term.multiprocessing` module contains helper classes that facilitate communication between the worker processes and the main process; the `QueueManager` provide a way to create a `LinesQueue` queue which can be shared between different processes. Refer to [example2b](https://github.com/soda480/pypbars/blob/main/examples/example2b.py) for how the helper methods can be used. \n\n**Note** the function being executed must accept a `LinesQueue` object that is used to write messages via its `write` method, this is the mechanism for how messages are sent from the worker processes to the main process, it is the main process that is displaying the messages to the terminal. The messages must be written using the format `{identifier}-\u003e{message}`, where {identifier} is a string that uniquely identifies a process, defined via the lookup argument to `ProgressBars`.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport time\nfrom pypbars import ProgressBars\nfrom list2term.multiprocessing import pool_map\nfrom list2term.multiprocessing import CONCURRENCY\n\ndef is_prime(num):\n    if num == 1:\n        return False\n    for i in range(2, num):\n        if (num % i) == 0:\n            return False\n    else:\n        return True\n\ndef count_primes(start, stop, logger):\n    workerid = f'{start}:{stop}'\n    logger.write(f'{workerid}-\u003eworker is {workerid}')\n    logger.write(f'{workerid}-\u003eprocessing total of {stop - start} items')\n    primes = 0\n    for number in range(start, stop):\n        if is_prime(number):\n            primes += 1\n        logger.write(f'{workerid}-\u003eprocessed {number}')\n    logger.write(f'{workerid}-\u003e{workerid} processing complete')\n    return primes\n\ndef main(number):\n    step = int(number / CONCURRENCY)\n    iterable = [(index, index + step) for index in range(0, number, step)]\n    lookup = [':'.join(map(str, item)) for item in iterable]\n    progress_bars = ProgressBars(lookup=lookup, show_prefix=False, show_fraction=False, use_color=True)\n    # print to screen with progress bars context\n    results = pool_map(count_primes, iterable, context=progress_bars)\n    # print to screen without progress bars context\n    # results = pool_map(count_primes, iterable)\n    # do not print to screen\n    # results = pool_map(count_primes, iterable, print_status=False)\n    return sum(results.get())\n\nif __name__ == '__main__':\n    start = time.perf_counter()\n    number = 50_000\n    result = main(number)\n    stop = time.perf_counter()\n    print(f\"Finished in {round(stop - start, 2)} seconds\\nTotal number of primes between 0-{number}: {result}\")\n```\n\n\u003c/details\u003e\n\n![example2](https://raw.githubusercontent.com/soda480/pypbars/main/docs/images/example2.gif)\n\n#### [example3 - resettable ProgressBars with multiprocessing Pool and Queue](https://github.com/soda480/pypbars/blob/main/examples/example3.py)\n\nThis example demonstrates how `pypbars` can be used to display progress bars from a small set processes executing in a [multiprocessing Pool](https://docs.python.org/3/library/multiprocessing.html#using-a-pool-of-workers) working on large amount of data defined in a shared work Queue. The workers will pop off the work from work queue and process it until there is no more work left in the work Queue. Since the workers are working on multiple sets the ProgressBar is reset everytime a worker begins work on a new set.  The ProgressBar maintains the number of iterations it has completed.\n\n\u003cdetails\u003e\u003csummary\u003eCode\u003c/summary\u003e\n\n```Python\nimport time, random, logging\nfrom multiprocessing import Queue\nfrom queue import Empty\nimport names\nfrom faker import Faker\nfrom multiprocessing import Pool\nfrom multiprocessing import get_context\nfrom multiprocessing import cpu_count\nfrom list2term.multiprocessing import LinesQueue\nfrom list2term.multiprocessing import QueueManager\nfrom queue import Empty\nfrom pypbars import ProgressBars\nlogger = logging.getLogger(__name__)\n\ndef prepare_queue(queue):\n    for _ in range(55):\n        queue.put({'total': random.randint(100, 150)})\n\ndef do_work(worker_id, total, logger):\n    logger.write(f'{worker_id}-\u003eworker is {names.get_full_name()}')\n    logger.write(f'{worker_id}-\u003eprocessing total of {total} items')\n    for index in range(total):\n        # simulate work by sleeping\n        time.sleep(random.choice([.001, .003, .008]))\n        logger.write(f'{worker_id}-\u003eprocessed {index}')\n    return total\n\ndef run_q(worker_id, queue, logger):\n    result = 0\n    while True:\n        try:\n            total = queue.get(timeout=1)['total']\n            result += do_work(worker_id, total, logger)\n            logger.write(f'{worker_id}-\u003ereset')\n        except Empty:\n            break\n    return result\n\ndef main(processes):\n    QueueManager.register('LinesQueue', LinesQueue)\n    QueueManager.register('Queue', Queue)\n    with QueueManager() as manager:\n        queue = manager.LinesQueue(ctx=get_context())\n        data_queue = manager.Queue()\n        prepare_queue(data_queue)\n        with Pool(processes) as pool:\n            print(f\"\u003e\u003e Adding {data_queue.qsize()} sets into a data queue that {processes} workers will work from until empty\")\n            process_data = [(Faker().name(), data_queue, queue) for index in range(processes)]\n            results = pool.starmap_async(run_q, process_data)\n            lookup = [f'{data[0]}' for data in process_data]\n            with ProgressBars(lookup=lookup, show_prefix=False, show_fraction=False, use_color=True, show_duration=True, clear_alias=True) as lines:\n                while True:\n                    try:\n                        item = queue.get(timeout=.1)\n                        if item.endswith('-\u003ereset'):\n                            index, message = lines.get_index_message(item)\n                            lines[index].reset(clear_alias=False)\n                        else:\n                            lines.write(item)\n                    except Empty:\n                        if results.ready():\n                            for index, _ in enumerate(lines):\n                                lines[index].complete = True\n                            break\n    return sum(results.get())\n\n\nif __name__ == '__main__':\n    processes = 3\n    results = main(processes)\n    print(f\"\u003e\u003e {processes} workers processed a total of {results} items\")\n```\n\n\u003c/details\u003e\n\n![example3](https://raw.githubusercontent.com/soda480/pypbars/main/docs/images/example3.gif)\n\n\n### Development\n\nClone the repository and ensure the latest version of Docker is installed on your development server.\n\nBuild the Docker image:\n```sh\ndocker image build \\\n-t pypbars:latest .\n```\n\nRun the Docker container:\n```sh\ndocker container run \\\n--rm \\\n-it \\\n-v $PWD:/code \\\npypbars:latest \\\nbash\n```\n\nExecute the build:\n```sh\npyb -X\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fpypbars","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoda480%2Fpypbars","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoda480%2Fpypbars/lists"}