{"id":37666036,"url":"https://github.com/dubovikmaster/parallelbar","last_synced_at":"2026-01-16T11:58:57.960Z","repository":{"id":47957425,"uuid":"392691245","full_name":"dubovikmaster/parallelbar","owner":"dubovikmaster","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-17T14:21:24.000Z","size":488,"stargazers_count":44,"open_issues_count":2,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-02T14:50:23.232Z","etag":null,"topics":["python3","tqdm"],"latest_commit_sha":null,"homepage":"https://dubovikmaster.github.io/parallelbar","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dubovikmaster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["dubovikmaster"]}},"created_at":"2021-08-04T13:04:46.000Z","updated_at":"2025-09-26T14:30:56.000Z","dependencies_parsed_at":"2023-02-19T17:45:43.115Z","dependency_job_id":"43d1aaef-9af4-40ac-84ca-2295f5e470d6","html_url":"https://github.com/dubovikmaster/parallelbar","commit_stats":{"total_commits":66,"total_committers":4,"mean_commits":16.5,"dds":0.5151515151515151,"last_synced_commit":"773ff6b0437b2a22be88035d573645bcc8c39ad0"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/dubovikmaster/parallelbar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dubovikmaster%2Fparallelbar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dubovikmaster%2Fparallelbar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dubovikmaster%2Fparallelbar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dubovikmaster%2Fparallelbar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dubovikmaster","download_url":"https://codeload.github.com/dubovikmaster/parallelbar/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dubovikmaster%2Fparallelbar/sbom","scorecard":{"id":358980,"data":{"date":"2025-08-11","repo":{"name":"github.com/dubovikmaster/parallelbar","commit":"b30fb8727afe44896abc9fe9950fb5e3ba300c23"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.9,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":9,"reason":"1 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-g7vv-2v7x-gj9p"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T10:18:36.433Z","repository_id":47957425,"created_at":"2025-08-18T10:18:36.433Z","updated_at":"2025-08-18T10:18:36.433Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python3","tqdm"],"created_at":"2026-01-16T11:58:57.871Z","updated_at":"2026-01-16T11:58:57.946Z","avatar_url":"https://github.com/dubovikmaster.png","language":"Python","funding_links":["https://github.com/sponsors/dubovikmaster"],"categories":[],"sub_categories":[],"readme":"\n# Parallelbar\n\n[![PyPI version fury.io](https://badge.fury.io/py/parallelbar.svg)](https://pypi.python.org/pypi/parallelbar/)\n[![PyPI license](https://img.shields.io/pypi/l/parallelbar.svg)](https://pypi.python.org/pypi/parallelbar/)\n[![PyPI download month](https://img.shields.io/pypi/dm/parallelbar.svg)](https://pypi.python.org/pypi/parallelbar/)\n\n## Table of contents\n* [Instalation](#Instalation)\n* [Usage](#Usage)\n* [Exception handling](#exception-handling)\n* [Changelog](#Changelog)\n   * [New in version 2.5](#new-in-version-2.5)\n   * [New in version 2.4](#new-in-version-2.4)\n   * [New in version 2.3](#new-in-version-2.3)\n   * [New in version 1.3](#new-in-version-1.3)\n   * [New in version 1.2](#new-in-version-1.2)\n   * [New in version 1.1](#new-in-version-1.1)\n   * [New in version 1.0](#new-in-version-1.0)\n   * [New in version 0.3](#new-in-version-0.3)\n* [Problems of the naive approach](#naive-approach)\n* [License](#license)\n\n**Parallelbar** displays the progress of tasks in the process pool for [**Pool**](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool) class methods such as `map`, `starmap` (since 1.2 version), `imap` and `imap_unordered`. Parallelbar is based on the [tqdm](https://github.com/tqdm/tqdm) module and the standard python [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) library. \nAlso, it is possible to handle exceptions that occur within a separate process, as well as set a timeout for the execution of a task by a process.\n\n\u003ca name=\"Installation\"\u003e\u003c/a\u003e\n## Installation\n```python\npip install parallelbar\n```\nor\n```python\npip install --user git+https://github.com/dubovikmaster/parallelbar.git\n```\n\n\n\u003ca name=\"Usage\"\u003e\u003c/a\u003e\n## Usage\n\n\n```python\nfrom parallelbar import progress_imap, progress_map, progress_imapu\nfrom parallelbar.tools import cpu_bench, fibonacci\n```\n\nLet's create a list of 100 numbers and test `progress_map` with default parameters on a toy function `cpu_bench`:\n\n\n```python\ntasks = range(10000)\n```\n```python\n%%time\nlist(map(cpu_bench, tasks))\n```\n```python\nWall time: 52.6 s\n```\n\nOk, by default this works on one core of my i7-9700F and it took 52 seconds. Let's parallelize the calculations for all 8 cores and look at the progress. This can be easily done by replacing standart function  **map** with **progress_map**.\n\n```python\nif __name__=='__main__':\n    progress_map(cpu_bench, tasks)\n```\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/first_bar_.gif)\n\nCore progress:\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/core_progress.gif)\n\nYou can also easily use **progress_imap** and **progress_imapu** analogs of the *imap* and *imap_unordered* methods of the **Pool()** class\n\n\n```python\n%%time\nif __name__=='__main__':\n    tasks = [20 + i for i in range(15)]\n    result = progress_imap(fibonacci, tasks, chunk_size=1, core_progress=False)\n```\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/one_bar_imap.gif)\n\n\u003ca name=\"exception-handling\"\u003e\u003c/a\u003e\n## Exception handling\nYou can handle exceptions and set timeouts for the execution of tasks by the process.   \nConsider the following toy example:\n\n```python\ndef foo(n):\n    if n==5 or n==17:\n        1/0\n    elif n==10:\n        time.sleep(2)\n    else:\n        time.sleep(1)\n    return n\nif __name__=='__main__':\n\tres = progress_map(foo, range(20), process_timeout=5, n_cpu=8)\n```\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/error_bar_2.gif)\n\nAs you can see, under the main progress bar, another progress bar has appeared that displays the number of tasks that ended unsuccessfully. At the same time, the main bar turned orange, as if signaling something went wrong\n```python\nprint(res)\n\t[0, 1, 2, 3, 4, ZeroDivisionError('division by zero'), 6, 7, 8, 9, 10, 11, 12,\n     13, 14, 15, 16, ZeroDivisionError('division by zero'), 18, 19]\n```\n In the resulting array, we have exceptions in the corresponding places. Also, we can see the exception traceback:\n```python\nprint(res[5].traceback)\nTraceback (most recent call last):\n  File \"/home/padu/anaconda3/envs/work/lib/python3.9/site-packages/pebble/common.py\", line 174, in process_execute\n    return function(*args, **kwargs)\n  File \"/home/padu/anaconda3/envs/work/lib/python3.9/site-packages/parallelbar/parallelbar.py\", line 48, in _process\n    result = func(task)\n  File \"/tmp/ipykernel_70395/285585760.py\", line 3, in foo\n    1/0\nZeroDivisionError: division by zero\n```\nFrom which concept at what place in the code the exception occurred. \nLet's add a timeout of 1.5 seconds for each process. If the process execution time exceeds 1.5 seconds, an appropriate exception will be raised and handled. In this case, the process will restart and continue to work (thanks to **pebble**)\n```python\nif __name__=='__main__':\n\tres = progress_map(foo, range(20), process_timeout=1.5, n_cpu=8)\n```\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/error_bar_1.gif)\n```python\nprint(res)\n\t[0, 1, 2, 3, 4, ZeroDivisionError('division by zero'), 6, 7, 8, 9, 'function foo took longer than 1.5 s.', \n\t11, 12, 13, 14, 15, 16, ZeroDivisionError('division by zero'), 18, 19]\n```\n\nException handling has also been added to methods **progress_imap** and **progress_imapu**.\n\u003ca name=\"Changelog\"\u003e\u003c/a\u003e\n## Changelog\n\n\u003ca name=\"new-in-version-2.5\"\u003e\u003c/a\u003e\n### Version 2.5\n- Introduced an optional `timeout` parameter to `progress_map` and `progress_starmap` for managing execution time limits.\n\n\u003ca name=\"new-in-version-2.4\"\u003e\u003c/a\u003e\n### Version 2.4\n- fixed [issue](https://github.com/dubovikmaster/parallelbar/issues/4)\n- For **Windows OS**, when using the `add_progress` decorator, the function being decorated no longer needs the `worker_queue` keyword argument.\n\n\n\u003ca name=\"new-in-version-2.3\"\u003e\u003c/a\u003e\n### New in version 2.3\n- added `wrappers` module with which contains decorators:\n  - `stop_it_after_timeout` - stops the function execution after the specified time (in seconds)\n  - `add_progress` - adds a progress bar to the function execution, exception handling and timeout.\n\nUsage example for UNIX systems:\n```python\nfrom parallelbar.wrappers import add_progress\nfrom parallelbar import progress_map\nimport time\n\n\n@add_progress(error_handling='coerce', timeout=.5)\ndef foo(n):\n    if n==5 or n==17:\n        1/0\n    elif n==10:\n        time.sleep(1)\n    else:\n        time.sleep(.1)\n    return n\n\ndef bar(x):\n    return [foo(i) for i in range(x)]\n\nif __name__=='__main__':\n    # you must specify the total number of tasks\n    res = progress_map(bar, [10, 20, 30, 40], n_cpu=4, total=100)\n```\nOut:\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/add_progress_example.gif)\n\nFor **Windows** systems you need to add the `worker_queue` parameter to the functions `foo` and `bar` and use the `used_add_progress_decorator` parameter in the `progress_map` function:\n```python\n@add_progress(error_handling='coerce', timeout=.5)\ndef foo(n):\n    if n==5 or n==17:\n        1/0\n    elif n==10:\n        time.sleep(1)\n    else:\n        time.sleep(.1)\n    return n\n\ndef bar(x, worker_queue=None):\n    return [foo(i, worker_queue=worker_queue) for i in range(x)]\n\nif __name__=='__main__':\n    res = progress_map(bar, [10, 20, 30, 40], n_cpu=4, total=100, used_add_progress_decorator=True)\n```\nOut:\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/add_progress_example.gif)\n\nYou can also use the `stopit_after_timeout` decorator separately:\n```python\nfrom parallelbar.wrappers import stopit_after_timeout\nfrom parallelbar import progress_map\nimport time\n\n\n@stopit_after_timeout(.5, raise_exception=True)\ndef foo(n):\n    if n==5:\n        time.sleep(1)\n    else:\n        time.sleep(.1)\n    return n\n\nif __name__=='__main__':\n    print(f'first result is: {foo(3)}')\n    print(f'second result is: {foo(5)}')\n```\nOut:\n```python\nfirst result is: 3\n\nTimeoutError                              Traceback (most recent call last)\nCell In[7], line 16\n     14 if __name__=='__main__':\n     15     print(foo(3))\n---\u003e 16     print(foo(5))\n\nFile /opt/conda/envs/user_response/lib/python3.10/site-packages/parallelbar/wrappers.py:38, in stopit_after_timeout.\u003clocals\u003e.actual_decorator.\u003clocals\u003e.wrapper(*args, **kwargs)\n     36     msg = f'function took longer than {s} s.'\n     37     if raise_exception:\n---\u003e 38         raise TimeoutError(msg)\n     39     result = msg\n     40 finally:\n\nTimeoutError: function took longer than 0.5 s.\n```\n- added `return_failed_tasks` keyword parameter to the `progress_map/starmap/imap/imapu` function (default=`False`) - if `True` then the result will include the tasks that failed with an exception.\n\n\u003ca name=\"new-in-version-1.3\"\u003e\u003c/a\u003e\n### New in version 1.3\n- added `maxtaskperchild` keyword parameter to the `progress_map/starmap/imap/imapu` function (default=`None`)\n\n\u003ca name=\"new-in-version-1.2\"\u003e\u003c/a\u003e\n### New in version 1.2\n\n - Added `progress_starmap` function. An extension of the [`starmap`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap) method of the `Pool` class.\n - Improved documentation.\n\n\u003ca name=\"new-in-version-1.1\"\u003e\u003c/a\u003e\n### New in version 1.1\n1. The `bar_step` keyword argument is no longer used and will be removed in a future version\n2. Added `need_serialize` boolean keyword argument to the `progress_map/imap/imapu` function (default `False`). Requires [dill](https://pypi.org/project/dill/) to be installed. If `True`\nthe target function is serialized using `dill` library. Thus, as a target function, you can now use lambda functions, class methods and other callable objects that `pickle` cannot serialize\n3. Added dynamic optimization of the progress bar refresh rate. This can significantly improve the performance of the `progress_map/imap/imapu` functions ror very long iterables and small execution time of one task by the objective function.\n\n\u003ca name=\"new-in-version-1.0\"\u003e\u003c/a\u003e\n### New in version 1.0\n1. The \"ignore\" value of the `error_behavior` key parameter is no longer supported.\n2. Default value of key parameter `error_behavior` changed to \"raise\".\n3. The [pebble](https://github.com/noxdafox/pebble) module is no longer used.\n4. Added key parameter `executor` in the functions `progress_map`, `progress_imap` and `progress_imapu`. Must be one of the values:\n   - \"threads\" - use thread pool\n   - \"processes\" - use processes pool (default)\n\n\u003ca name=\"new-in-version-0.3\"\u003e\u003c/a\u003e\n### New in version 0.3.0\n1. The `error_behavior` keyword argument has been added to the **progress_map**, **progress_imap** and **progress_imapu** methods. \nMust be one of the values: \"raise\", \"ignore\", \"coerce\". \n     - \"raise\" - raise an exception thrown in the process pool.\n     - \"ignore\" - ignore the exceptions that occur. Do not add anything to the result\n     - \"coerce\" - handle the exception. The result will include the value set by the parameter `set_error_value` (by default None - the traceback of the raised exception will be added to the result)\n2. The `set_error_value` keyword argument has been added to the **progress_map**, **progress_imap** and **progress_imapu** methods.\n\nExample of usage\n\n```python\nimport time\nimport resource as rs\nfrom parallelbar import progress_imap\n\n\ndef memory_limit(limit):\n    soft, hard = rs.getrlimit(rs.RLIMIT_AS)\n    rs.setrlimit(rs.RLIMIT_AS, (limit, hard))\n\n\ndef my_awesome_foo(n):\n    if n == 0:\n        s = 'a' * 10000000\n    elif n == 20:\n        time.sleep(100)\n    else:\n        time.sleep(1)\n    return n\n\n\nif __name__ == '__main__':\n    tasks = range(30)\n    start = time.monotonic()\n    result = progress_imap(my_awesome_foo, tasks, \n                           process_timeout=1.5, \n                           initializer=memory_limit, \n                           initargs=(100,),\n                           n_cpu=4,\n                           error_behavior='coerce',\n                           set_error_value=None,\n                           )\n    print(f'time took: {time.monotonic() - start:.1f}')\n    print(result)\n```\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/test-new.gif)\n```\ntime took: 8.2\n[MemoryError(), 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, \n16, 17, 18, 19, TimeoutError('function \"my_awesome_foo\" took longer than 1.5 s.'), 21, 22, 23, 24, 25, 26, 27, 28, 29]\n```\nSet NaN instead of tracebacks to the result of the pool operation:\n```python\nif __name__ == '__main__':\n    tasks = range(30)\n    start = time.monotonic()\n    result = progress_imap(my_awesome_foo, tasks, \n                           process_timeout=1.5, \n                           initializer=memory_limit, \n                           initargs=(100,),\n                           n_cpu=4,\n                           error_behavior='coerce',\n                           set_error_value=float('nan'),\n                           )\n    print(f'time took: {time.monotonic() - start:.1f}')\n    print(result)\n```\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/test-new.gif)\n```\ntime took: 8.0\n[nan, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, \n16, 17, 18, 19, nan, 21, 22, 23, 24, 25, 26, 27, 28, 29]\n```\nLet's ignore exception:\n```python\nif __name__ == '__main__':\n    tasks = range(30)\n    start = time.monotonic()\n    result = progress_imap(my_awesome_foo, tasks, \n                           process_timeout=1.5, \n                           initializer=memory_limit, \n                           initargs=(100,),\n                           n_cpu=4,\n                           error_behavior='ignore',\n                           set_error_value=None,\n                           )\n    print(f'time took: {time.monotonic() - start:.1f}')\n    print(result)\n```\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/test-new.gif)\n```\ntime took: 8.0\n[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, \n16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29]\n```\n\n\u003ca name=\"naive-approach\"\u003e\u003c/a\u003e\n## Problems of the naive approach\nWhy can't I do something simpler? Let's take the standard **imap** method and run through it in a loop with **tqdm** and take the results from the processes:\n```python\nfrom multiprocessing import Pool\nfrom tqdm.auto import tqdm\n```\n\n\n```python\nif __name__=='__main__':\n    with Pool() as p:\n        tasks = [20 + i for i in range(15)]\n        pool = p.imap(fibonacci, tasks)\n        result = []\n        for i in tqdm(pool, total=len(tasks)):\n            result.append(i)\n```\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/imap_naive_1.gif)\n\nIt looks good, doesn't it? But let's do the following, make the first task very difficult for the core. To do this, I will insert the number 38 at the beginning of the tasks list. Let's see what happens\n\n```python\nif __name__=='__main__':\n    with Pool() as p:\n        tasks = [20 + i for i in range(15)]\n        tasks.insert(0, 39)\n        pool = p.imap_unordered(fibonacci, tasks)\n        result = []\n        for i in tqdm(pool, total=len(tasks)):\n            result.append(i)\n```\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/imap_naive_2.gif)\n\nThis is a fiasco. Our progress hung on the completion of the first task and then at the end showed 100% progress.\nLet's try to do the same experiment only for the progress_imap function:\n\n```python\nif __name__=='__main__':\n    tasks = [20 + i for i in range(15)]\n    tasks.insert(0, 39)\n    result = progress_imap(fibonacci, tasks)\n```\n\n![](https://raw.githubusercontent.com/dubovikmaster/parallelbar/main/gifs/imap_naive_3.gif)\n\nThe progress_imap function takes care of collecting the result and closing the process pool for you.\nIn fact, the naive approach described above will work for the standard imap_unordered method. But it does not guarantee the order of the returned result. This is often critically important.\n\n\u003ca name=\"license\"\u003e\u003c/a\u003e\n## License\n\nMIT license\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubovikmaster%2Fparallelbar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdubovikmaster%2Fparallelbar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubovikmaster%2Fparallelbar/lists"}