{"id":15008904,"url":"https://github.com/luispedro/jug","last_synced_at":"2025-05-14T22:09:45.377Z","repository":{"id":573284,"uuid":"205237","full_name":"luispedro/jug","owner":"luispedro","description":"Parallel programming with Python","archived":false,"fork":false,"pushed_at":"2025-05-08T01:30:03.000Z","size":2361,"stargazers_count":452,"open_issues_count":1,"forks_count":62,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-05-08T01:37:08.241Z","etag":null,"topics":["hpc","parallel-computing","python","python-2","python-3","workflow","workflow-engine"],"latest_commit_sha":null,"homepage":"https://jug.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"go-gorp/gorp","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luispedro.png","metadata":{"files":{"readme":"README.rst","changelog":"ChangeLog","contributing":null,"funding":null,"license":"COPYING.MIT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2009-05-19T21:30:53.000Z","updated_at":"2025-05-08T01:30:08.000Z","dependencies_parsed_at":"2023-12-19T04:22:46.194Z","dependency_job_id":"372aaaef-2d03-430e-b3ce-c06dc25417dd","html_url":"https://github.com/luispedro/jug","commit_stats":{"total_commits":936,"total_committers":20,"mean_commits":46.8,"dds":0.5416666666666667,"last_synced_commit":"b557b60407ff1fbfe4f884a987be501bccb8b845"},"previous_names":[],"tags_count":67,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luispedro%2Fjug","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luispedro%2Fjug/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luispedro%2Fjug/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luispedro%2Fjug/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luispedro","download_url":"https://codeload.github.com/luispedro/jug/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235701,"owners_count":22036964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","parallel-computing","python","python-2","python-3","workflow","workflow-engine"],"created_at":"2024-09-24T19:21:33.679Z","updated_at":"2025-05-14T22:09:40.366Z","avatar_url":"https://github.com/luispedro.png","language":"Python","readme":"===========================================\nJug: A Task-Based Parallelization Framework\n===========================================\n\nJug allows you to write code that is broken up into\ntasks and run different tasks on different processors.\n\n.. image:: https://github.com/luispedro/jug/actions/workflows/python-package.yml/badge.svg\n       :target: https://github.com/luispedro/jug/actions/workflows/python-package.yml\n\n.. image:: https://zenodo.org/badge/205237.svg\n   :target: https://zenodo.org/badge/latestdoi/205237\n\n.. image:: https://img.shields.io/badge/install%20with-conda-brightgreen.svg?style=flat\n    :target: https://anaconda.org/conda-forge/jug\n\n.. image:: https://static.pepy.tech/personalized-badge/jug?period=total\u0026units=international_system\u0026left_color=black\u0026right_color=blue\u0026left_text=Downloads\n   :target: https://pepy.tech/project/jug\n\n.. image:: https://img.shields.io/badge/CITATION-doi.org%2F10.5334%2Fjors.161-green.svg\n   :target: https://doi.org/10.5334/jors.161\n\n\nIt uses the filesystem to communicate between processes and\nworks correctly over NFS, so you can coordinate processes on\ndifferent machines.\n\nJug is a pure Python implementation and should work on any platform.\n\nPython versions 3.5 and above are supported.\n\n*Website*: `http://luispedro.org/software/jug \u003chttp://luispedro.org/software/jug\u003e`__\n\n*Documentation*: `https://jug.readthedocs.org/ \u003chttps://jug.readthedocs.org/\u003e`__\n\n*Video*: On `vimeo \u003chttp://vimeo.com/8972696\u003e`__ or `showmedo\n\u003chttp://showmedo.com/videotutorials/video?name=9750000;fromSeriesID=975\u003e`__\n\n*Mailing List*: `https://groups.google.com/group/jug-users\n\u003chttps://groups.google.com/group/jug-users\u003e`__\n\nTestimonials\n------------\n\n\"I've been using jug with great success to distribute the running of a\nreasonably large set of parameter combinations\" - Andreas Longva\n\n\nInstall\n-------\n\nYou can install Jug with pip::\n\n    pip install Jug\n\nor use, if you are using `conda \u003chttps://anaconda.org/\u003e`__, you can install jug\nfrom `conda-forge \u003chttps://conda-forge.github.io/\u003e`__ using the following\ncommands::\n\n    conda config --add channels conda-forge\n    conda install jug\n\nCitation\n--------\n\nIf you use Jug to generate results for a scientific publication, please cite\n\n    Coelho, L.P., (2017). Jug: Software for Parallel Reproducible Computation in\n    Python. Journal of Open Research Software. 5(1), p.30.\n\n    https://doi.org/10.5334/jors.161\n\n\nShort Example\n-------------\n\nHere is a one minute example. Save the following to a file called ``primes.py``\n(if you have installed jug, you can obtain a slightly longer version of this\nexample by running ``jug demo`` on the command line)::\n\n    from jug import TaskGenerator\n    from time import sleep\n\n    @TaskGenerator\n    def is_prime(n):\n        sleep(1.)\n        for j in range(2,n-1):\n            if (n % j) == 0:\n                return False\n        return True\n\n    primes100 = [is_prime(n) for n in range(2,101)]\n\nThis is a brute-force way to find all the prime numbers up to 100. Of course,\nthis is only for didactical purposes, normally you would use a better method.\nSimilarly, the ``sleep`` function is so that it does not run too fast. Still,\nit illustrates the basic functionality of Jug for embarassingly parallel\nproblems.\n\nType ``jug status primes.py`` to get::\n\n    Task name                  Waiting       Ready    Finished     Running\n    ----------------------------------------------------------------------\n    primes.is_prime                  0          99           0           0\n    ......................................................................\n    Total:                           0          99           0           0\n\n\nThis tells you that you have 99 tasks called ``primes.is_prime`` ready to run.\nSo run ``jug execute primes.py \u0026``. You can even run multiple instances in the\nbackground (if you have multiple cores, for example). After starting 4\ninstances and waiting a few seconds, you can check the status again (with ``jug\nstatus primes.py``)::\n\n    Task name                  Waiting       Ready    Finished     Running\n    ----------------------------------------------------------------------\n    primes.is_prime                  0          63          32           4\n    ......................................................................\n    Total:                           0          63          32           4\n\n\nNow you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they\nwill all finish and you can inspect the results with ``jug shell primes.py``.\nThis will give you an ``ipython`` shell. The `primes100` variable is available,\nbut it is an ugly list of `jug.Task` objects. To get the actual value, you call\nthe `value` function::\n\n    In [1]: primes100 = value(primes100)\n\n    In [2]: primes100[:10]\n    Out[2]: [True, True, False, True, False, True, False, False, False, True]\n\nWhat's New\n----------\n\nVersion 2.3.1 (*5 November 2023*)\n\n- Update for Python 3.12\n\nVersion 2.3.0 (*25 June 2023*)\n\n- jug shell: Add ``get_filtered_tasks()``\n- jug: Fix ``jug --version`` (which had been broken in the refactoring to use subcommands)\n- jug shell: Fix message in jug shell when there are no dependencies (it would repeatedly print the message stating *this will only be run once*)\n- jug pack: Make it much faster to invalidate elements\n- file_store: ensure that the temporary directory exists\n\nVersion 2.2.3 (*26 May 2023*)\n- Fix ``jug shell`` for newer versions of IPython\n\nVersion 2.2.2 (*19 July 2022*)\n- Fix ``jug cleanup`` when packs are used (``jug pack``)\n\nVersion 2.2.1 (*19 May 2022*)\n- Fix bug with ``jug cleanup`` and the redis backend (`#86 \u003chttps://github.com/luispedro/jug/issues/86\u003e`__)\n\nVersion 2.2.0 (*3 May 2022*)\n\n- Add ``jug pack`` subcommand\n- Make ``get_tasks()`` return a copy of the tasks inside ``jug shell``\n- Remove ``six`` dependency\n\nVersion 2.1.1 (*18 March 2021*)\n\n- Include requirements files in distribution\n\nVersion 2.1.0 (*18 March 2021*)\n\n- Improvements to webstatus (by Robert Denham)\n- Removed Python 2.7 support\n- Fix output encoding for Python 3.8\n- Fix bug mixing ``mapreduce()`` \u0026 ``status --cache``\n- Make block_access (used in ``mapreduce()``) much faster (20x)\n- Fix important redis bug\n- More precise output in ``cleanup`` command\n\nVersion 2.0.2 (Thu Jun 11 2020)\n\n- Fix command line argument parsing\n\nVersion 2.0.1 (Thu Jun 11 2020)\n\n- Fix handling of ``JUG_EXIT_IF_FILE_EXISTS`` environmental variable\n- Fix passing an argument to ``jug.main()`` function\n- Extend ``--pdb`` to exceptions raised while importing the jugfile (issue #79)\n\nversion **2.0.0** (Fri Feb 21 2020)\n\n- jug.backend.base_store has 1 new method 'listlocks'\n- jug.backend.base_lock has 2 new methods 'fail' and 'is_failed'\n- Add 'jug execute --keep-failed' to preserve locks on failing tasks.\n- Add 'jug cleanup --failed-only' to remove locks from failed tasks\n- 'jug status' and 'jug graph' now display failed tasks\n- Check environmental exit variables by default (suggested by Renato Alves, issue #66)\n- Fix 'jug sleep-until' in the presence of barrier() (issue #71)\n\nFor older version see ``ChangeLog`` file or the `full history\n\u003chttps://jug.readthedocs.io/en/latest/history.html\u003e`__.\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluispedro%2Fjug","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluispedro%2Fjug","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluispedro%2Fjug/lists"}