{"id":13813554,"url":"https://github.com/csurfer/pypette","last_synced_at":"2025-04-05T23:08:20.745Z","repository":{"id":47093303,"uuid":"107827251","full_name":"csurfer/pypette","owner":"csurfer","description":"Ridiculously simple flow controller for building complex pipelines","archived":false,"fork":false,"pushed_at":"2022-12-09T05:43:24.000Z","size":582,"stargazers_count":264,"open_issues_count":1,"forks_count":9,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-29T22:09:10.026Z","etag":null,"topics":["multiprocessing","multithreading","python","threads"],"latest_commit_sha":null,"homepage":"https://csurfer.github.io/pypette","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/csurfer.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-22T00:55:55.000Z","updated_at":"2024-12-30T03:16:00.000Z","dependencies_parsed_at":"2023-01-25T12:00:49.269Z","dependency_job_id":null,"html_url":"https://github.com/csurfer/pypette","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csurfer%2Fpypette","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csurfer%2Fpypette/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csurfer%2Fpypette/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csurfer%2Fpypette/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/csurfer","download_url":"https://codeload.github.com/csurfer/pypette/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411234,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multiprocessing","multithreading","python","threads"],"created_at":"2024-08-04T04:01:21.214Z","updated_at":"2025-04-05T23:08:20.702Z","avatar_url":"https://github.com/csurfer.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/pypette\"\u003e\n        \u003cimg src=\"https://i.imgur.com/MBu5x0h.png\" width=\"60%\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/pypette\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/v/pypette.svg\" alt=\"pypiv\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/pypette\"\u003e\n        \u003cimg\n            src=\"https://img.shields.io/pypi/pyversions/pypette.svg\"\n            alt=\"pyv\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/csurfer/pypette/actions\"\u003e\n        \u003cimg\n            src=\"https://github.com/csurfer/pypette/actions/workflows/pytest.yml/badge.svg?branch=master\"\n            alt=\"Build Status\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codecov.io/gh/csurfer/pypette\"\u003e\n        \u003cimg\n            src=\"https://codecov.io/gh/csurfer/pypette/branch/master/graph/badge.svg?token=bzBkxPdbly\"\n            alt=\"Coverage Status\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://raw.githubusercontent.com/csurfer/pypette/master/LICENSE\"\u003e\n        \u003cimg\n            src=\"https://img.shields.io/badge/license-MIT-blue.svg\"\n            alt=\"License\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\npypette (to be read as pipette) is a module which makes building pipelines\nridiculously simple, allowing users to control the flow with minimal\ninstructions.\n\n## Features\n\n- Ridiculously simple interface.\n- Ability to view pipeline structure within the comfort of a terminal.\n- Run pipeline in exception resilient way if needed.\n- Create dependencies on pipelines easily.\n- Generate a easy to view/understand report within the comfort of a terminal.\n\n## Setup\n\n### Using pip\n\n```bash\npip install pypette\n```\n\n### Directly from the repository\n\n```bash\ngit clone https://github.com/csurfer/pypette.git\npython pypette/setup.py install\n```\n\n## Documentation\n\nDetailed documentation can be found at https://csurfer.github.io/pypette\n\n## Structures\n\n### Job\n\nThe basic unit of execution, say a python method or a callable.\n\n```python\nfrom pypette import Job\n\ndef print_hello():\n    print(\"Hello!\")\n\ndef print_hello_msg(msg):\n    print(\"Hello \" + msg + \"!\")\n\n# Job without arguments\nj1 = Job(print_hello)\n\n# Job with arguments specified as argument list\nj2 = Job(print_hello_msg, args=(\"pypette is simple\",))\n\n# Job with arguments specified as key word arguments\nj3 = Job(print_hello_msg, kwargs={\"msg\":\"pypette is simple\"})\n```\n\n### BashJob\n\nThe basic unit of execution, which runs a bash command.\n\n```python\nfrom pypette import BashJob\n\n# Job with bash commands\nb1 = BashJob(['ls', '-l'])\nb2 = BashJob(['pwd'])\n```\n\n### Pipe\n\nStructure to specify the flow in which the jobs need to be executed. The whole\ninterface consists of only 4 methods.\n\n```python\nfrom pypette import Pipe\n\n# 1. Create a new Pipe\np = Pipe('TestPipe')\n\n# 2. Add jobs to execute. (Assuming job_list is a list of python/bash jobs)\n\n# To run the jobs in job_list in order one after the other where each job waits\n# for the job before it to finish.\np.add_jobs(job_list)\n\n# To run the jobs in job_list parallelly and run the next step only after all\n# jobs in job list finish.\np.add_jobs(job_list, run_in_parallel=True)\n\n# Add jobs in a builder format.\np.add_stage(job1).add_stage(job2) # To add jobs in series.\np.add_stage(job1, job2) # To add jobs in parallel.\n```\n\n### Building complex pipelines\n\nJobs submitted to pipeline should be callables i.e. structures which can be run.\nThis means python methods, lambdas etc qualify.\n\nWhat about Pipe itself?\n\nOf course, it is a callable and you can submit a pipe object to be run along\nwith regular jobs. This way you can build small pipelines which achieve a\nspecific task and then combine them to create more complex pipelines.\n\n```python\nfrom pypette import BashJob, Job, Pipe\n\ndef welcome():\n    print(\"Welcome user!\")\n\ndef havefun():\n    print(\"Have fun!\")\n\ndef goodbye():\n    print(\"Goodbye!\")\n\n# Build a simple pipeline\np1 = Pipe('Fun')\np1.add_jobs([\n    Job(havefun),\n])\n\n# Include simple pipeline into a complicated pipeline\np2 = Pipe('Overall')\np2.add_jobs([\n    Job(welcome),\n    p1,\n    Job(goodbye),\n    BashJob(['ls', '-l']),\n    BashJob(['pwd'])\n])\n\np2.run() # This first runs welcome, then runs p1 pipeline then runs goodbye.\n```\n\n### Example pipeline\n\nAn example pipeline and its code are included in\n**[examples](https://github.com/csurfer/pypette/tree/master/examples)** folder.\n\n### Visualizing the pipeline using graph()\n\nPipeline objects have a method called `graph()` which helps visualize the\npipeline within the comfort of your terminal. The graph is recursive in nature\nand it visualizes everything that will be run if we call `run()` on the pipe\nobject.\n\nVisualizing the top-level pipeline in\n[examples/basic.py](https://github.com/csurfer/pypette/tree/master/examples/basic.py)\nled to the following visualization.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://i.imgur.com/1PaPlD3.png\" width=\"200px\"\u003e\n\u003c/p\u003e\n\n### Running the entire pipeline.\n\nThe only thing you need to do at this point to run the entire pipeline is to call\n`run()` on your pipeline object.\n\n### Reporting the entire pipeline.\n\nThe only thing you need to do at this point to get the report of entire pipeline\nis to call `report()` on your pipeline object.\n\n## Contributing\n\n### Bug Reports and Feature Requests\n\nPlease use [issue tracker](https://github.com/csurfer/pypette/issues) for\nreporting bugs or feature requests.\n\n### Development\n\n1. Checkout the repository.\n2. Make your changes and add/update relavent tests.\n3. Install **`poetry`** using **`pip install poetry`**.\n4. Run **`poetry install`** to create project's virtual environment.\n5. Run tests using **`poetry run tox`** (Any python versions which you don't have checked out will fail this). Fix failing tests and repeat.\n6. Make documentation changes that are relavant.\n7. Install **`pre-commit`** using **`pip install pre-commit`** and run **`pre-commit run --all-files`** to do lint checks.\n8. Generate documentation using **`poetry run sphinx-build -b html docs/ docs/_build/html`**.\n9. Generate **`requirements.txt`** for automated testing using **`poetry export --dev --without-hashes -f requirements.txt \u003e requirements.txt`**.\n10. Commit the changes and raise a pull request.\n\n### Buy the developer a cup of coffee!\n\nIf you found the utility helpful you can buy me a cup of coffee using\n\n[![Donate](https://www.paypalobjects.com/webstatic/en_US/i/btn/png/silver-pill-paypal-44px.png)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=3BSBW7D45C4YN\u0026lc=US\u0026currency_code=USD\u0026bn=PP%2dDonationsBF%3abtn_donate_SM%2egif%3aNonHosted)\n","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=3BSBW7D45C4YN\u0026lc=US\u0026currency_code=USD\u0026bn=PP%2dDonationsBF%3abtn_donate_SM%2egif%3aNonHosted"],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsurfer%2Fpypette","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcsurfer%2Fpypette","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsurfer%2Fpypette/lists"}