{"id":15628958,"url":"https://github.com/pwwang/pipen","last_synced_at":"2026-01-16T02:09:17.569Z","repository":{"id":37205098,"uuid":"80252164","full_name":"pwwang/pipen","owner":"pwwang","description":"A pipeline framework for python","archived":false,"fork":false,"pushed_at":"2025-03-14T05:42:47.000Z","size":6768,"stargazers_count":104,"open_issues_count":0,"forks_count":15,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-03T17:11:11.562Z","etag":null,"topics":["dot","jinja2","pipeline","pipeline-framework","python","workflow","workflow-engine"],"latest_commit_sha":null,"homepage":"https://pwwang.github.io/pipen/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pwwang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-27T22:39:34.000Z","updated_at":"2025-03-14T05:39:08.000Z","dependencies_parsed_at":"2024-01-25T19:29:19.715Z","dependency_job_id":"5bc69a16-47e8-43f3-bec5-6f0af6cbc6ff","html_url":"https://github.com/pwwang/pipen","commit_stats":{"total_commits":1071,"total_committers":9,"mean_commits":119.0,"dds":0.2717086834733894,"last_synced_commit":"09fb520867960d14d2ab680e1afc1e8f10541808"},"previous_names":["pwwang/pyppl"],"tags_count":119,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fpipen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fpipen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fpipen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pwwang%2Fpipen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pwwang","download_url":"https://codeload.github.com/pwwang/pipen/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590861,"owners_count":21129903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dot","jinja2","pipeline","pipeline-framework","python","workflow","workflow-engine"],"created_at":"2024-10-03T10:25:00.772Z","updated_at":"2025-12-30T03:28:37.782Z","avatar_url":"https://github.com/pwwang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./docs/rtd-logo.png\" width=\"320px\"\u003e\n\n**A pipeline framework for python**\n\n\u003c/div\u003e\n\n______________________________________________________________________\n\n[![Pypi][6]][7] [![Github][8]][9] ![Building][10] [![Docs and API][11]][1] [![Codacy][12]][13] [![Codacy coverage][14]][13] [![Deps][5]][23]\n\n[Documentation][1] | [ChangeLog][2] | [Examples][3] | [API][4]\n\n## Features\n\n- Easy to use\n- Nearly zero-configuration\n- Nice logging\n- Highly extendable\n- Cloud support naively\n\n## Installation\n\n```bash\npip install -U pipen\n```\n\n## Quickstart\n\n`example.py`\n\n```python\nfrom pipen import Proc, Pipen, run\n\nclass P1(Proc):\n    \"\"\"Sort input file\"\"\"\n    input = \"infile\"\n    input_data = [\"/tmp/data.txt\"]\n    output = \"outfile:file:intermediate.txt\"\n    script = \"cat {{in.infile}} | sort \u003e {{out.outfile}}\"\n\nclass P2(Proc):\n    \"\"\"Paste line number\"\"\"\n    requires = P1\n    input = \"infile:file\"\n    output = \"outfile:file:result.txt\"\n    script = \"paste \u003c(seq 1 3) {{in.infile}} \u003e {{out.outfile}}\"\n\n# class MyPipeline(Pipen):\n#     starts = P1\n\nif __name__ == \"__main__\":\n    # MyPipeline().run()\n    run(\"MyPipeline\", starts=P1)\n```\n\n```shell\n\u003e echo -e \"3\\n2\\n1\" \u003e /tmp/data.txt\n\u003e python example.py\n```\n\n```log\n04-17 16:19:35 I core                   _____________________________________   __\n04-17 16:19:35 I core                   ___  __ \\___  _/__  __ \\__  ____/__  | / /\n04-17 16:19:35 I core                   __  /_/ /__  / __  /_/ /_  __/  __   |/ /\n04-17 16:19:35 I core                   _  ____/__/ /  _  ____/_  /___  _  /|  /\n04-17 16:19:35 I core                   /_/     /___/  /_/     /_____/  /_/ |_/\n04-17 16:19:35 I core\n04-17 16:19:35 I core                               version: 1.0.2\n04-17 16:19:35 I core\n04-17 16:19:35 I core    ╔═══════════════════════════ MYPIPELINE ════════════════════════════╗\n04-17 16:19:35 I core    ║ My pipeline                                                       ║\n04-17 16:19:35 I core    ╚═══════════════════════════════════════════════════════════════════╝\n04-17 16:19:35 I core    plugins         : verbose v0.14.1\n04-17 16:19:35 I core    # procs         : 2\n04-17 16:19:35 I core    profile         : default\n04-17 16:19:35 I core    outdir          : /path/to/cwd/MyPipeline-output\n04-17 16:19:35 I core    cache           : True\n04-17 16:19:35 I core    dirsig          : 1\n04-17 16:19:35 I core    error_strategy  : ignore\n04-17 16:19:35 I core    forks           : 1\n04-17 16:19:35 I core    lang            : bash\n04-17 16:19:35 I core    loglevel        : info\n04-17 16:19:35 I core    num_retries     : 3\n04-17 16:19:35 I core    scheduler       : local\n04-17 16:19:35 I core    submission_batch: 8\n04-17 16:19:35 I core    template        : liquid\n04-17 16:19:35 I core    workdir         : /path/to/cwd/.pipen/MyPipeline\n04-17 16:19:35 I core    plugin_opts     :\n04-17 16:19:35 I core    template_opts   : filters={'realpath': \u003cfunction realpath at 0x7fc3eba12...\n04-17 16:19:35 I core                    : globals={'realpath': \u003cfunction realpath at 0x7fc3eba12...\n04-17 16:19:35 I core    Initializing plugins ...\n04-17 16:19:36 I core\n04-17 16:19:36 I core    ╭─────────────────────────────── P1 ────────────────────────────────╮\n04-17 16:19:36 I core    │ Sort input file                                                   │\n04-17 16:19:36 I core    ╰───────────────────────────────────────────────────────────────────╯\n04-17 16:19:36 I core    P1: Workdir: '/path/to/cwd/.pipen/MyPipeline/P1'\n04-17 16:19:36 I core    P1: \u003c\u003c\u003c [START]\n04-17 16:19:36 I core    P1: \u003e\u003e\u003e ['P2']\n04-17 16:19:36 I verbose P1: in.infile: /tmp/data.txt\n04-17 16:19:36 I verbose P1: out.outfile: /path/to/cwd/.pipen/MyPipeline/P1/0/output/intermediate.txt\n04-17 16:19:38 I verbose P1: Time elapsed: 00:00:02.051s\n04-17 16:19:38 I core\n04-17 16:19:38 I core    ╭═══════════════════════════════ P2 ════════════════════════════════╮\n04-17 16:19:38 I core    ║ Paste line number                                                 ║\n04-17 16:19:38 I core    ╰═══════════════════════════════════════════════════════════════════╯\n04-17 16:19:38 I core    P2: Workdir: '/path/to/cwd/.pipen/MyPipeline/P2'\n04-17 16:19:38 I core    P2: \u003c\u003c\u003c ['P1']\n04-17 16:19:38 I core    P2: \u003e\u003e\u003e [END]\n04-17 16:19:38 I verbose P2: in.infile: /path/to/cwd/.pipen/MyPipeline/P1/0/output/intermediate.txt\n04-17 16:19:38 I verbose P2: out.outfile: /path/to/cwd/MyPipeline-output/P2/result.txt\n04-17 16:19:41 I verbose P2: Time elapsed: 00:00:02.051s\n04-17 16:19:41 I core\n\n             MYPIPELINE: 100%|██████████████████████████████| 2/2 [00:06\u003c00:00, 0.35 procs/s]\n```\n\n```shell\n\u003e cat ./MyPipeline-output/P2/result.txt\n1       1\n2       2\n3       3\n```\n\n## Examples\n\nSee more examples at `examples/` and a more realcase example at:\n\n\u003chttps://github.com/pwwang/pipen-report/tree/master/example\u003e\n\n## Plugin gallery\n\nPlugins make `pipen` even better.\n\n- [`pipen-annotate`][26]: Use docstring to annotate pipen processes\n- [`pipen-args`][19]: Command line argument parser for pipen\n- [`pipen-board`][27]: Visualize configuration and running of pipen pipelines on the web\n- [`pipen-diagram`][18]: Draw pipeline diagrams for pipen\n- [`pipen-dry`][20]: Dry runner for pipen pipelines\n- [`pipen-filters`][17]: Add a set of useful filters for pipen templates.\n- [`pipen-lock`][25]: Process lock for pipen to prevent multiple runs at the same time.\n- [`pipen-log2file`][28]: Save running logs to file for pipen\n- [`pipen-poplog`][30]: Populate logs from jobs to running log of the pipeline\n- [`pipen-report`][16]: Generate report for pipen\n- [`pipen-runinfo`][29]: Save running information to file for pipen\n- [`pipen-verbose`][15]: Add verbosal information in logs for pipen.\n- [`pipen-gcs`][32]: A plugin for pipen to handle files in Google Cloud Storage.\n- [`pipen-deprecated`][34]: A pipen plugin to mark processes as deprecated.\n- [`pipen-cli-init`][21]: A pipen CLI plugin to create a pipen project (pipeline)\n- [`pipen-cli-ref`][31]: Make reference documentation for processes\n- [`pipen-cli-require`][24]: A pipen cli plugin check the requirements of a pipeline\n- [`pipen-cli-run`][22]: A pipen cli plugin to run a process or a pipeline\n- [`pipen-cli-gbatch`][33]: A pipen cli plugin to submit pipeline to Google Batch Jobs\n\n[1]: https://pwwang.github.io/pipen\n[2]: https://pwwang.github.io/pipen/CHANGELOG\n[3]: https://pwwang.github.io/pipen/examples\n[4]: https://pwwang.github.io/pipen/api/pipen\n[5]: https://img.shields.io/librariesio/release/pypi/pipen?style=flat-square\n[6]: https://img.shields.io/pypi/v/pipen?style=flat-square\n[7]: https://pypi.org/project/pipen/\n[8]: https://img.shields.io/github/v/tag/pwwang/pipen?style=flat-square\n[9]: https://github.com/pwwang/pipen\n[10]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/build.yml?style=flat-square\n[11]: https://img.shields.io/github/actions/workflow/status/pwwang/pipen/docs.yml?label=docs\u0026style=flat-square\n[12]: https://img.shields.io/codacy/grade/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square\n[13]: https://app.codacy.com/gh/pwwang/pipen\n[14]: https://img.shields.io/codacy/coverage/cf1c6c97e5c4480386a05b42dec10c6e?style=flat-square\n[15]: https://github.com/pwwang/pipen-verbose\n[16]: https://github.com/pwwang/pipen-report\n[17]: https://github.com/pwwang/pipen-filters\n[18]: https://github.com/pwwang/pipen-diagram\n[19]: https://github.com/pwwang/pipen-args\n[20]: https://github.com/pwwang/pipen-dry\n[21]: https://github.com/pwwang/pipen-cli-init\n[22]: https://github.com/pwwang/pipen-cli-run\n[23]: https://libraries.io/github/pwwang/pipen#repository_dependencies\n[24]: https://github.com/pwwang/pipen-cli-require\n[25]: https://github.com/pwwang/pipen-lock\n[26]: https://github.com/pwwang/pipen-annotate\n[27]: https://github.com/pwwang/pipen-board\n[28]: https://github.com/pwwang/pipen-log2file\n[29]: https://github.com/pwwang/pipen-runinfo\n[30]: https://github.com/pwwang/pipen-poplog\n[31]: https://github.com/pwwang/pipen-cli-ref\n[32]: https://github.com/pwwang/pipen-gcs\n[33]: https://github.com/pwwang/pipen-cli-gbatch\n[34]: https://github.com/pwwang/pipen-deprecated\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwwang%2Fpipen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpwwang%2Fpipen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpwwang%2Fpipen/lists"}