{"id":47676153,"url":"https://github.com/joarhal/piperig","last_synced_at":"2026-04-02T13:31:27.598Z","repository":{"id":346413811,"uuid":"1189864329","full_name":"joarhal/piperig","owner":"joarhal","description":"Declarative pipeline runner for shell scripts. Define steps, params, and loops in YAML — piperig expands and executes.","archived":false,"fork":false,"pushed_at":"2026-03-24T15:15:18.000Z","size":433,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-24T16:58:55.920Z","etag":null,"topics":["automation","cli","devops","go","pipeline","task-runner","yaml"],"latest_commit_sha":null,"homepage":"https://joarhal.github.io/piperig/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joarhal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-23T18:37:55.000Z","updated_at":"2026-03-24T16:41:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/joarhal/piperig","commit_stats":null,"previous_names":["joarhal/piperig"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/joarhal/piperig","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joarhal%2Fpiperig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joarhal%2Fpiperig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joarhal%2Fpiperig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joarhal%2Fpiperig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joarhal","download_url":"https://codeload.github.com/joarhal/piperig/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joarhal%2Fpiperig/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31307132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cli","devops","go","pipeline","task-runner","yaml"],"created_at":"2026-04-02T13:31:26.573Z","updated_at":"2026-04-02T13:31:27.593Z","avatar_url":"https://github.com/joarhal.png","language":"Go","readme":"# piperig\n\n[![CI](https://github.com/joarhal/piperig/actions/workflows/ci.yml/badge.svg)](https://github.com/joarhal/piperig/actions/workflows/ci.yml)\n[![Release](https://img.shields.io/github/v/release/joarhal/piperig)](https://github.com/joarhal/piperig/releases)\n[![Go](https://img.shields.io/github/go-mod/go-version/joarhal/piperig)](https://go.dev/)\n[![License](https://img.shields.io/github/license/joarhal/piperig)](LICENSE)\n\nRun your scripts as declarative YAML pipelines — with loops, date ranges, retries, and scheduling. Single binary, no runtime dependencies.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/banner.png\" alt=\"piperig terminal output\" width=\"560\"\u003e\n\u003c/p\u003e\n\npiperig picks the interpreter by extension (`.py` → python, `.sh` → bash, `.js` → node, `.ts` → npx tsx, `.rb` → ruby). No extension means direct exec. Add custom ones in [`.piperig.yaml`](#project-config).\n\n## Contents\n\n- [Install](#install)\n- [Usage](#usage)\n- [Demo](#demo)\n- [Parameters](#parameters)\n- [Time expressions](#time-expressions)\n- [Iteration](#iteration)\n- [Execution control](#execution-control)\n- [Nested pipes](#nested-pipes)\n- [Structured output](#structured-output)\n- [Input modes](#input-modes)\n- [Scheduling](#scheduling)\n- [Interactive picker](#interactive-picker)\n- [CLI](#cli)\n- [Reference](#reference)\n\n## Install\n\n```bash\nbrew install joarhal/tap/piperig          # Homebrew\ngo install github.com/joarhal/piperig/cmd/piperig@latest   # Go\n```\n\n## Usage\n\n```bash\npiperig new pipe pipes/daily/images    # scaffold a .pipe.yaml\npiperig run pipes/daily/images.pipe.yaml             # run a pipe\npiperig run pipes/daily/images.pipe.yaml quality=90   # run with override\npiperig check pipes/daily/images.pipe.yaml            # preview the call plan\npiperig run                                           # interactive picker\n```\n\n**Create with AI.** Ask your LLM to run `piperig llm` to read the full documentation, then ask it to create `.pipe.yaml` files for your scripts.\n\n## Demo\n\n```yaml\n# news.pipe.yaml\ndescription: Collect and summarize news for last 3 days\n\nwith:\n  output_dir: ./news\n\nloop:\n  date: -3d..-1d\n\neach:\n  - { source: hackernews, topic: programming }\n  - { source: reddit,     topic: tech }\n\nsteps:\n  - job: scripts/fetch_articles.py\n    retry: 2\n    retry_delay: 3s\n  - job: scripts/summarize.py\n  - job: scripts/send_digest.sh\n    each: false\n    loop: false\n    allow_failure: true\n```\n\n```bash\n$ piperig check news.pipe.yaml\n\nPipe: news.pipe.yaml (Collect and summarize news for last 3 days)\n\n  Step 1: scripts/fetch_articles.py × 2 each × 3 date = 6 calls\n    1. output_dir=./news  source=hackernews  topic=programming  date=2026-03-18\n    2. output_dir=./news  source=hackernews  topic=programming  date=2026-03-19\n    3. output_dir=./news  source=hackernews  topic=programming  date=2026-03-20\n    4. output_dir=./news  source=reddit      topic=tech         date=2026-03-18\n    ...\n\n  Step 2: scripts/summarize.py × 2 each × 3 date = 6 calls\n\n  Step 3: scripts/send_digest.sh = 1 call\n\n  Total: 13 calls\n```\n\n`piperig check` shows the plan. `piperig run` executes it. That's the whole idea.\n\n## Parameters\n\nThe simplest pipe — one script with fixed parameters:\n\n```yaml\nsteps:\n  - job: scripts/resize.py\n    with:\n      src: /data/photos\n      quality: 80\n```\n\npiperig passes params as environment variables by default (`SRC=/data/photos QUALITY=80 python scripts/resize.py`).\n\n`description` is optional — shown in `piperig check` and the interactive picker.\n\n**Shared parameters.** When multiple steps need the same values, move `with` to the pipe level:\n\n```yaml\nwith:\n  src: /data/photos\n  dest: /data/output\n\nsteps:\n  - job: scripts/download.sh\n  - job: scripts/resize.py\n    with:\n      quality: 80           # added to src + dest\n  - job: scripts/upload.sh\n```\n\nTop-level `with` merges into every step. Step values win on conflict.\n\n**CLI overrides.** Override anything at runtime — no file edits needed:\n\n```bash\npiperig run resize.pipe.yaml quality=95 dest=/tmp/test\n```\n\n**Environment variables.** Use `$VAR` or `${VAR}` in `with` values to pull from the process environment — keep secrets and host-specific config out of pipe files:\n\n```yaml\nwith:\n  db_host: $DB_HOST\n  bucket: s3://${S3_BUCKET}/output\n```\n\nUnset variables expand to empty string. Works in `with` and `each`, not in `loop`.\n\n**Templates.** Use `{key}` to reference other parameters. Substitution pulls from the full parameter pool — `with`, `each`, `loop`, and overrides:\n\n```yaml\nwith:\n  dest: /data/output\n\neach:\n  - { label: fullhd, size: 1920x1080 }\n\nsteps:\n  - job: scripts/resize.py\n    with:\n      output: {dest}/{label}.jpg  # → /data/output/fullhd.jpg\n```\n\n## Time expressions\n\npiperig recognizes time expressions in `with`, `loop`, and `each` values and resolves them before passing to jobs:\n\n| Expression | Result | Meaning |\n|---|---|---|\n| `-1d` | `2026-03-20` | yesterday |\n| `0d` | `2026-03-21` | today |\n| `-2h` | `2026-03-21T09:00:00` | 2 hours ago, rounded to hour |\n| `-30m` | `2026-03-21T11:13:00` | 30 min ago, rounded to minute |\n| `-1w` | `2026-03-16` | last Monday |\n| `-7d..-1d` | 7 values | last 7 days (in `loop`) |\n\nRounding guarantees idempotency — run at 11:15 or 11:59, `-2h` always gives `09:00`.\n\n## Iteration\n\n### `loop` — repeat a step over a range\n\n```yaml\nsteps:\n  - job: scripts/report.py\n    loop:\n      date: -7d..-1d\n```\n\n7 days → 7 calls. Each call gets `date` set to one value from the range.\n\nLoop values: time ranges (`-7d..-1d`), numeric ranges (`1..5`), lists (`[eu, us, asia]`).\n\n### `each` — repeat a step over parameter sets\n\n```yaml\nsteps:\n  - job: scripts/resize.py\n    each:\n      - { size: 1920x1080, label: fullhd }\n      - { size: 1280x720,  label: hd }\n      - { size: 128x128,   label: thumb }\n```\n\n3 sets → 3 calls. Unlike `loop`, `each` lets you pass multiple related params per iteration.\n\n### Pipe-level iteration\n\nWhen all steps should iterate, move `loop`/`each` to the pipe level:\n\n```yaml\nloop:\n  date: -3d..-1d\n\neach:\n  - { label: fullhd }\n  - { label: thumb }\n\nsteps:\n  - job: scripts/download.sh\n  - job: scripts/resize.py\n  - job: scripts/upload.sh\n```\n\nEvery step gets 3 dates x 2 labels = **6 calls**.\n\n### Cartesian product\n\nMultiple keys in `loop` multiply. Add `each` and they multiply again:\n\n```yaml\nloop:\n  date: -3d..-1d\n  region: [eu, us]\n\neach:\n  - { size: 1920x1080, label: fullhd }\n  - { size: 128x128,   label: thumb }\n\nsteps:\n  - job: scripts/resize.py\n```\n\n3 dates x 2 regions x 2 sizes = **12 calls**.\n\n### Disabling per step\n\nWhen iteration is at the pipe level, some steps may not need it. Use `false` to opt out:\n\n```yaml\nloop:\n  date: -2d..-1d\n\neach:\n  - { label: fullhd }\n  - { label: thumb }\n\nsteps:\n  - job: scripts/download.sh        # 2 calls (each: false → only loop)\n    each: false\n\n  - job: scripts/resize.py          # 4 calls (each × loop)\n\n  - job: scripts/upload.sh          # 1 call  (both disabled)\n    each: false\n    loop: false\n```\n\n## Execution control\n\n### Retry\n\n```yaml\nretry: 2                  # pipe-level: all steps get 2 retries\n\nsteps:\n  - job: scripts/upload.sh\n    retry: 3              # override: 3 retries for this step\n    retry_delay: 5s       # pause between attempts (default: 1s)\n  - job: scripts/notify.sh\n    retry: false          # disable inherited retry\n```\n\n### Timeout\n\n```yaml\nsteps:\n  - job: scripts/resize.py\n    timeout: 10m          # killed after 10 minutes\n```\n\n### Allow failure\n\n```yaml\nsteps:\n  - job: scripts/notify.sh\n    allow_failure: true   # pipe continues even if this fails\n```\n\nAll three can be set at **pipe level** (applies to all steps) or **step level** (overrides).\n\n## Nested pipes\n\nWhen `job` points to a `.pipe.yaml`, piperig runs it as a child pipeline:\n\n```yaml\nsteps:\n  - job: scripts/prepare.sh\n  - job: pipes/images.pipe.yaml\n    with:\n      quality: 90           # overrides child's with\n  - job: scripts/cleanup.sh\n```\n\nParent `with` overrides child `with`. The child's own `loop`/`each` work as written.\n\n`loop` and `each` work on nested pipe steps — the child pipe is invoked once per combination:\n\n```yaml\nsteps:\n  - job: pipes/kpi/dau.pipe.yaml\n    each:\n      - { project: ds }\n      - { project: hn2 }\n      - { project: br }\n```\n\n3 projects = 3 invocations of the child pipe, each with a different `project` override.\n\n## Structured output\n\nPrint JSON lines to stdout for structured logs. Declare `log` fields at pipe or step level — piperig extracts them and formats as a table:\n\n```yaml\nlog:\n  - label\n  - file\n  - size\n\nsteps:\n  - job: scripts/resize.py\n```\n\n```python\nprint(json.dumps({\"label\": \"fullhd\", \"file\": \"photo.jpg\", \"size\": \"1920x1080\"}))\n```\n\n```\n09:15:32 → scripts/resize.py\n           ▸ fullhd | photo.jpg | 1920x1080\n           ▸ thumb  | photo.jpg | 128x128\n09:15:32 ✓ scripts/resize.py  0.3s\n```\n\nPlain text output still works and can be mixed freely with JSON.\n\n## Input modes\n\nControl how parameters reach your scripts:\n\n```yaml\ninput: json              # pipe-level default\n\nsteps:\n  - job: scripts/process.py          # json (from pipe)\n  - job: scripts/deploy.sh\n    input: args                      # override: --key value\n  - job: scripts/notify.py\n    input: env                       # override: KEY=value\n```\n\n| Mode | Delivery |\n|---|---|\n| `env` (default) | `SRC=/data python script.py` |\n| `json` | `{\"src\":\"/data\"}` on stdin |\n| `args` | `python script.py --src /data` |\n\n## Scheduling\n\nRun pipes on a schedule with `piperig serve`:\n\n```yaml\n# schedule.yaml\n- name: daily-images\n  cron: \"0 5 * * *\"\n  run:\n    - pipes/daily/\n  with:\n    quality: 80\n\n- name: healthcheck\n  every: 10m\n  run:\n    - pipes/healthcheck.pipe.yaml\n```\n\n```bash\npiperig serve schedule.yaml        # daemon mode\npiperig serve schedule.yaml --now  # run everything once, then exit\n```\n\nEach entry uses `cron` or `every` (not both). `with` overrides pipe parameters — same as CLI `key=value`.\n\n## Interactive picker\n\nRun `piperig run` without arguments to browse all pipes in your project:\n\n```\n  ╭──────────╮\n  │ piperig  │\n  ╰──────────╯\n\n   run   check   ←/→\n\n  ▸ pipes/daily/images.pipe.yaml — Resize images for the last 2 days\n    pipes/daily/reports.pipe.yaml — Weekly sales report\n    pipes/maintenance/backup.pipe.yaml — Database backup\n    pipes/daily/\n    pipes/maintenance/\n\n  ↑/↓ move  •  ←/→ mode  •  type to filter  •  Enter run  •  q quit\n```\n\nType to filter by path. Toggle between **run** and **check** with arrow keys.\n\nPipes with `hidden: true` are excluded from the picker but can still be run directly or used as nested pipes.\n\n## CLI\n\n| Command | Description |\n|---|---|\n| `piperig run \u003cfile\u003e` | Run a pipe |\n| `piperig run \u003cfile\u003e key=value` | Run with overrides |\n| `piperig run \u003cdirectory/\u003e` | Run all pipes in directory |\n| `piperig run` | Interactive picker |\n| `piperig check \u003cfile\u003e` | Show call plan |\n| `piperig check \u003cfile\u003e key=value` | Check with overrides |\n| `piperig serve \u003cschedule.yaml\u003e` | Cron scheduler |\n| `piperig serve \u003cschedule.yaml\u003e --now` | Run schedule once and exit |\n| `piperig run/check/serve --no-color` | Disable colors (for CI/logs) |\n| `piperig list [directory]` | List all pipes |\n| `piperig init` | Create `.piperig.yaml` |\n| `piperig new pipe \u003cname\u003e` | Scaffold a `.pipe.yaml` |\n| `piperig new schedule \u003cname\u003e` | Scaffold a `schedule.yaml` |\n| `piperig version` | Print version |\n\n## Reference\n\n\u003cdetails\u003e\n\u003csummary\u003eDotenv\u003c/summary\u003e\n\nOptional `.env` file in the project root is loaded automatically:\n\n```\nDB_HOST=localhost\nDB_PASSWORD=secret123\nS3_BUCKET=my-bucket\n```\n\nPriority (weakest → strongest): `.env` \u003c system environment \u003c `.piperig.yaml env:` \u003c `with` parameters.\n\n`.env` variables are available for `$VAR` interpolation in `with`, `each`, and `loop`. Does not require `.piperig.yaml`.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eProject config\u003c/summary\u003e\n\nOptional `.piperig.yaml` at the project root:\n\n```yaml\ninterpreters:\n  .py: python3.11\n  .php: php\n  .lua: lua\n\nenv:\n  PYTHONPATH: .\n  NODE_ENV: production\n```\n\n**interpreters** — custom script runners for non-standard extensions. Defaults: `.py` → python, `.sh` → bash, `.js` → node, `.ts` → npx tsx, `.rb` → ruby.\n\n**env** — environment variables added to every subprocess. Useful for `PYTHONPATH`, `NODE_ENV`, API keys, and other runtime config. Config values override system environment.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eExit codes\u003c/summary\u003e\n\n| Code | Meaning |\n|---|---|\n| `0` | success |\n| `1` | pipe failed (non-zero exit, timeout, retries exhausted) |\n| `2` | validation error (bad YAML, missing files, unknown keys) |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eValidation\u003c/summary\u003e\n\npiperig validates **before** execution — no jobs run until everything checks out:\n\n- Unknown YAML keys → error (catches typos like `rerty: 3`)\n- Job files and nested `.pipe.yaml` must exist on disk\n- Extensions must be supported (built-in or `.piperig.yaml`)\n- `loop`/`each` on nested pipe steps — supported, produces multiple invocations\n- `input` must be `env`, `json`, or `args`\n- Time expressions must parse correctly\n- Templates `{key}` must resolve from available parameters\n- `with` values must be scalars (no nested objects or lists)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eParameter priority\u003c/summary\u003e\n\nWeakest → strongest:\n\npipe `with` → `each` item → `loop` value → step `with` → **CLI `key=value`**\n\n\u003c/details\u003e\n\n## Contributing\n\nIssues and pull requests welcome at [github.com/joarhal/piperig](https://github.com/joarhal/piperig).\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoarhal%2Fpiperig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoarhal%2Fpiperig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoarhal%2Fpiperig/lists"}