{"id":51423029,"url":"https://github.com/true-monte-kristo/bulk-post","last_synced_at":"2026-07-05T01:01:51.807Z","repository":{"id":365799979,"uuid":"1264077268","full_name":"true-monte-kristo/bulk-post","owner":"true-monte-kristo","description":"Templated HTTP request runner CLI — fire bulk or parallel requests, one per CSV row, filling {{placeholder}} slots from each row. Supports multi-step workflows with response-chaining variables, bearer/basic auth, retries, and a  live pause/resume terminal UI. Near-stdlib Python cli","archived":false,"fork":false,"pushed_at":"2026-06-30T09:52:33.000Z","size":239,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-30T11:18:44.611Z","etag":null,"topics":["api-testing","bulk-requests","cli","command-line-tool","csv","http","http-client","load-testing","parallel","python","rest-api","templating","workflow"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/bulk-post/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/true-monte-kristo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-09T14:37:17.000Z","updated_at":"2026-06-30T09:52:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/true-monte-kristo/bulk-post","commit_stats":null,"previous_names":["true-monte-kristo/bulk-post"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/true-monte-kristo/bulk-post","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-monte-kristo%2Fbulk-post","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-monte-kristo%2Fbulk-post/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-monte-kristo%2Fbulk-post/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-monte-kristo%2Fbulk-post/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/true-monte-kristo","download_url":"https://codeload.github.com/true-monte-kristo/bulk-post/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/true-monte-kristo%2Fbulk-post/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35140189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-testing","bulk-requests","cli","command-line-tool","csv","http","http-client","load-testing","parallel","python","rest-api","templating","workflow"],"created_at":"2026-07-05T01:01:50.816Z","updated_at":"2026-07-05T01:01:51.786Z","avatar_url":"https://github.com/true-monte-kristo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bulk-post\n\n[![CI](https://github.com/true-monte-kristo/bulk-post/actions/workflows/ci.yml/badge.svg)](https://github.com/true-monte-kristo/bulk-post/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)\n[![Security: report a vulnerability](https://img.shields.io/badge/security-report%20a%20vulnerability-red.svg)](https://github.com/true-monte-kristo/bulk-post/security/advisories/new)\n\nA near-stdlib Python CLI that fires templated HTTP requests driven by CSV data. You define the request — URL, method, body, headers — with `{{placeholder}}` slots, and each CSV row supplies the values that fill them: one request per row, or a multi-step request workflow per row in `--workflow` mode. Supports bearer or basic auth (default: no auth) with automatic 401 re-prompt, a live terminal UI with pause/resume, parallel execution, and a redrive file for failed rows. Third-party dependencies: PyYAML and jsonpath-ng — both always installed, but lazily imported (PyYAML only on the `--workflow` code path, jsonpath-ng only on the workflow-variables code path).\n\n## Requirements\n\n- Python 3.12+\n- [PyYAML](https://pypi.org/project/PyYAML/) — runtime dependency; lazily imported, only exercised in `--workflow` mode\n- [jsonpath-ng](https://pypi.org/project/jsonpath-ng/) — runtime dependency; lazily imported, only exercised when a workflow declares variables\n\n## Installation\n\n### From PyPI\n\nInstall from [PyPI](https://pypi.org/project/bulk-post/):\n\n```bash\nuv tool install bulk-post   # or: pipx install bulk-post  /  pip install bulk-post\nbulk-post --help\n```\n\n### With Homebrew\n\nInstall from the [tap](https://github.com/true-monte-kristo/homebrew-tap):\n\n```bash\nbrew tap true-monte-kristo/tap\nbrew trust true-monte-kristo/tap   # Homebrew 6.x requires trusting third-party taps once\nbrew install bulk-post\nbulk-post --help\n```\n\n### From source\n\nInstall globally from a checkout with [uv](https://docs.astral.sh/uv/):\n\n```bash\nuv tool install .\nuv tool install . --reinstall   # re-install after changing the code\n```\n\nOr run directly without installing (from the repo root):\n\n```bash\npython -m bulk_post --help          # ensure pyyaml and jsonpath-ng are installed (uv run handles this)\n```\n\n## Usage\n\n```\nbulk-post -u \u003curl-template\u003e -c \u003ccsv-file\u003e [options]\n```\n\n### Examples\n\nCancel every invoice in a CSV using DELETE:\n\n```bash\nbulk-post \\\n  -u \"https://api.example.com/invoices/{{id}}/cancel\" \\\n  -c invoices.csv \\\n  -m DELETE\n```\n\nPATCH with a JSON body, 200 ms between requests, verbose output:\n\n```bash\nbulk-post \\\n  -u \"https://api.example.com/invoices/{{id}}/status\" \\\n  -c invoices.csv \\\n  -m PATCH \\\n  -b '{\"status\": \"cancelled\", \"reason\": \"{{reason}}\"}' \\\n  -d 200 \\\n  -v\n```\n\nPOST form-encoded data:\n\n```bash\nbulk-post \\\n  -u \"https://api.example.com/items\" \\\n  -c items.csv \\\n  -m POST \\\n  -b \"id={{id}}\u0026status={{status}}\" \\\n  -C \"application/x-www-form-urlencoded\"\n```\n\nResume after a failure at row 47:\n\n```bash\nbulk-post -u \"https://api.example.com/items/{{id}}\" -c items.csv -o 47\n```\n\n## CLI flags\n\n| Flag | Short | Default | Description |\n|------|-------|---------|-------------|\n| `--url` | `-u` | required* | URL template; `{{col}}` is replaced with the value from that CSV column. *Provide either `--url` or `--workflow` (mutually exclusive) |\n| `--csv` | `-c` | required | Path to the input CSV file |\n| `--method` | `-m` | `POST` | HTTP method (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, …) |\n| `--body` | `-b` | — | Request body; supports `{{col}}` placeholders |\n| `--content-type` | `-C` | `application/json` | `Content-Type` header sent with the request body; ignored when no body is provided. When set to a JSON or XML type, the body template is validated once at startup before any requests are sent — the script exits immediately with an error if the template is structurally invalid |\n| `--auth-type` | `-a` | `none` | Auth method: `bearer`, `basic`, or `none` |\n| `--token` | `-t` | — | Bearer token; used with `--auth-type bearer` (see [Auth](#auth) below) |\n| `--user` | `-U` | — | Basic auth credentials as `user:pass`; used with `--auth-type basic` (see [Auth](#auth) below) |\n| `--delay` | `-d` | `0` | Milliseconds to wait between requests |\n| `--offset` | `-o` | `0` | Skip the first N data rows (useful for resuming after a failure) |\n| `--timeout` | `-T` | `30` | Per-request timeout in seconds |\n| `--redrive-file` | `-r` | `\u003cstem\u003e_failed.csv` | Where to write rows that failed; auto-named from the CSV path if omitted |\n| `--verbose` | `-v` | false | Print URL, request/response headers (Authorization masked), body, status, and timing for every row |\n| `--header` | `-H` | — | Add a custom request header in `Name: value` format; repeatable. Values support `{{col}}` placeholders |\n| `--parallel` | `-p` | false | Process rows concurrently using multiple threads; `--delay` is ignored in this mode |\n| `--concurrency-level` | `-n` | CPU count | Number of worker threads; only used with `--parallel` |\n| `--debug` | `-D` | false | Print worker thread name on each row log line and show a live debug bar with queue depth, active thread count, and ok/fail counters; only meaningful with `--parallel` |\n| `--workflow` | `-w` | — | Path to a workflow YAML file; mutually exclusive with `--url` |\n| `--retry-on` | `-R` | — (off) | Comma-separated HTTP status codes to retry on (e.g. `503,429`); enables retries |\n| `--retry-backoff` | `-B` | `fixed` | `fixed` or `exponential`; requires `--retry-on` |\n| `--max-retries` | `-M` | `5` | Retries after the initial attempt (5 ⇒ up to 6 requests); requires `--retry-on` |\n| `--retry-delay` | `-y` | `200` | Milliseconds; fixed delay, or initial delay for exponential; requires `--retry-on` |\n| `--multiplier` | `-x` | `1.5` | Exponential backoff multiplier; requires `--retry-backoff exponential` |\n| `--max-retry-delay` | `-Y` | `30000` | Millisecond hard cap on any single retry wait; requires `--retry-on` |\n| `--version` | `-V` | — | Print version and exit |\n\n## CSV format\n\nThe CSV must have a header row. Column names are used as placeholder names in `--url`, `--body`, and `--header` values. Every `{{placeholder}}` in the URL, body, or header values must match a column name; the script exits with an error if any are missing.\n\nThe input delimiter is detected automatically (comma, semicolon, tab, or pipe), falling back to comma when it can't be determined. The failed-rows redrive CSV is written with the same delimiter as the input. There is no delimiter flag.\n\n```csv\nid,reason\n1001,duplicate\n1002,customer_request\n```\n\n## Auth\n\nSelect the auth method with `--auth-type` / `-a` (default: `none`):\n\n### Bearer token\n\nPass `--auth-type bearer` (or `-a bearer`). Token resolution order: `--token` / `-t` flag → `BULK_TOKEN` env var → interactive prompt at startup.\n\nIf the server returns **401** mid-run, the script pauses, prompts for a fresh token, and retries the failed row automatically. This is handy when tokens are short-lived SSO bearer tokens that must be copied manually (e.g. from browser DevTools) and cannot be fetched programmatically.\n\n### Basic auth\n\nPass `--auth-type basic` (or `-a basic`). Credentials (`user:pass`) are resolved in the same order: `--user` / `-U` flag → `BULK_USER` env var → interactive prompt. On **401**, the script prompts for new credentials and retries.\n\n### No auth (default)\n\nThe default when `--auth-type` is omitted (or pass `--auth-type none` / `-a none` explicitly). No `Authorization` header is sent.\n\n## Retries\n\nRetries are disabled by default. Pass `--retry-on` with a comma-separated list of HTTP status codes to enable them:\n\n```bash\nbulk-post -u \"https://api.example.com/{{id}}\" -c rows.csv -R 503,429 -B exponential\n```\n\nOnly the exact listed status codes trigger a retry — network errors and timeouts do not. If the server returns a `Retry-After` or `X-Retry-After` header, the wait is extended to at least that value before the next attempt. All waits are capped by `--max-retry-delay`. Each retry prints a short `[RETRY]` notice; `--verbose` adds the elapsed time and body snippet of the failed attempt.\n\n**Flags:**\n\n| Flag | Short | Default | Description |\n|------|-------|---------|-------------|\n| `--retry-on` | `-R` | — | Comma-separated status codes (e.g. `503,429`); required to enable retries |\n| `--retry-backoff` | `-B` | `fixed` | `fixed` or `exponential` |\n| `--max-retries` | `-M` | `5` | Retries after the first attempt (5 ⇒ up to 6 total requests) |\n| `--retry-delay` | `-y` | `200` | ms; fixed wait between retries, or starting delay for exponential |\n| `--multiplier` | `-x` | `1.5` | Growth factor for exponential backoff |\n| `--max-retry-delay` | `-Y` | `30000` | ms hard cap on any single retry wait |\n\n**Notes:**\n\n- `401` cannot be listed in `--retry-on` when `--auth-type` is `bearer` or `basic` — the 401 auth-refresh flow owns that status code.\n- Retry sleeps are interruptible: `/pause` freezes the countdown, `/exit` abandons the wait and routes the row to the redrive file.\n- In workflow mode, retry flags are not available; use `retry_policy:` in the workflow YAML instead (see [Workflow mode](#workflow-mode)).\n\n## Terminal UI\n\nWhen running in an interactive terminal, a live bottom bar shows:\n\n- **Progress bar** — `current / total` rows with a visual fill bar\n- **Command input** — type a command and press Enter; Tab autocompletes\n\nAvailable commands:\n\n| Command | Effect |\n|---------|--------|\n| `/pause` | Pause sending; script waits until you resume |\n| `/resume` | Resume after a pause |\n| `/exit` | Stop after the current row and print a summary |\n\nIn non-TTY mode (piped input, CI, test environments) the bottom bar is skipped and no interactive commands are available.\n\n## Redrive file\n\nRows that fail (network error, non-2xx response, or substitution error) are written to the redrive file. By default this is `\u003ccsv-stem\u003e_failed.csv` next to the input file. Re-run with `-c \u003cstem\u003e_failed.csv` to redrive only those rows.\n\nIf no rows fail, the redrive file is deleted automatically.\n\n## Workflow mode\n\nInstead of a single `--url` template, you can define a multi-step workflow in a YAML file and run it with `--workflow` / `-w`.\n\nEach CSV row fires all steps in document order. Steps within a row are always sequential; `--parallel` controls per-row concurrency across rows.\n\n### Workflow YAML format\n\n```yaml\nworkflow:\n  persist_context: false           # optional; default false — see \"Redrive and resume\" below\n  description: Optional human-readable description  # skipped at runtime\n\n  groupA:                          # logical grouping for shared auth\n    auth:\n      type: bearer                 # bearer | basic | none\n      token: some_token            # optional — prompted if omitted\n    endpoints:\n      - step-name:                 # user-chosen name; unique within the group\n          url: https://api.example.com/{{id}}\n          method: POST             # default POST\n          headers:\n            Content-Type: application/json\n            X-Custom: value\n          body: '{\"key\": \"{{col}}\"}'\n          on_error: stop           # stop (default) | continue\n          auth:                    # step-level auth overrides group auth\n            type: bearer\n            token: override_token\n\n  groupB:\n    auth:\n      type: basic\n      user: alice\n      password: secret\n    endpoints:\n      - another-step:\n          url: https://other.example.com/{{id}}\n          method: DELETE\n\n  groupC:                          # no auth\n    endpoints:\n      - no-auth-step:\n          url: https://public.example.com/{{id}}\n          method: GET\n```\n\nKey rules:\n\n- **Execution order** — steps fire in the order they appear in the document (top to bottom across all groups).\n- **Group auth** — all steps in a group inherit the group's auth unless they declare their own.\n- **`on_error`** — `stop` (default) halts remaining steps for that row and writes it to the redrive file; `continue` logs the failure, writes the row, and proceeds to the next step.\n- **Placeholders** — `{{col}}` in `url`, `body`, and header values is replaced with the matching CSV column value, same as in single-URL mode.\n\n### Workflow retries\n\nEach group and each endpoint may carry an optional `retry_policy:` block. An endpoint-level `retry_policy` **replaces** the group's entirely — there is no merging. `retry_policy: none` on an endpoint disables any inherited group policy. `on_error` evaluates only the final post-retries result.\n\n```yaml\nworkflow:\n  groupA:\n    auth:\n      type: bearer\n    retry_policy:           # group-level policy, inherited by all endpoints in the group\n      retry_on: 503,429     # required: comma-separated string or YAML list\n      backoff: exponential  # optional: fixed (default) or exponential\n      multiplier: 1.5       # optional: exponential only; defaults to 1.5\n      delay: 200            # optional: fixed delay / initial delay in ms; defaults to 200\n      max_retries: 5        # optional: retries after the initial attempt; defaults to 5\n      max_delay: 30000      # optional: hard cap on any single retry wait in ms; defaults to 30000\n    endpoints:\n      - call-api:\n          url: https://api.example.com/{{id}}\n          method: POST\n\n  groupB:\n    endpoints:\n      - call-other:\n          url: https://other.example.com/{{id}}\n          method: DELETE\n          retry_policy:     # endpoint-level policy replaces any group policy entirely\n            retry_on: 503,429\n            backoff: fixed\n            delay: 200\n            max_retries: 5\n\n  groupC:\n    retry_policy:           # group has a policy …\n      retry_on: 503\n    endpoints:\n      - no-retry-step:\n          retry_policy: none  # … but this endpoint opts out\n          url: https://public.example.com/{{id}}\n          method: GET\n```\n\n### Redrive and resume\n\nMid-workflow resume is opt-in via a `persist_context: true` key at the top of the `workflow:` mapping (default `false`).\n\n- **`persist_context: false` (default):** failed rows are written to the redrive file with only the original input columns. Re-running always starts each row from step 1. If the input CSV contains `_bulk_post_step` or `_bulk_post_var/…` context columns (e.g. from a previous run with `persist_context: true`), they are ignored, with a startup warning printed to stderr.\n\n- **`persist_context: true`:** when a step fails, the row is written to the redrive file with an extra column `_bulk_post_step` set to the path of the first failed step (e.g. `groupA/step-name`) plus any persisted variable columns. Re-running with that redrive CSV skips all steps before the failed one, resuming mid-workflow automatically.\n\n\u003e **Security note:** `persist_context: true` writes response-derived data (potentially sensitive) to disk in plaintext. Do not share or commit redrive CSVs produced with this option enabled.\n\n### Workflow variables\n\nA step can capture a value from an earlier step's JSON response and pass it as a `{{$name}}` placeholder into that step's `url`, `headers`, or `body`. This lets you chain steps — for example, create a resource in step 1 and use the returned `id` in the URL of step 2.\n\nVariables are declared under a `variables:` key at group level (inherited by all endpoints in the group) and/or at endpoint level (overrides the group on name conflict):\n\n```yaml\nworkflow:\n  groupA:\n    auth:\n      type: bearer\n    endpoints:\n      - create-item:\n          url: https://api.example.com/items\n          method: POST\n          body: '{\"name\": \"{{name}}\"}'\n\n  groupB:\n    endpoints:\n      - delete-item:\n          # Capture the id from create-item's response at endpoint level\n          variables:\n            $id:\n              source: .workflow.groupA.create-item  # leading dot and \"workflow.\" prefix are optional\n              jsonPath: $.id                        # JSONPath; first match is used\n              nullable: false                       # fail this step if id is missing\n          url: https://api.example.com/items/{{$id}}\n          method: DELETE\n```\n\n**Variable rules:**\n\n- Names must start with `$` (e.g. `$id`) and are referenced as `{{$id}}` in URL, headers, and body.\n- `source` is written as `.workflow.\u003cgroup\u003e.\u003cendpoint\u003e` (or just `\u003cgroup\u003e/\u003cendpoint\u003e`). It must refer to an endpoint that runs before the current step — forward and self references are rejected at startup.\n- `jsonPath` uses full JSONPath syntax (powered by [`jsonpath-ng`](https://pypi.org/project/jsonpath-ng/)). Only the first match is used. A match that is an object or array (non-scalar) fails the step.\n- `nullable` defaults to `true`. When `false`, a null value or no-match fails the step (row written to redrive file); when `true`, it resolves to an empty string.\n- Variable values are scoped to a single CSV row and are never shared across rows.\n- All variable declarations are validated at startup — undefined references, bad names, unreachable sources, and invalid JSONPath expressions all cause an immediate exit with a clear error.\n\n**Resume/redrive with variables:**\n\nRequires `persist_context: true` at the top of the `workflow:` mapping. When enabled, on row failure any resolved variable values are persisted into reserved redrive-CSV columns named `_bulk_post_var/\u003csource_path\u003e/\u003cname\u003e`. Re-running the redrive CSV skips completed steps and reads these persisted values for variables whose source step was skipped. When `persist_context` is `false` (the default), variable values are not persisted and re-runs always start from step 1.\n\n\u003e **Security note:** redrive CSVs may contain response-derived data (potentially sensitive) in plaintext. Do not share or commit redrive CSVs that were produced from workflows using variables.\n\n### Example\n\n```bash\nbulk-post -w workflow.yaml -c rows.csv\n```\n\n## Running tests\n\n```bash\nuv run python -m unittest discover tests/\n```\n\nTests use stdlib `unittest`. PyYAML and jsonpath-ng are runtime dependencies (always installed); the workflow-parsing cases use PyYAML and the workflow-variable cases use jsonpath-ng. `uv run` provides both from `uv.lock` automatically; if you run `python -m unittest` directly, do so inside a virtualenv that has both packages.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrue-monte-kristo%2Fbulk-post","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrue-monte-kristo%2Fbulk-post","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrue-monte-kristo%2Fbulk-post/lists"}