Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/true-monte-kristo/bulk-post

Templated HTTP request runner CLI — fire bulk or parallel requests, one per CSV row, filling {{placeholder}} slots from each row. Supports multi-step workflows with response-chaining variables, bearer/basic auth, retries, and a live pause/resume terminal UI. Near-stdlib Python cli
https://github.com/true-monte-kristo/bulk-post

api-testing bulk-requests cli command-line-tool csv http http-client load-testing parallel python rest-api templating workflow

Last synced: about 2 hours ago
JSON representation

Templated HTTP request runner CLI — fire bulk or parallel requests, one per CSV row, filling {{placeholder}} slots from each row. Supports multi-step workflows with response-chaining variables, bearer/basic auth, retries, and a live pause/resume terminal UI. Near-stdlib Python cli

Awesome Lists containing this project

README 

          
# bulk-post



[![CI](https://github.com/true-monte-kristo/bulk-post/actions/workflows/ci.yml/badge.svg)](https://github.com/true-monte-kristo/bulk-post/actions/workflows/ci.yml)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)

[![Security: report a vulnerability](https://img.shields.io/badge/security-report%20a%20vulnerability-red.svg)](https://github.com/true-monte-kristo/bulk-post/security/advisories/new)



A near-stdlib Python CLI that fires templated HTTP requests driven by CSV data. You define the request — URL, method, body, headers — with `{{placeholder}}` slots, and each CSV row supplies the values that fill them: one request per row, or a multi-step request workflow per row in `--workflow` mode. Supports bearer or basic auth (default: no auth) with automatic 401 re-prompt, a live terminal UI with pause/resume, parallel execution, and a redrive file for failed rows. Third-party dependencies: PyYAML and jsonpath-ng — both always installed, but lazily imported (PyYAML only on the `--workflow` code path, jsonpath-ng only on the workflow-variables code path).



## Requirements



- Python 3.12+

- [PyYAML](https://pypi.org/project/PyYAML/) — runtime dependency; lazily imported, only exercised in `--workflow` mode

- [jsonpath-ng](https://pypi.org/project/jsonpath-ng/) — runtime dependency; lazily imported, only exercised when a workflow declares variables



## Installation



### From PyPI



Install from [PyPI](https://pypi.org/project/bulk-post/):



```bash

uv tool install bulk-post   # or: pipx install bulk-post  /  pip install bulk-post

bulk-post --help

```



### With Homebrew



Install from the [tap](https://github.com/true-monte-kristo/homebrew-tap):



```bash

brew tap true-monte-kristo/tap

brew trust true-monte-kristo/tap   # Homebrew 6.x requires trusting third-party taps once

brew install bulk-post

bulk-post --help

```



### From source



Install globally from a checkout with [uv](https://docs.astral.sh/uv/):



```bash

uv tool install .

uv tool install . --reinstall   # re-install after changing the code

```



Or run directly without installing (from the repo root):



```bash

python -m bulk_post --help          # ensure pyyaml and jsonpath-ng are installed (uv run handles this)

```



## Usage



```

bulk-post -u  -c  [options]

```



### Examples



Cancel every invoice in a CSV using DELETE:



```bash

bulk-post \

  -u "https://api.example.com/invoices/{{id}}/cancel" \

  -c invoices.csv \

  -m DELETE

```



PATCH with a JSON body, 200 ms between requests, verbose output:



```bash

bulk-post \

  -u "https://api.example.com/invoices/{{id}}/status" \

  -c invoices.csv \

  -m PATCH \

  -b '{"status": "cancelled", "reason": "{{reason}}"}' \

  -d 200 \

  -v

```



POST form-encoded data:



```bash

bulk-post \

  -u "https://api.example.com/items" \

  -c items.csv \

  -m POST \

  -b "id={{id}}&status={{status}}" \

  -C "application/x-www-form-urlencoded"

```



Resume after a failure at row 47:



```bash

bulk-post -u "https://api.example.com/items/{{id}}" -c items.csv -o 47

```



## CLI flags



| Flag | Short | Default | Description |

|------|-------|---------|-------------|

| `--url` | `-u` | required* | URL template; `{{col}}` is replaced with the value from that CSV column. *Provide either `--url` or `--workflow` (mutually exclusive) |

| `--csv` | `-c` | required | Path to the input CSV file |

| `--method` | `-m` | `POST` | HTTP method (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, …) |

| `--body` | `-b` | — | Request body; supports `{{col}}` placeholders |

| `--content-type` | `-C` | `application/json` | `Content-Type` header sent with the request body; ignored when no body is provided. When set to a JSON or XML type, the body template is validated once at startup before any requests are sent — the script exits immediately with an error if the template is structurally invalid |

| `--auth-type` | `-a` | `none` | Auth method: `bearer`, `basic`, or `none` |

| `--token` | `-t` | — | Bearer token; used with `--auth-type bearer` (see [Auth](#auth) below) |

| `--user` | `-U` | — | Basic auth credentials as `user:pass`; used with `--auth-type basic` (see [Auth](#auth) below) |

| `--delay` | `-d` | `0` | Milliseconds to wait between requests |

| `--offset` | `-o` | `0` | Skip the first N data rows (useful for resuming after a failure) |

| `--timeout` | `-T` | `30` | Per-request timeout in seconds |

| `--redrive-file` | `-r` | `_failed.csv` | Where to write rows that failed; auto-named from the CSV path if omitted |

| `--verbose` | `-v` | false | Print URL, request/response headers (Authorization masked), body, status, and timing for every row |

| `--header` | `-H` | — | Add a custom request header in `Name: value` format; repeatable. Values support `{{col}}` placeholders |

| `--parallel` | `-p` | false | Process rows concurrently using multiple threads; `--delay` is ignored in this mode |

| `--concurrency-level` | `-n` | CPU count | Number of worker threads; only used with `--parallel` |

| `--debug` | `-D` | false | Print worker thread name on each row log line and show a live debug bar with queue depth, active thread count, and ok/fail counters; only meaningful with `--parallel` |

| `--workflow` | `-w` | — | Path to a workflow YAML file; mutually exclusive with `--url` |

| `--retry-on` | `-R` | — (off) | Comma-separated HTTP status codes to retry on (e.g. `503,429`); enables retries |

| `--retry-backoff` | `-B` | `fixed` | `fixed` or `exponential`; requires `--retry-on` |

| `--max-retries` | `-M` | `5` | Retries after the initial attempt (5 ⇒ up to 6 requests); requires `--retry-on` |

| `--retry-delay` | `-y` | `200` | Milliseconds; fixed delay, or initial delay for exponential; requires `--retry-on` |

| `--multiplier` | `-x` | `1.5` | Exponential backoff multiplier; requires `--retry-backoff exponential` |

| `--max-retry-delay` | `-Y` | `30000` | Millisecond hard cap on any single retry wait; requires `--retry-on` |

| `--version` | `-V` | — | Print version and exit |



## CSV format



The CSV must have a header row. Column names are used as placeholder names in `--url`, `--body`, and `--header` values. Every `{{placeholder}}` in the URL, body, or header values must match a column name; the script exits with an error if any are missing.



The input delimiter is detected automatically (comma, semicolon, tab, or pipe), falling back to comma when it can't be determined. The failed-rows redrive CSV is written with the same delimiter as the input. There is no delimiter flag.



```csv

id,reason

1001,duplicate

1002,customer_request

```



## Auth



Select the auth method with `--auth-type` / `-a` (default: `none`):



### Bearer token



Pass `--auth-type bearer` (or `-a bearer`). Token resolution order: `--token` / `-t` flag → `BULK_TOKEN` env var → interactive prompt at startup.



If the server returns **401** mid-run, the script pauses, prompts for a fresh token, and retries the failed row automatically. This is handy when tokens are short-lived SSO bearer tokens that must be copied manually (e.g. from browser DevTools) and cannot be fetched programmatically.



### Basic auth



Pass `--auth-type basic` (or `-a basic`). Credentials (`user:pass`) are resolved in the same order: `--user` / `-U` flag → `BULK_USER` env var → interactive prompt. On **401**, the script prompts for new credentials and retries.



### No auth (default)



The default when `--auth-type` is omitted (or pass `--auth-type none` / `-a none` explicitly). No `Authorization` header is sent.



## Retries



Retries are disabled by default. Pass `--retry-on` with a comma-separated list of HTTP status codes to enable them:



```bash

bulk-post -u "https://api.example.com/{{id}}" -c rows.csv -R 503,429 -B exponential

```



Only the exact listed status codes trigger a retry — network errors and timeouts do not. If the server returns a `Retry-After` or `X-Retry-After` header, the wait is extended to at least that value before the next attempt. All waits are capped by `--max-retry-delay`. Each retry prints a short `[RETRY]` notice; `--verbose` adds the elapsed time and body snippet of the failed attempt.



**Flags:**



| Flag | Short | Default | Description |

|------|-------|---------|-------------|

| `--retry-on` | `-R` | — | Comma-separated status codes (e.g. `503,429`); required to enable retries |

| `--retry-backoff` | `-B` | `fixed` | `fixed` or `exponential` |

| `--max-retries` | `-M` | `5` | Retries after the first attempt (5 ⇒ up to 6 total requests) |

| `--retry-delay` | `-y` | `200` | ms; fixed wait between retries, or starting delay for exponential |

| `--multiplier` | `-x` | `1.5` | Growth factor for exponential backoff |

| `--max-retry-delay` | `-Y` | `30000` | ms hard cap on any single retry wait |



**Notes:**



- `401` cannot be listed in `--retry-on` when `--auth-type` is `bearer` or `basic` — the 401 auth-refresh flow owns that status code.

- Retry sleeps are interruptible: `/pause` freezes the countdown, `/exit` abandons the wait and routes the row to the redrive file.

- In workflow mode, retry flags are not available; use `retry_policy:` in the workflow YAML instead (see [Workflow mode](#workflow-mode)).



## Terminal UI



When running in an interactive terminal, a live bottom bar shows:



- **Progress bar** — `current / total` rows with a visual fill bar

- **Command input** — type a command and press Enter; Tab autocompletes



Available commands:



| Command | Effect |

|---------|--------|

| `/pause` | Pause sending; script waits until you resume |

| `/resume` | Resume after a pause |

| `/exit` | Stop after the current row and print a summary |



In non-TTY mode (piped input, CI, test environments) the bottom bar is skipped and no interactive commands are available.



## Redrive file



Rows that fail (network error, non-2xx response, or substitution error) are written to the redrive file. By default this is `_failed.csv` next to the input file. Re-run with `-c _failed.csv` to redrive only those rows.



If no rows fail, the redrive file is deleted automatically.



## Workflow mode



Instead of a single `--url` template, you can define a multi-step workflow in a YAML file and run it with `--workflow` / `-w`.



Each CSV row fires all steps in document order. Steps within a row are always sequential; `--parallel` controls per-row concurrency across rows.



### Workflow YAML format



```yaml

workflow:

  persist_context: false           # optional; default false — see "Redrive and resume" below

  description: Optional human-readable description  # skipped at runtime



  groupA:                          # logical grouping for shared auth

    auth:

      type: bearer                 # bearer | basic | none

      token: some_token            # optional — prompted if omitted

    endpoints:

      - step-name:                 # user-chosen name; unique within the group

          url: https://api.example.com/{{id}}

          method: POST             # default POST

          headers:

            Content-Type: application/json

            X-Custom: value

          body: '{"key": "{{col}}"}'

          on_error: stop           # stop (default) | continue

          auth:                    # step-level auth overrides group auth

            type: bearer

            token: override_token



  groupB:

    auth:

      type: basic

      user: alice

      password: secret

    endpoints:

      - another-step:

          url: https://other.example.com/{{id}}

          method: DELETE



  groupC:                          # no auth

    endpoints:

      - no-auth-step:

          url: https://public.example.com/{{id}}

          method: GET

```



Key rules:



- **Execution order** — steps fire in the order they appear in the document (top to bottom across all groups).

- **Group auth** — all steps in a group inherit the group's auth unless they declare their own.

- **`on_error`** — `stop` (default) halts remaining steps for that row and writes it to the redrive file; `continue` logs the failure, writes the row, and proceeds to the next step.

- **Placeholders** — `{{col}}` in `url`, `body`, and header values is replaced with the matching CSV column value, same as in single-URL mode.



### Workflow retries



Each group and each endpoint may carry an optional `retry_policy:` block. An endpoint-level `retry_policy` **replaces** the group's entirely — there is no merging. `retry_policy: none` on an endpoint disables any inherited group policy. `on_error` evaluates only the final post-retries result.



```yaml

workflow:

  groupA:

    auth:

      type: bearer

    retry_policy:           # group-level policy, inherited by all endpoints in the group

      retry_on: 503,429     # required: comma-separated string or YAML list

      backoff: exponential  # optional: fixed (default) or exponential

      multiplier: 1.5       # optional: exponential only; defaults to 1.5

      delay: 200            # optional: fixed delay / initial delay in ms; defaults to 200

      max_retries: 5        # optional: retries after the initial attempt; defaults to 5

      max_delay: 30000      # optional: hard cap on any single retry wait in ms; defaults to 30000

    endpoints:

      - call-api:

          url: https://api.example.com/{{id}}

          method: POST



  groupB:

    endpoints:

      - call-other:

          url: https://other.example.com/{{id}}

          method: DELETE

          retry_policy:     # endpoint-level policy replaces any group policy entirely

            retry_on: 503,429

            backoff: fixed

            delay: 200

            max_retries: 5



  groupC:

    retry_policy:           # group has a policy …

      retry_on: 503

    endpoints:

      - no-retry-step:

          retry_policy: none  # … but this endpoint opts out

          url: https://public.example.com/{{id}}

          method: GET

```



### Redrive and resume



Mid-workflow resume is opt-in via a `persist_context: true` key at the top of the `workflow:` mapping (default `false`).



- **`persist_context: false` (default):** failed rows are written to the redrive file with only the original input columns. Re-running always starts each row from step 1. If the input CSV contains `_bulk_post_step` or `_bulk_post_var/…` context columns (e.g. from a previous run with `persist_context: true`), they are ignored, with a startup warning printed to stderr.



- **`persist_context: true`:** when a step fails, the row is written to the redrive file with an extra column `_bulk_post_step` set to the path of the first failed step (e.g. `groupA/step-name`) plus any persisted variable columns. Re-running with that redrive CSV skips all steps before the failed one, resuming mid-workflow automatically.



> **Security note:** `persist_context: true` writes response-derived data (potentially sensitive) to disk in plaintext. Do not share or commit redrive CSVs produced with this option enabled.



### Workflow variables



A step can capture a value from an earlier step's JSON response and pass it as a `{{$name}}` placeholder into that step's `url`, `headers`, or `body`. This lets you chain steps — for example, create a resource in step 1 and use the returned `id` in the URL of step 2.



Variables are declared under a `variables:` key at group level (inherited by all endpoints in the group) and/or at endpoint level (overrides the group on name conflict):



```yaml

workflow:

  groupA:

    auth:

      type: bearer

    endpoints:

      - create-item:

          url: https://api.example.com/items

          method: POST

          body: '{"name": "{{name}}"}'



  groupB:

    endpoints:

      - delete-item:

          # Capture the id from create-item's response at endpoint level

          variables:

            $id:

              source: .workflow.groupA.create-item  # leading dot and "workflow." prefix are optional

              jsonPath: $.id                        # JSONPath; first match is used

              nullable: false                       # fail this step if id is missing

          url: https://api.example.com/items/{{$id}}

          method: DELETE

```



**Variable rules:**



- Names must start with `$` (e.g. `$id`) and are referenced as `{{$id}}` in URL, headers, and body.

- `source` is written as `.workflow..` (or just `/`). It must refer to an endpoint that runs before the current step — forward and self references are rejected at startup.

- `jsonPath` uses full JSONPath syntax (powered by [`jsonpath-ng`](https://pypi.org/project/jsonpath-ng/)). Only the first match is used. A match that is an object or array (non-scalar) fails the step.

- `nullable` defaults to `true`. When `false`, a null value or no-match fails the step (row written to redrive file); when `true`, it resolves to an empty string.

- Variable values are scoped to a single CSV row and are never shared across rows.

- All variable declarations are validated at startup — undefined references, bad names, unreachable sources, and invalid JSONPath expressions all cause an immediate exit with a clear error.



**Resume/redrive with variables:**



Requires `persist_context: true` at the top of the `workflow:` mapping. When enabled, on row failure any resolved variable values are persisted into reserved redrive-CSV columns named `_bulk_post_var//`. Re-running the redrive CSV skips completed steps and reads these persisted values for variables whose source step was skipped. When `persist_context` is `false` (the default), variable values are not persisted and re-runs always start from step 1.



> **Security note:** redrive CSVs may contain response-derived data (potentially sensitive) in plaintext. Do not share or commit redrive CSVs that were produced from workflows using variables.



### Example



```bash

bulk-post -w workflow.yaml -c rows.csv

```



## Running tests



```bash

uv run python -m unittest discover tests/

```



Tests use stdlib `unittest`. PyYAML and jsonpath-ng are runtime dependencies (always installed); the workflow-parsing cases use PyYAML and the workflow-variable cases use jsonpath-ng. `uv run` provides both from `uv.lock` automatically; if you run `python -m unittest` directly, do so inside a virtualenv that has both packages.