{"id":50325092,"url":"https://github.com/dunnock/pqls","last_synced_at":"2026-05-29T05:30:24.400Z","repository":{"id":359672142,"uuid":"1246684769","full_name":"dunnock/pqls","owner":"dunnock","description":"CLI for listing parquet tables ","archived":false,"fork":false,"pushed_at":"2026-05-22T21:39:09.000Z","size":149,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-22T23:31:44.184Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dunnock.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-22T12:52:27.000Z","updated_at":"2026-05-22T21:39:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dunnock/pqls","commit_stats":null,"previous_names":["dunnock/pqls"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dunnock/pqls","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dunnock%2Fpqls","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dunnock%2Fpqls/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dunnock%2Fpqls/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dunnock%2Fpqls/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dunnock","download_url":"https://codeload.github.com/dunnock/pqls/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dunnock%2Fpqls/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33639050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T05:30:23.706Z","updated_at":"2026-05-29T05:30:24.394Z","avatar_url":"https://github.com/dunnock.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pqls\n\n![Release](https://github.com/dunnock/pqls/actions/workflows/release.yml/badge.svg)\n![Latest Release](https://img.shields.io/github/v/release/dunnock/pqls)\n![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue)\n\nA command-line tool for listing the contents and metadata of Apache Parquet files and partitioned parquet datasets, modelled on HDF5's `h5ls`.\n\n## Install\n\n```sh\ncurl -fsSL https://github.com/dunnock/pqls/releases/latest/download/install.sh | sh\n```\n\n### Or install with cargo:\n\n```sh\ncargo install pqls\n```\n\n## Examples\n\n**Inspect a single file:**\n```sh\npqls data.parquet\n```\n\n**Detailed stats (per-column min/max/nulls):**\n```sh\npqls -d data.parquet\n```\n\n**Dump as CSV:**\n```sh\npqls --csv data.parquet\npqls --csv --head 100 data.parquet\n```\n\n**List a partitioned dataset:**\n```sh\npqls /path/to/dataset/\npqls -d -r /path/to/dataset/\n```\n\n**Machine-readable output:**\n```sh\npqls -q data.parquet\n```\n\n## CLI\n\n```\npqls [OPTIONS] \u003cPATH\u003e [PATH_B]\n\nARGS:\n  \u003cPATH\u003e            path to a .parquet file or directory to inspect\n  [PATH_B]          second .parquet file for schema diff (required by --diff)\n\nOPTIONS:\n      --diff                compare schemas of two files; exits 0 if identical, 1 if different\n  -d, --detail              show per-row-group column statistics (min/max/nulls)\n  -r, --recursive           recurse into a directory and list all .parquet files\n      --csv                 dump rows as CSV to stdout\n      --head \u003cN\u003e            limit output to the first N rows (applies to --csv and --ndjson)\n  -q, --quiet               suppress human-readable headers; emit tab-separated summary lines\n      --schema              print schema only (column names and types)\n      --json                emit output as JSON (works with --schema, --kv-meta, --check, --partition-stats, --diff)\n      --ndjson              stream rows as newline-delimited JSON (NDJSON)\n      --sample \u003cN\u003e          emit N randomly-sampled rows; requires --ndjson or --csv\n      --columns \u003cCOLS\u003e      comma-separated list of column names to project (e.g. id,ts,value)\n      --kv-meta             print Parquet key-value metadata (writer version, custom properties)\n      --scan-stats          scan the full file to compute per-column min/max/nulls/n_distinct; requires -d\n      --partition-stats     aggregate row counts and file sizes across a Hive-partitioned directory; requires -r\n      --check               verify file integrity by reading the footer and all row groups\n      --deep                with --check: read every data page (slower but catches corrupt column data)\n  -h, --help                print help\n  -V, --version             print version\n```\n\n## Why pqls?\n\n**Static binary.** No JVM, no Python interpreter, no `pip install`. Drop the binary on\nany Linux box and it runs — sub-100ms startup on the critical path of a data pipeline.\n\n**Composable.** Stdout is always clean (data only; warnings go to stderr). Pipe anywhere:\n\n```sh\npqls --csv file.parquet | xsv stats\npqls --schema file.parquet | diff - expected.schema\n```\n\n**Agent-friendly.** Machine-readable `--schema --json` and `--ndjson` output let code\nagents inspect schema and rows without parsing human text. See `SKILL.md` for patterns.\n\n**One-liner install:**\n\n```sh\ncurl -fsSL https://github.com/dunnock/pqls/releases/latest/download/install.sh | sh\n```\n\n**Fast:**\n\n| Tool | Runtime | Startup | Schema dump | Stats | Pipe-composable |\n|------|---------|---------|-------------|-------|-----------------|\n| **pqls** | none (static) | ~50ms | `--schema --json` | `--scan-stats` | yes |\n| parquet-tools | JVM | ~2s | text only | yes | no |\n| DuckDB | Go binary | ~200ms | SQL only | SQL | no |\n| fastparquet | Python | ~500ms | Python API | Python API | no |\n\n## How pqls compares\n\n| | pqls | parquet-cli (Apache) | pqrs | DuckDB |\n|---|---|---|---|---|\n| Static binary, no JVM/Python | **yes** | no (JAR) | yes | yes |\n| `--schema --json` for agents | **yes** | no (text only) | no | via SQL |\n| NDJSON rows (`--ndjson`) | **yes** | no | cat -f json | via SQL |\n| Column projection (`--columns`) | **yes** | yes | no | via SQL |\n| Random sampling (`--sample N`) | **yes** | no | yes | ORDER BY random() |\n| Key-value metadata (`--kv-meta`) | **yes** | footer cmd | no | parquet_kv_metadata() |\n| Directory / partition listing | **yes** | no | no | no |\n| SKILL.md for code agents | **yes** | no | no | no |\n| Composable (stdin/stdout clean) | **yes** | no | partial | no |\n\npqls is the only static binary in this list that produces JSON schema output and\nNDJSON rows without requiring SQL. It is designed for shell pipelines and agent\ntooling where DuckDB's startup time or SQL syntax is overhead.\n\n## Agent usage\n\npqls is designed to be called by code agents (Claude, Codex, Cursor, etc.)\nwithout any human at the terminal.\n\n### Discover schema\n\n```shell\npqls --schema --json /path/to/foo.parquet\n```\n\nReturns a JSON object — safe to parse with `jq` or Python `json.loads`.\nField `logical_type` tells you `DATE`, `TIMESTAMP_MICROS`, `DECIMAL(10,2)`, etc.\n\n### Sample rows to understand data\n\n```shell\npqls --ndjson --sample 50 foo.parquet\n```\n\n50 rows, one JSON object per line. Pipe to `jq` for field inspection.\n\n### Project specific columns\n\n```shell\npqls --ndjson --columns user_id,amount --sample 20 foo.parquet\n```\n\n### Check embedded metadata (Spark / Pandas schema)\n\n```shell\npqls --kv-meta --json foo.parquet | jq '.[\"pandas\"]'\n```\n\n### Composable pipeline example\n\n```shell\n# Find which files in a partitioned dataset have more than 1M rows\npqls -q --recursive /data/events/ \\\n  | awk -F'\\t' '$2 \u003e 1000000 { print $1 }'\n```\n\n### Exit code contract\n\nScripts should test `$?`:\n- `0` — success, output on stdout\n- `1` — file/path error or schema mismatch (with --diff)\n- `2` — corrupt or invalid parquet, or bad flag combination\n\n## Releases\n\nPre-built binaries are attached to every [GitHub release](https://github.com/dunnock/pqls/releases):\n\n| Platform | Asset |\n|----------|-------|\n| Linux x86_64 | `pqls-linux-x86_64.tar.gz` |\n| Linux aarch64 | `pqls-linux-aarch64.tar.gz` |\n| macOS Intel (x86_64) | `pqls-darwin-x86_64.tar.gz` |\n| macOS Apple Silicon (aarch64) | `pqls-darwin-aarch64.tar.gz` |\n| Windows x86_64 | `pqls-windows-x86_64.zip` |\n\n**One-liner install (Linux and macOS):**\n```sh\ncurl -fsSL https://github.com/dunnock/pqls/releases/latest/download/install.sh | sh\n```\n\n**Install from crates.io:**\n```sh\ncargo install pqls\n```\nThis compiles from source and works on any platform with a Rust toolchain. No runtime\ndependencies — pqls is a fully static binary on Linux (musl not required; glibc is fine).\n\n**System requirements:** none beyond a standard Linux/macOS/Windows environment.\npqls does not require hugepages, elevated privileges, or any kernel tuning.\n\n**`CRATES_IO_TOKEN` secret (maintainers only):** the release workflow publishes to\ncrates.io automatically on tag push. Add the secret once in the GitHub repo settings\nunder *Settings → Secrets and variables → Actions* with the name `CRATES_IO_TOKEN`.\n\n## S3 paths\n\npqls accepts `s3://bucket/key` and `s3://bucket/prefix/` paths directly.\n\n```sh\n# inspect schema of a single S3 object (no full download)\npqls s3://my-bucket/events/2024/data.parquet\n\n# JSON schema for agents / pipelines\npqls --schema --json s3://my-bucket/events/2024/data.parquet\n\n# list all .parquet files under a prefix with brief schema per file\npqls s3://my-bucket/events/2024/\n\n# machine-readable listing\npqls --json s3://my-bucket/events/2024/\n```\n\n### AWS auth\n\npqls uses the standard AWS credential provider chain — no new CLI flags.\nSet credentials via environment variables, `~/.aws/credentials`, an IAM\ninstance role, or SSO:\n\n```sh\n# env vars\nexport AWS_ACCESS_KEY_ID=...\nexport AWS_SECRET_ACCESS_KEY=...\nexport AWS_REGION=us-east-1\n\n# named profile\nexport AWS_PROFILE=my-profile\n\n# IAM role / ECS task role / IRSA — no config needed\n```\n\n### Trade-offs\n\n- Schema inspection uses S3 range gets (last 64 KiB) — no whole-file\n  download.\n- No row data access over S3 (`--csv`, `--ndjson` are not supported for\n  S3 paths).\n- No local cache — every `pqls s3://...` issues fresh range gets.\n\n## Releasing\n\n```sh\n# default: minor bump (e.g. 0.5.1 → 0.6.0)\nmake release\n\n# or override the bump type:\nmake release BUMP=patch\nmake release BUMP=major\n\n# preview what would happen — no side effects:\nmake release-dry-run\n```\n\n**Requirements:** run from your host (not a container), on the `main` branch with a clean\nworking tree that is in sync with `origin/main`.\n\n`.github/workflows/release.yml` fires on tag push, builds multi-platform binaries, creates\nthe GitHub release, and publishes to crates.io — you do **not** need to do any of that\nmanually. `make release` commits, tags, and pushes; the workflow does the rest.\n\nRecovery helpers (normally not needed):\n- `make release-resume` — push + publish if the local commit/tag exist but push previously failed\n- `make publish` — re-run `cargo publish` only (tag already on GitHub)\n\n## License\n\nLicensed under either of [MIT](LICENSE-MIT) or [Apache-2.0](LICENSE-APACHE) at your option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdunnock%2Fpqls","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdunnock%2Fpqls","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdunnock%2Fpqls/lists"}