{"id":29616661,"url":"https://github.com/jimyag/parquet-tools","last_synced_at":"2026-07-15T20:35:56.209Z","repository":{"id":240065528,"uuid":"787979002","full_name":"jimyag/parquet-tools","owner":"jimyag","description":"Utility to pretty inspect Parquet files.  Supported reading from: http/https URL, local file, s3/s3a URL . Converting a parquet schema to a go structure.","archived":false,"fork":false,"pushed_at":"2024-11-25T04:00:57.000Z","size":72,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-25T05:17:34.243Z","etag":null,"topics":["golang","parquet","parquet-tools","parquet-viewer","pretty"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jimyag.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-17T14:45:41.000Z","updated_at":"2024-11-25T04:00:16.000Z","dependencies_parsed_at":"2024-11-25T05:17:37.060Z","dependency_job_id":"53056007-2519-443a-8284-294fa98656da","html_url":"https://github.com/jimyag/parquet-tools","commit_stats":null,"previous_names":["jimyag/parquet-tools"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/jimyag/parquet-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimyag%2Fparquet-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimyag%2Fparquet-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimyag%2Fparquet-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimyag%2Fparquet-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jimyag","download_url":"https://codeload.github.com/jimyag/parquet-tools/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jimyag%2Fparquet-tools/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266222295,"owners_count":23894992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","parquet","parquet-tools","parquet-viewer","pretty"],"created_at":"2025-07-21T01:02:00.722Z","updated_at":"2026-07-15T20:35:56.202Z","avatar_url":"https://github.com/jimyag.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# parquet-tools\n\nUtility to pretty inspect Parquet files.\n\nSupported reading from: http/https URL, local file, s3/s3a URL\n\n## Features\n\n- **cat**: Print the first N records from a file (`--format jsonl/csv/json`, `--columns`)\n- **tail**: Print the last N records from a file (`--format jsonl/csv/json`, `--columns`)\n- **rowcount**: Print the number of rows in a file\n- **validate**: Validate a file's footer and optionally all data (`--full`)\n- **meta**: Pretty print a Parquet file's metadata\n- **schema**: Pretty print the Avro schema for a file\n- **struct**: Print the Go struct for a file\n- **footer**: Print the Parquet file footer in JSON format\n- **diff**: Diff two Parquet files' schemas\n\n## Install\n\n``` bash\ngo install github.com/jimyag/parquet-tools@latest\n```\n\n## Usage\n\n``` bash\nparquet-tools -h\n```\n\npretty print the Avro schema for a file\n\n``` bash\nparquet-tools meta https://github.com/jimyag/parquet-tools/raw/main/testdata/v0.7.1.parquet\n+-------------------------+------------------------------------------+\n| KEY                     | VALUE                                    |\n+-------------------------+------------------------------------------+\n|         filename        | https://github.com/jimyag/parquet-tools/ |\n|                         |     raw/main/testdata/v0.7.1.parquet     |\n+-------------------------+------------------------------------------+\n|         version         |                   v1.0                   |\n+-------------------------+------------------------------------------+\n|        created by       |    parquet-cpp version 1.3.2-SNAPSHOT    |\n+-------------------------+------------------------------------------+\n|         num rows        |                    10                    |\n+-------------------------+------------------------------------------+\n| key value file metadata |                     1                    |\n+-------------------------+------------------------------------------+\n|          pandas         | {\"index_columns\": [\"__index_level_0__\"], |\n|                         |  \"column_indexes\": [{\"name\": null, \"pand |\n|                         | as_type\": \"string\", \"numpy_type\": \"objec |\n|                         | t\", \"metadata\": null}], \"columns\": [{\"na |\n|                         |  me\": \"carat\", \"pandas_type\": \"float64\", |\n|                         | \"numpy_type\": \"float64\", \"metadata\": nul |\n|                         | l}, {\"name\": \"cut\", \"pandas_type\": \"unic |\n|                         | ode\", \"numpy_type\": \"object\", \"metadata\" |\n|                         | : null}, {\"name\": \"color\", \"pandas_type\" |\n|                         | : \"unicode\", \"numpy_type\": \"object\", \"me |\n|                         | tadata\": null}, {\"name\": \"clarity\", \"pan |\n|                         | das_type\": \"unicode\", \"numpy_type\": \"obj |\n|                         | ect\", \"metadata\": null}, {\"name\": \"depth |\n|                         | \", \"pandas_type\": \"float64\", \"numpy_type |\n|                         | \": \"float64\", \"metadata\": null}, {\"name\" |\n|                         | : \"table\", \"pandas_type\": \"float64\", \"nu |\n|                         | mpy_type\": \"float64\", \"metadata\": null}, |\n|                         |  {\"name\": \"price\", \"pandas_type\": \"int64 |\n|                         | \", \"numpy_type\": \"int64\", \"metadata\": nu |\n|                         | ll}, {\"name\": \"x\", \"pandas_type\": \"float |\n|                         | 64\", \"numpy_type\": \"float64\", \"metadata\" |\n|                         | : null}, {\"name\": \"y\", \"pandas_type\": \"f |\n|                         | loat64\", \"numpy_type\": \"float64\", \"metad |\n|                         | ata\": null}, {\"name\": \"z\", \"pandas_type\" |\n|                         | : \"float64\", \"numpy_type\": \"float64\", \"m |\n|                         | etadata\": null}, {\"name\": \"__index_level |\n|                         | _0__\", \"pandas_type\": \"int64\", \"numpy_ty |\n|                         | pe\": \"int64\", \"metadata\": null}], \"panda |\n|                         |           s_version\": \"0.20.1\"}          |\n+-------------------------+------------------------------------------+\n|   number of row groups  |                     1                    |\n+-------------------------+------------------------------------------+\n|  number of real columns |                    11                    |\n+-------------------------+------------------------------------------+\n|    number of columns    |                    11                    |\n+-------------------------+------------------------------------------+\n\n--- row group:  0  begin ---\n+----------------+------+\n| total bytes    | 1327 |\n+----------------+------+\n| number of rows |   10 |\n+----------------+------+\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n| COLUMN     | COUNT | MIN  | MAX  | NULLS | DISTI | COMPR | ENCOD | UNCOMPRESSED | COMPRESSED |\n|            | S     |      |      |       | NCT   | ESSIO | INGS  |              |            |\n|            |       |      |      |       |       | N     |       |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|    carat   |   10  | 0.21 | 0.31 |   0   |   0   | SNAPP | PLAIN | 125          | 129        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|     cut    |   10  |   -  |   -  |   0   |   0   | SNAPP | PLAIN | 115          | 119        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|    color   |   10  |   -  |   -  |   0   |   0   | SNAPP | PLAIN | 73           | 77         |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|   clarity  |   10  |   -  |   -  |   0   |   0   | SNAPP | PLAIN | 104          | 100        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|    depth   |   10  | 56.9 | 65.1 |   0   |   0   | SNAPP | PLAIN | 152          | 132        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|    table   |   10  |  55  |  65  |   0   |   0   | SNAPP | PLAIN | 109          | 105        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|    price   |   10  |  326 |  338 |   0   |   0   | SNAPP | PLAIN | 125          | 111        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|      x     |   10  | 3.87 | 4.34 |   0   |   0   | SNAPP | PLAIN | 145          | 143        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|      y     |   10  | 3.78 | 4.35 |   0   |   0   | SNAPP | PLAIN | 145          | 143        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n|      z     |   10  | 2.31 | 2.75 |   0   |   0   | SNAPP | PLAIN | 145          | 144        |\n|            |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n| __index_le |   10  |   0  |   9  |   0   |   0   | SNAPP | PLAIN | 152          | 124        |\n|   vel_0__  |       |      |      |       |       |   Y   | _DICT |              |            |\n|            |       |      |      |       |       |       | IONAR |              |            |\n|            |       |      |      |       |       |       | Y PLA |              |            |\n|            |       |      |      |       |       |       | IN RL |              |            |\n|            |       |      |      |       |       |       |   E   |              |            |\n+------------+-------+------+------+-------+-------+-------+-------+--------------+------------+\n--- row group:  0  end ---\n\nrequired group field_id=-1 schema {\n  optional double field_id=-1 carat;\n  optional byte_array field_id=-1 cut (String);\n  optional byte_array field_id=-1 color (String);\n  optional byte_array field_id=-1 clarity (String);\n  optional double field_id=-1 depth;\n  optional double field_id=-1 table;\n  optional int64 field_id=-1 price;\n  optional double field_id=-1 x;\n  optional double field_id=-1 y;\n  optional double field_id=-1 z;\n  optional int64 field_id=-1 __index_level_0__;\n}\n```\n\nread from http or https\n\n``` bash\nparquet-tools schema https://github.com/jimyag/parquet-tools/raw/main/testdata/v0.7.1.parquet\nrequired group field_id=-1 schema {\n  optional double field_id=-1 carat;\n  optional byte_array field_id=-1 cut (String);\n  optional byte_array field_id=-1 color (String);\n  optional byte_array field_id=-1 clarity (String);\n  optional double field_id=-1 depth;\n  optional double field_id=-1 table;\n  optional int64 field_id=-1 price;\n  optional double field_id=-1 x;\n  optional double field_id=-1 y;\n  optional double field_id=-1 z;\n  optional int64 field_id=-1 __index_level_0__;\n}\n```\n\nprint the Go struct for a file\n\n```bash\nparquet-tools struct https://github.com/jimyag/parquet-tools/raw/main/testdata/v0.7.1.parquet\ntype schema struct {\n  Carat any `parquet:\"carat\"`\n  Cut string `parquet:\"cut\"`\n  Color string `parquet:\"color\"`\n  Clarity string `parquet:\"clarity\"`\n  Depth any `parquet:\"depth\"`\n  Table any `parquet:\"table\"`\n  Price any `parquet:\"price\"`\n  X any `parquet:\"x\"`\n  Y any `parquet:\"y\"`\n  Z any `parquet:\"z\"`\n  IndexLevel0 any `parquet:\"__index_level_0__\"`\n}\n```\n\ndiff two Parquet files schema\n\n```bash\nparquet-tools diff v0.7.1.parquet v0.7.2.parquet\n```\n\nprint row count\n\n```bash\nparquet-tools rowcount v0.7.1.parquet\n10\n\n# multiple files: filename\u003ctab\u003ecount\nparquet-tools rowcount v0.7.1.parquet v0.7.2.parquet\nv0.7.1.parquet  10\nv0.7.2.parquet  10\n```\n\nvalidate a file (shallow — footer only)\n\n```bash\nparquet-tools validate v0.7.1.parquet\nOK  v0.7.1.parquet\n```\n\nvalidate with full data read\n\n```bash\nparquet-tools validate --full --verbose v0.7.1.parquet\nOK  v0.7.1.parquet — 10 rows, 1 row groups, 11 columns (all data read)\n```\n\nprint first 3 records as JSON Lines (default)\n\n```bash\nparquet-tools cat -n 3 v0.7.1.parquet\n{\"carat\":0.23,\"cut\":\"Ideal\",\"color\":\"E\",\"clarity\":\"SI2\",...}\n```\n\nprint as CSV\n\n```bash\nparquet-tools cat -n 3 --format csv v0.7.1.parquet\ncarat,cut,color,...\n0.23,Ideal,E,...\n```\n\nprint as a JSON array\n\n```bash\nparquet-tools cat -n 3 --format json v0.7.1.parquet\n[\n{\"carat\":0.23,...},\n...\n]\n```\n\nfilter specific columns\n\n```bash\nparquet-tools cat -n 3 --columns carat,cut v0.7.1.parquet\n{\"carat\":0.23,\"cut\":\"Ideal\"}\n```\n\nprint the last 5 records\n\n```bash\nparquet-tools tail -n 5 v0.7.1.parquet\n```\n\nread from local file\n\n``` bash\nparquet-tools schema v0.7.1.parquet\nrequired group field_id=-1 schema {\n  optional double field_id=-1 carat;\n  optional byte_array field_id=-1 cut (String);\n  optional byte_array field_id=-1 color (String);\n  optional byte_array field_id=-1 clarity (String);\n  optional double field_id=-1 depth;\n  optional double field_id=-1 table;\n  optional int64 field_id=-1 price;\n  optional double field_id=-1 x;\n  optional double field_id=-1 y;\n  optional double field_id=-1 z;\n  optional int64 field_id=-1 __index_level_0__;\n}\n```\n\nread from s3 bucket\n\n``` bash\ncat s3.toml\n[[s3]]\nregion = \"xxxx\"\naccess_key = \"ak\"\nsecret_key = \"sk\"\nendpoint = \"endpoint\"\ndisable_ssl = false\nforce_path_style = true\n```\n\n``` bash\nparquet-tools schema s3://jimyag/parquet-tools/testdata/v0.7.1.parquet --s3-config s3.toml\nrequired group field_id=-1 schema {\n  optional double field_id=-1 carat;\n  optional byte_array field_id=-1 cut (String);\n  optional byte_array field_id=-1 color (String);\n  optional byte_array field_id=-1 clarity (String);\n  optional double field_id=-1 depth;\n  optional double field_id=-1 table;\n  optional int64 field_id=-1 price;\n  optional double field_id=-1 x;\n  optional double field_id=-1 y;\n  optional double field_id=-1 z;\n  optional int64 field_id=-1 __index_level_0__;\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjimyag%2Fparquet-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjimyag%2Fparquet-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjimyag%2Fparquet-tools/lists"}