{"id":18012864,"url":"https://github.com/tonivade/pq","last_synced_at":"2025-09-10T16:32:02.095Z","repository":{"id":176445220,"uuid":"657989372","full_name":"tonivade/pq","owner":"tonivade","description":"The objetive is create a tool similar to jq but for parquet files","archived":false,"fork":false,"pushed_at":"2025-09-01T08:44:22.000Z","size":1033,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-01T10:53:40.908Z","etag":null,"topics":["cli","java","native-image","parquet"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonivade.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-06-24T12:25:29.000Z","updated_at":"2025-09-01T08:44:19.000Z","dependencies_parsed_at":"2023-10-15T14:31:45.341Z","dependency_job_id":"3ded1921-8c49-495e-8b88-47e94811cd43","html_url":"https://github.com/tonivade/pq","commit_stats":null,"previous_names":["tonivade/pq"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/tonivade/pq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonivade%2Fpq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonivade%2Fpq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonivade%2Fpq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonivade%2Fpq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonivade","download_url":"https://codeload.github.com/tonivade/pq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonivade%2Fpq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274487231,"owners_count":25294480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-10T02:00:12.551Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","java","native-image","parquet"],"created_at":"2024-10-30T03:19:20.686Z","updated_at":"2025-09-10T16:32:02.038Z","avatar_url":"https://github.com/tonivade.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"pq\n===============\n\nParquet query tool\n\n# Objetive\n\nCreate a tool similar to jq but for parquet files.\n\n# Installation\n\nThe preferred instalation process is download one of the precompiled binary files that can be downloaded from releases page in github. You will find these different files:\n\n- [pq-linux-amd64](https://github.com/tonivade/pq/releases/download/0.7.0/pq-linux-amd64): native image for linux x86_64. Compressed using upx.\n- [pq-linux-arm64](https://github.com/tonivade/pq/releases/download/0.7.0/pq-linux-arm64): native image for linux aarch64. Compressed using upx.\n- [pq-darwin-arm64](https://github.com/tonivade/pq/releases/download/0.7.0/pq-darwin-arm64): native image for macos aarch64.\n- [pq-darwin-amd64](https://github.com/tonivade/pq/releases/download/0.7.0/pq-darwin-amd64): native image for macos x86_64.\n- [pq-windows-amd64.exe](https://github.com/tonivade/pq/releases/download/0.7.0/pq-windows-amd64.exe): native image for windows x86_64.\n- [pq.jar](https://github.com/tonivade/pq/releases/download/0.7.0/pq.jar): fat jar with all the needed classes\n\nNative images has been generated using graalvm-ce 23.0.2.\n\nCurrent version is 0.7.0.\n\n# Usage\n\nYou will see all the available subcommands using help subcommand:\n\n```sh\n$ ./pq help\nUsage: pq [-v] [COMMAND]\nparquet query tool\n  -v, --verbose   enable debug logs\nCommands:\n  count     print total number of rows in parquet file\n  schema    print schema of parquet file\n  read      print content of parquet file in json format\n  metadata  print metadata of parquet file\n  write     create a parquet file from a jsonl stream and a schema\n  help      Display help information about the specified command.\nCopyright(c) 2023 by @tonivade\n```\n\n## count\n\nPrint number of rows in file:\n\n```sh\n$ ./pq help count\nUsage: pq count [-v] [--filter=PREDICATE] FILE\nprint total number of rows in parquet file\n      FILE                 parquet file\n      --filter=PREDICATE   predicate to apply to the rows\n  -v, --verbose            enable debug logs\n```\n\nExample:\n\n```sh\n$ ./pq count example.parquet\n1000\n```\n\n### Filter rows\n\nYou can get the count or rows that match a filter this way:\n\n```sh\n$ ./pq count --filter 'gender == \"Male\"' example.parquet\n451\n```\n\n## schema\n\nPrint parquet schema.\n\n```sh\n$ ./pq help schema\nUsage: pq schema [-v] [--select=COLUMN[,COLUMN...]]... FILE\nprint schema of parquet file\n      FILE        parquet file\n      --select=COLUMN[,COLUMN...]\n                  list of columns to select\n  -v, --verbose   enable debug logs\n```\n\nExample:\n\n```sh\n$ ./pq schema example.parquet\nmessage spark_schema {\n  optional int32 id;\n  optional binary first_name (STRING);\n  optional binary last_name (STRING);\n  optional binary email (STRING);\n  optional binary gender (STRING);\n  optional binary ip_address (STRING);\n  optional binary cc (STRING);\n  optional binary country (STRING);\n  optional binary birthdate (STRING);\n  optional double salary;\n  optional binary title (STRING);\n  optional binary comments (STRING);\n}\n```\n\n### Select columns\n\nYou can select the columns you want to include in the schema using the `--select` option.\n\n```sh\n$ ./pq schema --select id,first_name,email example.parquet\nmessage spark_schema {\n  optional int32 id;\n  optional binary first_name (STRING);\n  optional binary email (STRING);\n}\n```\n\n## read\n\nPrint file content.\n\n```sh\n$ ./pq help read\nUsage: pq read [-v] [--index] [--filter=PREDICATE] [--format=JSON|CSV]\n               [--get=ROW] [--head=ROWS] [--skip=ROWS] [--tail=ROWS]\n               [--select=COLUMN[,COLUMN...]]... FILE\nprint content of parquet file in json format\n      FILE                 parquet file\n      --filter=PREDICATE   predicate to apply to the rows\n      --format=JSON|CSV    output format, json or csv\n      --get=ROW            print just the row with given index\n      --head=ROWS          get the first N number of rows\n      --index              print row index\n      --select=COLUMN[,COLUMN...]\n                           list of columns to select\n      --skip=ROWS          skip a number N of rows\n      --tail=ROWS          get the last N number of rows\n  -v, --verbose            enable debug logs\n```\n\nExample:\n\n```sh\n$ ./pq read example.parquet\n{\"id\":1,\"first_name\":\"Amanda\",\"last_name\":\"Jordan\",\"email\":\"ajordan0@com.com\",\"gender\":\"Female\",\"ip_address\":null,\"cc\":\"6759521864920116\",\"country\":\"Indonesia\",\"birthdate\":\"3/8/1971\",\"salary\":49756.53,\"title\":\"Internal Auditor\",\"comments\":\"1E+02\"}\n...\n```\n\n### Select columns\n\nYou can select the columns you want to include in the output using the `--select` option:\n\n```sh\n$ ./pq read --select id,first_name,email example.parquet\n{\"id\":1,\"first_name\":\"Amanda\",\"email\":\"ajordan0@com.com\"}\n...\n```\n\n### Filter rows\n\nYou can filter the rows that match a filter this way:\n\n```sh\n$ ./pq read --filter 'gender == \"Male\"' example.parquet\n{\"id\":2,\"first_name\":\"Albert\",\"last_name\":\"Freeman\",\"email\":\"afreeman1@is.gd\",\"gender\":\"Male\",\"ip_address\":\"218.111.175.34\",\"cc\":\"\",\"country\":\"Canada\",\"birthdate\":\"1/16/1968\",\"salary\":150280.17,\"title\":\"Accountant IV\",\"comments\":\"\"}\n...\n```\n\n## metadata\n\nPrint file metadata.\n\n```sh\n$ ./pq help metadata\nUsage: pq metadata [-v] [--show-blocks] FILE\nprint metadata of parquet file\n      FILE            parquet file\n      --show-blocks   show block metadata info\n  -v, --verbose       enable debug logs\n```\n\nExample:\n\n```sh\n$ ./pq metadata example.parquet\n\"createdBy\":parquet-mr version 1.8.3 (build aef7230e114214b7cc962a8f3fc5aeed6ce80828)\n\"count\":1000\n```\n\n## write\n\nCreates a parquet file from a jsonl/csv file and a shema.\n\n```sh\n$ ./pq help write\nUsage: pq write [-v] [--format=JSON|CSV] [--schema=FILE] FILE\ncreate a parquet file from a jsonl stream and a schema\n      FILE                destination parquet file\n      --format=JSON|CSV   input format, json or csv\n      --schema=FILE       file with schema definition\n  -v, --verbose           enable debug logs\n```\n\n# License\n\nThis project is released under MIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonivade%2Fpq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonivade%2Fpq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonivade%2Fpq/lists"}