{"id":39464617,"url":"https://github.com/ktrueda/go-parquet-tools","last_synced_at":"2026-01-18T04:48:58.351Z","repository":{"id":56854387,"uuid":"519459373","full_name":"ktrueda/go-parquet-tools","owner":"ktrueda","description":"Faster parquet-tools","archived":false,"fork":false,"pushed_at":"2022-08-30T15:14:53.000Z","size":113,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-20T15:45:31.268Z","etag":null,"topics":["parquet","parquet-tools"],"latest_commit_sha":null,"homepage":"","language":"Thrift","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ktrueda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-30T08:19:49.000Z","updated_at":"2023-07-27T09:27:49.000Z","dependencies_parsed_at":"2022-09-06T08:01:39.017Z","dependency_job_id":null,"html_url":"https://github.com/ktrueda/go-parquet-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ktrueda/go-parquet-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktrueda%2Fgo-parquet-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktrueda%2Fgo-parquet-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktrueda%2Fgo-parquet-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktrueda%2Fgo-parquet-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ktrueda","download_url":"https://codeload.github.com/ktrueda/go-parquet-tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktrueda%2Fgo-parquet-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28530297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parquet","parquet-tools"],"created_at":"2026-01-18T04:48:57.717Z","updated_at":"2026-01-18T04:48:58.341Z","avatar_url":"https://github.com/ktrueda.png","language":"Thrift","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-parquet-tools\n\nAlternative to [pypi parquet-tools](https://pypi.org/project/parquet-tools/) in Golang.\n\nYou can show content/schema of parquet file(s) on local disk or on Amazon S3. It is incompatible with original parquet-tools. go-parquet-tools is faster because this is implemented in golang.\n\n## Install\n\n```bash\ngo install github.com/ktrueda/go-parquet-tools@latest\n```\n\n## Usage\n\n```bash\ngo-parquet-tools csv test_resources/test1.parquet\none,two,three\n-1,foo,true\n\u003cnil\u003e,bar,false\n2.5,baz,true\n```\n\n```bash\ngo-parquet-tools show --nil None \"test_resources/*\"\n+------+-----+-------+\n| one  | two | three |\n+------+-----+-------+\n| -1   | foo | true  |\n| None | bar | false |\n| 2.5  | baz | true  |\n| -1   | foo | true  |\n| None | bar | false |\n| 2.5  | baz | true  |\n+------+-----+-------+\n```\n\n```bash\ngo-parquet-tools show s3://foo/test1.parquet\nDownloaded s3://foo/test.parquet to /var/folders/f3/9l_qwscs3z94m3yw255bw4l40000gn/T/9ed16365-58e2-40f2-a492-e8477b418a0f.parquet .\n+-------+-----+-------+\n| one   | two | three |\n+-------+-----+-------+\n| -1    | foo | true  |\n| \u003cnil\u003e | bar | false |\n| 2.5   | baz | true  |\n+-------+-----+-------+\n```\n\n```bash\ngo-parquet-tools inspect test_resources/test1.parquet\n```\n\n\u003cdetails\u003e\n\u003csummary\u003einsepct output\u003c/summary\u003e\n\n```bash\nVersion:  1\nSchema:\n        ######### schema #########\n        Type:  \u003cnil\u003e\n        TypeLength:  \u003cnil\u003e\n        RepetitionType:  REQUIRED\n        Name:  schema\n        NumChildren:  0xc0000288d8\n        ConvertedType:  \u003cnil\u003e\n        Scale:  \u003cnil\u003e\n        Precision:  \u003cnil\u003e\n        FieldID:  \u003cnil\u003e\n        LogicalType:  \u003cnil\u003e\n        ######### one #########\n        Type:  DOUBLE\n        TypeLength:  \u003cnil\u003e\n        RepetitionType:  OPTIONAL\n        Name:  one\n        NumChildren:  \u003cnil\u003e\n        ConvertedType:  \u003cnil\u003e\n        Scale:  \u003cnil\u003e\n        Precision:  \u003cnil\u003e\n        FieldID:  \u003cnil\u003e\n        LogicalType:  \u003cnil\u003e\n        ######### two #########\n        Type:  BYTE_ARRAY\n        TypeLength:  \u003cnil\u003e\n        RepetitionType:  OPTIONAL\n        Name:  two\n        NumChildren:  \u003cnil\u003e\n        ConvertedType:  UTF8\n        Scale:  \u003cnil\u003e\n        Precision:  \u003cnil\u003e\n        FieldID:  \u003cnil\u003e\n        LogicalType:  LogicalType({STRING:StringType({}) MAP:\u003cnil\u003e LIST:\u003cnil\u003e ENUM:\u003cnil\u003e DECIMAL:\u003cnil\u003e DATE:\u003cnil\u003e TIME:\u003cnil\u003e TIMESTAMP:\u003cnil\u003e INTEGER:\u003cnil\u003e UNKNOWN:\u003cnil\u003e JSON:\u003cnil\u003e BSON:\u003cnil\u003e UUID:\u003cnil\u003e})\n        ######### three #########\n        Type:  BOOLEAN\n        TypeLength:  \u003cnil\u003e\n        RepetitionType:  OPTIONAL\n        Name:  three\n        NumChildren:  \u003cnil\u003e\n        ConvertedType:  \u003cnil\u003e\n        Scale:  \u003cnil\u003e\n        Precision:  \u003cnil\u003e\n        FieldID:  \u003cnil\u003e\n        LogicalType:  \u003cnil\u003e\nNumRows:  3\nRowGroups:\n        Columns:\n                #########\n                FilePath  \u003cnil\u003e\n                FileOffset  108\n                MetaData.Type  DOUBLE\n                MetaData.Encodings  [PLAIN_DICTIONARY PLAIN RLE]\n                MetaData.PathInSchema  [one]\n                MetaData.Codec  SNAPPY\n                MetaData.NumValues  3\n                MetaData.TotalUncompressedSize  100\n                MetaData.TotalCompressedSize  104\n                MetaData.KeyValueMetadata  []\n                MetaData.DataPageOffset  36\n                MetaData.IndexPageOffset  \u003cnil\u003e\n                MetaData.DictionaryPageOffset  0xc000028930\n                MetaData.Statistics  Statistics({Max:[0 0 0 0 0 0 4 64] Min:[0 0 0 0 0 0 240 191] NullCount:0xc000028938 DistinctCount:\u003cnil\u003e MaxValue:[0 0 0 0 0 0 4 64] MinValue:[0 0 0 0 0 0 240 191]})\n                MetaData.EncodingStats  [PageEncodingStats({PageType:DICTIONARY_PAGE Encoding:PLAIN_DICTIONARY Count:1}) PageEncodingStats({PageType:DATA_PAGE Encoding:PLAIN_DICTIONARY Count:1})]\n                MetaData.BloomFilterOffset  \u003cnil\u003e\n                OffsetIndexOffset  \u003cnil\u003e\n                OffsetIndexLength  \u003cnil\u003e\n                ColumnIndexOffset  \u003cnil\u003e\n                ColumnIndexLength  \u003cnil\u003e\n                CryptoMeatadata  \u003cnil\u003e\n                EncryptedColumnMetadata  []\n                #########\n                FilePath  \u003cnil\u003e\n                FileOffset  281\n                MetaData.Type  BYTE_ARRAY\n                MetaData.Encodings  [PLAIN_DICTIONARY PLAIN RLE]\n                MetaData.PathInSchema  [two]\n                MetaData.Codec  SNAPPY\n                MetaData.NumValues  3\n                MetaData.TotalUncompressedSize  76\n                MetaData.TotalCompressedSize  80\n                MetaData.KeyValueMetadata  []\n                MetaData.DataPageOffset  238\n                MetaData.IndexPageOffset  \u003cnil\u003e\n                MetaData.DictionaryPageOffset  0xc000028948\n                MetaData.Statistics  Statistics({Max:[] Min:[] NullCount:0xc000028950 DistinctCount:\u003cnil\u003e MaxValue:[102 111 111] MinValue:[98 97 114]})\n                MetaData.EncodingStats  [PageEncodingStats({PageType:DICTIONARY_PAGE Encoding:PLAIN_DICTIONARY Count:1}) PageEncodingStats({PageType:DATA_PAGE Encoding:PLAIN_DICTIONARY Count:1})]\n                MetaData.BloomFilterOffset  \u003cnil\u003e\n                OffsetIndexOffset  \u003cnil\u003e\n                OffsetIndexLength  \u003cnil\u003e\n                ColumnIndexOffset  \u003cnil\u003e\n                ColumnIndexLength  \u003cnil\u003e\n                CryptoMeatadata  \u003cnil\u003e\n                EncryptedColumnMetadata  []\n                #########\n                FilePath  \u003cnil\u003e\n                FileOffset  388\n                MetaData.Type  BOOLEAN\n                MetaData.Encodings  [PLAIN RLE]\n                MetaData.PathInSchema  [three]\n                MetaData.Codec  SNAPPY\n                MetaData.NumValues  3\n                MetaData.TotalUncompressedSize  40\n                MetaData.TotalCompressedSize  42\n                MetaData.KeyValueMetadata  []\n                MetaData.DataPageOffset  346\n                MetaData.IndexPageOffset  \u003cnil\u003e\n                MetaData.DictionaryPageOffset  \u003cnil\u003e\n                MetaData.Statistics  Statistics({Max:[1] Min:[0] NullCount:0xc000028970 DistinctCount:\u003cnil\u003e MaxValue:[1] MinValue:[0]})\n                MetaData.EncodingStats  [PageEncodingStats({PageType:DATA_PAGE Encoding:PLAIN Count:1})]\n                MetaData.BloomFilterOffset  \u003cnil\u003e\n                OffsetIndexOffset  \u003cnil\u003e\n                OffsetIndexLength  \u003cnil\u003e\n                ColumnIndexOffset  \u003cnil\u003e\n                ColumnIndexLength  \u003cnil\u003e\n                CryptoMeatadata  \u003cnil\u003e\n                EncryptedColumnMetadata  []\n        TotalByteSize:  226\n        NumRows:  3\n        SotringColumns:  []\n        FileOffset:  0xc000028978\n        TotalCompressedSize:  0xc000028980\n        Ordinal:  0xc000028988\nKeyValueMetaData:  [KeyValue({Key:pandas Value:0xc000063330}) KeyValue({Key:ARROW:schema Value:0xc000063340})]\nCreatedBy:  0xc000063350\nColumnOrders:  [ColumnOrder({TYPE_ORDER:TypeDefinedOrder({})}) ColumnOrder({TYPE_ORDER:TypeDefinedOrder({})}) ColumnOrder({TYPE_ORDER:TypeDefinedOrder({})})]\nEncryptionAlgorithm:  \u003cnil\u003e\nFooterSigningKeyMetadata:  []\n```\n\n\u003c/details\u003e\n\n## Benchmark result\n\ngo-parquet-tools is 100x faster than pypi parquet-tools.\n\n| Command                                             |    Mean [ms] | Min [ms] | Max [ms] | Relative |\n| :-------------------------------------------------- | -----------: | -------: | -------: | -------: |\n| `parquet-tools csv test_resources/test1.parquet`    | 702.8 ± 19.9 |    676.2 |    739.4 |     1.00 |\n| `go-parquet-tools csv test_resources/test1.parquet` |    6.6 ± 0.4 |      6.2 |      7.3 |     1.00 |\n\nhttps://github.com/sharkdp/hyperfine\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktrueda%2Fgo-parquet-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fktrueda%2Fgo-parquet-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktrueda%2Fgo-parquet-tools/lists"}