{"id":13412817,"url":"https://github.com/parsyl/parquet","last_synced_at":"2025-03-14T18:32:22.852Z","repository":{"id":39611791,"uuid":"168235171","full_name":"parsyl/parquet","owner":"parsyl","description":"A library for reading and writing parquet files.","archived":false,"fork":false,"pushed_at":"2023-08-30T14:00:32.000Z","size":984,"stargazers_count":103,"open_issues_count":1,"forks_count":12,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-07-31T20:51:29.647Z","etag":null,"topics":["dremel","golang","parquet","reader","writer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parsyl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-01-29T21:52:30.000Z","updated_at":"2024-07-27T07:57:07.000Z","dependencies_parsed_at":"2023-01-29T04:16:00.499Z","dependency_job_id":"8379c421-40dd-4a15-8f95-20e8686dcc7c","html_url":"https://github.com/parsyl/parquet","commit_stats":null,"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsyl%2Fparquet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsyl%2Fparquet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsyl%2Fparquet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parsyl%2Fparquet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parsyl","download_url":"https://codeload.github.com/parsyl/parquet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221495311,"owners_count":16832457,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dremel","golang","parquet","reader","writer"],"created_at":"2024-07-30T20:01:29.597Z","updated_at":"2024-10-26T04:31:14.713Z","avatar_url":"https://github.com/parsyl.png","language":"Go","funding_links":[],"categories":["Libraries","File Handling","Files","文件处理","Relational Databases","文件处理`处理文件和文件系统操作的库`"],"sub_categories":["Go","Search and Analytic Databases","Advanced Console UIs","检索及分析资料库","SQL 查询语句构建库"],"readme":"# Parquet\n\nParquet generates a parquet reader and writer based on a struct.  The struct\ncan be defined by you or it can be generated by reading an existing parquet file.\n\nWe (Parsyl) will respond to pull requests and issues to the best of our\nabilities.  However, sometimes we will have higher priorities and the response\nmight not be immediate.\n\nNOTE: If you generate the code based on a parquet file there are quite a few\nlimitations.  The PageType of each PageHeader must be DATA_PAGE and the Codec\n(defined in ColumnMetaData) must be PLAIN or SNAPPY. Also, the parquet file's\nschema must consist of the currently [supported types](#supported-types).  But\nwait, there's more!  Some of the encodings, like DELTA_BINARY_PACKED, BIT_PACKED,\nPLAIN_DICTIONARY, and DELTA_BYTE_ARRAY are also not supported.  I would guess\nthere are other parquet options that will cause problems since there are so many\npossibilities.\n\n## Installation\n    \n    go get -u github.com/parsyl/parquet/...\n\nThis will also install parquet's only two dependencies: thift and snappy\n\n## Usage\n\nFirst define a struct for the data to be written to parquet:\n\n```go\ntype Person struct {\n  \tID  int32  `parquet:\"id\"`\n\tAge *int32 `parquet:\"age\"`\n}\n```\n\nNext, add a go:generate comment somewhere (in this example all code lives\nin main.go):\n\n```go\n// go:generate parquetgen -input main.go -type Person -package main\n```\n\nGenerate the code for the reader and writer:\n\n```console\n$ go generate\n```\n\nA new file (parquet.go) has now been written that defines ParquetWriter\nand ParquetReader.  Next, make use of the writer and reader:\n\n```go\npackage main\n\nimport (\n    \"bytes\"\n    \"encoding/json\"\n)\n\nfunc main() {\n    var buf bytes.Buffer\n    w, err := NewParquetWriter(\u0026buf)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    w.Add(Person{ID: 1, Age: getAge(30)})\n    w.Add(Person{ID: 2})\n\n    // Each call to write creates a new parquet row group.\n    if err := w.Write(); err != nil {\n        log.Fatal(err)\n    }\n\n    // Close must be called when you are done.  It writes\n    // the parquet metadata at the end of the file.\n    if err := w.Close(); err != nil {\n        log.Fatal(err)\n    }\n\n    r, err := NewParquetReader(bytes.NewReader(buf.Bytes()))\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    enc := json.NewEncoder(os.Stdout)\n    for r.Next() {\n        var p Person\n        r.Scan(\u0026p)\n        enc.Encode(p)\n    }\n\n    if err := r.Error(); err != nil {\n        log.Fatal(err)\n    }\n}\n\nfunc getAge(a int32) *int32 { return \u0026a }\n```\n\nNewParquetWriter has a couple of optional arguments available: MaxPageSize,\nUncompressed, and Snappy.  For example, the following sets the page size (number\nof rows in a page before a new one is created) and sets the page data compression\nto snappy:\n\n```go\nw, err := NewParquetWriter(\u0026buf, MaxPageSize(10000), Snappy)\n```\n\nSee [this](./_examples/people) for a complete example of how to generate the code\nbased on an existing struct.\n\nSee [this](./_examples/via_parquet) for a complete example of how to generate the code\nbased on an existing parquet file.\n\n## Supported Types \n\nThe struct used to define the parquet data can have the following types:\n\n```\nint32\nuint32\nint64\nuint64\nfloat32\nfloat64\nstring\nbool\n```\n\nEach of these types may be a pointer to indicate that the data is optional.  The\nstruct can also embed another struct:\n\n```go\ntype Being struct {\n\tID  int32  `parquet:\"id\"`\n\tAge *int32 `parquet:\"age\"`\n}\n\ntype Person struct {\n\tBeing\n\tUsername string `parquet:\"username\"`\n}\n```\n\nNested and repeated structs are supported too:\n\n```go\ntype Being struct {\n\tID  int32  `parquet:\"id\"`\n\tAge *int32 `parquet:\"age\"`\n}\n\ntype Person struct {\n\tBeing    Being\n\tUsername string `parquet:\"username\"`\n\tFriends  []Being\n}\n```\n\nIf you want a field to be excluded from parquet you can tag\nit with a dash or make it unexported like so:\n\n```go\ntype Being struct {\n  \tID  int32  `parquet:\"id\"`\n\tPassword string`parquet:\"-\"` //will not be written to parquet\n\tage int32                    //will not be written to parquet\n}\n```\n\n## Parquetgen\n\nParquetgen is the command that go generate should call in\norder to generate the code for your custom type.  It also can\nprint the page headers and file metadata from a parquet file:\n\n```console\n$ parquetgen --help\nUsage of parquetgen:\n  -ignore\n        ignore unsupported fields in -type, otherwise log.Fatal is called when an unsupported type is encountered (default true)\n  -import string\n        import statement of -type if it doesn't live in -package\n  -input string\n        path to the go file that defines -type\n  -metadata\n        print the metadata of a parquet file (-parquet) and exit\n  -output string\n        name of the file that is produced, defaults to parquet.go (default \"parquet.go\")\n  -package string\n        package of the generated code\n  -pageheaders\n        print the page headers of a parquet file (-parquet) and exit (also prints the metadata)\n  -parquet string\n        path to a parquet file (if you are generating code based on an existing parquet file or printing the file metadata or page headers)\n  -struct-output string\n        name of the file that is produced, defaults to parquet.go (default \"generated_struct.go\")\n  -type string\n        name of the struct that will used for writing and reading\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparsyl%2Fparquet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparsyl%2Fparquet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparsyl%2Fparquet/lists"}