{"id":17968443,"url":"https://github.com/travisbrown/parquetry","last_synced_at":"2025-03-25T09:30:52.372Z","repository":{"id":195274914,"uuid":"691160945","full_name":"travisbrown/parquetry","owner":"travisbrown","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-14T16:07:55.000Z","size":197,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-20T00:41:12.011Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/travisbrown.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-13T16:09:12.000Z","updated_at":"2025-02-14T16:07:59.000Z","dependencies_parsed_at":"2024-02-25T19:33:53.100Z","dependency_job_id":"ef0a4ff2-2fe5-4ebc-9df7-91680c8283a0","html_url":"https://github.com/travisbrown/parquetry","commit_stats":null,"previous_names":["travisbrown/parquetry"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fparquetry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fparquetry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fparquetry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fparquetry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/travisbrown","download_url":"https://codeload.github.com/travisbrown/parquetry/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245435049,"owners_count":20614818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T14:21:23.955Z","updated_at":"2025-03-25T09:30:52.361Z","avatar_url":"https://github.com/travisbrown.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parquet code generation for Rust\n\n[![Rust build status](https://img.shields.io/github/actions/workflow/status/travisbrown/parquetry/ci.yaml?branch=main)](https://github.com/travisbrown/parquetry/actions)\n[![Coverage status](https://img.shields.io/codecov/c/github/travisbrown/parquetry/main.svg)](https://codecov.io/github/travisbrown/parquetry)\n\nThis project provides tools for generating Rust code to work with [Parquet][parquet] files using the Rust implementation of [Arrow][arrow].\nIt includes both a code generation crate (`parquetry-gen`) and a small runtime library required by the generated code (`parquetry`).\n\nPlease note that this software is **not** \"open source\",\nbut the source is available for use and modification by individuals, non-profit organizations, and worker-owned businesses\n(see the [license section](#license) below for details).\n\n## Table of contents\n\n* [Example](#example)\n* [Dependencies](#dependencies)\n* [Usage](#usage)\n* [Testing](#testing)\n* [Status and scope](#status-and-scope)\n* [Warnings](#warnings)\n* [License](#license)\n\n## Example\n\nGiven a schema like this:\n\n```\nmessage user {\n    required int64 id (integer(64, false));\n    required int64 ts (timestamp(millis, true));\n    optional int32 status;\n\n    optional group user_info {\n        required byte_array screen_name (string);\n\n        optional group user_name_info {\n            required byte_array name (string);\n\n            optional group user_profile_info {\n                required int64 created_at (timestamp(millis, true));\n                required byte_array location (string);\n                required byte_array description (string);\n                optional byte_array url (string);\n\n                required int32 followers_count;\n                required int32 friends_count;\n                required int32 favourites_count;\n                required int32 statuses_count;\n\n                optional group withheld_in_countries (list) {\n                    repeated group list {\n                        required byte_array element (string);\n                    }\n                }\n            }\n        }\n    }\n}\n```\n\nThe code generator will produce the following Rust structs:\n\n```rust\n#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]\npub struct User {\n    pub id: u64,\n    #[serde(with = \"chrono::serde::ts_milliseconds\")]\n    pub ts: chrono::DateTime\u003cchrono::Utc\u003e,\n    pub status: Option\u003ci32\u003e,\n    pub user_info: Option\u003cUserInfo\u003e,\n}\n\n#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]\npub struct UserInfo {\n    pub screen_name: String,\n    pub user_name_info: Option\u003cUserNameInfo\u003e,\n}\n\n#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]\npub struct UserNameInfo {\n    pub name: String,\n    pub user_profile_info: Option\u003cUserProfileInfo\u003e,\n}\n\n#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]\npub struct UserProfileInfo {\n    #[serde(with = \"chrono::serde::ts_milliseconds\")]\n    pub created_at: chrono::DateTime\u003cchrono::Utc\u003e,\n    pub location: String,\n    pub description: String,\n    pub url: Option\u003cString\u003e,\n    pub followers_count: i32,\n    pub friends_count: i32,\n    pub favourites_count: i32,\n    pub statuses_count: i32,\n    pub withheld_in_countries: Option\u003cVec\u003cString\u003e\u003e,\n}\n```\n\nIt will also generate an instance of the `parquetry::Schema` trait for `User` with the code for reading and writing values to Parquet files.\n\n## Dependencies\n\nAll usage requires the use of the `parquetry`, [`parquet`][rust-parquet], [`chrono`][chrono], and [`lazy_static`][lazy-static] crates as runtime dependencies.\n\nIf the `serde_support` flag is enabled in configuration (which it is by default),\nyou will also need a dependency on [`serde`][serde] with the `derive` feature enabled.\n\nIf the `tests` flag is enabled in configuration (also the default),\nyou will need to add [`bincode`][bincode] (a recent `2.0.0-rc.x` with the `serde` feature enabled),\n[`tempfile`][tempfile], and [`quickcheck`][quickcheck] to your `dev-dependencies`.\n\n## Usage\n\nThe `example` directory provides a fairly minimal example, and the generated code is checked in there.\nIn most cases a `build.rs` like the following should be all you need:\n\n```rust\nuse std::{fs::File, io::Write};\n\nfn main() -\u003e Result\u003c(), parquetry_gen::error::Error\u003e {\n    for schema in parquetry_gen::ParsedFileSchema::open_dir(\n        \"src/schemas/\",\n        Default::default(),\n        Some(\".parquet.txt\"),\n    )? {\n        println!(\"cargo:rerun-if-changed={}\", schema.absolute_path_str()?);\n        let mut output = File::create(format!(\"src/{}.rs\", schema.name))?;\n        write!(output, \"{}\", schema.code()?)?;\n    }\n\n    Ok(())\n}\n```\n\nBy default the generated code is formatted with [`prettyplease`][prettyplease] and is annotated to indicate that it should not be formatted by Rustfmt,\nbut if you'd prefer to use Rustfmt yourself, you can set `format` to false in the configuration.\n\n## Testing\n\nThe default configuration will generate test code that uses [QuickCheck][quickcheck] to generate arbitrary values and confirm that they serialize and deserialize correctly.\n\nThe test code generation currently does not support some types.\nSpecifically, sequences of floating point numbers, fixed-length byte arrays, and timestamps are not supported.\nIf your schema contains any of these types, the generated test code will simply not compile, and you will have to disable the `tests` flag in your configuration\n(you can also open an issue if you'd like).\n\nThe generated test code does not produce `NaN` values for floating point types.\nIf you want to confirm that your system handles these values correctly, you'll have to do that manually.\n\nBy default the generated arbitrary values for your types may be very large.\nYou may want to set the `QUICKCHECK_GENERATOR_SIZE` environment variable to a small value (e.g. `10` or `20`) if your tests are too slow.\n\n## Status and scope\n\nThese tools support schemas with most physical and logical types, and with arbitrary nestings of lists, optional fields, and structures.\n\nMissing features that I might add at some point:\n\n* 8- and 16-bit logical integer types (trivial, I just haven't needed them)\n* `DATE`, `TIME`, `INTERVAL`, and `UUID` (same as previous)\n* `DECIMAL` (at least for the `INT32` and `INT64` representations)\n* `ENUM` (not really useful in this context since the schema doesn't enumerate the variants?)\n* Maps (a little more work but probably worth having)\n\nFeatures that will probably never be supported:\n\n* The `INT96` physical type (which has been [deprecated](https://issues.apache.org/jira/browse/PARQUET-323))\n* Timestamps with [local semantics](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#local-semantics-timestamps-not-normalized-to-utc) (`isAdjustedToUTC = false`)\n* [Legacy list shapes](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules)\n* [Legacy map shapes](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1)\n* A way to avoid the [`chrono`][chrono] dependency or support for other time libraries\n\nThis project differs from [`parquet_derive`][parquet-derive] in a few ways:\n\n* Both generate reading and writing code, but this project generates Rust structs from the schema, as opposed to the reverse.\n* This project does not use `parquet::record::RecordWriter` (it just didn't seem all that useful and I wanted more flexibility).\n* This project supports nested structures.\n\nIn general the two projects have different use cases, and if you just want to store some Rust values in Parquet, I'd recommend choosing `parquet_derive`.\n\n## Warnings\n\n### Name collisions\n\nThere's currently no special handling for field names that collide with Rust keywords, names from the standard library, etc.\nGroup names should also be unique within a schema.\nThe code generator also produces non-public structs with names that could in theory collide with user-generated code.\n\nIn most of these cases the problems should be immediately obvious, since the generated code will generally just not compile.\nIt wouldn't be hard to check for these collisions and provide better errors, or to allow more user control over naming to avoid these issues,\nbut this hasn't been a priority for me.\n\n### Constructors\n\nThe generated code includes `fn new` constructors for each struct that will truncate the precision of any `DateTime\u003cUtc\u003e` to the number of subsecond digits\nsupported by the column representation. These constructors will also check whether any string arguments contain null bytes, and will return an error if they do.\n\nIf you don't want either behavior, you can construct the structs manually, as all fields are always public.\n\n### Serde instances\n\nBy default the generated code will include derived Serde instances for serialization.\nThese instances will use the time unit specified by the schema (millisecond or microsecond) for fields with the type `DateTime\u003cUtc\u003e` or `Option\u003cDateTime\u003cUtc\u003e\u003e`, \nbut fields of type `Vec\u003cDateTime\u003cUtc\u003e\u003e` will use the default Serde serialization encoding for `DateTime\u003cUtc\u003e`.\n\nThere's no particular reason for this beyond the fact that [`chrono::serde`][chrono-serde] only provides e.g. `ts_milliseconds` and `ts_milliseconds_option` functions,\nand the runtime library could easily provide its own `ts_milliseconds_vec` that would be used in these cases.\n\n### Performance\n\nI haven't made any attempt to optimize the generated code and probably won't until performance becomes an issue for my use cases.\n\n## License\n\nThis software is published under the [Anti-Capitalist Software License][acsl] (v. 1.4).\n\n[acsl]: https://anticapitalist.software/\n[arrow]: https://arrow.apache.org/\n[arrow-rs]: https://github.com/apache/arrow-rs\n[bincode]: https://docs.rs/bincode/latest/bincode/\n[chrono]: https://docs.rs/chrono/latest/chrono/\n[chrono-serde]: https://docs.rs/chrono/latest/chrono/serde/index.html\n[lazy-static]: https://docs.rs/lazy_static/latest/lazy_static/\n[parquet]: https://parquet.apache.org/\n[parquet-derive]: https://crates.io/crates/parquet_derive\n[prettyplease]: https://github.com/dtolnay/prettyplease\n[quickcheck]: https://docs.rs/quickcheck/latest/quickcheck/\n[rust-parquet]: https://docs.rs/parquet/latest/parquet/\n[serde]: https://serde.rs/\n[tempfile]: https://docs.rs/tempfile/latest/tempfile/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravisbrown%2Fparquetry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftravisbrown%2Fparquetry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravisbrown%2Fparquetry/lists"}