{"id":33938567,"url":"https://github.com/loro-dev/columnar","last_synced_at":"2026-04-09T03:31:59.590Z","repository":{"id":165407986,"uuid":"557625895","full_name":"loro-dev/columnar","owner":"loro-dev","description":"ergonomic columnar storage encoding crate","archived":false,"fork":false,"pushed_at":"2025-09-23T12:56:24.000Z","size":329,"stargazers_count":27,"open_issues_count":6,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-29T21:04:21.814Z","etag":null,"topics":["backward-compatibility","columnar","columnar-storage","compression","forward-compatible","rle","serde"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loro-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-10-26T02:16:36.000Z","updated_at":"2025-12-07T16:09:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"41d574e0-8b0c-4a31-b7f5-7dd86723d379","html_url":"https://github.com/loro-dev/columnar","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/loro-dev/columnar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loro-dev%2Fcolumnar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loro-dev%2Fcolumnar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loro-dev%2Fcolumnar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loro-dev%2Fcolumnar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loro-dev","download_url":"https://codeload.github.com/loro-dev/columnar/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loro-dev%2Fcolumnar/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31584569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"online","status_checked_at":"2026-04-09T02:00:06.848Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backward-compatibility","columnar","columnar-storage","compression","forward-compatible","rle","serde"],"created_at":"2025-12-12T15:02:23.293Z","updated_at":"2026-04-09T03:31:59.585Z","avatar_url":"https://github.com/loro-dev.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `serde_columnar`\r\n\r\n`serde_columnar` is an ergonomic columnar storage encoding crate that offers forward and backward compatibility.\r\n\r\nIt allows the contents that need to be serialized and deserialized to be encoded into binary using columnar storage, all by just employing simple macro annotations.\r\n\r\nFor more detailed introduction, please refer to this `Notion` link: [Serde-Columnar](https://www.notion.so/loro-dev/Serde-Columnar-Ergonomic-columnar-storage-encoding-crate-7b0c86d6f8d24e4da45a1e2ebd86741c?pvs=4).\r\n\r\n![Image](https://github.com/user-attachments/assets/48475c7d-61c8-4903-892d-70d702137dba)\r\n\r\n## Features 🚀\r\n\r\n`serde_columnar` comes with several remarkable features:\r\n\r\n- 🗜️ Utilizes columnar storage in conjunction with various compression strategies to significantly reduce the size of the encoded content.\r\n- 🔄 Built-in forward and backward compatibility solutions, eliminating the need for maintaining additional version codes.\r\n- 🌳 Supports nested columnar storage.\r\n- 📦 Supports list and map containers\r\n- 🔄 Supports deserialization using iterator format.\r\n\r\n## How to use\r\n\r\n### Install\r\n\r\n```shell\r\ncargo add serde_columnar\r\n```\r\n\r\nOr edit your `Cargo.toml` and add `serde_columnar` as dependency:\r\n\r\n```toml\r\n[dependencies]\r\nserde_columnar = \"0.3\"\r\n```\r\n\r\n### Container Attribute\r\n\r\n- `vec`:\r\n  - Declare this struct will be rows of a vec-like container\r\n  - Automatically derive [`RowSer`](https://docs.rs/serde_columnar/latest/serde_columnar/trait.RowSer.html) trait if set `ser` at the same time\r\n  - Automatically derive [`RowDe`](https://docs.rs/serde_columnar/latest/serde_columnar/trait.RowDe.html) trait if set `de` at the same time\r\n- `map`:\r\n  - Declare this struct will be rows of a map-like container\r\n  - Automatically derive [`KeyRowSer`](https://docs.rs/serde_columnar/latest/serde_columnar/trait.KeyRowSer.html) trait if set `ser` at the same time\r\n  - Automatically derive [`KeyRowDe`](https://docs.rs/serde_columnar/latest/serde_columnar/trait.KeyRowDe.html) trait if set `de` at the same time\r\n- `ser`:\r\n  - Automatically derive `Serialize` trait for this struct\r\n- `de`:\r\n  - Automatically derive `Deserialize` trait for this struct\r\n- `iterable`:\r\n  - Declare this struct will be iterable\r\n  - Only available for `row` struct\r\n  - [Iterable](https://github.com/loro-dev/columnar#Iterable) for more details\r\n\r\n### Field Attribute\r\n\r\n- `strategy`:\r\n  - The columnar compression strategy applied to this field.\r\n  - Optional value: `Rle`/`DeltaRle`/`BoolRle`/`DeltaOfDelta`.\r\n  - Only available for `row` struct.\r\n- `class`:\r\n  - Declare this field is a container for rows. The field's type is usually `Vec` or `HashMap` and their variants.\r\n  - Optional value: `vec` or `map`.\r\n  - Only available for `table` struct.\r\n- `skip`:\r\n  - Same as [`#[serde(skip)]`](https://serde.rs/field-attrs.html#skip), do not serialize or deserialize this field.\r\n- `borrow`:\r\n  - Same as [`#[serde(borrow)]`](https://serde.rs/field-attrs.html#borrow), borrow data for this field from the deserializer by using zero-copy deserialization.\r\n  - use `#[columnar(borrow=\"'a + 'b\")]` to specify explicitly which lifetimes should be borrowed.\r\n  - Only available for `table` struct for now.\r\n- `iter`:\r\n  - Declare the iterable row type when deserializing using iter mode.\r\n  - Only available for field marked `class`.\r\n  - Only available for `class=\"vec\"`.\r\n- `optional` \u0026 `index`:\r\n  - In order to achieve forward and backward compatibility, some fields that may change can be marked as `optional`.\r\n  - And in order to avoid the possibility of errors in the future, such as change the order of optional fields, it is necessary to mark the `index`.\r\n  - All `optional` fields must be after other fields.\r\n  - The `index` is the unique identifier of the optional field, which will be encoded into the result. If the corresponding identifier cannot be found during deserialization, `Default` will be used.\r\n  - `optional` fields can be added or removed in future versions. The compatibility premise is that the field type of the same index does not change or the encoding format is compatible (such as changing `u32` to `u64`).\r\n\r\n### Examples\r\n\r\n```rust\r\nuse serde_columnar::{columnar, from_bytes, to_vec};\r\n\r\n#[columnar(vec, ser, de)]                // this struct can be a row of vec-like container\r\nstruct RowStruct {\r\n    name: String,\r\n    #[columnar(strategy = \"DeltaRle\")]   // this field will be encoded by `DeltaRle`\r\n    id: u64,\r\n    #[columnar(strategy = \"Rle\")]        // this field will be encoded by `Rle`\r\n    gender: String,\r\n    #[columnar(strategy = \"BoolRle\")]    // this field will be encoded by `BoolRle`\r\n    married: bool\r\n    #[columnar(optional, index = 0)]     // This field is optional, which means that this field can be added in this version or deleted in a future version\r\n    future: String\r\n    #[columnar(strategy = \"DeltaOfDelta\")] // this field will be encoded by `DeltaOfDelta`\r\n    time: i64\r\n}\r\n\r\n#[columnar(ser, de)]                    // derive `Serialize` and `Deserialize`\r\nstruct TableStruct\u003c'a\u003e {\r\n    #[columnar(class = \"vec\")]          // this field is a vec-like table container\r\n    pub data: Vec\u003cRowStruct\u003e,\r\n    #[columnar(borrow)]                 // the same as `#[serde(borrow)]`\r\n    pub text: Cow\u003c'a, str\u003e\r\n    #[columnar(skip)]                   // the same as `#[serde(skip)]`\r\n    pub ignore: u8\r\n    #[columnar(optional, index = 0)]    // table container also supports optional field\r\n    pub other_data: u64\r\n\r\n}\r\n\r\nlet table = TableStruct::new(...);\r\nlet bytes = serde_columnar::to_vec(\u0026table).unwrap();\r\nlet table_from_bytes = serde_columnar::from_bytes::\u003cTableStruct\u003e(\u0026bytes).unwrap();\r\n\r\n```\r\n\r\nYou can find more examples of `serde_columnar` in `examples` and `tests`.\r\n\r\n### Iterable\r\n\r\nWhen we use columnar for compression encoding, there is a premise that the field is iterable. So we can completely borrow the encoded bytes to obtain all the data in the form of iterator during deserialization without directly allocating the memory of all the data. This implementation can also be achieved completely through macros.\r\n\r\nTo use iter mode when deserializing, you only need to do 3 things:\r\n\r\n1. mark all row struct with `iterable`\r\n2. mark the field of row container with `iter=\"...\"`\r\n3. use `serde_columnar::iter_from_bytes` to deserialize\r\n\r\n```rust\r\n#[columnar(vec, ser, de, iterable)]\r\nstruct Row{\r\n  #[columnar(strategy=\"Rle\")]\r\n  rle: String\r\n  #[columnar(strategy=\"DeltaRle\")]\r\n  delta_rle: u64\r\n  other: u8\r\n}\r\n\r\n#[columnar(ser, de)]\r\nstruct Table{\r\n  #[columnar(class=\"vec\", iter=\"Row\")]\r\n  vec: Vec\u003cRow\u003e,\r\n  other: u8\r\n}\r\n\r\nlet table = Table::new(...);\r\nlet bytes = serde_columnar::to_vec(\u0026table).unwrap();\r\nlet table_iter = serde_columnar::iter_from_bytes::\u003cTable\u003e(\u0026bytes).unwrap();\r\n\r\n```\r\n\r\n## Acknowledgements\r\n\r\n- [serde](https://github.com/serde-rs/serde): Serialization framework for Rust.\r\n- [postcard](https://github.com/jamesmunns/postcard): Postcard is a #![no_std] focused serializer and deserializer for Serde. We use it as serializer and deserializer in order to provide VLE and ZigZag encoding.\r\n- [Automerge](https://github.com/automerge/automerge): Automerge is an excellent crdt framework, we reused the code related to RLE Encoding in it.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floro-dev%2Fcolumnar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floro-dev%2Fcolumnar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floro-dev%2Fcolumnar/lists"}