{"id":28419967,"url":"https://github.com/cjermain/capnp2arrow","last_synced_at":"2025-06-28T18:32:00.573Z","repository":{"id":296606000,"uuid":"646619623","full_name":"cjermain/capnp2arrow","owner":"cjermain","description":"Cap'N Proto to Arrow data transfer","archived":false,"fork":false,"pushed_at":"2024-04-15T19:21:30.000Z","size":38,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-01T08:33:42.738Z","etag":null,"topics":["arrow","arrow2","capnp","capnproto","polars","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cjermain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-29T00:35:00.000Z","updated_at":"2024-05-25T17:53:59.000Z","dependencies_parsed_at":"2025-06-01T08:36:47.897Z","dependency_job_id":"3ca5072e-747b-4c27-8d61-96869f971238","html_url":"https://github.com/cjermain/capnp2arrow","commit_stats":null,"previous_names":["cjermain/capnp2arrow"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cjermain/capnp2arrow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjermain%2Fcapnp2arrow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjermain%2Fcapnp2arrow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjermain%2Fcapnp2arrow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjermain%2Fcapnp2arrow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cjermain","download_url":"https://codeload.github.com/cjermain/capnp2arrow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjermain%2Fcapnp2arrow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262476192,"owners_count":23317231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","arrow2","capnp","capnproto","polars","rust"],"created_at":"2025-06-04T22:44:37.521Z","updated_at":"2025-06-28T18:32:00.560Z","avatar_url":"https://github.com/cjermain.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# capnp2arrow\n\nThis is a work-in-progress demonstration of reading a series of Cap'N Proto messages into Arrow. The dynamic value Reader is used to flexibly traverse arbitrary schemas, allowing the library to be schema-agnostic.\n\n## Setup\n\n```\nrustup override set nightly\nsudo apt install capnproto  # install compiler\n```\n\nGenerate an id: `capnp id`\n\n## Demo\n\nCreate a JSON Lines file with a new-line separated list of points:\n```\ncat \u003c\u003c EOF \u003e points.jsonl\n{\"values\": [{\"x\": 0, \"y\": 1}, {\"x\": -1, \"y\": 2}]}\n{\"values\": [{\"x\": 0, \"y\": 0}]}\n{\"values\": [{\"x\": -2, \"y\": 3}]}\nEOF\n```\n\nConvert the JSONL to binary Cap'N Proto messages based on the schema:\n```\ncat points.jsonl | capnp convert json:binary ./src/schema/point.capnp Points \u003e points.bin\n```\n\nRun the binary messages through the demo:\n```\n$ cat points.bin | cargo run\nshape: (3, 1)\n┌─────────────────────────┐\n│ values                  │\n│ ---                     │\n│ list[struct[2]]         │\n╞═════════════════════════╡\n│ [{0.0,1.0}, {-1.0,2.0}] │\n│ [{0.0,0.0}]             │\n│ [{-2.0,3.0}]            │\n└─────────────────────────┘\n```\n\n## Tests\n\n```\ncargo test\n```\n\nThe test schema is from the capnproto-rust repo:\n```\nwget -qO- https://raw.githubusercontent.com/capnproto/capnproto-rust/master/capnpc/test/test.capnp \u003e tests/test.capnp\n```\n\n## Debug\n\n```\nrust-gdb -q target/debug/capnp2arrow\n(gdb) b rust_panic\n(gdb) r \u003c points.bin\n```\n\n\n## References\n\n1. Reflection based `Debug` implementation: https://github.com/capnproto/capnproto-rust/blob/f7c86befe11b27f33c2a45957d402abff2b9e347/capnp/src/stringify.rs\n\n2. Reflection based example: https://github.com/capnproto/capnproto-rust/blob/master/example/fill_random_values/src/lib.rs\n\n3. Cap'N Proto `TypeVariant`: https://docs.rs/capnp/latest/capnp/introspect/enum.TypeVariant.html\n\n4. Arrow2 `DataTypes`: https://docs.rs/arrow2/latest/arrow2/datatypes/enum.DataType.html\n\n5. Cap'N Proto Language Reference: https://capnproto.org/language.html\n\n6. Cap'N Proto test schema: https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/capnp/test.capnp\n\n7. Cap'N Proto test JSON: https://github.com/capnproto/capnproto/blob/master/c%2B%2B/src/capnp/testdata/pretty.json\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcjermain%2Fcapnp2arrow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcjermain%2Fcapnp2arrow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcjermain%2Fcapnp2arrow/lists"}