{"id":34723925,"url":"https://github.com/spiraldb/streamvbyte-zig","last_synced_at":"2025-12-25T02:16:34.088Z","repository":{"id":237292378,"uuid":"697370691","full_name":"spiraldb/streamvbyte-zig","owner":"spiraldb","description":"Zig port of Stream VByte encoding","archived":false,"fork":false,"pushed_at":"2023-09-27T21:56:23.000Z","size":10,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"develop","last_synced_at":"2025-10-21T19:40:57.894Z","etag":null,"topics":["compression","zig"],"latest_commit_sha":null,"homepage":"","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spiraldb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-27T15:34:58.000Z","updated_at":"2025-06-17T02:55:47.000Z","dependencies_parsed_at":"2024-05-07T14:47:48.198Z","dependency_job_id":null,"html_url":"https://github.com/spiraldb/streamvbyte-zig","commit_stats":null,"previous_names":["spiraldb/streamvbyte-zig"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/spiraldb/streamvbyte-zig","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spiraldb%2Fstreamvbyte-zig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spiraldb%2Fstreamvbyte-zig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spiraldb%2Fstreamvbyte-zig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spiraldb%2Fstreamvbyte-zig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spiraldb","download_url":"https://codeload.github.com/spiraldb/streamvbyte-zig/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spiraldb%2Fstreamvbyte-zig/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28017198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-25T02:00:05.988Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","zig"],"created_at":"2025-12-25T02:16:31.920Z","updated_at":"2025-12-25T02:16:34.069Z","avatar_url":"https://github.com/spiraldb.png","language":"Zig","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zig StreamVByte\n\nA Zig port of [Stream VByte](https://github.com/lemire/streamvbyte).\n\nThis project started as an experiment to explore the feasibility and\nDevX of implementing compression codecs in Zig.\n\nIn particular,\n* Leveraging comptime to avoid writing (or generating) repetitive code.\n* Using the Zig @Vector API for portable SIMD.\n\n### Comptime Lookup Tables\n\nStream VByte leverages lookup tables (LUTs) for pre-computing shuffle\nmasks as well as the lengths of compressed quads.\n\nZig's comptime feature allows us to generate static LUTs at compile-time\nby defining them in regular Zig. This makes it possible for the reader\nto understand the origin of what are otherwise typically opaque \"magic\"\nvalues.\n\n### Zig SIMD\n\nZig leans on LLVM for its `@Vector` SIMD API. While Zig offers a `@shuffle`\nbuiltin to operate on vectors, LLVM (and therefore Zig) require the shuffle\nmask to be comptime known.\n\nSince the shuffle mask is a lookup based on runtime control bits, it isn't\npossible to use the `@shuffle` builtin.\n\n\u003e It is worth noting at this point that we use Zig 0.11.0.\n\nWe _were_ able to generate functions for each of the shuffle masks, and then\nstore these function pointers in a LUT. But since these functions cannot be inlined, the overhead was far too much.\n\nWe also tried to `@cImport` the relevant instrinsics headers. But found that\nZig is currently unable to inline `callconv(.C)` extern functions.\n\nThis left us to implement a shuffle function ourselves using `pshufb`\n(and `tbl.16b`) with Zig's inline ASM. Whilst this isn't ideal, it doesn't\nseem unreasonable for these instructions to be supported by the builtin\n`@shuffle` operator when the operand is only runtime known.\n\n## Benchmarks\n\nWe spent most of our time playing around with shuffle LUTs, so\nthe remainder of the code isn't particularly heavily optimized.\nNor have we run these benchmarks on a variety of architectures.\nTake everything with a handful of salt. Especially given how\nmuch variance there is in the \"benchmarks\".\n\n```bash \"M2 Macbook Air\"\n$ zig build test -Doptimize=ReleaseFast\nZig StreamVByte\n\tEncode 10000000 ints between 0 and 10000 in mean 3467410ns\n\t=\u003e 2883 million ints per second\n\n\tDecode 10000000 ints between 0 and 10000 in mean 1681129ns\n\t=\u003e 5948 million ints per second\n\nOriginal C StreamVByte\n\tEncode 10000000 ints between 0 and 10000 in mean 2065012ns\n\t=\u003e 4842 million ints per second\n\n\tDecode 10000000 ints between 0 and 10000 in mean 1669633ns\n\t=\u003e 5989 million ints per second\n```\n\n---\n\nThe source code is released under the [MIT license](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspiraldb%2Fstreamvbyte-zig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspiraldb%2Fstreamvbyte-zig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspiraldb%2Fstreamvbyte-zig/lists"}