{"id":51770642,"url":"https://github.com/afernandes/zstd-prefix-validation","last_synced_at":"2026-07-20T00:05:53.654Z","repository":{"id":364381257,"uuid":"1267560553","full_name":"afernandes/zstd-prefix-validation","owner":"afernandes","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-12T19:29:19.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T21:21:55.360Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/afernandes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-12T16:47:27.000Z","updated_at":"2026-06-12T19:29:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/afernandes/zstd-prefix-validation","commit_stats":null,"previous_names":["afernandes/zstd-prefix-validation"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/afernandes/zstd-prefix-validation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afernandes%2Fzstd-prefix-validation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afernandes%2Fzstd-prefix-validation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afernandes%2Fzstd-prefix-validation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afernandes%2Fzstd-prefix-validation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/afernandes","download_url":"https://codeload.github.com/afernandes/zstd-prefix-validation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afernandes%2Fzstd-prefix-validation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35668475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-19T02:00:06.923Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-20T00:05:49.534Z","updated_at":"2026-07-20T00:05:53.645Z","avatar_url":"https://github.com/afernandes.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zstandard `SetPrefix` validation — dotnet/runtime#129214\n\nMeasurement harness and raw results behind the numbers posted in\n[dotnet/runtime#129214](https://github.com/dotnet/runtime/issues/129214)\n(*[API Proposal]: Expose HashLog and ChainLog on ZstandardCompressionOptions*).\n\nIt measures how effective `ZstandardEncoder.SetPrefix` (delta compression / \"patch-from\") is per\nquality level, on synthetic self-delta probes and on real version pairs, through both the .NET 11\nmanaged API and vanilla native libzstd 1.5.7 (where `ZSTD_c_hashLog` — the knob the API proposal\nexposes — can be set).\n\n## Findings (all measured, every real-pair delta round-trip-verified via SHA-256)\n\n1. **Quality ≤ 15 is unaffected by the prefix-truncation issue** at any tested size: the\n   self-delta probe stays at 0.011% with 32 MB … 1280 MB prefixes (`results/expA`, `expB`).\n   On real version pairs the cost of staying at q15 instead of bt levels + `hashLog` is ~1.7–2.2×.\n2. **The probe's catastrophic cliff (q16–19, ≥48 MB → 100%) is content-specific, not\n   size-specific.** Self-delta with *real* content (a slice of `linux-6.12.92.tar` as its own\n   prefix, same parameters) gives 0.017–0.019% where the random-content probe gives 100.002%\n   (`expG` vs `expA`/`expF`). On real pairs, managed q19/q22 delta a 1.5 GB prefix just fine —\n   aligned or with all 92,418 tar entries reversed (`expC`, `expE2`).\n3. **LDM is what rescues real content.** With LDM force-disabled (`ZSTD_c_enableLongDistanceMatching = 2`,\n   native-only — the managed `false` maps to 0 = auto = still ON at `windowLog ≥ 27`), native q19 with\n   default tables collapses to 9.35% on the linux pair and 47.18% on a production pair; with\n   `hashLog = 28` it recovers to 0.11% **without LDM** (`expE2`). The knob makes prefix referencing\n   robust independently of whether LDM can find the matches.\n4. **`hashLog` specifically unlocks the bt strategies**: at q15 it changes essentially nothing\n   (2,737,868 → 2,728,695 bytes on the linux pair, −0.3%), consistent with the truncation\n   `1 \u003c\u003c MAX(hashLog+3, chainLog+1)` applying under bt match-finders.\n5. The managed wrapper is a faithful pass-through: native runs at default parameters produce\n   **byte-identical** deltas to the managed API (delta SHA-256 match at q19 and q15, `expJ`).\n\nHeadline table — `linux-6.12.92.tar → linux-6.12.93.tar` (1,548,513,280 → 1,548,564,480 bytes,\nthe same pair used by the maintainer in the issue thread; `WindowLog=31`, LDM on, single thread):\n\n| config | delta bytes | % of target | wall time |\n|---|--:|--:|--:|\n| managed q3 | 2,581,560 | 0.1667% | 3.2 s |\n| managed q15 | 2,737,868 | 0.1768% | 20.2 s |\n| managed q19 | 1,573,757 | 0.1016% | 169.7 s |\n| managed q22 | 1,276,452 | 0.0824% | 384.4 s |\n| native q19 + `hashLog=28` | 1,387,510 | 0.0896% | 536.1 s |\n| native q15 + `hashLog=28` | 2,728,695 | 0.1762% | 267.8 s |\n\n## Layout\n\n- `Probe/` — .NET 11 console driver. Subcommands:\n  - `matrix \u003cqualities\u003e \u003csizesMi\u003e [wlog]` — synthetic self-delta (random buffer, seed 42, compressed\n    against itself as prefix; an effective prefix ⇒ ~0% output). Literal repro from the issue.\n  - `real \u003cbase\u003e \u003ctarget\u003e managed \u003cq\u003e` — managed patch-from on real files + SHA-256 round-trip.\n  - `real \u003cbase\u003e \u003ctarget\u003e native \u003cq\u003e \u003chashLog\u003e [ldm]` — native libzstd patch-from\n    (`ZSTD_CCtx_refPrefix` + `ZSTD_compressStream2`), optional `hashLog`, LDM switch (0 auto / 1 on /\n    2 force-off) + SHA-256 round-trip.\n  - `selfslice \u003cfile\u003e \u003csizeMi\u003e \u003cq\u003e` — self-delta with real content.\n  - `revtar \u003cin.tar\u003e \u003cout.tar\u003e` — same entries, reverse order (matches at non-constant offsets,\n    so repeat-offsets can't chain).\n- `Tests/` — xunit; one test per numeric fact cited in the thread (96 MiB scale).\n- `results/` — raw outputs of every run cited in the issue (see `results/README.md` for the map).\n- `setup.ps1` — downloads the datasets and native libzstd, SHA-256-verified.\n- `run-experiments.ps1` — reproduces everything in order (~1.5–2 h total).\n\n## Requirements\n\n- .NET SDK `11.0.100-preview.4.26230.115` (pinned by `global.json`) — same as the issue report.\n- Windows x64 (the native runs load the official `libzstd.dll` from the\n  [zstd v1.5.7 release](https://github.com/facebook/zstd/releases/tag/v1.5.7),\n  SHA-256 `8f07e1112ed283e5cd2798833e9a3c32d8961381bc36da04af57a1b0ca9bd40b`).\n  On Linux the `DllImport(\"libzstd\")` resolves a system `libzstd.so` — use 1.5.7 for comparable\n  numbers (untested here).\n- ≥ 16 GB RAM for the 1.5 GB pairs (prefix + target + output + tables are all in memory).\n\n## Datasets\n\n| file | bytes | SHA-256 |\n|---|--:|---|\n| `data/linux-6.12.92.tar` | 1,548,513,280 | `6357d098952cad2d4fb5179da50f3cfe883085fec50fdcaf169f2415b7e01b63` |\n| `data/linux-6.12.93.tar` | 1,548,564,480 | `5c9cd2d34b009dd237489906cd2beba122aa2a17680048fb3746817d06d5c345` |\n\nBoth come from `cdn.kernel.org` (`.tar.xz` decompressed); the byte sizes match the listing the\nmaintainer posted in the issue, so the pairs are identical.\n\nThe **production pair** cited in the issue (two adjacent releases of a 1.35 GB install-tree tar:\n.NET assemblies + APK + MSIX) is proprietary and not redistributable; its raw result lines are\npreserved in `results/` so the numbers can be inspected, but those runs cannot be re-executed from\nthis repository.\n\n## Methodology notes\n\n- `WindowLog` = smallest value covering prefix+input, capped at zstd's max (31). For the 1.5 GB\n  pairs that cap is hit; matches at the whole-file-aligned distance (≈ the prefix length) remain\n  reachable throughout, and every delta decodes correctly with `refPrefix` (verified).\n- LDM (`EnableLongDistanceMatching`) is on in all runs unless stated; note the managed `false`\n  cannot force-disable it at `windowLog ≥ 27` (native `0` = auto), which is why LDM-off controls\n  use the native path with value `2` (`ZSTD_ps_disable`).\n- Timing is `Stopwatch` around the compress call only (file I/O excluded), single-threaded,\n  one process per configuration, on an i7-12700H / 64 GB. Output sizes are deterministic;\n  times are single-run and indicative. BenchmarkDotNet was deliberately not used: runs take\n  seconds to minutes each and the primary metric (output size) is deterministic, so statistical\n  repetition would multiply machine-hours without changing any conclusion.\n- `results/expE-controls.txt` is the first, partially invalid attempt of the reversed-tar\n  experiment (a `revtar` crash left a 0-byte tar; lines with `targetBytes=0` are garbage).\n  It is kept for transparency; `expE2` is the valid rerun.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafernandes%2Fzstd-prefix-validation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fafernandes%2Fzstd-prefix-validation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafernandes%2Fzstd-prefix-validation/lists"}