{"id":51562911,"url":"https://github.com/eser/sw3z","last_synced_at":"2026-07-10T12:01:29.865Z","repository":{"id":345835202,"uuid":"1187590033","full_name":"eser/sw3z","owner":"eser","description":"SW³ Zipped Format Specification","archived":false,"fork":false,"pushed_at":"2026-03-21T00:45:18.000Z","size":34,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-13T19:41:29.524Z","etag":null,"topics":["idtech3","lz4-compression","pk3","zip-archive"],"latest_commit_sha":null,"homepage":"https://software3.org","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eser.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":null},"created_at":"2026-03-20T22:47:29.000Z","updated_at":"2026-03-21T15:41:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/eser/sw3z","commit_stats":null,"previous_names":["eser/sw3z"],"tags_count":0,"template":false,"template_full_name":"eser/template","purl":"pkg:github/eser/sw3z","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eser%2Fsw3z","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eser%2Fsw3z/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eser%2Fsw3z/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eser%2Fsw3z/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eser","download_url":"https://codeload.github.com/eser/sw3z/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eser%2Fsw3z/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35330738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["idtech3","lz4-compression","pk3","zip-archive"],"created_at":"2026-07-10T12:01:28.978Z","updated_at":"2026-07-10T12:01:29.848Z","avatar_url":"https://github.com/eser.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# SW3Z — SW³ Zipped Archive Format\n\nSW3Z is a read-optimized game asset archive format for id Tech 3 derived\nengines. It is a modernized replacement for the `.pk3` container format,\ndesigned around the realities of contemporary hardware: NVMe storage,\nVulkan/D3D12 GPU pipelines, high-resolution texture sets, and per-file\nstreaming.\n\n---\n\n## Quick Reference\n\n| Property            | Value                                  |\n| ------------------- | -------------------------------------- |\n| Magic               | `SW3Z` (0x53 0x57 0x33 0x5A)           |\n| Index position      | Start of file                          |\n| Default compression | LZ4 Frame Format                       |\n| Checksum            | CRC32C (Castagnoli)                    |\n| Signing             | Ed25519 (optional)                     |\n| Entry size          | 40 bytes                               |\n| Max archives        | Unlimited (stack/override by priority) |\n\nSee [SPEC.md](./SPEC.md) for the full format specification.\n\n---\n\n## Rationale\n\n### Why not just use pk3?\n\nThe `.pk3` format is a renamed ZIP file using DEFLATE compression. It has served\nthe Quake modding community well since 1999, but it shows its age in several\nways:\n\n- **DEFLATE is slow to decompress** — roughly 300–500 MB/s on a single core.\n  Modern games load dozens of assets per frame; decompression becomes a CPU\n  bottleneck.\n- **Central Directory is at the end** — a reader must seek to the end of the\n  file to discover what is inside, then seek back to read any asset. On local\n  storage this costs two seeks per archive open.\n- **No per-file compression convention** — pk3 tooling applies DEFLATE\n  uniformly. Already-compressed assets (JPEG, OGG, DXT) are re-compressed for no\n  benefit, wasting CPU at load time.\n- **No alignment guarantees** — Vulkan and D3D12 require texture data to be\n  aligned to specific boundaries (256 B, 4 KiB) for direct GPU upload. ZIP\n  provides no mechanism for this.\n- **Weak runtime integrity** — ZIP includes per-file CRC32, but pk3 loaders\n  rarely validate it at load time. Corruption surfaces as visual glitches or\n  crashes with no indication of which file is affected.\n\nSW3Z addresses all of these while preserving what made pk3 great: a simple\nstackable archive system where later archives override earlier ones, enabling\nmods and patches without touching base content.\n\n---\n\n### Container format: why not TAR?\n\nTAR and ZIP are the two obvious starting points. The comparison came down to one\nquestion: **where is the index?**\n\nTAR has no index at all. File headers are inline, immediately before each file's\ndata. Finding a specific asset requires a linear scan from the beginning — O(n).\nFor a game engine that needs to load a texture 60 times per second, this is\nunacceptable.\n\nZIP has a Central Directory at the _end_ of the file. This was a deliberate 1989\ndesign choice to enable sequential single-pass writing — you write files as you\ngo, then append the directory at the end. For network partial downloads, this\nalso allows fetching just the directory via an HTTP Range request, then\nselectively downloading only needed files.\n\nSW3Z takes the best of both and discards the rest:\n\n- **Index at the start**, like neither TAR nor ZIP — enabling a single seek to\n  load the full index on archive open.\n- **Per-file metadata** from ZIP — every entry knows its own offset, sizes, and\n  checksum without scanning.\n- **Simple flat layout** from TAR — no ZIP spec complexity, no local vs. central\n  header duplication.\n\nThe tradeoff is that the writer must know all offsets before writing the header.\nIn practice this means a two-pass write or building the index in memory. Since\nSW3Z is written offline by Go tooling (not streamed), this is a non-issue.\n\n---\n\n### Compression: why LZ4?\n\nOur specific scenario is important here: **assets are converted from pk3 on the\nplayer's local machine at install time**, not downloaded in the SW3Z format.\nThis changes the compression calculus significantly.\n\n| Algorithm      | Decompress  | Compress     | Ratio     | License     |\n| -------------- | ----------- | ------------ | --------- | ----------- |\n| DEFLATE (zlib) | ~400 MB/s   | slow         | good      | free        |\n| LZ4            | ~4,000 MB/s | fast         | moderate  | free        |\n| zstd           | ~1,500 MB/s | configurable | very good | free        |\n| Oodle Kraken   | ~1,800 MB/s | slow         | excellent | proprietary |\n\nBecause compression happens once at install time, compress speed is irrelevant.\nOnly **decompress speed** and **disk footprint** matter at runtime.\n\nLZ4 was chosen because:\n\n1. **Decompress speed is unmatched among free algorithms** — ~4 GB/s approaches\n   RAM bandwidth. This means the CPU is almost never the bottleneck when\n   streaming assets; the GPU pipeline or disk I/O will saturate first.\n2. **Background streaming is central to the engine** — with many small assets\n   loading across frames, LZ4's near-zero CPU overhead keeps the decompression\n   thread from interfering with render and simulation threads.\n3. **No patents, no license fees** — BSD 2-Clause. The same author (Yann Collet,\n   now at Meta) also wrote zstd.\n4. **LZ4 Frame Format is binary-compatible** between Go\n   (`github.com/pierrec/lz4/v4`) and C (`lz4frame.h`), with no glue code\n   required.\n\n**What about zstd?** zstd is also free, also patent-free, and decompresses at\n~1,500 MB/s — fast enough for almost any scenario. It is included in the spec as\ncompression algorithm `0x02` for cases where disk space is a concern (e.g., very\nlarge texture sets). Writers may choose zstd per-file; the engine handles both\ntransparently. Think of LZ4 as the performance default and zstd as the\nspace-saving option.\n\n**What about Oodle?** Excellent ratios and speeds, but requires a commercial\nlicense outside of Unreal Engine. Not appropriate for an open-source format.\n\n---\n\n### Checksum: why CRC32C and not xxHash or SHA-256?\n\nSeveral algorithms were considered for the per-entry integrity checksum.\n\n**SHA-256** was rejected immediately — it is a cryptographic hash, slow by\ndesign, and the threat it defends against (intentional collision) is already\ncovered by Ed25519 archive signing. Using SHA-256 for a corruption check is\nusing a sledgehammer to hang a picture.\n\n**xxHash32 / xxHash64** are excellent non-cryptographic hashes — fast,\nwell-distributed, and widely used. However, Go's standard library has no xxHash\nsupport, requiring an external dependency for what is ultimately a simple\ncorruption check.\n\n**CRC32C (Castagnoli)** was chosen for three reasons:\n\n1. **Zero additional dependencies** — `hash/crc32` is in Go's standard library.\n   In C, hardware intrinsics (x86 SSE4.2 `_mm_crc32_u*`, ARM `__crc32c*`)\n   provide single-instruction implementations; lightweight software fallbacks\n   exist for older CPUs.\n2. **Hardware acceleration is universal** — on modern CPUs, CRC32C runs at\n   near-memcpy speed when using SSE4.2 or ARM CRC instructions. Developers who\n   need maximum throughput can drop in the hardware path with no spec changes.\n3. **Consistency with LZ4** — the LZ4 Frame Format uses CRC32C for its own\n   internal content checksum. Using the same polynomial across both layers\n   avoids having two different checksum algorithms in the same code path.\n\nCRC32C is not cryptographic and cannot detect intentional tampering — nor does\nit need to. That responsibility belongs to the Ed25519 archive signature.\nPer-entry corruption detection uses CRC32C instead — a much better fit for the\ntask.\n\n---\n\n### Archive signing: why Ed25519?\n\nArchive-level integrity (CRC32C per entry) answers \"was this file corrupted?\"\nEd25519 signing answers a different question: \"did a trusted party produce this\narchive?\"\n\nThis matters for:\n\n- **Anti-cheat** — a server can refuse to load unsigned or incorrectly signed\n  archives in competitive play.\n- **Official content verification** — distinguishing first-party releases from\n  community modifications.\n- **Tamper detection** — detecting post-distribution modification of an archive.\n\nEd25519 was chosen over ECDSA P-256 or RSA because:\n\n- 64-byte signatures — minimal EOF overhead\n- Fast verification — relevant when loading many archives at startup\n- Small key sizes — 32-byte public key, easily embedded in engine config\n- Modern, well-audited — no known practical attacks\n\nSHA-256 was explicitly rejected as a signing method. A hash without a private\nkey proves nothing about authorship — anyone can recompute the same hash.\n\nPublic key distribution is intentionally out of scope. Engine integrations are\nexpected to embed or configure trusted public keys independently.\n\n---\n\n### Index structure: why FNV-1a for path hashing?\n\nPath lookup at runtime must be fast. String comparison across thousands of\nentries on every asset load would be unacceptable. A hash map keyed on a 64-bit\npath hash reduces lookup to O(1) with a single integer comparison in the common\ncase.\n\nFNV-1a 64-bit was chosen because:\n\n- Trivially implementable in ~10 lines of C with no dependencies\n- The same hash is already used elsewhere in the codebase for path normalization\n- 64-bit output makes accidental collisions astronomically unlikely in practice\n- In the rare case of a collision, the spec mandates fallback to full string\n  comparison via the string table\n\nPaths are normalized to lowercase with forward slashes before hashing. Directory\npaths include the trailing `/` — `textures/gothic/` and `textures/gothic` are\ndistinct entries with distinct hashes. This rule is explicit in the spec to\nprevent silent cross-implementation incompatibilities.\n\n---\n\n### Alignment: why power-of-2 exponent?\n\nModern GPU APIs (Vulkan, D3D12) and NVMe direct I/O require asset data to begin\nat specific byte boundaries. Rather than a fixed global alignment, SW3Z supports\nper-entry alignment as a power-of-2 exponent stored in one byte.\n\n```c\nuint32_t alignment = 1u \u003c\u003c entry.alignment_exp;  // single instruction\n```\n\nThe exponent encoding was preferred over an enum for a simple reason: it\neliminates the lookup table. Any power-of-2 alignment is representable, and\ndecoding is a single bit-shift. Valid exponents range from 4 (16 bytes, SIMD\nminimum) to 16 (64 KiB, large page / GPU texture alignment). Values outside this\nrange are treated as malformed entries.\n\nPadding bytes are zeroed and not counted in `compressed_size`. Readers MUST use\n`data_offset` to locate assets — never assume contiguous layout.\n\n---\n\n### What was deliberately left out\n\nSeveral features were proposed and rejected for v1.0:\n\n- **File timestamps** — no use case in a game engine; asset freshness is\n  determined by archive priority order, not modification time.\n- **Directory entries as required** — directories are implicit from path\n  strings. Explicit directory entries are supported but optional.\n- **SHA-256 as a signing option** — conflates integrity with authenticity;\n  removed in favor of the cleaner CRC32C + Ed25519 separation.\n- **Arbitrary compression enums** — only three values defined (none, LZ4, zstd).\n  Unknown values cause the entry to be skipped, not the archive to be rejected.\n- **Content-addressable storage** — asset aliasing (multiple paths pointing to\n  the same `data_offset`) covers the primary deduplication use case without the\n  complexity of a full CAS layer.\n\nA machine-readable format description (Kaitai Struct `.ksy`) is planned\npost-v1.0 finalization.\n\n---\n\n## Contributing\n\nSee [CONTRIBUTING.md](./.github/CONTRIBUTING.md) for guidelines.\n\n## License\n\n[Apache-2.0](./LICENSE)\n\n---\n\n_SW3Z named after [SW³](https://software3.org)._\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feser%2Fsw3z","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feser%2Fsw3z","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feser%2Fsw3z/lists"}