{"id":13496646,"url":"https://github.com/elfshaker/elfshaker","last_synced_at":"2025-04-11T11:48:20.205Z","repository":{"id":38237782,"uuid":"417082765","full_name":"elfshaker/elfshaker","owner":"elfshaker","description":"elfshaker stores binary objects efficiently","archived":false,"fork":false,"pushed_at":"2024-09-13T15:21:52.000Z","size":706,"stargazers_count":2302,"open_issues_count":19,"forks_count":46,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-04-03T22:05:48.338Z","etag":null,"topics":["binary-storage","executable-packing","version-control"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elfshaker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-14T10:15:44.000Z","updated_at":"2025-03-22T10:41:45.000Z","dependencies_parsed_at":"2024-09-21T02:01:53.515Z","dependency_job_id":"edc50459-5981-444d-8123-9969f623df68","html_url":"https://github.com/elfshaker/elfshaker","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elfshaker%2Felfshaker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elfshaker%2Felfshaker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elfshaker%2Felfshaker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elfshaker%2Felfshaker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elfshaker","download_url":"https://codeload.github.com/elfshaker/elfshaker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248388609,"owners_count":21095418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-storage","executable-packing","version-control"],"created_at":"2024-07-31T19:01:54.336Z","updated_at":"2025-04-11T11:48:20.179Z","avatar_url":"https://github.com/elfshaker.png","language":"Rust","funding_links":[],"categories":["Rust","others"],"sub_categories":[],"readme":"# `elfshaker`\n\n## 400 GiB -\u003e 100 MiB, with 1s access time†; [when applied to clang builds](https://github.com/elfshaker/manyclangs).\n\nelfshaker is a low-footprint, high-performance version control system fine-tuned for binaries.\n\n- elfshaker is a CLI tool written in the [Rust programming language](https://www.rust-lang.org/).\n\n- It stores snapshots of directories into highly-compressed pack files and provides fast on-demand access to the stored files. It is particularly good for storing lots of similar files, for example object files from an incremental build.\n\n- It allows few-second access to any commit of clang with the [manyclangs project](https://github.com/elfshaker/manyclangs). For example, this accelerates bisection of [LLVM](https://llvm.org/) by a factor of 60x! This is done by extracting builds of LLVM on-demand from locally stored elfshaker packs, each of which contains ~1,800 builds and is about 100 MiB in size, even though the full originals would take TiBs to store! Extracting a single build takes 2-4s on modern hardware.\n\n## †Applicability\n\nOr, \"how on earth do you get such a phenomenal result?\".\n\nIt works particularly well for our [presented use case](https://github.com/elfshaker/manyclangs) because storing pre-link object files has these properties:\n\n* There are many files,\n* Most of them don't change very often so there are a lot of duplicate files,\n* When they do change, the deltas of the binaries are not huge.\n\nWe achieve this in manyclangs by compiling object code with the `-ffunction-sections` and `-fdata-sections` compiler flags. This has the effect that if you 'insert' a function into a translation unit, the insertion does not cause all of the addresses to change across the whole object file.\n\nIf you looked at the binary delta on the linked executable from such a change, it will be large, because all of the absolute addresses after the insertion will change, and references to those addresses will change. These address changes are not handled well by compression algorithms, resulting in a poor compression ratio. The effect of this is large: if you compress many revisions of clang executables together, you will see a compression ratio of something like 20%. This is pretty good! But elfshaker achieves a ratio of something closer to 0.01% (or 10,000x), amortized across many builds.\n\n\n## [Installation guide](docs/users/installing.md)\n\n## [Usage guide](docs/users/usage.md)\n\n## Quickstart\n\n1. Consult our [installation](docs/users/installing.md) and [usage guide](docs/users/usage.md), make sure you know what you're doing.\n2. `elfshaker store \u003csnapshot\u003e` -- capture the state of the current working directory into a named snapshot `\u003csnapshot\u003e`.\n3. `elfshaker pack \u003cpack name\u003e` -- capture all 'loose' snapshots into a single pack file (this is what gets you the compression win).\n4. `elfshaker extract \u003csnapshot\u003e` -- restore the state of a previous snapshot into the current working directory.\n\nFor more detail, take a look at our [workflow documentation](https://github.com/elfshaker/elfshaker/blob/main/docs/users/usage.md#workflow) and watch the talk from the 2021 LLVM Developers' Meeting:\n\n\u003ca href=\"https://www.youtube.com/watch?v=l2KWOqEGRwg\"\u003e\n  \u003cimg src=\"https://img.youtube.com/vi/l2KWOqEGRwg/0.jpg\" width=\"240\" alt=\"2021 LLVM Dev Mtg “Manyclangs: Fast bisection with a small storage cost\"\u003e\n\u003c/a\u003e\n\n## System Compatibility\n\nThe following platforms are used for our CI tests:\n\n- Ubuntu 20.04 LTS\n\nBut we aim to support all popular Linux platforms, macOS and Windows in production.\n\nWe officially support the following architectures:\n- AArch64\n- x86-64\n\n## Current Status\n\nThe file format and directory structure is stable. We intend that pack files created with the current elfshaker version will remain compatible with future versions. Please kick the tyres and do your own validation, and file bugs if you find any. We have done a fair amount of validation for our use cases but there may be things we haven't needed yet, so please start a discussion and file issues/patches.\n\n\n## Contributing\n\n*Contributions are highly appreciated.*\nRefer to our [Contributing guide](docs/contributors/contributing.md).\n\n\u003c!-- TODO(veselink1): 🛈 View the [elfshaker API reference](https://elfshaker.github.io/docs). --\u003e\n\n## Contact\n\nThe best way to reach us to join the [elfshaker/community](https://gitter.im/elfshaker/community) on Gitter.\nThe original authors of elfshaker are Peter Waller ([@peterwaller-arm](https://github.com/peterwaller-arm)) \\\u003cpeter.waller@arm.com\\\u003e and Vesko Karaganev ([@veselink1](https://github.com/veselink1)) \\\u003cvesko.karaganev@gmail.com\\\u003e and you may also contact us via email.\n\n## Security\n\nRefer to our [Security policy](SECURITY.md).\n\n## License\n\nelfshaker is licensed under the [Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felfshaker%2Felfshaker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felfshaker%2Felfshaker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felfshaker%2Felfshaker/lists"}