{"id":13563730,"url":"https://github.com/jubako/arx","last_synced_at":"2025-04-03T20:31:32.116Z","repository":{"id":142352178,"uuid":"609217717","full_name":"jubako/arx","owner":"jubako","description":"  Store files and directory in an archive. Like tar, but faster and with direct random access.","archived":false,"fork":false,"pushed_at":"2025-03-18T07:48:46.000Z","size":815,"stargazers_count":155,"open_issues_count":14,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-18T08:37:42.245Z","etag":null,"topics":["archive","compression","file-format","fs","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jubako.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE-MIT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["jubako"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":"jubako","issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2023-03-03T16:12:32.000Z","updated_at":"2025-03-15T03:12:17.000Z","dependencies_parsed_at":"2024-08-28T10:50:32.743Z","dependency_job_id":"30b98b30-9d8f-4f2e-a9ca-e3d16c76c05c","html_url":"https://github.com/jubako/arx","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jubako%2Farx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jubako%2Farx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jubako%2Farx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jubako%2Farx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jubako","download_url":"https://codeload.github.com/jubako/arx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247074679,"owners_count":20879298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive","compression","file-format","fs","rust"],"created_at":"2024-08-01T13:01:22.705Z","updated_at":"2025-04-03T20:31:32.108Z","avatar_url":"https://github.com/jubako.png","language":"Rust","funding_links":["https://github.com/sponsors/jubako","https://liberapay.com/jubako","https://liberapay.com/jubako/donate","https://buymeacoffee.com/jubako"],"categories":["Rust"],"sub_categories":[],"readme":"# arx: A Fast, Mountable File Archive\n\nArx is a high-performance file archive format built upon the [Jubako container format](https://github.com/jubako/jubako).\nIt offers a compelling alternative to traditional archive formats like zip and tar, providing significant speed advantages,\nespecially for large archives and random access operations. Arx archives can even be mounted as read-only filesystems.\n\n## Key Features\n\n* **Fast Creation and Extraction:**  Arx leverages optimized compression algorithms and a structured data layout for significantly faster archive creation and extraction times compared to traditional methods, particularly for larger datasets.\n* **Random Access:**  Access individual files within the archive without needing to decompress the entire archive.  This is particularly beneficial for large archives.\n* **Read-Only Mounting (Linux and MacoOS):** Mount Arx archives as read-only filesystems using FUSE, allowing you to directly access and work with files within the archive without decompression.\n* **Versatile Compression:** Supports various compression algorithms, including zstd (default), lz4, and lzma, allowing you to choose the best option for your data and performance needs.\n* **Comprehensive CLI Tool:** A command-line interface simplifies archive creation, extraction, listing, and mounting.\n* **Python Bindings:**  A Python wrapper facilitates integration with Python projects.\n\n\n## Installation\n\n### Using Cargo\n\nFist, make sure you have the following dependencies installed:\n\n* fuse3: you need the `fuse3.pc` pkgconfig file, depending on your OS it should\ncome with the development package of the `fuse3` library\n\nThe easiest way to install `arx` is via Cargo, Rust's package manager:\n\n```bash\ncargo install arx\n```\n\n### Pre-built Binaries\n\nPre-built binaries for Windows, macOS, and Linux are available for each release on [GitHub Releases](https://github.com/jubako/arx/releases). Download the appropriate binary for your operating system and add it to your system's `PATH` environment variable.\n\n## Usage Examples\n\n**Create an Archive:**\n\nCreate an archive named `my_archive.arx` from the directory `my_directory`:\n\n```bash\narx create -o my_archive.arx -r my_directory\n```\nThe `-r` flag indicates recursive inclusion of subdirectories.  You can omit this for non-recursive creation.\n\nTo strip a common prefix from the file paths within the archive, use the `--strip-prefix` option:\n\n```bash\narx create -o my_archive.arx -r --strip-prefix /home/user/documents /home/user/documents/my_directory\n```\n\n**Extract an Archive:**\n\nExtract the contents of `my_archive.arx` to the directory `my_output_dir`:\n\n```bash\narx extract my_archive.arx -C my_output_dir\n```\n\nThe `-C` flag specifies the output directory. If omitted, extraction happens in the current directory.\n\n**List Archive Contents:**\n\nList the files and directories within `my_archive.arx`:\n\n```bash\narx list my_archive.arx\n```\n\nFor a more machine-readable output suitable for scripting, use the `--stable-output` option:\n\n```bash\narx list --stable-output my_archive.arx\n```\n\n**Dump a Single File:**\n\nDump the contents of a specific file (`my_directory/my_file.txt`) within the archive to standard output:\n\n```bash\narx dump my_archive.arx my_directory/my_file.txt\n```\n\nTo redirect the output to a file, use redirection:\n\n```bash\narx dump my_archive.arx my_directory/my_file.txt my_file.txt\n```\n\n**Mount the Archive (Linux and MacOS):**\n\nMount `my_archive.arx` to a mount point (requires `libfuse-dev` on Linux and `macfuse` on macOS):\n\n```bash\nmkdir mount_point\narx mount my_archive.arx mount_point\n```\n\nUnmount using the standard `umount` command. If `mount_point` is not provided, a temporary mount point will be created.\nThe `arx mount` command runs in the background by default. Use the `--foreground` flag to keep it in the foreground.\n\n**Convert Zip/Tar Archives:**\n\n\nConvert a zip archive (`my_archive.zip`) or a tar archive (`my_archive.tar.gz`) to an Arx archive:\n\n```bash\nzip2arx -o my_archive.arx my_archive.zip\ntar2arx -o my_archive.arx my_archive.tar.gz\n```\n\nYou may need to install `zip2arx` and `tar2arx` tools, the same you have installed `arx` tool.\n\nRemote tar archives can also be converted using `tar2arx`:\n\n```bash\ntar2arx -o my_archive.arx https://example.com/my_archive.tar.gz\n```\n\n## Performance\n\nThe following tables compare the performance of Arx to different archive formats.\nTests were conducted on various datasets (the entire Linux kernel, its drivers directory, and its documentation directory) stored on an SSD.\nAll tests were run on a tmpfs (archive and extracted files stored in memory).\nMount diff time measures the time to diff the mounted archive with the source directory using `diff -r`.\nMounting of tar and zip archives was performed using the `archivemount` tool.\nArx mount is implemented using the fuse API.\nSquashfs was mounted using the kernel; SquashfsFuse was mounted using the fuse API; Only `Mount diff` differs between the two.\n\n\"Mount diff\" times for tar and zip are significantly longer and may not always be fully measured depending on the dataset and system specifications.\n\nThe comparaison script is available at [script/compare_archive.py](https://github.com/jubako/arx/blob/main/script/compare_archive.py)\n\n\n**Linux doc (Documentation directory only of Linux source code):**\n\n|     Type     |  Creation  |   Size   |  Extract   |  Listing   | Mount diff |    Dump    |\n| ------------ | ---------- | -------- | ---------- | ---------- | ---------- | ---------- |\n|          Arx | 150ms963μs | 11.10 MB | 038ms395μs | 004ms051μs | 299ms764μs | 005ms618μs |\n|           FS | 150ms639μs | 38.45 MB | 106ms821μs | 006ms962μs | 077ms414μs |      498μs |\n|     Squashfs | 103ms076μs | 10.60 MB | 098ms787μs | 005ms365μs | 261ms533μs | 002ms088μs |\n| SquashfsFuse | 097ms863μs | 10.60 MB |          - |          - | 748ms597μs |          - |\n|          Tar | 141ms079μs |  9.68 MB | 065ms744μs | 041ms015μs |     02m41s | 042ms143μs |\n|          Zip |   01s083ms | 15.22 MB | 388ms720μs | 037ms044μs |     03m06s | 014ms088μs |\n\n**Ratio `\u003cArchive\u003e time / Arx time` (A ratio \u003e 100% means Arx is better):**\n\n|     Type     | Creation | Size | Extract | Listing | Mount diff | Dump |\n| ------------ | -------- | ---- | ------- | ------- | ---------- | ---- |\n|           FS |     100% | 346% |    278% |    172% |        26% |   9% |\n|     Squashfs |      68% |  95% |    257% |    132% |        87% |  37% |\n| SquashfsFuse |      65% |  95% |       - |       - |       250% |    - |\n|          Tar |      93% |  87% |    171% |   1012% |     53997% | 750% |\n|          Zip |     718% | 137% |   1012% |    914% |     62350% | 251% |\n\n\n**Linux Driver (Driver directory only of Linux source code):**\n\n|     Type     |  Creation  |   Size    |  Extract   |  Listing   | Mount diff |    Dump    |\n| ------------ | ---------- | --------- | ---------- | ---------- | ---------- | ---------- |\n|          Arx |   01s060ms |  98.23 MB | 241ms699μs | 009ms516μs |   01s290ms | 007ms193μs |\n|           FS | 778ms095μs | 799.02 MB | 523ms191μs | 021ms578μs | 467ms559μs |      495μs |\n|     Squashfs | 829ms886μs | 121.70 MB | 435ms851μs | 012ms289μs |   01s629ms | 002ms190μs |\n| SquashfsFuse | 829ms237μs | 121.70 MB |          - |          - |   03s823ms |          - |\n|          Tar | 911ms042μs |  97.96 MB | 515ms178μs | 472ms060μs |          - | 504ms231μs |\n|          Zip |   20s498ms | 141.91 MB |   03s665ms | 098ms194μs |          - | 034ms481μs |\n\n**Ratio `\u003cArchive\u003e time / Arx time` (A ratio \u003e 100% means Arx is better):**\n\n|     Type     | Creation | Size | Extract | Listing | Mount diff | Dump  |\n| ------------ | -------- | ---- | ------- | ------- | ---------- | ----- |\n|           FS |      73% | 813% |    216% |    227% |        36% |    7% |\n|     Squashfs |      78% | 124% |    180% |    129% |       126% |   30% |\n| SquashfsFuse |      78% | 124% |       - |       - |       296% |     - |\n|          Tar |      86% | 100% |    213% |   4961% |          - | 7010% |\n|          Zip |    1932% | 144% |   1516% |   1032% |          - |  479% |\n\n\n**Linux Source Code (Entire Linux source code):**\n\n|     Type     | Creation |   Size    |  Extract   |  Listing   | Mount diff |    Dump    |\n| ------------ | -------- | --------- | ---------- | ---------- | ---------- | ---------- |\n|          Arx | 02s104ms | 170.97 MB | 435ms846μs | 022ms238μs |   02s829ms | 010ms613μs |\n|           FS | 01s605ms |   1.12 GB |   01s046ms | 043ms358μs | 943ms546μs |      493μs |\n|     Squashfs | 01s430ms | 201.43 MB | 725ms532μs | 024ms050μs |   03s272ms | 002ms374μs |\n| SquashfsFuse | 01s417ms | 201.43 MB |          - |          - |   13s864ms |          - |\n|          Tar | 01s479ms | 168.77 MB | 938ms758μs | 799ms550μs |          - | 802ms427μs |\n|          Zip | 31s810ms | 252.96 MB |   06s260ms | 256ms137μs |          - | 045ms722μs |\n\n**Ratio `\u003cArchive\u003e time / Arx time` (A ratio \u003e 100% means Arx is better):**\n\n|     Type     | Creation | Size | Extract | Listing | Mount diff | Dump  |\n| ------------ | -------- | ---- | ------- | ------- | ---------- | ----- |\n|           FS |      76% | 674% |    240% |    195% |        33% |    5% |\n|     Squashfs |      68% | 118% |    166% |    108% |       116% |   22% |\n| SquashfsFuse |      67% | 118% |       - |       - |       490% |     - |\n|          Tar |      70% |  99% |    215% |   3595% |          - | 7561% |\n|          Zip |    1511% | 148% |   1436% |   1152% |          - |  431% |\n\n**Kernel Compilation Time (Time needed to compile the whole kernel with default configuration `-j8`):**\n\n| Type | Compilation |\n| ---- | ----------- |\n|  Arx |         40m |\n|   FS |         32m |\n\n\nArx archives are slightly larger (about 1%) than tar.zst archives but 15% smaller than squashfs. Creation and full extraction times are comparable to other formats, but listing files and accessing individual files from the archive are much faster using arx or squashfs. Access time is almost constant independently of the archive size, unlike tar, where access time increases significantly with archive size. Mounting an arx archive makes the archive usable without extraction.\n\n\n## Contributing\n\nContributions are welcome! Please open an issue or submit a pull request.\n\n## Sponsoring\n\nI ([@mgautierfr](https://github.com/mgautierfr)) am a freelance developer. All jubako projects are created in my free time, which competes with my paid work.\nIf you want me to be able to spend more time on Jubako projects, please consider [sponsoring me](https://github.com/sponsors/jubako).\nYou can also donate on [liberapay](https://liberapay.com/jubako/donate) or [buy me a coffee](https://buymeacoffee.com/jubako).\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE-MIT](LICENSE-MIT) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjubako%2Farx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjubako%2Farx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjubako%2Farx/lists"}