{"id":20673699,"url":"https://github.com/cpg314/squashfs-async","last_synced_at":"2025-04-13T17:13:34.405Z","repository":{"id":255083847,"uuid":"597706052","full_name":"cpg314/squashfs-async","owner":"cpg314","description":"Parsing and reading of SquashFS archives, on top of any implementor of the tokio::io::AsyncRead and tokio::io::AsyncSeek traits.","archived":false,"fork":false,"pushed_at":"2025-02-25T22:30:08.000Z","size":57,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T08:04:49.229Z","etag":null,"topics":["async","rust","squashfs","tokio"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpg314.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-05T11:48:46.000Z","updated_at":"2025-02-25T22:30:12.000Z","dependencies_parsed_at":"2024-08-27T23:15:33.597Z","dependency_job_id":"4409cea2-1d57-47d6-bdd7-38d7a1cfb15b","html_url":"https://github.com/cpg314/squashfs-async","commit_stats":null,"previous_names":["cpg314/squashfs-async"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsquashfs-async","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsquashfs-async/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsquashfs-async/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fsquashfs-async/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpg314","download_url":"https://codeload.github.com/cpg314/squashfs-async/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248519555,"owners_count":21117761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","rust","squashfs","tokio"],"created_at":"2024-11-16T20:42:17.975Z","updated_at":"2025-04-13T17:13:34.381Z","avatar_url":"https://github.com/cpg314.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"Parsing and reading of [SquashFS](https://en.wikipedia.org/wiki/SquashFS) archives, on top of any\nimplementor of the [`tokio::io::AsyncRead`] and [`tokio::io::AsyncSeek`] traits.\n\nMore precisely, this crate provides:\n\n- A [`SquashFs`] structure to read SquashFS archives on top of any asynchronous reader.\n- An implementation of [`fuser_async::Filesystem`] on [`SquashFs`], allowing to easily build [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) filesystems using SquashFS archives.\n- A `squashfuse-rs` binary for mounting SquashFS images via FUSE, with async IO and multithreaded decompression.\n\n## Motivation: multithreaded/async SquashFS reading\n\nThe main motivation was to provide a [`squashfuse`](https://github.com/vasi/squashfuse/pull/70#issuecomment-1249788158) implementation that could:\n\n- Decompress blocks in parallel.\n- Benefit from async I/O when relevant (mostly with the case of a networked backend in mind), with easy integration with [`tokio::io`].\n\nTo the author's understanding, `squashfuse` [uses a single-threaded FUSE loop](https://github.com/hpc/charliecloud/issues/1157), and while the kernel driver does multithreaded decompression (when compiled with this option), it doesn't support parallel reads. Note that a [patch](https://github.com/vasi/squashfuse/pull/70) exists to add multi-threading to the low-level squashfuse see [Benchmarks](#Benchmarks).\n\n### Example: squashfs-on-S3\n\nThis crate has been used to implement a FUSE filesystem providing transparent access to squashfs images hosted on an S3 API, using the S3 example in [`fuser_async`]. With a local [MinIO](https://github.com/minio/minio) server, throughput of 365 MB/s (resp. 680 MB/s) are achieved for sequential (resp. parallel) access to zstd1-compressed images with 20 MB files.\n\n## `squashfuse-rs` binary\n\nThe `squashfuse-rs` binary is an example that implements an analogue to `squashfuse` using this crate, allowing to mount squashfs images via FUSE.\n\n```console\n$ squashfuse-rs --help\nUSAGE:\nsquashfuse-rs [OPTIONS] \u003cINPUT\u003e \u003cMOUNTPOINT\u003e\n\nARGS:\n   \u003cINPUT\u003e         Input squashfs image\n   \u003cMOUNTPOINT\u003e    Mountpoint\n\nOPTIONS:\n       --backend \u003cBACKEND\u003e              [default: memmap] [possible values: tokio, async-fs, memmap]\n       --cache-mb \u003cCACHE_MB\u003e            Cache size (MB) [default: 100]\n   -d, --debug\n       --direct-limit \u003cDIRECT_LIMIT\u003e    Limit (B) for fetching small files with direct access [default: 0]\n   -h, --help                           Print help information\n       --readers \u003cREADERS\u003e              Number of readers [default: 4]\n\n```\n\n## Benchmarks\n\nThe following benchmarks (see `tests/`) compute the mean and standard deviation of 10 runs, dropping caches after each run, with the following variations:\n\n- Sequential or parallel (with 4 threads) read.\n- Compressed archive (gzip and zstd1) or not.\n- Different backends for reading the underlying file in `squashfuse-rs`.\n- The archives are either:\n  - Case A: Containing sixteen random files of 20 MB each, generated by `tests/testdata.rs` (note that given that the files are random, the zstd compression has minimal effect on the data blocks).\n  - Case B: Containing three hundred 20 MB images (with a compression ratio of 1.1 with zstd-1).\n\nEntries are normalized by `(case, comp)` pair (i.e. pairs of rows) with respect to the duration of the sequential `squashfuse` run. Number smaller than 1 indicate faster results than this baseline. The last 3 columns are `squashfuse-rs` with different backends ([`MemMap`](`pools::LocalReadersPoolMemMap`) being the most performant).\n\n| Case |     | Comp. | `squashfuse` | [`squashfuse_ll_mt`](https://github.com/vasi/squashfuse/pull/70) | [`MemMap`](`pools::LocalReadersPoolMemMap`) | [`Tokio`](`pools::LocalReadersPool`) | [`AsyncFs`](`pools::LocalReadersPoolAsyncFs`) |\n| ---- | --- | ----- | ------------ | ---------------------------------------------------------------- | ------------------------------------------- | ------------------------------------ | --------------------------------------------- |\n| A    | Seq | -     | 1            | 1.16                                                             | 1.01                                        | 1.93                                 | 1.56                                          |\n|      | Par | -     | 1.8          | 0.5                                                              | 0.54                                        | 0.8                                  | 0.76                                          |\n|      | Seq | gzip  | 1            | 0.92                                                             | 0.94                                        | 1.79                                 | 1.48                                          |\n|      | Par | gzip  | 2.07         | 0.46                                                             | 0.51                                        | 0.75                                 | 0.71                                          |\n|      | Seq | zstd1 | 1            | 0.96                                                             | 1.04                                        | 1.78                                 | 1.47                                          |\n|      | Par | zstd1 | 2.35         | 0.48                                                             | 0.51                                        | 0.76                                 | 0.71                                          |\n| B    | Seq | -     | 1            | 0.89                                                             | 0.93                                        | 2.08                                 | 1.43                                          |\n|      | Par | -     | 1.6          | 0.54                                                             | 0.6                                         | 0.89                                 | 0.91                                          |\n|      | Seq | zstd1 | 1            | 0.59                                                             | 0.65                                        | 0.98                                 | 0.87                                          |\n|      | Par | zstd1 | 1.07         | 0.3                                                              | 0.35                                        | 0.3                                  | 0.54                                          |\n\n\u003csmall\u003eSmaller numbers are better; numbers smaller than 1 denote an improvement over the baseline\u003c/small\u003e\n\n\u003e [!WARNING]  \n\u003e These should be updated with the latest versions of the code and of `squashfuse`.\n\nTo execute the tests (case A), `cargo` needs to run with root privileges to be able to clear caches between runs, e.g.\n\n```console\n$ N_RUNS=10 CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER='sudo -E' cargo test -r --test main -- --nocapture\n```\n\n## Differences with similar crates\n\n- [`squashfs`](https://crates.io/crates/squashfs) is a work in progress that only supports parsing some structures (superblock, fragment table, uid/gid table).\n- [`backhand`](https://crates.io/crates/backhand) and this crate were implemented indendently at roughly the same time. Some differences are (see also [Limitations](#Limitations) below):\n  - The primary goal of this crate was to allow mounting squashfs images with FUSE, with async IO and multithreaded decompression. `backend` uses a synchronous [`std::io::Read`]/[`std::io::Seek`] backend, while this crate uses a [`tokio::io::AsyncRead`]/[`tokio::io::AsyncSeek`] backend.\n  - This crates provide caching for decompressed blocks.\n  - `backhand` supports write operations, while `squashfs-async` doesn't.\n- [`squashfs-ng-rs`](https://crates.io/crates/squashfs-ng) wraps the C API, while this crate is a pure Rust implementation.\n\n## References\n\n- \u003chttps://dr-emann.github.io/squashfs/\u003e\n- \u003chttps://dr-emann.github.io/squashfs/squashfs.html\u003e\n\n## Limitations/TODOs\n\n- For now, only file and directory inodes are supported.\n- The tables are loaded into memory on initial parsing for caching, rather than being accessed lazily.\n- ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fsquashfs-async","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpg314%2Fsquashfs-async","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fsquashfs-async/lists"}