{"id":50465855,"url":"https://github.com/threefoldtech/zos_rfs","last_synced_at":"2026-06-01T07:04:33.644Z","repository":{"id":37091661,"uuid":"172350020","full_name":"threefoldtech/zos_rfs","owner":"threefoldtech","description":"a dedupe filesystem in rust","archived":false,"fork":false,"pushed_at":"2026-05-27T08:46:42.000Z","size":4596,"stargazers_count":3,"open_issues_count":15,"forks_count":1,"subscribers_count":2,"default_branch":"development","last_synced_at":"2026-05-27T10:22:02.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/threefoldtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-02-24T14:48:33.000Z","updated_at":"2026-05-27T09:43:48.000Z","dependencies_parsed_at":"2023-11-06T11:38:12.376Z","dependency_job_id":"ac7632ab-cf58-4736-8cfa-68c97b15ff71","html_url":"https://github.com/threefoldtech/zos_rfs","commit_stats":null,"previous_names":["threefoldtech/mycofs","threefoldtech/rfs"],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/threefoldtech/zos_rfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fzos_rfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fzos_rfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fzos_rfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fzos_rfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/threefoldtech","download_url":"https://codeload.github.com/threefoldtech/zos_rfs/tar.gz/refs/heads/development","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2Fzos_rfs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33763681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-01T07:04:33.570Z","updated_at":"2026-06-01T07:04:33.629Z","avatar_url":"https://github.com/threefoldtech.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ZOS RFS - Remote File System\n\n[![Test](https://github.com/threefoldtech/zos_rfs/actions/workflows/tests.yaml/badge.svg?branch=master)](https://github.com/threefoldtech/zos_rfs/actions/workflows/tests.yaml)\n\nZOS RFS (Remote File System) is a command-line tool written in Rust that enables efficient storage, distribution, and mounting of filesystems across different storage backends. It separates filesystem metadata from content, allowing files to be downloaded on demand rather than up front. ZOS RFS is particularly useful for distributing container images, large datasets, and web content with minimal bandwidth and storage overhead.\n\n## What this is\n\nZOS RFS solves file distribution by storing filesystem metadata in compact \"flist\" files while keeping actual content in pluggable backends. When an flist is mounted, files are fetched lazily — only the blocks that are actually accessed are downloaded. Identical files across different flists are stored only once through content deduplication.\n\n## What this repository contains\n\n- **`rfs` CLI binary** with commands for packing, mounting, unpacking, Docker conversion, cloning, and server operation\n- **Flist metadata format** — compact descriptions of filesystems without embedded content\n- **Multiple storage backends** — directory, ZDB, S3, and HTTP\n- **Web server mode** for browser-based flist management\n- **Comprehensive documentation** in the [docs](./docs) directory covering tutorials, user guides, and architecture\n\n## Key Features\n\n- **Flists**: Compact metadata files that describe filesystems without containing the actual data\n- **On-demand access**: Files are only downloaded when accessed, saving bandwidth and storage\n- **Content deduplication**: Identical files are stored only once, even across different flists\n- **Multiple storage backends**: Store content in directory, ZDB, S3, or HTTP backends\n- **Docker conversion**: Convert Docker images to flists for efficient distribution\n- **Server functionality**: Run a server for web-based flist management\n- **Sharding and replication**: Distribute and replicate content across multiple storage backends\n\n## Role in the stack\n\nZOS RFS is used by ZOS / Zero-OS, the operating system layer used to run and manage nodes, for container image distribution and filesystem provisioning. When a workload is deployed on a node, the required filesystem is fetched on demand via ZOS RFS, minimizing startup time and storage consumption. It integrates with the content-addressed storage layer to ensure efficient deduplication and distribution.\n\n## Relation to ThreeFold\n\nThis technology is used within the ThreeFold ecosystem and was first deployed on the ThreeFold Grid. The component itself is designed as reusable infrastructure technology and should be understood by its technical function first, independent of any specific deployment.\n\n## Ownership\n\nThis repository is owned and maintained by TF-Tech NV, a Belgian company responsible for the development and maintenance of this technology.\n\n## Common Use Cases\n\n### Efficient Docker Image Distribution\n\nConvert Docker images to flists for more efficient distribution and usage:\n\n```bash\n# Convert a Docker image to an flist\nrfs docker -i nginx:latest -s dir:///tmp/store\n\n# Mount the resulting flist\nsudo rfs mount -m nginx-latest.fl /mnt/nginx\n```\n\n### Distributing Large Filesystems\n\nPackage and distribute large filesystems efficiently:\n\n```bash\n# Create an flist from a directory\nrfs pack -m myapp.fl -s dir:///tmp/store /path/to/myapp\n\n# Share the flist (typically \u003c1MB) instead of the entire content\n# Recipients can mount it and access files on-demand\nsudo rfs mount -m myapp.fl /mnt/myapp\n```\n\n### Web Content Distribution\n\nPublish and distribute web content efficiently:\n\n```bash\n# Create an flist from a website directory\nrfs pack -m website.fl -s s3://user:pass@s3.example.com:9000/bucket /path/to/website\n\n# Mount the website on a server\nsudo rfs mount -m website.fl /var/www/html\n```\n\n### Application Deployment\n\nDeploy applications with all their dependencies:\n\n```bash\n# Convert an application Docker image to an flist\nrfs docker -i myapp:latest -s zdb://zdb.example.com:9900/namespace\n\n# Deploy the application on multiple servers by mounting the flist\nsudo rfs mount -m myapp-latest.fl /opt/myapp\n```\n\n## Quick Start\n\n### Installation\n\n```bash\n# Install dependencies\nsudo apt-get install -y build-essential fuse libfuse-dev\n\n# Install Rust\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\nsource $HOME/.cargo/env\n\n# Clone the repository\ngit clone https://github.com/threefoldtech/zos_rfs.git\ncd zos_rfs\n\n# Build ZOS RFS\nrustup target add x86_64-unknown-linux-musl\ncargo build --features build-binary --release --target=x86_64-unknown-linux-musl\n\n# Install the binary\nsudo cp ./target/x86_64-unknown-linux-musl/release/rfs /usr/local/bin/\n```\n\n### Creating Your First Flist\n\n```bash\n# Create a directory to use as a store\nmkdir -p ~/rfs-store\n\n# Create an flist from a directory\nrfs pack -m ~/test.fl -s dir://~/rfs-store ~/my-directory\n\n# Mount the flist\nmkdir -p ~/mount-point\nsudo rfs mount -m ~/test.fl ~/mount-point\n\n# Access the files\nls -la ~/mount-point\n\n# Unmount when done\nsudo umount ~/mount-point\n```\n\n### Converting a Docker Image\n\n```bash\n# Create a directory to use as a store\nmkdir -p ~/rfs-store\n\n# Convert a Docker image to an flist\nrfs docker -i alpine:latest -s dir://~/rfs-store\n\n# Mount the resulting flist\nmkdir -p ~/mount-point\nsudo rfs mount -m alpine-latest.fl ~/mount-point\n\n# Run commands in the mounted filesystem\nsudo chroot ~/mount-point /bin/sh -c \"ls -la\"\n\n# Unmount when done\nsudo umount ~/mount-point\n```\n\n### Running the Server\n\n```bash\n# Create a configuration file\ncat \u003e config.toml \u003c\u003c EOF\nhost = \"localhost\"\nport = 3000\nstore_url = [\"dir:///tmp/store\"]\nflist_dir = \"flists\"\n\njwt_secret = \"your-secret-key\"\njwt_expire_hours = 5\n\n[[users]]\nusername = \"admin\"\npassword = \"admin-password\"\nEOF\n\n# Create the flists directory\nmkdir -p flists/admin\n\n# Run the server\nrfs server --config-path config.toml\n```\n\n## Command Reference\n\nThe `rfs` command provides all the functionality you need to work with flists:\n\n- `rfs pack` - Create flists from directories\n- `rfs mount` - Mount flists as filesystems\n- `rfs unpack` - Extract flist contents to a directory\n- `rfs docker` - Convert Docker images to flists\n- `rfs server` - Run the ZOS RFS server for web-based management\n- `rfs config` - Manage flist metadata and stores\n- `rfs clone` - Copy data between stores\n- `rfs flist tree` - Display flist contents as a tree structure\n- `rfs flist inspect` - Inspect file details within an flist\n- And more...\n\nFor detailed information about each command, use the `--help` flag:\n\n```bash\nrfs --help\nrfs pack --help\nrfs mount --help\n```\n\n## Documentation\n\nComprehensive documentation is available in the [docs](./docs) directory:\n\n### Getting Started\n\n- [Installation and Basic Usage](./docs/tutorials/getting-started.md)\n\n### Core Functionality\n\n- [Creating Flists](./docs/tutorials/creating-flists.md)\n- [Mounting Flists](./docs/tutorials/mounting-flists.md)\n- [End-to-End Flist Workflow](./docs/tutorials/end-to-end-flist-workflow.md)\n\n### Docker Integration\n\n- [Converting Docker Images](./docs/tutorials/docker-conversion.md)\n- [End-to-End Docker Workflow](./docs/tutorials/end-to-end-docker-workflow.md)\n\n### Server and Distribution\n\n- [Server Setup](./docs/tutorials/server-setup.md)\n- [Website Publishing](./docs/tutorials/website-publishing.md)\n- [Syncing Files](./docs/tutorials/syncing-files.md)\n\n### User Guides\n\n- [ZOS RFS Command Reference](./docs/user-guides/rfs-cli.md)\n- [ZOS RFS Server Setup and Usage](./docs/user-guides/fl-server.md)\n- [Web Interface Guide](./docs/user-guides/frontend.md)\n- [Performance Tuning](./docs/user-guides/performance-tuning.md)\n- [Troubleshooting](./docs/user-guides/troubleshooting.md)\n\n### Concepts and Architecture\n\n- [Understanding Flists](./docs/concepts/flists.md)\n- [Storage Backends](./docs/concepts/stores.md)\n- [Caching](./docs/concepts/caching.md)\n- [Sharding and Replication](./docs/concepts/sharding.md)\n- [System Architecture](./docs/architecture/overview.md)\n\n## Community and Support\n\n- **GitHub Issues**: Report bugs or request features on the [GitHub repository](https://github.com/threefoldtech/zos_rfs/issues)\n- **Documentation**: Comprehensive documentation is available in the [docs](./docs) directory\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2Fzos_rfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthreefoldtech%2Fzos_rfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2Fzos_rfs/lists"}