{"id":27185044,"url":"https://github.com/threefoldtech/0-db-fs","last_synced_at":"2025-08-11T00:35:22.206Z","repository":{"id":41147517,"uuid":"317621779","full_name":"threefoldtech/0-db-fs","owner":"threefoldtech","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-12T03:19:26.000Z","size":246,"stargazers_count":2,"open_issues_count":17,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-02T14:22:16.323Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/threefoldtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-12-01T17:45:46.000Z","updated_at":"2024-06-02T14:22:16.324Z","dependencies_parsed_at":"2023-10-12T13:08:37.186Z","dependency_job_id":null,"html_url":"https://github.com/threefoldtech/0-db-fs","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db-fs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db-fs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db-fs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threefoldtech%2F0-db-fs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/threefoldtech","download_url":"https://codeload.github.com/threefoldtech/0-db-fs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074976,"owners_count":21043490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T17:10:13.514Z","updated_at":"2025-04-09T17:10:14.176Z","avatar_url":"https://github.com/threefoldtech.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zdbfs (0-db filesystem)\n\nzdbfs is a fuse filesystem using [0-db](https://github.com/threefoldtech/0-db) as backend.\nThis filesystem uses some optimization specific needed to keep 0-db backend stable and small, while\nproviding good performance. 0-db backend provide snapshot, history and runtime backup nearly out-of-box.\n\n# Specifications\n\nzdbfs uses 3 namespaces to keep filesystem state: `metadata`, `datablock`, `temporary`\n\nMetadata (meta) are where inode are written. For now, inode contains metadata about a file or a directory\nand their data locations. Directory contains map files/inode aswell. Files contains list of blocks.\n\nDatablock (data) contains the real payload of a file, only files payload are stored there, nothing else.\nEach block can be large up-to blocksize, but it's not fixed, non-full blocks are smaller.\n\nTemporary (temp) namespace is a volatile namespace which will get some blocks pushed if the cache\nis full, in order to keep memory cache low. Since each insertion create a new entry, namespace\ncan grow up quickly if too many overwrites occures. This temporary namespace is made to be flushed\nwhen empty, in order to keep is small. This namespace **requires** a password, in order\nto use `FLUSH` command, which 0-db enforce to be protected by password.\n\n# Cache\n\nThere is a rudimentary cache implementation, used **mainly** for 0-db optimization, not for performance.\nSince 0-db is a always append database, sending a datablock while it's updated will result of a lot of\ndata added in the backend, there is no overwrite. The same applies for inode. If cache is disabled, creating\n10 directories in the same directory will store 10 times the directory content in the backend. This can quickly\ngrow up to unmaintainable namespace.\n\nCache introduced keep files and directories opened in memory, waiting for operations to be\ncompleted or waiting some time before flushing pending changes to the backend.\n\n# Dependencies\n\nOnly `libfuse 3` and `hiredis 1.0` are required to build `zdbfs`, on Linux.\nThere is for now a hard dependency to `libunwind` for debug purpose. Could become optional later.\n\nNote that, only `gcc` or `clang` are supported as C compiler.\n\n# Build\n\nYou can build a debug version with simple `make` command.\n\nTo produce a release version (no debug message), you can use `make release` command.\n\n# Options\n\nYou can configure zdbfs via runtime arguments to pass via `-o` during mount, eg: `zdbfs -o mh=1.1.1.1,ts=newpwd /mnt/temp`\n\nAvailable options (with their default value):\n```\nhost=(not set)      unique hostname to use for meta, data and temp\nport=(not set)      unique port to use for meta, data and temp\nunix=(not set)      unique unix socket path to use for meta, data and temp\n\nmh=localhost        metadata zdb hostname\nmp=9900             metadata zdb port\nmu=(not set)        metadata zdb unix socket path (optional)\nmn=zdbfs-meta       metadata namespace name\nms=(not set)        metadata namespace password (optional)\n\ndh=localhost        datablock zdb hostname\ndp=9900             datablock zdb port\ndu=(not set)        datablock zdb unix socket path (optional)\ndn=zdbfs-data       datablock namespace name\nds=(not set)        datablock namespace password (optional)\n\nth=localhost        temporary zdb hostname\ntp=9900             temporary zdb port\ntu=(not set)        temporary zdb unix socket path (optional)\ntn=zdbfs-temp       temporary namespace name\nts=hello            temporary namespace password (mandatory)\n\nsize=(10G)          runtime filesystem reported size (for debug purpose)\n                    size needs to be in bytes, default is 10737418240 (10G)\n\nnocache             disable runtime cache (for debug purpose)\nautons              try to create required namespace on runtime\nbackground          run in background when filesystem is ready\nlogfile=(not set)   write operations to specified logfile\n                    note: this make lot of resolv request and can\n                    reduce performance, this could generate large logfile\n                    aswell if you do lot of operations\n```\n\n# Quick Setup\n\n## Automatic\n\nStart a 0-db locally, in sequential mode:\n```\nzdb --mode seq\n```\n\nThen start `zdbfs` with `autons` option, to mount filesystem on `/mnt/zdbfs`:\n```\n./zdbfs -o autons /mnt/zdbfs\n```\n\nThat's it.\n\n## Manual\n\nStart a 0-db locally, in sequential mode:\n```\nzdb --mode seq\n```\n\nThen create required namespaces:\n```\ncat \u003c\u003c EOF | redis-cli -p 9900\nNSNEW zdbfs-meta\nNSNEW zdbfs-data\nNSNEW zdbfs-temp\nNSSET zdbfs-temp password hello\nNSSET zdbfs-temp public 0\nEOF\n```\n\nYou can now run `./zdbfs /mnt/zdbfs` to mount the filesystem on `/mnt/zdbfs` target.\n\n# Performance\n\nUsing 0-db on a basic SSD, 100 MB/s linear write can be achieved easily.\n\nThe filesystem can be used as storage to build a Linux kernel with `defconfig` target, with\na fully working result file and compilation process.\n\nMore information soon :)\n\n# Support\n\nAny workload should be supported on this filesystem, with some exceptions:\n- Opening a file in 'always append mode' will not have the expected behavior\n- There is no support of `O_TMPFILE` by fuse layer, which is a feature required by\noverlayfs, thus this is not supported. Overlayfs is used by Docker for example.\n\n# Specification\n\nSupported operations:\n - `create`\n - `open`\n   - Support truncate\n - `lookup`\n - `read`\n - `write`\n - `unlink`\n - `rename`\n - `stat`\n - `statfs`\n   - Basic values\n - `opendir`\n - `closedir`\n - `readdir`\n - `link`\n - `symlink`\n - `readlink`\n - `mkdir`\n - `rmdir`\n   - Does not delete non-empty directories\n - `ftruncate`\n - `getattr`\n - `setattr`\n   - `chmod`\n   - `chown`\n   - `time`\n - Hole read/write\n - Random read/write\n\nKnown unsupported yet:\n - `open`: `O_APPEND` read flag\n - `open`: `O_RD_ONLY`, `O_WRONLY`, `O_RDWR` limitation, everything is allowed\n - User/group permissions internal checking, workaround with fuse default options\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2F0-db-fs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthreefoldtech%2F0-db-fs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreefoldtech%2F0-db-fs/lists"}