https://github.com/threefoldtech/quantum-storage
https://github.com/threefoldtech/quantum-storage
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/threefoldtech/quantum-storage
- Owner: threefoldtech
- License: apache-2.0
- Created: 2021-01-06T16:20:38.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2025-03-26T00:39:14.000Z (about 2 months ago)
- Last Synced: 2025-04-09T17:09:54.074Z (about 1 month ago)
- Language: Shell
- Size: 159 KB
- Stars: 6
- Watchers: 9
- Forks: 4
- Open Issues: 16
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Quantum Storage Filesystem
Quantum Storage is a FUSE filesystem that uses mechanisms of forward error correction (Reed Solomon codes) to make sure data (files and metadata) are stored in multiple remote places in a way that we can afford losing `x` number of locations without losing the data. There is other factors that are involved into this operation like encryption. Please check [0-stor](https://github.com/threefoldtech/0-stor_v2) documentations for details.
The aim is to support unlimited local storage with remote backends for offload and backup which cannot be broken, even by a quantum computer.
## Overview
To have a working qsfs filesystem there are multiple components that need to work together to make it work. These components are
- [0-db-fs](https://github.com/threefoldtech/0-db-fs) this is what creates the `FUSE` mount (the actual user facing filesystem) this component does not know about qsfs or forward error correction. It's main job is to expose the fuse filesystem and store it's data in a local zdb instacne
- [0-db](https://github.com/threefoldtech/0-db) is a local `cache` db. this is what is used by the `0-db-fs` to store the actual data of the filesystem. This means that any read/write operations triggered by the `0-db-fs` directly access this (single) instance of `0-db` for the data blocks
- [0-stor](https://github.com/threefoldtech/0-stor_v2) zero stor is listening to `0-db` events (with a hooks system) to upload and/or download zdb data files segments to remote locations. that's where the encryption and forward error correction happens.
Since zdb is an `append-only` database, the local db will just keep growing linearly with each write (and delete) operation. ZDB will then create db segment files that are granted to **not** change in the future. What happens once a segment file is closed (it hit it's max file size) a hook is triggered which in return will trigger `0-stor` to chunk and upload this file to the remote locations (zdbs).
The segment file will then be deleted (at some point) in the future when the number of segment files reaches a certain number, older files will get deleted.
If the filesystem then is trying to access a piece of old data, it will make a read call to the local `zdb`. If the zdb is trying to access an old segment of the db that is no longer on disk, another hook is triggered to `0-stor` to download that segment. 0-stor then will re-download the required segment from the remote locations and re-build it.
Once the zdb segment file is restored, the read operation continues.
## Bootstrap
You can use the bootstrap (`bootstrap/bootstrap.v`) to download therequired components and start everything. The default configuration uses everything localy. You can pass a specific zstor configuration file to use a real backend.
Everything will be installed in `~/.threefold` and nowhere else.
This bootstrap will spawn two `zdb`'s', one `zstor daemon` and the `zdbfs` fuse system.
## Other documentation
- [Set up qsfs manually](docs/manual_setup.md).
- [Docker image documentation](docker/README.md).
- [Operations guide](docs/operations.md)