Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/KonstantinSchubert/zero
Local file system transparently swapping to the cloud
https://github.com/KonstantinSchubert/zero
Last synced: 3 months ago
JSON representation
Local file system transparently swapping to the cloud
- Host: GitHub
- URL: https://github.com/KonstantinSchubert/zero
- Owner: KonstantinSchubert
- License: other
- Created: 2018-04-12T21:15:05.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-02-26T01:10:34.000Z (over 2 years ago)
- Last Synced: 2024-04-20T14:52:45.452Z (7 months ago)
- Language: Python
- Homepage:
- Size: 157 KB
- Stars: 775
- Watchers: 30
- Forks: 23
- Open Issues: 5
-
Metadata Files:
- Readme: Readme.md
- License: LICENSE
Awesome Lists containing this project
README
# Zero
Fuse filesystem backed by backblaze cloud storage and transparent local persistent hard-drive cache.
Ideally, it feels like an infinite local file system because it keeps those files local that are used while moving those files to the remote storage that haven't been accessed in a long time.
## Hard links are not supported
Zero operates on a abstraction of files being addressed by paths.
In order to offer hard good hard link support, one would need to instead choose an inode abstraction level, where multiple paths map to the same inode, which then addresses the file.
Unfortunately, fuse does not seem to support a true inode-level API. For example an inode-level API would not require to implement a `move` operation, as this would be an operation handled on the path->inode mapping level.
One could imagine building an adapter that translates from fuse into an inode-level API.
This may be a project for a later date.
## State of Development
**This is WORK IN PROGRESS**
There are a few _known issues and race conditions_. For example, the ctime on the files may be newer than the correct value. Also, writes and reads are _slow_.
_Do not use in production._
## Setup
Create a config.yml:
```
accountId: [...]
applicationKey: [...]
bucketId: [...]
sqliteFileLocation: [...]
targetDiskUsage: [...]
```
and save in `~/.config/zero/`
Here, `accountId`, `applicationKey` and `bucketId` are the corresponding backblaze settings and `sqliteFileLocation` is simply the path to a place where the sqlite databases containing the state of the virtual file system can be stored.
`targetDiskUsage` is the amount of disk space (in GB) that you would like to use for local caching.
Install with `python setup.py develop`
## Usage
Run
zero-fuse
to mount the file system.
You will also need to run
zero-worker
to start the worker process that moves files between disk and cloud.
## Testing
Run the tests with `pytest` like this:
`py.test`
There are 3 categories of tests:
- Unit tests are testing the code on the function level
- Integration tests are testing the interactions between bigger parts of the code and outside APIs. They may use the internet.
- System tests are testing the software as a black box. They should continue working through a big refactor or even if everything was re-written in a different programming language. (There are no system tests yet as of now.)
To run the system tests, first clear all data and start zero like this:
```
debug-delete-everything ~/playground/mountpoint ~/playground/to && zero ~/playground/mountpoint ~/playground/to
```
Then run the tests:
```
py.test zero/tests/system_tests
```