Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/snowtrack/snowfs
SnowFS - a fast, scalable version control file storage for graphic files :art:
https://github.com/snowtrack/snowfs
apfs binary-manager graphic-files storage version-control version-control-system version-manager version-tracker
Last synced: about 2 months ago
JSON representation
SnowFS - a fast, scalable version control file storage for graphic files :art:
- Host: GitHub
- URL: https://github.com/snowtrack/snowfs
- Owner: snowtrack
- License: gpl-3.0
- Created: 2021-01-25T04:28:08.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-25T13:06:31.000Z (11 months ago)
- Last Synced: 2024-07-16T08:40:53.775Z (2 months ago)
- Topics: apfs, binary-manager, graphic-files, storage, version-control, version-control-system, version-manager, version-tracker
- Language: TypeScript
- Homepage: https://www.snowtrack.io
- Size: 805 KB
- Stars: 1,284
- Watchers: 33
- Forks: 42
- Open Issues: 28
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
![snowfs][snowfs_banner]
# SnowFS - a fast, scalable version control file storage for graphic files
[](https://github.com/Snowtrack/SnowFS/releases/tag/v0.9.2)
[](https://coveralls.io/github/Snowtrack/SnowFS)
[](https://github.com/snowtrack/SnowFS/actions)
[](https://www.gnu.org/licenses/gpl-3.0)
[](https://deepsource.io/gh/Snowtrack/SnowFS/?ref=repository-badge)
[](https://www.codacy.com/gh/Snowtrack/SnowFS/dashboard?utm_source=github.com&utm_medium=referral&utm_content=Snowtrack/SnowFS&utm_campaign=Badge_Grade)
## Overview
SnowFS is a high-performance command-line application and node library for *Windows*, *macOS* and *Linux* with a focus on binary file versioning. It is made for the graphics industry and was initially developed for [Snowtrack](https://www.snowtrack.io).
**Disclaimer:** This project is in beta and actively developed.
![terminal][terminal_preview]
## Feature highlights
- Supports Branches
- Asynchronous File Hashing
- Project open to file-content awareness (e.g: `*.psd`, `*.blend`, `*.c4d`, `..`)
- Super-fast-detection of modifications in large binaries
- Support for instant snapshots\*\*
- Support for instant rollback\*\*
- Support for files bigger >4TB
- Block-cloning and Copy-on-Write support for [APFS](https://developer.apple.com/documentation/foundation/file_system/about_apple_file_system), [ReFS](https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview) and [Btrfs](https://en.wikipedia.org/wiki/Btrfs)
- Support for removing single versions and/or binaries
- Primarily I/O bound through [libuv](https://github.com/libuv/libuv)
- Feature XYZ made by you!
\*\* If the underlying filesystem supports it (e.g. APFS, ReFS and Btrfs)
## Made with SnowFS
- [Snowtrack](https://www.snowtrack.io)
- [Snow for Blender](https://snowtrack.io/snow-for-blender.html)
- [Snow for Cinema 4D](https://snowtrack.io/snow-for-cinema-4d.html)
- [Snow for Unity](https://github.com/daniellanner/snow-for-unity-facade)
## Why not Git/Git-LFS, libgit2, or SVN?
First and foremost, the implementations of Git - namely [Git](https://git-scm.com/)/[Git-LFS](https://github.com/git-lfs/git-lfs)
and [libgit2](https://libgit2.org/) are excellent implementations of version control systems.
But due to their focus on the software development lifecycle they are not suitable to version binaries or graphic files.
`SnowFS` addresses the technical challenges for graphic files by its core design.
### Git/Git-LFS:
**Advantages:**
- Support on all major platforms
- Supported by hosting platforms like GitHub, GitLab, and BitBucket.
- Fast diff-operation for text-files
**Disadvantages:**
- If not properly tracked, binaries become accidentally part of "base" history
- Removing older commits is cumbersome due to Gits commit hashing integrity
- Complicated *rewriting history* procedure
- Issues with binaries >4GB on Windows as reported [here](https://github.com/git-lfs/git-lfs/issues/2434), [here](https://confluence.atlassian.com/bitbucketserverkb/files-larger-than-4-gb-are-not-correctly-handled-on-windows-935385144.html), and [here](https://stackoverflow.com/questions/49018053/how-large-does-a-large-file-have-to-be-to-benefit-from-git-lfs)
- Slow in binary modification detection
### libgit2
**Advantages:**
- Faster zipping, packing, and delta-compression than the reference implementation *Git*
- Supports [custom backends](https://git-scm.com/book/it/v2/Appendix-B%3A-Embedding-Git-in-your-Applications-Libgit2)
**Disadvantages:**
- No native support for Git-LFS without custom backends
- Custom backends break compatibility with Git
## TypeScript / C++ backport
`SnowFS` is currently written in TypeScript. It is a great language to write powerful and performant
I/O bound prototypes. We are aware of the demand for a C++ implementation, and due to
our roots as C++ developers, we are very interested in a backport as well. It would make SnowFS easier
to integrate into other projects. If you have comments, ideas or recommendations, please let us know.
## Running benchmarks
We have also implemented a comparison benchmark between **SnowFS** vs. **git-lfs**.
After executing the [build instructions](#build-instructions) for a development build, the benchmarks can be executed with the following command:
```bash
$ npm run benchmark
```
Example run on a Macbook Pro (2020) with an APFS formatted SSD to check-in, delete and restore a 4GB Photoshop File.
```
...
git lfs track *.psd
git add texture.psd: 20164ms
snow add texture.psd: 4596ms
git rm texture.psd: 575ms
snow rm texture.psd: 111ms
git checkout HEAD~1: 9739ms
snow checkout HEAD~1: 77ms <-- Yeah!
```
## Examples
### Code
You can find the best and up-to-date code examples in the `test/` directory. Given below these are simply **"Hello World!"** examples to get you started.
```typescript
import * as fse from "fs-extra";
import { join } from "path";
import { Index } from "./src";
import { Repository } from "./src/repository";
export async function main() {
const repoPath = "/path/to/a/non/existing/directory";
// Create a new repository.
let repo: Repository = Repository.initExt(repoPath);
// Copy a file called 'texture.psd' to the working directory.
await fse.copyFile("/path/to/texture.psd", join(repoPath, "texture.psd"));
// Retrieve the main index.
const index: Index = repo.ensureMainIndex();
// Mark 'texture.psd' as a new file in the index.
index.addFiles(["texture.psd"]);
// Write all files of the index to the object database.
await index.writeFiles();
// Create a commit for the index.
await repo.createCommit(index, "This is my first commit");
}
main();
```
### Command line interface
The CLI of `SnowFS` offers some basic functionality and is subject to enhancements. See [Report #90](https://github.com/Snowtrack/SnowFS/issues/90)
[](https://github.com/Snowtrack/SnowFS/releases/tag/v0.9.2)
```
$ snow init foo
$ cp /path/to/texture.psd foo/texture.psd
$ cd foo
$ snow add .
$ snow commit -m "My first texture"
$ snow log
$ snow checkout -b MyNewBranch
$ snow log
```
## Build Instructions
1. To build `SnowFS` install [node.js](https://nodejs.org/en/) for your specific platform.
1.1. On Windows, Visual Studio is required. We recommend the [Visual Studio 2019 Community Edition](https://visualstudio.microsoft.com/downloads/). During the installation, please enable Desktop development with C++.
1.2. On MacOS, [XCode](https://developer.apple.com/xcode/) is required.
2. To build a **development build** execute:
```bash
$ git clone https://github.com/Snowtrack/snowfs.git
$ cd snowfs
$ npm install
$ npm run test
```
3. To build a **production build including the CLI**, execute the commands above, and continue with the commands below:
```bash
$ npm run tsc
$ npm run build
$ cd dist
```
### How To Debug
For the development of SnowFS we recommend **VSCode**. The repository contains a [launch.json](.vscode/launch.json) file with pre-defined runner configurations. For more information, please visit [this](https://github.com/Snowtrack/SnowFS/pull/49#issue-579026705) pull-request.
## Versioning
Starting with version 1.0.0 `SnowFS` follows the [semantic versioning](http://semver.org/)
scheme. The API change and backward compatibility rules are those indicated by
SemVer.
## Licensing
`SnowFS` is licensed under the **GPL v3** license, please review the [LICENSE file](LICENSE).
Excluded from the license are images, artworks, and logos. Please file a request by mail, if you have any questions.
## Community
- [Discord](https://discord.gg/RDKPuH8dkA)
- [Support](https://github.com/Snowtrack/snowfs/labels/question)
- [Email](mailto:[email protected])
### Other resources
The [tests and benchmarks](https://github.com/snowtrack/snowfs/tree/main/test) also serve as API specification and usage examples.
## Supported Platforms
Currently, Windows, macOS, and Linux are supported. `SnowFS` works on plain filesystems like FAT, NTFS, HFS+ and has extended support for APFS, ReFS and Btrfs.
## How can I contribute?
See the [guidelines for contributing][].
[node.js]: http://nodejs.org/
[terminal_preview]: https://github.com/snowtrack/snowfs/raw/main/img/terminal.gif
[guidelines for contributing]: https://github.com/snowtrack/snowfs/blob/main/CONTRIBUTING.md
[snowfs_banner]: https://github.com/snowtrack/snowfs/raw/main/img/banner.png
