https://github.com/mkmik/imsy
simple incremental pull of immutable large files
https://github.com/mkmik/imsy
cdc incremental large-files rabin-fingerprint replication
Last synced: about 1 year ago
JSON representation
simple incremental pull of immutable large files
- Host: GitHub
- URL: https://github.com/mkmik/imsy
- Owner: mkmik
- License: apache-2.0
- Created: 2019-05-22T00:36:42.000Z (about 7 years ago)
- Default Branch: main
- Last Pushed: 2022-06-21T08:04:32.000Z (about 4 years ago)
- Last Synced: 2025-05-27T13:54:59.053Z (about 1 year ago)
- Topics: cdc, incremental, large-files, rabin-fingerprint, replication
- Language: Go
- Size: 17.6 KB
- Stars: 10
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# imsy
## overview
`imsy` shows the underlying principle of file replication
mechanism suitable for large immutable files, such as VM images.
The core idea is to split a file in chunks using a Content Defined Chunking (CDC) mechanism,
and save chunks in a Content Addressed Store (CAS), where each chunk is identified by its hash (e.g. SHA256)
The file can now be fully recovered by knowing the list of hashes of its constituent chunks, in order.
## usage
First check out and run `go build`.
Then get hold of a couple of big files that are different but related, e.g. two VM images. Squashed uncompressed docker images would work too. Let's call them `vm1.img` and `vm2.img`.
Then run:
```
$ imsy -dir server1data prepare