Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/makew0rld/merkdir

Create Merkle trees from your directories.
https://github.com/makew0rld/merkdir

Last synced: about 1 month ago
JSON representation

Create Merkle trees from your directories.

Awesome Lists containing this project

README 

        
# merkdir



`merkdir` creates [Merkle trees](https://en.wikipedia.org/wiki/Merkle_tree) from your directories, and can generate inclusion proofs for any files in those directories.



## Use case



Let's say you have a bunch of files in your `~/Documents` folder, some public, some private. Run `merkdir` on the directory, and you get a root hash which anonymously represents all your documents. Now you can publish that, sign it, [timestamp](https://www.makeworld.space/2023/09/time_for_timestamping.html) it, whatever you like.



At some later date, you can publish one of these (unchanged) documents along with the inclusion proof generated by `merkdir`. This will provably connect the file back to root hash that you published/signed/etc in the past.



Using the power of Merkle trees, `merkdir` allows you to prove things about millions of files at once. It can make signing or timestamping all the files in your home directory efficient and secure.



The real world applications are slim, but interesting. Maybe I'm a whistleblower with thousands of files, but I don't want to release them all right now. This tool allows you to easily prove that a file you release later was part of the original collection.



Another use case might be an organization that has a large archive of important files. This tool allows them to efficiently sign all of these files by just signing the root hash. 



## Install



Pre-built binaries can be installed from the releases page. Otherwise you can build from source by using [`just`](https://github.com/casey/just). The Go compiler is of course required.



If you use [Homebrew](https://brew.sh/), you can install `merkdir` through the my personal tap.

```

brew tap makew0rld/tap

brew install merkdir

```

You can update it with:

```

brew upgrade merkdir

```



## Usage



```bash

# Create a Merkle tree that holds hashes of all your documents

$ merkdir gen -o documents_tree.merkdir ~/Documents

Finding files...

Found 2339 files. Starting hashing...

 100% |████████████████████████████████████████| (1.6/1.6 GB, 5.0 GB/s)        

Root hash: 3e1db8e48dd101bed67ccd117ad011fa76aca26c38ce1ab1612010d5140618b1



# Now publish that hash, sign it, etc

# If you need it again:

$ merkdir root --hex documents_tree.merkdir

3e1db8e48dd101bed67ccd117ad011fa76aca26c38ce1ab1612010d5140618b1



# Generate a proof for one file

$ merkdir inclusion -t documents_tree.merkdir -f "name/of/file.txt" -o my_proof.merkdir

# Now publish that along with the file itself

# This proves the file is a part of the tree represented by the root hash



# Verify that a file on disk hasn't changed since the tree was generated

$ merkdir verify-file -t my_merkle_tree.merkdir -n "name/of/file.txt"



# Verify an inclusion proof you received

# You get the root hash as output, and must compare it to the expected root hash

$ merkdir verify-inclusion -p some_inclusion_proof.bin -f path/to/file.pdf --hex

3e1db8e48dd101bed67ccd117ad011fa76aca26c38ce1ab1612010d5140618b1



# Or compare it to a root hash directly:

$ merkdir verify-inclusion -p some_inclusion_proof.bin -f path/to/file.pdf --hash "abc123..."

OK: proof and file match given root hash



# Get info on a tree file

$ merkdir info documents_tree.merkdir

Root hash: 3e1db8e48dd101bed67ccd117ad011fa76aca26c38ce1ab1612010d5140618b1

FS root: /home/makeworld/Documents

Num. of files: 2339

Creation time: 2023-12-26 19:33:29 -0500 EST



# Or for a proof, relative to the tree it's from

$ merkdir info -p some_proof.merkdir documents_tree.merkdir

File index: 2137

File name: My Document.ods

Nonce: bc0d0ba51259d0ba0475f754bc4a7cf4

Proof length: 10 hashes

```



All merkdir output files are [CBOR](https://cbor.io/), so they can be easily used by other tools.



## Security



`merkdir` uses the fast and secure BLAKE3 hash algorithm.



Inclusion proofs are designed to be shared publicly, and so don't expose filenames or even direct file hashes. The only information relevant to your filesystem that they reveal is the number of files in the original Merkle tree.



## Alternatives



Other tools like [IPFS](https://github.com/ipfs/kubo) or [Iroh](https://iroh.computer/) also create Merkle trees over directories. The main difference is the `merkdir` is singularly focused on doing this, and crucially makes generating inclusion proofs easy. Currently I'm not aware of a way to generate inclusion proofs with these tools.



## License



`merkdir` is licensed under the GPL v3.0. See the [LICENSE](./LICENSE) file for details.