https://github.com/drpaulbrewer/verify-md5

Command line client to manage md5 verification of files in local directories and Google Cloud Storage [tm] buckets
https://github.com/drpaulbrewer/verify-md5

command-line-tool files google-cloud-storage integrity-monitoring md5 nodejs race-detection

Last synced: 7 months ago
JSON representation

Command line client to manage md5 verification of files in local directories and Google Cloud Storage [tm] buckets

• Host: GitHub
• URL: https://github.com/drpaulbrewer/verify-md5
• Owner: DrPaulBrewer
• License: other
• Created: 2017-04-03T23:49:29.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2024-05-29T03:47:12.000Z (over 1 year ago)
• Last Synced: 2024-05-29T16:49:05.852Z (over 1 year ago)
• Topics: command-line-tool, files, google-cloud-storage, integrity-monitoring, md5, nodejs, race-detection
• Language: JavaScript
• Size: 122 KB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 11
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

          
# verify-md5

A standalone program to compare a file of md5 hash signatures in `/path/to/md5.json` to the md5 hashes of the contents of the listed files in that same directory

## Installation

To install globally:

    npm i verify-md5 -g

## Prerequisites

1. `nodejs` is installed and accesible by invoking `node`

1. An `md5.json` file consisting of an object with filenames as keys and base64 md5 as value for each key.

Example md5.json file:


{

  "data.csv": "fi5F17hHxxqZ9E6gpUlvfA==",

  "profit.csv": "NbLa4M1gKG5Mi1gREVtwWg==",

  "trade.csv": "PTIe1aHM/ww85vzA6UbMAg==",

  "volume.csv": "z8pesXd3MJ7PjmQq+MhNrw==",

  "secretcode.js": "ho5ejYGt7uYwtEZSg8M7QQ==",

  "someotherfile.json": "7Vl0EF+dhruo4YhBFlaW8g=="

}



## Usage

If the path begins with `gs://` it refers to a Google Cloud Storage[tm] bucket, e.g. `gs://bucketName/path/to/md5.json`

    verify-md5 --key /path/to/secret/myserviceaccountkey.json gs://bucketName/path/to/md5.json

    

If the path is a filesystem path, the local filesystem is used:

    verify-md5 /path/to/md5.json

`--json` enables json output:

`json` output is a 4 element array:

`[true|false, goodFiles, badFiles, failErrors, dirname]`

Otherwise, "human-readable" output is used.

### Examples

Success example, using storage bucket.


verify-md5 --key secret.json gs://somebucket/path/to/md5.json

bucket is somebucket

path is path/to/md5.json

OK gs://somebucket/path/to/md5.json

ok:    effalloc.csv

ok:    ohlc.csv

ok:    profit.csv

ok:    sim.json

ok:    trade.csv

ok:    volume.csv



Failure example, with local file system.


verify-md5 /tmp/test/md5.json 

FAIL /tmp/test/md5.json

ok:    effalloc.csv

ok:    ohlc.csv

ok:    sim.json

ok:    profit.csv

ok:    trade.csv

fail:  volume.csv md5 File Verification Error on file: /tmp/test/volume.csv : expected: z8pesXd3MJ7PjmQq+MhNrw== actual: 8VMdzVVyB0zpNxwmtJRwUw==




verify-md5 --json /tmp/test/md5.json 

[

  false,

  [

    "effalloc.csv",

    "ohlc.csv",

    "sim.json",

    "profit.csv",

    "trade.csv"

  ],

  [

    "volume.csv"

  ],

  {

    "volume.csv": {

      "name": "MD5FileVerificationError",

      "message": "md5 File Verification Error on file: /tmp/test/volume.csv : expected: z8pesXd3MJ7PjmQq+MhNrw== actual: 8VMdzVVyB0zpNxwmtJRwUw==",

      "stack": "Error\n    at new MD5FileVerificationError (/usr/local/lib/node_modules/verify-md5/node_modules/verify-fsdir-md5/index.js:17:19)\n    at hasha.fromFile.then (/usr/local/lib/node_modules/verify-md5/node_modules/verify-fsdir-md5/index.js:50:19)\n    at process._tickCallback (internal/process/next_tick.js:103:7)",

      "actual": "8VMdzVVyB0zpNxwmtJRwUw==",

      "expected": "z8pesXd3MJ7PjmQq+MhNrw==",

      "file": "/tmp/test/volume.csv"

    }

  },

  "/tmp/test/"

]



## Authentication / Authorization

`verify-md5` on a filesystem path runs with the permissions of the current user.

`verify-md5` on a `gs://` path will attempt to authenticate with the Google Storage API.

Unless you run in the Google Cloud, you will need to include a `--key keyfilename` to access your buckets.

If you need to generate a key for your project, look in the IAM section of the Google Cloud Platform Console and generate  either: 

* a service account key for role: Storage Object Viewer

* or, for granularity, a new service account with no permissions, added as a reader to an individual bucket acl.

Important: It is solely your responsibility to keep API keys secret and manage related security and financial risks.

## Storage API usage

Only the bucket's `md5.json` file is downloaded with a `storage.bucket.file.download` call.  

Then `storage.bucket.file.get` calls are used to access md5 metadata for each file, not `.download()` calls.  

Creating the necessary md5 metadata in the bucket is normally automatic when objects are stored.

Creating the `md5.json` file in the bucket to enable these comparisons is your responsibility.

## Help

    verify-md5 --help

prints this reminder message



  Usage: verify-md5 [options] 

  Options:

    -h, --help       output usage information

    -V, --version    output the version number

    --key [keyfile]  keyfile to access Google Cloud Storage API

    --json           output test results in json



## Copyright

Copyright 2017 Paul Brewer, Economic and Financial Technology Consulting LLC 

## License

The MIT License

### Trademarks

Google Cloud Storage[tm] is a trademark of Google, Inc.

This software is not a product of Google, Inc.