https://github.com/vessl-ai/tvault
Quickly compare PyTorch models in a local, lightweight registry
https://github.com/vessl-ai/tvault
deep-learning machine-learning mlops pytorch
Last synced: 10 months ago
JSON representation
Quickly compare PyTorch models in a local, lightweight registry
- Host: GitHub
- URL: https://github.com/vessl-ai/tvault
- Owner: vessl-ai
- License: mit
- Created: 2023-04-13T08:19:13.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-04-27T04:26:56.000Z (about 3 years ago)
- Last Synced: 2025-02-17T07:17:02.642Z (over 1 year ago)
- Topics: deep-learning, machine-learning, mlops, pytorch
- Language: Python
- Homepage:
- Size: 89.8 KB
- Stars: 23
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Quickly compare PyTorch models in a local, lightweight registry
----
`tvault` is designed to help academic researchers iterate their models faster without logging overheads.
Many of the academic researchers we encounter simply want to *get going* with minimum setup and configurations. This often means using local codebase as opposed to integrating tools like [VESSL Experiments](https://docs.vessl.ai/api-reference/python-sdk/utils/vessl.log) and Weights & Biases to git-committed codes.
`tvault` is "git diff for machine learning" — a simple, lightweight framework for quickly tracking and comparing ML experiments in a local model registry.
* Track and version models locally with `tvault.log_all()`
* Get a birds-eye differences of two experiments with `tvault --diff_flag`
Get started with pip install:
```
pip install tvault
```
For those who are already using [VESSL Python SDK and CLI](https://docs.vessl.ai/api-reference/what-is-the-vessl-cli-sdk),
```
pip install "vessl[tvault]"
```
Follow our guide below with our [MNIST example code](https://github.com/vessl-ai/tvault/blob/1a6b5e038ff3fd4780a186bec7a555215a5e3c31/assets/mnist-train.py).
----
## Getting started with `tvault.log()`
Insert `tvault.log()` in your code's training loop with the metrics you want to track as tags.
https://github.com/vessl-ai/tvault/blob/4bebfe48594b00597889b2696dcec532d72b33d4/assets/mnist-train.py#L145-L146
`tvault.log()` will automatically create the following:
* A folder `model_log` under your current directory - a tracking dashboard or model registry for your code
* A unique hash for the model and model ID for each training run with the key metrics
## Look up experiments with `tvault --find_flag`
`tvault`'s `find_flag` option allows you to look up different experiments with a simple CLI command. `find_flag` offers three different ways of exploring results:
1. Search by hash — The command below shows all experiments with the hash value of `2ba4adf`.
```
tvault --find_flag --condition hash --hash 2ba4adf
```
2. Search by result — The command below shows all experiments with the result between `50` and `100`.
```
tvault --find_flag --condition result --min 90 --max 100
```
3. Search by tags — The command below shows all experiments tagged as `0.5x`.
```
tvault --find_flag --condition tag --tag_type size --tag 0.5x
```
## Compare models with `tvault --diff_flag`
`tvault`'s `diff_flag` option allows you to look up the difference of two models by specifying the model hash and index. `tvault` automatically detects and displays the changes in functions while removing git diffs that are not related to the model.
This is useful when you have a baseline model that you want to iterate with different hyperparameters and higher-level architectures without digging through your code line-by-line. `tvault` can also get the diff related to the optimizers.
```
tvault --diff_flag --sha1 f407ed0 --index1 0 --sha2 737b47a --index2 0
```
## Issues, feature requests, and questions
We are excited to hear your feedback!
* For issues and feature requests, please open a GitHub issue.
* For questions, please use GitHub Discussions.
* For general discussions, join our community Slack.
## VESSL for Academics
Our free academic plan is dedicated to help graduate students set up a modern ML research workflow with zero maintenance overheads. [Learn more](https://vesslai.notion.site/VESSL-for-Academics-fa47bf5e69b44e92b5daaead758cb057) about VESSL's full-fledged, fully-managed job scheduler and experiment tracking.