https://github.com/neuro-inc/mlops-wandb-bucket-ref

https://github.com/neuro-inc/mlops-wandb-bucket-ref

mlops neuro s3 wandb weights-and-biases

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/neuro-inc/mlops-wandb-bucket-ref
• Owner: neuro-inc
• License: apache-2.0
• Created: 2021-03-25T09:31:04.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2024-02-22T15:41:07.000Z (7 months ago)
• Last Synced: 2024-07-23T05:25:42.306Z (2 months ago)
• Topics: mlops, neuro, s3, wandb, weights-and-biases
• Language: Python
• Homepage:
• Size: 113 KB
• Stars: 1
• Watchers: 6
• Forks: 1
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.D/.gitignore
  • License: LICENSE

Awesome Lists containing this project

README

        
# WaBucketRef

_Neu.ro platform integration with [Weights and Biases](https://wandb.ai/site) to store artifact binaries in Neu.ro buckets and refer them in WandB Artifact for lineage_

## Installation

You could install library directly from the GitHub.

All Github tags are tested. The backward compatibility is not guaranteed yet:

```bash

export WABUCKET_VERSION=v22.5.5

pip install "wabucketref @ git+https://github.com/neuro-inc/mlops-wandb-bucket-ref.git@$WABUCKET_VERSION"

```

## Usage

WaBucketRef could be used as a CLI application, or Python module to **upload** or **download** [W&B artifacts](https://docs.wandb.ai/guides/artifacts/construct-an-artifact#add-a-uri-reference) stored in Neu.ro platform Buckets. In this case, BLOBs are stored in Neu.ro Buckets backend system, while articat metadata is managed by W&B.

### Console usage

Refer to the dedicated [CLI.md](./CLI.md) usage file.

### SDK usage

You could import WaBucket python API client to manage artifacts and utilize W&B SDK in your script for more granular control on W&B runs, metrics and artifacts.

Here is an example, where one downloads artifact from the platform bucket "bucket-name". Afterwards, the model training happens which creates model metrics object and, serialises model at the `model_path` path at local file system.

Finally, those metrics are logged in W&B system, and the model binary is stored under the platform bucket.

```python

from wabucketref import WaBucketRefAPI

api = WaBucketRefAPI(

    bucket="bucket-name",

    project_name="my-w&b-project",

)

folder = api.download_artifact("name", "type", "version")

metrics, model_path = do_train(artifact=folder)

wandb.run.log(metrics) # Run is created by WaBucketRefAPI automatically at `download_artifact` method call.

api.upload_artifact(model_path, "my-model", "model")

api.close() # Do not forget to release the resources!

```