https://github.com/lambdazy/lzy

Platform for a hybrid execution of ML workflows that transparently integrates local and remote runtimes
https://github.com/lambdazy/lzy

distributed hybrid kubernetes machine-learning mlops orchestration-engine python workflow

Last synced: 11 days ago
JSON representation

Platform for a hybrid execution of ML workflows that transparently integrates local and remote runtimes

• Host: GitHub
• URL: https://github.com/lambdazy/lzy
• Owner: lambdazy
• License: other
• Created: 2021-10-14T12:43:36.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2024-05-24T11:56:51.000Z (about 2 years ago)
• Last Synced: 2025-04-06T08:06:22.089Z (about 1 year ago)
• Topics: distributed, hybrid, kubernetes, machine-learning, mlops, orchestration-engine, python, workflow
• Language: Java
• Homepage:
• Size: 13.3 MB
• Stars: 70
• Watchers: 8
• Forks: 3
• Open Issues: 3
• Metadata Files:
  • Readme: readme.md
  • Contributing: CONTRIBUTING
  • License: LICENSE
  • Authors: AUTHORS

Awesome Lists containing this project

• awesome-java - ʎzy

README

          
[![Pypi version](https://img.shields.io/pypi/v/pylzy)](https://pypi.org/project/pylzy/)

[![Tests](https://github.com/lambda-zy/lzy/actions/workflows/pull-tests.yaml/badge.svg)](https://github.com/lambda-zy/lzy/actions/workflows/pull-tests.yaml)

[![Java tests coverage](https://gist.githubusercontent.com/mrMakaronka/be651155cb12a8006cecdee948ce1a0a/raw/master-java-coverage.svg)]()

[![Python tests coverage](https://gist.githubusercontent.com/mrMakaronka/0095e900fb0fcbe5575ddc3c717fb65b/raw/master-coverage.svg)](https://github.com/lambdazy/lzy/tree/master/pylzy/tests)

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pylzy.svg)](https://pypi.org/project/pylzy/)

[![Telegram chat](https://img.shields.io/badge/chat-on%20Telegram-2ba2d9.svg)](https://t.me/+ad3OA-J96b9jYWJi)

# ʎzy

ʎzy is a platform for a hybrid execution of ML workflows that transparently integrates local and remote runtimes

with the following properties:

- Python-native SDK

- Automatic env (pip/conda) sync

- K8s-native runtime

- Resources allocation on-demand

- Env-independent results storage

## Quick start

ʎzy allows running any python functions on a cluster by annotating them with `@op` decorator:

```python

@op(gpu_count=1, gpu_type=GpuType.V100.name)

def train(data_set: Bunch) -> CatBoostClassifier:

    cb_model = CatBoostClassifier(iterations=1000, task_type="GPU", devices='0:1', train_dir='/tmp/catboost')

    cb_model.fit(data_set.data, data_set.target, verbose=True)

    return cb_model

# local python function call

model = train(data_set)

# remote call on a cluster

lzy = Lzy()

with lzy.workflow("training"):

    model = train(data_set)

```

Please read the [tutorial](https://github.com/lambdazy/lzy/tree/master/docs/tutorials/0-contents.md) for details.

## Runtime

Check out our [key concepts](https://github.com/lambdazy/lzy/tree/master/docs/arch/key-concepts.md) and [architecture intro](https://github.com/lambdazy/lzy/tree/master/docs/arch/intro_en.md).

## Community

Join our chat [on telegram](https://t.me/+ad3OA-J96b9jYWJi)!

## Development

Development [guide](https://github.com/lambdazy/lzy/tree/master/docs/development.md).

## Deployment

Deployment guide.

* [YCloud](https://github.com/lambdazy/lzy/tree/master/docs/deployment_ycloud.md)