Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/apache/submarine

Submarine is Cloud Native Machine Learning Platform.
https://github.com/apache/submarine

ai deep-learning docker kubernetes machine-learning notebook

Last synced: 3 months ago
JSON representation

Submarine is Cloud Native Machine Learning Platform.

Host: GitHub
URL: https://github.com/apache/submarine
Owner: apache
License: apache-2.0
Archived: true
Created: 2019-09-19T04:00:17.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-04-03T00:59:35.000Z (9 months ago)
Last Synced: 2024-09-26T16:01:21.561Z (3 months ago)
Topics: ai, deep-learning, docker, kubernetes, machine-learning, notebook
Language: Java
Homepage: https://submarine.apache.org/
Size: 29.3 MB
Stars: 691
Watchers: 59
Forks: 254
Open Issues: 52
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

![Colored_logo_with_text](website/static/img/icons/color_logo_with_text.png)

![Submarine workflow](https://github.com/apache/submarine/actions/workflows/master.yml/badge.svg?branch=master) ![python-sdk workflow](https://github.com/apache/submarine/actions/workflows/python.yml/badge.svg?branch=master) [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) [![PyPI version](https://badge.fury.io/py/apache-submarine.svg)](https://badge.fury.io/py/apache-submarine)

# What is Apache Submarine?

**Apache Submarine** (Submarine for short) is an **End-to-End Machine Learning Platform** to allow data scientists to create end-to-end machine learning workflows. On **Submarine**, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.

## Why Submarine?

Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?

### Problems

1. Many platforms lack easy-to-use user interfaces (API, SDK, and IDE, etc.)
2. In the same company, data scientists in different teams usually spend much time on developments of existing feature sets and models.
3. Data scientists put emphasis on domain-specific tasks (e.g. Click-Through-Rate), but they need to implement their models from scratch with SDKs provided by existing platforms.
4. Many platforms lack a unified workbench to manage each component in the ML lifecycle.

_Theodore Levitt_ once said:

```
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
```

### Goals of Submarine

#### Model Training (Experiment)

- Run/Track distributed training `experiment` on prem or cloud via easy-to-use UI/API/SDK.
- Easy for data scientists to manage versions of `experiment` and dependencies of `environment`.
- Support popular machine learning frameworks, including **TensorFlow**, **PyTorch**, **Horovod**, and **MXNet**
- Provide pre-defined **template** for data scientists to implement domain-specific tasks easily (e.g. using DeepFM template to build a CTR prediction model)
- Support many compute resources (e.g. CPU and GPU, etc.)
- Support **Kubernetes** and **YARN**
- Pipeline is also on the backlog, we will look into pipeline for training in the future.

#### Notebook Service

- Submarine aims to provide a notebook service (e.g. Jupyter notebook) which allows users to manage notebook instances running on the cluster.

#### Model Management (Serving/versioning/monitoring, etc.)

- Model management for model-serving/versioning/monitoring is on the roadmap.

## Easy-to-use User Interface

As mentioned above, Submarine attempts to provide **Data-Scientist-friendly** UI to make data scientists have a good user experience. Here're some examples.

### Example: Submit a distributed Tensorflow experiment via Submarine Python SDK

#### Run a Tensorflow Mnist experiment

```python

# New a submarine client of the submarine server
submarine_client = submarine.ExperimentClient(host='http://localhost:8080')

# The experiment's environment, could be Docker image or Conda environment based
environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0')

# Specify the experiment's name, framework it's using, namespace it will run in,
# the entry point. It can also accept environment variables. etc.
# For PyTorch job, the framework should be 'Pytorch'.
experiment_meta = ExperimentMeta(name='mnist-dist',
namespace='default',
framework='Tensorflow',
cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100')
# 1 PS task of 2 cpu, 1GB
ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
replicas=1)
# 1 Worker task
worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
replicas=1)

# Wrap up the meta, environment and task specs into an experiment.
# For PyTorch job, the specs would be "Master" and "Worker".
experiment_spec = ExperimentSpec(meta=experiment_meta,
environment=environment,
spec={'Ps':ps_spec, 'Worker': worker_spec})

# Submit the experiment to submarine server
experiment = submarine_client.create_experiment(experiment_spec=experiment_spec)

# Get the experiment ID
id = experiment['experimentId']

```

#### Query a specific experiment

```python
submarine_client.get_experiment(id)
```

#### Wait for finish

```python
submarine_client.wait_for_finish(id)
```

#### Get the experiment's log

```python
submarine_client.get_log(id)
```

#### Get all running experiment

```python
submarine_client.list_experiments(status='running')
```

For a quick-start, see [Submarine On K8s](https://submarine.apache.org/docs/gettingStarted/quickstart)

### Example: Submit a pre-defined experiment template job

### Example: Submit an experiment via Submarine UI

(Available on 0.5.0, see Roadmap)

## Architecture, Design and requirements

If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on [Architecture-and-requirement](https://submarine.apache.org/docs/designDocs/architecture-and-requirements)

Detailed design documentation, implementation notes can be found at: [Implementation notes](https://submarine.apache.org/docs/designDocs/implementation-notes)

## Apache Submarine Community

Read the [Apache Submarine Community Guide](https://submarine.apache.org/docs/community/README)

How to contribute [Contributing Guide](https://submarine.apache.org/docs/community/contributing)

Login Submarine slack channel: [https://join.slack.com/t/asf-submarine/shared_invite](https://join.slack.com/t/asf-submarine/shared_invite/zt-18614cyqs-UhspdUOneiyg~ZPiVomDqw)

Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE

## User Document

See [User Guide Home Page](https://submarine.apache.org/docs/)

## Developer Document

See [Developer Guide Home Page](https://submarine.apache.org/docs/devDocs/Development/)

## Roadmap

What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap

## Changelog

From [here](https://submarine.apache.org/versions/), you can know the changelog and the issue tracker of different version of Apache Submarine.

## Resources

[Apache submarine: a unified machine learning platform made simple](https://dl.acm.org/doi/abs/10.1145/3517207.3526984) at EuroMLSys '22

## License

The Apache Submarine project is licensed under the Apache 2.0 License. See the [LICENSE](./LICENSE) file for details.