Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/apache/submarine
Submarine is Cloud Native Machine Learning Platform.
https://github.com/apache/submarine
ai deep-learning docker kubernetes machine-learning notebook
Last synced: about 1 month ago
JSON representation
Submarine is Cloud Native Machine Learning Platform.
- Host: GitHub
- URL: https://github.com/apache/submarine
- Owner: apache
- License: apache-2.0
- Archived: true
- Created: 2019-09-19T04:00:17.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-04-03T00:59:35.000Z (7 months ago)
- Last Synced: 2024-09-26T16:01:21.561Z (about 1 month ago)
- Topics: ai, deep-learning, docker, kubernetes, machine-learning, notebook
- Language: Java
- Homepage: https://submarine.apache.org/
- Size: 29.3 MB
- Stars: 691
- Watchers: 59
- Forks: 254
- Open Issues: 52
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
  [](https://www.apache.org/licenses/LICENSE-2.0.html) [](https://badge.fury.io/py/apache-submarine)
# What is Apache Submarine?
**Apache Submarine** (Submarine for short) is an **End-to-End Machine Learning Platform** to allow data scientists to create end-to-end machine learning workflows. On **Submarine**, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.
## Why Submarine?
Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?
### Problems
1. Many platforms lack easy-to-use user interfaces (API, SDK, and IDE, etc.)
2. In the same company, data scientists in different teams usually spend much time on developments of existing feature sets and models.
3. Data scientists put emphasis on domain-specific tasks (e.g. Click-Through-Rate), but they need to implement their models from scratch with SDKs provided by existing platforms.
4. Many platforms lack a unified workbench to manage each component in the ML lifecycle.
_Theodore Levitt_ once said:
```
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
```
### Goals of Submarine
#### Model Training (Experiment)
- Run/Track distributed training `experiment` on prem or cloud via easy-to-use UI/API/SDK.
- Easy for data scientists to manage versions of `experiment` and dependencies of `environment`.
- Support popular machine learning frameworks, including **TensorFlow**, **PyTorch**, **Horovod**, and **MXNet**
- Provide pre-defined **template** for data scientists to implement domain-specific tasks easily (e.g. using DeepFM template to build a CTR prediction model)
- Support many compute resources (e.g. CPU and GPU, etc.)
- Support **Kubernetes** and **YARN**
- Pipeline is also on the backlog, we will look into pipeline for training in the future.
#### Notebook Service
- Submarine aims to provide a notebook service (e.g. Jupyter notebook) which allows users to manage notebook instances running on the cluster.
#### Model Management (Serving/versioning/monitoring, etc.)
- Model management for model-serving/versioning/monitoring is on the roadmap.
## Easy-to-use User Interface
As mentioned above, Submarine attempts to provide **Data-Scientist-friendly** UI to make data scientists have a good user experience. Here're some examples.
### Example: Submit a distributed Tensorflow experiment via Submarine Python SDK
#### Run a Tensorflow Mnist experiment
```python
# New a submarine client of the submarine server
submarine_client = submarine.ExperimentClient(host='http://localhost:8080')
# The experiment's environment, could be Docker image or Conda environment based
environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0')
# Specify the experiment's name, framework it's using, namespace it will run in,
# the entry point. It can also accept environment variables. etc.
# For PyTorch job, the framework should be 'Pytorch'.
experiment_meta = ExperimentMeta(name='mnist-dist',
namespace='default',
framework='Tensorflow',
cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100')
# 1 PS task of 2 cpu, 1GB
ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
replicas=1)
# 1 Worker task
worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
replicas=1)
# Wrap up the meta, environment and task specs into an experiment.
# For PyTorch job, the specs would be "Master" and "Worker".
experiment_spec = ExperimentSpec(meta=experiment_meta,
environment=environment,
spec={'Ps':ps_spec, 'Worker': worker_spec})
# Submit the experiment to submarine server
experiment = submarine_client.create_experiment(experiment_spec=experiment_spec)
# Get the experiment ID
id = experiment['experimentId']
```
#### Query a specific experiment
```python
submarine_client.get_experiment(id)
```
#### Wait for finish
```python
submarine_client.wait_for_finish(id)
```
#### Get the experiment's log
```python
submarine_client.get_log(id)
```
#### Get all running experiment
```python
submarine_client.list_experiments(status='running')
```
For a quick-start, see [Submarine On K8s](https://submarine.apache.org/docs/gettingStarted/quickstart)
### Example: Submit a pre-defined experiment template job
### Example: Submit an experiment via Submarine UI
(Available on 0.5.0, see Roadmap)
## Architecture, Design and requirements
If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on [Architecture-and-requirement](https://submarine.apache.org/docs/designDocs/architecture-and-requirements)
Detailed design documentation, implementation notes can be found at: [Implementation notes](https://submarine.apache.org/docs/designDocs/implementation-notes)
## Apache Submarine Community
Read the [Apache Submarine Community Guide](https://submarine.apache.org/docs/community/README)
How to contribute [Contributing Guide](https://submarine.apache.org/docs/community/contributing)
Login Submarine slack channel: [https://join.slack.com/t/asf-submarine/shared_invite](https://join.slack.com/t/asf-submarine/shared_invite/zt-18614cyqs-UhspdUOneiyg~ZPiVomDqw)
Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE
## User Document
See [User Guide Home Page](https://submarine.apache.org/docs/)
## Developer Document
See [Developer Guide Home Page](https://submarine.apache.org/docs/devDocs/Development/)
## Roadmap
What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap
## Changelog
From [here](https://submarine.apache.org/versions/), you can know the changelog and the issue tracker of different version of Apache Submarine.
## Resources
[Apache submarine: a unified machine learning platform made simple](https://dl.acm.org/doi/abs/10.1145/3517207.3526984) at EuroMLSys '22
## License
The Apache Submarine project is licensed under the Apache 2.0 License. See the [LICENSE](./LICENSE) file for details.