{"id":13487410,"url":"https://github.com/apache/submarine","last_synced_at":"2026-01-10T04:51:27.325Z","repository":{"id":37799420,"uuid":"209459144","full_name":"apache/submarine","owner":"apache","description":"Submarine is Cloud Native Machine Learning Platform.","archived":true,"fork":false,"pushed_at":"2024-04-03T00:59:35.000Z","size":30721,"stargazers_count":695,"open_issues_count":52,"forks_count":252,"subscribers_count":59,"default_branch":"master","last_synced_at":"2024-12-03T16:16:30.354Z","etag":null,"topics":["ai","deep-learning","docker","kubernetes","machine-learning","notebook"],"latest_commit_sha":null,"homepage":"https://submarine.apache.org/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-19T04:00:17.000Z","updated_at":"2024-12-02T05:59:37.000Z","dependencies_parsed_at":"2024-01-23T17:30:30.693Z","dependency_job_id":"d67bedfc-b36a-4201-9685-6affaabf3478","html_url":"https://github.com/apache/submarine","commit_stats":{"total_commits":1050,"total_committers":91,"mean_commits":"11.538461538461538","dds":0.8847619047619047,"last_synced_commit":"7a1d551798c6785fc68fe028fc46f74c3ee6976d"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fsubmarine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fsubmarine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fsubmarine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fsubmarine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/submarine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235669382,"owners_count":19026815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","deep-learning","docker","kubernetes","machine-learning","notebook"],"created_at":"2024-07-31T18:00:58.928Z","updated_at":"2025-10-08T01:30:26.969Z","avatar_url":"https://github.com/apache.png","language":"Java","funding_links":[],"categories":["Java","人工智能"],"sub_categories":[],"readme":"\u003c!---\n  Licensed under the Apache License, Version 2.0 (the \"License\");\n  you may not use this file except in compliance with the License.\n  You may obtain a copy of the License at\n\n   http://www.apache.org/licenses/LICENSE-2.0\n\n  Unless required by applicable law or agreed to in writing, software\n  distributed under the License is distributed on an \"AS IS\" BASIS,\n  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n  See the License for the specific language governing permissions and\n  limitations under the License. See accompanying LICENSE file.\n--\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n![Colored_logo_with_text](website/static/img/icons/color_logo_with_text.png)\n\n![Submarine workflow](https://github.com/apache/submarine/actions/workflows/master.yml/badge.svg?branch=master) ![python-sdk workflow](https://github.com/apache/submarine/actions/workflows/python.yml/badge.svg?branch=master) [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) [![PyPI version](https://badge.fury.io/py/apache-submarine.svg)](https://badge.fury.io/py/apache-submarine)\n\n\u003c/div\u003e\n\n# What is Apache Submarine?\n\n**Apache Submarine** (Submarine for short) is an **End-to-End Machine Learning Platform** to allow data scientists to create end-to-end machine learning workflows. On **Submarine**, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.\n\n## Why Submarine?\n\nSome open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?\n\n### Problems\n\n1. Many platforms lack easy-to-use user interfaces (API, SDK, and IDE, etc.)\n2. In the same company, data scientists in different teams usually spend much time on developments of existing feature sets and models.\n3. Data scientists put emphasis on domain-specific tasks (e.g. Click-Through-Rate), but they need to implement their models from scratch with SDKs provided by existing platforms.\n4. Many platforms lack a unified workbench to manage each component in the ML lifecycle.\n\n_Theodore Levitt_ once said:\n\n```\n“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”\n```\n\n### Goals of Submarine\n\n#### Model Training (Experiment)\n\n- Run/Track distributed training `experiment` on prem or cloud via easy-to-use UI/API/SDK.\n- Easy for data scientists to manage versions of `experiment` and dependencies of `environment`.\n- Support popular machine learning frameworks, including **TensorFlow**, **PyTorch**, **Horovod**, and **MXNet**\n- Provide pre-defined **template** for data scientists to implement domain-specific tasks easily (e.g. using DeepFM template to build a CTR prediction model)\n- Support many compute resources (e.g. CPU and GPU, etc.)\n- Support **Kubernetes** and **YARN**\n- Pipeline is also on the backlog, we will look into pipeline for training in the future.\n\n#### Notebook Service\n\n- Submarine aims to provide a notebook service (e.g. Jupyter notebook) which allows users to manage notebook instances running on the cluster.\n\n#### Model Management (Serving/versioning/monitoring, etc.)\n\n- Model management for model-serving/versioning/monitoring is on the roadmap.\n\n## Easy-to-use User Interface\n\nAs mentioned above, Submarine attempts to provide **Data-Scientist-friendly** UI to make data scientists have a good user experience. Here're some examples.\n\n### Example: Submit a distributed Tensorflow experiment via Submarine Python SDK\n\n#### Run a Tensorflow Mnist experiment\n\n```python\n\n# New a submarine client of the submarine server\nsubmarine_client = submarine.ExperimentClient(host='http://localhost:8080')\n\n# The experiment's environment, could be Docker image or Conda environment based\nenvironment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0')\n\n# Specify the experiment's name, framework it's using, namespace it will run in,\n# the entry point. It can also accept environment variables. etc.\n# For PyTorch job, the framework should be 'Pytorch'.\nexperiment_meta = ExperimentMeta(name='mnist-dist',\n                                 namespace='default',\n                                 framework='Tensorflow',\n                                 cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100')\n# 1 PS task of 2 cpu, 1GB\nps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',\n                             replicas=1)\n# 1 Worker task\nworker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',\n                                 replicas=1)\n\n# Wrap up the meta, environment and task specs into an experiment.\n# For PyTorch job, the specs would be \"Master\" and \"Worker\".\nexperiment_spec = ExperimentSpec(meta=experiment_meta,\n                                 environment=environment,\n                                 spec={'Ps':ps_spec, 'Worker': worker_spec})\n\n# Submit the experiment to submarine server\nexperiment = submarine_client.create_experiment(experiment_spec=experiment_spec)\n\n# Get the experiment ID\nid = experiment['experimentId']\n\n```\n\n#### Query a specific experiment\n\n```python\nsubmarine_client.get_experiment(id)\n```\n\n#### Wait for finish\n\n```python\nsubmarine_client.wait_for_finish(id)\n```\n\n#### Get the experiment's log\n\n```python\nsubmarine_client.get_log(id)\n```\n\n#### Get all running experiment\n\n```python\nsubmarine_client.list_experiments(status='running')\n```\n\nFor a quick-start, see [Submarine On K8s](https://submarine.apache.org/docs/gettingStarted/quickstart)\n\n### Example: Submit a pre-defined experiment template job\n\n### Example: Submit an experiment via Submarine UI\n\n(Available on 0.5.0, see Roadmap)\n\n## Architecture, Design and requirements\n\nIf you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on [Architecture-and-requirement](https://submarine.apache.org/docs/designDocs/architecture-and-requirements)\n\nDetailed design documentation, implementation notes can be found at: [Implementation notes](https://submarine.apache.org/docs/designDocs/implementation-notes)\n\n## Apache Submarine Community\n\nRead the [Apache Submarine Community Guide](https://submarine.apache.org/docs/community/README)\n\nHow to contribute [Contributing Guide](https://submarine.apache.org/docs/community/contributing)\n\nLogin Submarine slack channel: [https://join.slack.com/t/asf-submarine/shared_invite](https://join.slack.com/t/asf-submarine/shared_invite/zt-18614cyqs-UhspdUOneiyg~ZPiVomDqw)\n\nIssue Tracking: https://issues.apache.org/jira/projects/SUBMARINE\n\n## User Document\n\nSee [User Guide Home Page](https://submarine.apache.org/docs/)\n\n## Developer Document\n\nSee [Developer Guide Home Page](https://submarine.apache.org/docs/devDocs/Development/)\n\n## Roadmap\n\nWhat to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap\n\n## Changelog\n\nFrom [here](https://submarine.apache.org/versions/), you can know the changelog and the issue tracker of different version of Apache Submarine.\n\n## Resources\n\n[Apache submarine: a unified machine learning platform made simple](https://dl.acm.org/doi/abs/10.1145/3517207.3526984) at EuroMLSys '22\n\n## License\n\nThe Apache Submarine project is licensed under the Apache 2.0 License. See the [LICENSE](./LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fsubmarine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fsubmarine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fsubmarine/lists"}