https://github.com/pingsutw/submarine-sdk

A python SDK for Apache/Submarine
https://github.com/pingsutw/submarine-sdk

Last synced: 19 days ago
JSON representation

A python SDK for Apache/Submarine

• Host: GitHub
• URL: https://github.com/pingsutw/submarine-sdk
• Owner: pingsutw
• License: apache-2.0
• Created: 2019-08-30T17:16:55.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2023-07-06T21:35:43.000Z (over 1 year ago)
• Last Synced: 2024-10-17T05:09:09.106Z (about 1 month ago)
• Language: Python
• Homepage: https://pypi.org/project/pysubmarine/
• Size: 49.8 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


# What is Hadoop Submarine?  [![Build Status](https://travis-ci.org/pingsutw/Submarine-SDK.svg?branch=master)](https://travis-ci.org/pingsutw/Submarine-SDK)

Submarine is a new subproject of Apache Hadoop.

Submarine is a project which allows infra engineer / data scientist to run

*unmodified* Tensorflow or PyTorch programs on YARN or Kubernetes.

Goals of Submarine:

- It allows jobs easy access data/models in HDFS and other storages.

- Can launch services to serve Tensorflow/PyTorch models.

- Support run distributed Tensorflow jobs with simple configs.

- Support run user-specified Docker images.

- Support specify GPU and other resources.

- Support launch tensorboard for training jobs if user specified.

- Support customized DNS name for roles (like tensorboard.$user.$domain:6006)

# Submarine-SDK

- Allow data scients to track distributed ML job 

- Support store ML parameters and metrics in Submarine-server

- Support store ML job output (e.g. csv,images)

- Support hdfs,S3 and mysql 

- (Submarine-DB) metric and param instance in submarine-server database 

- (Submarine-DB) Support REST Api for submarine server 

- (WEB) Metric tracking ui in submarine-web

- (WEB) Metric graphical display in submarine-web