Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/yahoo/tensorflowonspark

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
https://github.com/yahoo/tensorflowonspark

cluster featured machine-learning python scala spark tensorflow yahoo

Last synced: about 2 hours ago
JSON representation

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

Host: GitHub
URL: https://github.com/yahoo/tensorflowonspark
Owner: yahoo
License: apache-2.0
Created: 2017-01-20T18:15:57.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2023-07-10T10:34:11.000Z (about 1 year ago)
Last Synced: 2024-09-24T08:33:31.440Z (about 9 hours ago)
Topics: cluster, featured, machine-learning, python, scala, spark, tensorflow, yahoo
Language: Python
Homepage:
Size: 9.03 MB
Stars: 3,872
Watchers: 282
Forks: 938
Open Issues: 15
Metadata Files:
- Readme: README.md
- Contributing: Contributing.md
- License: LICENSE
- Code of conduct: Code-of-Conduct.md

Awesome Lists containing this project

README

        
# TensorFlowOnSpark

> _TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark

clusters._

[![Build Status](https://cd.screwdriver.cd/pipelines/6384/badge)](https://cd.screwdriver.cd/pipelines/6384)

[![Package](https://img.shields.io/badge/package-pypi-blue.svg)](https://pypi.org/project/tensorflowonspark/)

[![Downloads](https://img.shields.io/pypi/dm/tensorflowonspark.svg)](https://img.shields.io/pypi/dm/tensorflowonspark.svg)

[![Documentation](https://img.shields.io/badge/Documentation-latest-blue.svg)](https://yahoo.github.io/TensorFlowOnSpark/)

By combining salient features from the [TensorFlow](https://www.tensorflow.org) deep learning framework with [Apache Spark](http://spark.apache.org) and [Apache Hadoop](http://hadoop.apache.org), TensorFlowOnSpark enables distributed

deep learning on a cluster of GPU and CPU servers.

It enables both distributed TensorFlow training and

inferencing on Spark clusters, with a goal to minimize the amount

of code changes required to run existing TensorFlow programs on a

shared grid.  Its Spark-compatible API helps manage the TensorFlow

cluster with the following steps:

1. **Startup** - launches the Tensorflow main function on the executors, along with listeners for data/control messages.

1. **Data ingestion**

   - **InputMode.TENSORFLOW** - leverages TensorFlow's built-in APIs to read data files directly from HDFS.

   - **InputMode.SPARK** - sends Spark RDD data to the TensorFlow nodes via a `TFNode.DataFeed` class.  Note that we leverage the [Hadoop Input/Output Format](https://github.com/tensorflow/ecosystem/tree/master/hadoop) to access TFRecords on HDFS.

1. **Shutdown** - shuts down the Tensorflow workers and PS nodes on the executors.

## Table of Contents

- [Background](#background)

- [Install](#install)

- [Usage](#usage)

- [API](#api)

- [Contribute](#contribute)

- [License](#license)

## Background

TensorFlowOnSpark was developed by Yahoo for large-scale distributed

deep learning on our Hadoop clusters in Yahoo's private cloud.

TensorFlowOnSpark provides some important benefits (see [our

blog](https://developer.yahoo.com/blogs/157196317141/))

over alternative deep learning solutions.

   * Easily migrate existing TensorFlow programs with <10 lines of code change.

   * Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard.

   * Server-to-server direct communication achieves faster learning when available.

   * Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow.

   * Easily integrate with your existing Spark data processing pipelines.

   * Easily deployed on cloud or on-premise and on CPUs or GPUs.

## Install

TensorFlowOnSpark is provided as a pip package, which can be installed on single machines via:

```

# for tensorflow>=2.0.0

pip install tensorflowonspark

# for tensorflow<2.0.0

pip install tensorflowonspark==1.4.4

```

For distributed clusters, please see our [wiki site](../../wiki) for detailed documentation for specific environments, such as our getting started guides for [single-node Spark Standalone](https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_Standalone), [YARN clusters](../../wiki/GetStarted_YARN) and [AWS EC2](../../wiki/GetStarted_EC2).  Note: the Windows operating system is not currently supported due to [this issue](https://github.com/yahoo/TensorFlowOnSpark/issues/36).

## Usage

To use TensorFlowOnSpark with an existing TensorFlow application, you can follow our [Conversion Guide](../../wiki/Conversion-Guide) to describe the required changes.  Additionally, our [wiki site](../../wiki) has pointers to some presentations which provide an overview of the platform.

**Note: since TensorFlow 2.x breaks API compatibility with TensorFlow 1.x, the examples have been updated accordingly.  If you are using TensorFlow 1.x, you will need to checkout the `v1.4.4` tag for compatible examples and instructions.**

## API

[API Documentation](https://yahoo.github.io/TensorFlowOnSpark/) is automatically generated from the code.

## Contribute

Please join the [TensorFlowOnSpark user group](https://groups.google.com/forum/#!forum/TensorFlowOnSpark-users) for discussions and questions.  If you have a question, please review our [FAQ](../../wiki/Frequently-Asked-Questions) before posting.

Contributions are always welcome.  For more information, please see our [guide for getting involved](Contributing.md).

## License

The use and distribution terms for this software are covered by the Apache 2.0 license.

See [LICENSE](LICENSE) file for terms.