https://github.com/yahoo/TensorFlowOnSpark

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
https://github.com/yahoo/TensorFlowOnSpark

cluster featured machine-learning python scala spark tensorflow yahoo

Last synced: 8 months ago
JSON representation

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

• Host: GitHub
• URL: https://github.com/yahoo/TensorFlowOnSpark
• Owner: yahoo
• License: apache-2.0
• Created: 2017-01-20T18:15:57.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2023-07-10T10:34:11.000Z (over 2 years ago)
• Last Synced: 2025-03-18T22:24:14.917Z (8 months ago)
• Topics: cluster, featured, machine-learning, python, scala, spark, tensorflow, yahoo
• Language: Python
• Homepage:
• Size: 9.03 MB
• Stars: 3,876
• Watchers: 278
• Forks: 943
• Open Issues: 16
• Metadata Files:
  • Readme: README.md
  • Contributing: Contributing.md
  • License: LICENSE
  • Code of conduct: Code-of-Conduct.md

Awesome Lists containing this project

• awesome-tensorflow - TensorFlowOnSpark - initiative from Yahoo! to enable distributed TensorFlow with Apache Spark. (Libraries)
• awesome-list - TensorFlowOnSpark - Brings TensorFlow programs to Apache Spark clusters (Deep Learning Framework / High-Level DL APIs)
• awesome-tensorflow - TensorFlowOnSpark - initiative from Yahoo! to enable distributed TensorFlow with Apache Spark. (Libraries)
• awesome-python-machine-learning-resources - GitHub - 2% open · ⏱️ 21.04.2022): (分布式机器学习)
• awesome-production-machine-learning - TensorFlowOnSpark - TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters. (Computation Load Distribution)
• StarryDivineSky - yahoo/TensorFlowOnSpark
• fucking-awesome-tensorflow - TensorFlowOnSpark - initiative from Yahoo! to enable distributed TensorFlow with Apache Spark. (Libraries)
• Awesome-TensorFlow-Chinese - TensorFlowOnSpark - initiative from Yahoo! to enable distributed TensorFlow with Apache Spark. (库 / 微信群)

README

          

# TensorFlowOnSpark

> _TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark

clusters._

[![Build Status](https://cd.screwdriver.cd/pipelines/6384/badge)](https://cd.screwdriver.cd/pipelines/6384)

[![Package](https://img.shields.io/badge/package-pypi-blue.svg)](https://pypi.org/project/tensorflowonspark/)

[![Downloads](https://img.shields.io/pypi/dm/tensorflowonspark.svg)](https://img.shields.io/pypi/dm/tensorflowonspark.svg)

[![Documentation](https://img.shields.io/badge/Documentation-latest-blue.svg)](https://yahoo.github.io/TensorFlowOnSpark/)

By combining salient features from the [TensorFlow](https://www.tensorflow.org) deep learning framework with [Apache Spark](http://spark.apache.org) and [Apache Hadoop](http://hadoop.apache.org), TensorFlowOnSpark enables distributed

deep learning on a cluster of GPU and CPU servers.

It enables both distributed TensorFlow training and

inferencing on Spark clusters, with a goal to minimize the amount

of code changes required to run existing TensorFlow programs on a

shared grid.  Its Spark-compatible API helps manage the TensorFlow

cluster with the following steps:

1. **Startup** - launches the Tensorflow main function on the executors, along with listeners for data/control messages.

1. **Data ingestion**

   - **InputMode.TENSORFLOW** - leverages TensorFlow's built-in APIs to read data files directly from HDFS.

   - **InputMode.SPARK** - sends Spark RDD data to the TensorFlow nodes via a `TFNode.DataFeed` class.  Note that we leverage the [Hadoop Input/Output Format](https://github.com/tensorflow/ecosystem/tree/master/hadoop) to access TFRecords on HDFS.

1. **Shutdown** - shuts down the Tensorflow workers and PS nodes on the executors.

## Table of Contents

- [Background](#background)

- [Install](#install)

- [Usage](#usage)

- [API](#api)

- [Contribute](#contribute)

- [License](#license)

## Background

TensorFlowOnSpark was developed by Yahoo for large-scale distributed

deep learning on our Hadoop clusters in Yahoo's private cloud.

TensorFlowOnSpark provides some important benefits (see [our

blog](https://developer.yahoo.com/blogs/157196317141/))

over alternative deep learning solutions.

   * Easily migrate existing TensorFlow programs with <10 lines of code change.

   * Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard.

   * Server-to-server direct communication achieves faster learning when available.

   * Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow.

   * Easily integrate with your existing Spark data processing pipelines.

   * Easily deployed on cloud or on-premise and on CPUs or GPUs.

## Install

TensorFlowOnSpark is provided as a pip package, which can be installed on single machines via:

```

# for tensorflow>=2.0.0

pip install tensorflowonspark

# for tensorflow<2.0.0

pip install tensorflowonspark==1.4.4

```

For distributed clusters, please see our [wiki site](../../wiki) for detailed documentation for specific environments, such as our getting started guides for [single-node Spark Standalone](https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_Standalone), [YARN clusters](../../wiki/GetStarted_YARN) and [AWS EC2](../../wiki/GetStarted_EC2).  Note: the Windows operating system is not currently supported due to [this issue](https://github.com/yahoo/TensorFlowOnSpark/issues/36).

## Usage

To use TensorFlowOnSpark with an existing TensorFlow application, you can follow our [Conversion Guide](../../wiki/Conversion-Guide) to describe the required changes.  Additionally, our [wiki site](../../wiki) has pointers to some presentations which provide an overview of the platform.

**Note: since TensorFlow 2.x breaks API compatibility with TensorFlow 1.x, the examples have been updated accordingly.  If you are using TensorFlow 1.x, you will need to checkout the `v1.4.4` tag for compatible examples and instructions.**

## API

[API Documentation](https://yahoo.github.io/TensorFlowOnSpark/) is automatically generated from the code.

## Contribute

Please join the [TensorFlowOnSpark user group](https://groups.google.com/forum/#!forum/TensorFlowOnSpark-users) for discussions and questions.  If you have a question, please review our [FAQ](../../wiki/Frequently-Asked-Questions) before posting.

Contributions are always welcome.  For more information, please see our [guide for getting involved](Contributing.md).

## License

The use and distribution terms for this software are covered by the Apache 2.0 license.

See [LICENSE](LICENSE) file for terms.