https://github.com/flink-extended/dl-on-flink
Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc) to enable distributed deep learning training and inference on a Flink cluster.
https://github.com/flink-extended/dl-on-flink
deep-learning flink python pytorch tensorflow
Last synced: 6 months ago
JSON representation
Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc) to enable distributed deep learning training and inference on a Flink cluster.
- Host: GitHub
- URL: https://github.com/flink-extended/dl-on-flink
- Owner: flink-extended
- License: apache-2.0
- Created: 2019-06-17T07:44:14.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-11-12T19:57:19.000Z (about 1 year ago)
- Last Synced: 2025-06-16T11:18:15.259Z (6 months ago)
- Topics: deep-learning, flink, python, pytorch, tensorflow
- Language: Java
- Homepage:
- Size: 53.8 MB
- Stars: 689
- Watchers: 40
- Forks: 202
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-java - Deep Learning Flink
README
# Deep Learning on Flink
Deep Learning on Flink aims to integrate Flink and deep learning frameworks
(e.g. TensorFlow, PyTorch, etc.) to enable distributed deep learning training and
inference on a Flink cluster.
It runs the deep learning tasks inside a Flink operator so that Flink can help
establish a distributed environment, manage the resource, read/write the data
with the rich connectors in Flink and handle the failures.
Currently, Deep Learning on Flink supports TensorFlow.
## Supported Operating System
Deep Learning on Flink is tested and supported on the following 64-bit systems:
- Ubuntu 18.04
- macOS 10.15
## Support Framework Version
- TensorFlow: 1.15.x & 2.4.x
- PyTorch: 1.11.x
- Flink: 1.14.x
## Getting Started
Deep learning on Flink currently works with Tensorflow and PyTorch. You can see
the following pages for the usage and examples.
- [DL on Flink with Tensorflow](doc/tensorflow-index.md)
- [DL on Flink with PyTorch](doc/pytorch-index.md)
## Build From Source
**Requirements**
- python: 3.7
- cmake >= 3.6
- java 1.8
- maven >=3.3.0
Deep Learning on Flink requires Java and Python works together. Thus, we need
to build for both Java and Python.
### Initializing Submodules before Building Deep Learning on Flink from Source
Please use the following command to initialize submodules before building
from source.
```bash
git submodule update --init --recursive
```
### Build Java
```shell
mvn -DskipTests clean install
```
After finish, you can find the target distribution in the `dl-on-flink-dist/target`
folder.
### Build Python
#### Install from Source
You can run the following commands to install the Python packages from source
```sh
# Install dl-on-flink-framework first
pip install dl-on-flink-framework/python
# Note that you should only install one of the following as they require
# different versions of Tensorflow
# For tensorflow 1.15.x
pip install dl-on-flink-tensorflow/python
# For tensorflow 2.4.x
pip install dl-on-flink-tensorflow-2.x/python
```
#### Build wheels
We provide a script to build wheels for Python packages, you can run the
following command.
```sh
bash tools/build_wheel.sh
```
After finish, you can find the wheels at `tools/dist`. Then you can install the
python package with the wheels.
```sh
pip install tools/dist/
```
# For More Information
[Design document](doc/design.md)
# License
[Apache License 2.0](LICENSE)