{"id":28763233,"url":"https://github.com/flink-extended/dl-on-flink","last_synced_at":"2025-12-25T00:01:49.010Z","repository":{"id":36970232,"uuid":"192298972","full_name":"flink-extended/dl-on-flink","owner":"flink-extended","description":"Deep Learning on Flink aims to integrate Flink and deep learning frameworks (e.g. TensorFlow, PyTorch, etc) to enable distributed deep learning training and inference on a Flink cluster.","archived":false,"fork":false,"pushed_at":"2024-11-12T19:57:19.000Z","size":56464,"stargazers_count":689,"open_issues_count":5,"forks_count":202,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-06-16T11:18:15.259Z","etag":null,"topics":["deep-learning","flink","python","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flink-extended.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-17T07:44:14.000Z","updated_at":"2025-06-12T13:20:20.000Z","dependencies_parsed_at":"2024-11-15T07:37:57.248Z","dependency_job_id":null,"html_url":"https://github.com/flink-extended/dl-on-flink","commit_stats":{"total_commits":690,"total_committers":22,"mean_commits":"31.363636363636363","dds":0.6681159420289855,"last_synced_commit":"ab209907c4a4ed7a017d00d079519fb01fd4045d"},"previous_names":["alibaba/flink-ai-extended"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/flink-extended/dl-on-flink","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fdl-on-flink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fdl-on-flink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fdl-on-flink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fdl-on-flink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flink-extended","download_url":"https://codeload.github.com/flink-extended/dl-on-flink/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fdl-on-flink/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260148731,"owners_count":22965956,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","flink","python","pytorch","tensorflow"],"created_at":"2025-06-17T09:07:53.778Z","updated_at":"2025-12-25T00:01:43.989Z","avatar_url":"https://github.com/flink-extended.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":[],"readme":"# Deep Learning on Flink\n\nDeep Learning on Flink aims to integrate Flink and deep learning frameworks\n(e.g. TensorFlow, PyTorch, etc.) to enable distributed deep learning training and\ninference on a Flink cluster.\n\nIt runs the deep learning tasks inside a Flink operator so that Flink can help\nestablish a distributed environment, manage the resource, read/write the data\nwith the rich connectors in Flink and handle the failures.\n\nCurrently, Deep Learning on Flink supports TensorFlow.\n\n## Supported Operating System\nDeep Learning on Flink is tested and supported on the following 64-bit systems:\n\n- Ubuntu 18.04\n- macOS 10.15\n\n## Support Framework Version\n- TensorFlow: 1.15.x \u0026 2.4.x\n- PyTorch: 1.11.x\n- Flink: 1.14.x\n \n## Getting Started\nDeep learning on Flink currently works with Tensorflow and PyTorch. You can see\nthe following pages for the usage and examples.\n\n- [DL on Flink with Tensorflow](doc/tensorflow-index.md)\n- [DL on Flink with PyTorch](doc/pytorch-index.md)\n\n## Build From Source\n\n**Requirements**\n- python: 3.7\n- cmake \u003e= 3.6\n- java 1.8\n- maven \u003e=3.3.0\n\nDeep Learning on Flink requires Java and Python works together. Thus, we need \nto build for both Java and Python.\n\n### Initializing Submodules before Building Deep Learning on Flink from Source\n\nPlease use the following command to initialize submodules before building \nfrom source.\n\n```bash\ngit submodule update --init --recursive\n```\n\n### Build Java\n\n```shell \nmvn -DskipTests clean install\n```\n\nAfter finish, you can find the target distribution in the `dl-on-flink-dist/target`\nfolder.\n\n### Build Python\n\n#### Install from Source\nYou can run the following commands to install the Python packages from source\n\n```sh\n# Install dl-on-flink-framework first\npip install dl-on-flink-framework/python\n\n# Note that you should only install one of the following as they require\n# different versions of Tensorflow \n# For tensorflow 1.15.x\npip install dl-on-flink-tensorflow/python\n# For tensorflow 2.4.x\npip install dl-on-flink-tensorflow-2.x/python\n```\n\n#### Build wheels\nWe provide a script to build wheels for Python packages, you can run the\nfollowing command.\n\n```sh\nbash tools/build_wheel.sh\n```\n\nAfter finish, you can find the wheels at `tools/dist`. Then you can install the\npython package with the wheels.\n\n```sh\npip install tools/dist/\u003cwheel\u003e\n```\n\n# For More Information\n\n[Design document](doc/design.md)\n\n# License\n[Apache License 2.0](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflink-extended%2Fdl-on-flink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflink-extended%2Fdl-on-flink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflink-extended%2Fdl-on-flink/lists"}