{"id":13569785,"url":"https://github.com/flink-extended/flink-remote-shuffle","last_synced_at":"2025-06-17T09:10:01.645Z","repository":{"id":38021217,"uuid":"432925755","full_name":"flink-extended/flink-remote-shuffle","owner":"flink-extended","description":"Remote Shuffle Service for Flink","archived":false,"fork":false,"pushed_at":"2023-01-06T05:50:02.000Z","size":68163,"stargazers_count":189,"open_issues_count":22,"forks_count":56,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-04T06:36:07.517Z","etag":null,"topics":["flink","shuffle"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flink-extended.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-29T06:12:32.000Z","updated_at":"2025-01-13T09:40:28.000Z","dependencies_parsed_at":"2023-02-05T11:31:52.361Z","dependency_job_id":null,"html_url":"https://github.com/flink-extended/flink-remote-shuffle","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/flink-extended/flink-remote-shuffle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fflink-remote-shuffle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fflink-remote-shuffle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fflink-remote-shuffle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fflink-remote-shuffle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flink-extended","download_url":"https://codeload.github.com/flink-extended/flink-remote-shuffle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flink-extended%2Fflink-remote-shuffle/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260326793,"owners_count":22992388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flink","shuffle"],"created_at":"2024-08-01T14:00:44.248Z","updated_at":"2025-06-17T09:10:01.615Z","avatar_url":"https://github.com/flink-extended.png","language":"Java","funding_links":[],"categories":["Java","大数据"],"sub_categories":["Spring Cloud框架"],"readme":"# Remote Shuffle Service for Flink\n\n- [Overview](#overview)\n- [Supported Flink Version](#supported-flink-version)\n- [Document](#document)\n- [Building from Source](#building-from-source)\n- [Example](#example)\n- [How to Contribute](#how-to-contribute)\n- [Support](#support)\n- [Acknowledge](#acknowledge)\n\n## Overview\n\nThis project implements a remote shuffle service for batch data processing\nof [Flink](https://flink.apache.org/). By adopting the storage and compute separation architecture,\nit brings several important benefits:\n\n1. The scale up/down of computing resources and storage resources is now decoupled which means you\n   can scale each up/down on demand freely.\n\n2. Compute and storage stability never influence each other anymore. The remote shuffle service is\n   free of user-code which can improve shuffle stability. For example, the termination\n   of `TaskExecutor`s will not lead to data loss and the termination of remote `ShuffleWorker`s is\n   tolerable.\n\n3. By offloading the data shuffle work to the remote shuffle service, the computation resources can\n   be released immediately after the upstream map tasks finish which can save resources.\n\nIn addition, the remote shuffle implementation borrows some good designs from Flink which can\nbenefit both stability and performance, for example:\n\n1. Managed memory is preferred. Both the storage and network memory are managed which can\n   significantly solve the OutOfMemory issue.\n\n2. The credit-based backpressure mechanism is adopted which is good for both network stability and\n   performance.\n\n3. The zero-copy network data transmission is implemented which can save memory and is also good for\n   stability and performance.\n\nBesides, there are other important optimizations like load balancing and better sequential IO (\nbenefiting from the centralized service per node), tcp connection reuse, shuffle data compression,\nadaptive execution (together with FLIP-187), etc.\n\nBefore going open source, this project has been used in production widely and behaves well on both\nstability and performance. Hope you enjoy it.\n\n## Supported Flink Version\n\nThe remote shuffle service works together with Flink 1.14+. Some patches are needed to be applied to\nFlink to support lower Flink versions. If you need any help on that, please let us know, we can\noffer some help to prepare the patches for the Flink version you use.\n\n## Document\n\nThe remote shuffle service supports standalone, yarn and k8s deployment. You can find the full documents [here](https://github.com/flink-extended/flink-remote-shuffle/wiki).\nIn the future, more internal implementation detail specifications will be supplemented.\n\n## Building from Source\n\nTo build this flink remote shuffle project from source, you should first clone the project:\n\n```bash\ngit clone git@github.com:flink-extended/flink-remote-shuffle.git\n```\n\nThen you can build the project using maven (Maven and Java 8 required):\n\n```bash\ncd flink-remote-shuffle # switch to the remote shuffle project home directory\nmvn package -DskipTests\n```\n\nAfter finish, you can find the target distribution in the build-target folder. Note that if you want to run tests locally, we suggest you to run `mvn install -DskipTests` first to avoid potential failures.\n\nFor k8s deployment, you can run the following command to build the docker image (Docker required):\n\n```bash\ncd flink-remote-shuffle # switch to the remote shuffle project home directory\nsh ./tools/build_docker_image.sh\n```\n\nYou can also publish the docker image by running the following command. The script that publishes\nthe docker image takes three arguments: the first one is the registry address (default value is\n'docker.io'), the second one is the namespace (default value is 'flinkremoteshuffle'), the third one\nis the repository name (default value is 'flink-remote-shuffle').\n\n```bash\ncd flink-remote-shuffle # switch to the remote shuffle project home directory\nsh ./tools/publish_docker_image.sh REGISTRY NAMESPACE REPOSITORY\n```\n\n## Example\n\nAfter building the code from source, you can start and run a demo flink batch job using the remote\nshuffle service locally (Flink 1.14+ required):\n\nAs the first step, you can download the Flink distribution from the\nFlink's [download page](https://flink.apache.org/downloads.html#apache-flink-1140), for example,\nApache Flink 1.14.0 for Scala 2.11:\n\n```bash\nwget https://dlcdn.apache.org/flink/flink-1.14.0/flink-1.14.0-bin-scala_2.11.tgz\ntar zxvf flink-1.14.0-bin-scala_2.11.tgz\n```\n\nThen after building the remote shuffle project from source, you can copy the shuffle plugin jar file\nfrom build-target/lib directory (for example, build-target/lib/shuffle-plugin-1.0-SNAPSHOT.jar) to\nthe Flink lib directory and copy the build-in example job jar file to the flink home directory (\nflink-1.14.0):\n\n```bash\ncp flink-remote-shuffle/build-target/lib/shuffle-plugin-1.0-SNAPSHOT.jar flink-1.14.0/lib/\ncp flink-remote-shuffle/build-target/examples/BatchJobDemo.jar flink-1.14.0/\n```\n\nAfter that, you can start a local remote shuffle cluster by running the following command:\n\n```bash\ncd flink-remote-shuffle # switch to the remote shuffle project home directory\ncd build-target # run after building from source\n./bin/start-cluster.sh -D remote-shuffle.storage.local-data-dirs=\"[HDD]/tmp/\" -D remote-shuffle.memory.data-writing-size=256m -D remote-shuffle.memory.data-reading-size=256m\n```\n\nThen you can start a local Flink cluster and config Flink to use the remote shuffle service by\nrunning the following command:\n\n```bash\ncd flink-1.14.0 # switch to the flink home directory\n./bin/start-cluster.sh -D shuffle-service-factory.class=com.alibaba.flink.shuffle.plugin.RemoteShuffleServiceFactory -D remote-shuffle.manager.rpc-address=127.0.0.1\n```\n\nFinally, you can run the demo batch job:\n\n```bash\ncd flink-1.14.0 # switch to the flink home directory\nbin/flink run -c com.alibaba.flink.shuffle.examples.BatchJobDemo ./BatchJobDemo.jar\n```\n\nTo stop the local cluster, you can just run the stop-cluster.sh script in the bin directory:\n\n```bash\ncd flink-1.14.0 # switch to the flink home directory\nbin/stop-cluster.sh\n```\n\n```bash\ncd flink-remote-shuffle # switch to the remote shuffle project home directory\nbin/stop-cluster.sh\n```\n\n## How to Contribute\n\nAny feedback of this project is highly appreciated. You can report a bug by opening an issue on\nGitHub. You can also contribute any new features or improvements. See\nthe [contribution guide](./docs/contribution.md)\nfor more information.\n\n## Support\n\nWe provide free support for users using this project. You can join the Slack channel or scan the\nfollowing QR code to join the [DingTalk](https://www.dingtalk.com/) user support group for further\nhelp and collaboration:\n\nEnglish:\n\nPlease join the Slack channel by clicking this [invitation](https://join.slack.com/t/slack-5xu7894/shared_invite/zt-ykp807ok-1JXMcE6HS~NCplRp2T31fQ).\n\nChinese:\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./docs/imgs/support-zh.jpeg\" width=\"30%\"/\u003e\n\u003c/div\u003e\n\n\n## Acknowledge\n\nThis is a Flink ecosystem project. Apache Flink is an excellent unified stateful data processing\nengine. This project borrows some good designs (e.g. the credit-based backpressure) and building\nblocks (e.g. rpc and high availability) from Flink.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflink-extended%2Fflink-remote-shuffle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflink-extended%2Fflink-remote-shuffle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflink-extended%2Fflink-remote-shuffle/lists"}