{"id":13569734,"url":"https://github.com/apache/incubator-uniffle","last_synced_at":"2025-03-10T02:48:13.667Z","repository":{"id":41379036,"uuid":"504360264","full_name":"apache/incubator-uniffle","owner":"apache","description":"Uniffle is a high performance, general purpose Remote Shuffle Service.","archived":false,"fork":false,"pushed_at":"2025-03-06T02:25:19.000Z","size":13075,"stargazers_count":402,"open_issues_count":289,"forks_count":154,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-08T06:48:55.362Z","etag":null,"topics":["mapreduce","remote-shuffle-service","rss","shuffle","spark","tez"],"latest_commit_sha":null,"homepage":"https://uniffle.apache.org/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-17T01:47:53.000Z","updated_at":"2025-03-06T02:25:23.000Z","dependencies_parsed_at":"2023-10-12T17:22:37.098Z","dependency_job_id":"917ef029-955f-49c7-bc30-78785d175c97","html_url":"https://github.com/apache/incubator-uniffle","commit_stats":{"total_commits":989,"total_committers":83,"mean_commits":11.91566265060241,"dds":0.8574317492416582,"last_synced_commit":"41d78aafac933e7c7ffa5dddd440b7465a3bb63e"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-uniffle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-uniffle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-uniffle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-uniffle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/incubator-uniffle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242778801,"owners_count":20183832,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mapreduce","remote-shuffle-service","rss","shuffle","spark","tez"],"created_at":"2024-08-01T14:00:43.465Z","updated_at":"2025-03-10T02:48:13.649Z","avatar_url":"https://github.com/apache.png","language":"Java","funding_links":[],"categories":["Java","大数据"],"sub_categories":[],"readme":"\u003c!--\n  ~ Licensed to the Apache Software Foundation (ASF) under one or more\n  ~ contributor license agreements.  See the NOTICE file distributed with\n  ~ this work for additional information regarding copyright ownership.\n  ~ The ASF licenses this file to You under the Apache License, Version 2.0\n  ~ (the \"License\"); you may not use this file except in compliance with\n  ~ the License.  You may obtain a copy of the License at\n  ~\n  ~    http://www.apache.org/licenses/LICENSE-2.0\n  ~\n  ~ Unless required by applicable law or agreed to in writing, software\n  ~ distributed under the License is distributed on an \"AS IS\" BASIS,\n  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n  ~ See the License for the specific language governing permissions and\n  ~ limitations under the License.\n  --\u003e\n\n# Apache Uniffle (Incubating)\n\nUniffle is a high performance, general purpose remote shuffle service for distributed computing engines.\nIt provides the ability to push shuffle data into centralized storage service,\nchanging the shuffle style from \"local file pull-like style\" to \"remote block push-like style\".\nIt brings in several advantages like supporting disaggregated storage deployment,\nsuper large shuffle jobs, and high elasticity.\nCurrently it supports [Apache Spark][1], [Apache Hadoop MapReduce][2] and [Apache Tez][3].\n\n[1]: https://spark.apache.org\n[2]: https://hadoop.apache.org\n[3]: https://tez.apache.org\n\n[![Build](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml/badge.svg?branch=master\u0026event=push)](https://github.com/apache/incubator-uniffle/actions/workflows/build.yml)\n[![Codecov](https://codecov.io/gh/apache/incubator-uniffle/branch/master/graph/badge.svg)](https://codecov.io/gh/apache/incubator-uniffle)\n[![License](https://img.shields.io/github/license/apache/incubator-uniffle)](https://github.com/apache/incubator-uniffle/blob/master/LICENSE)\n[![Release](https://img.shields.io/github/v/release/apache/incubator-uniffle)](https://github.com/apache/incubator-uniffle/releases)\n[![Slack](https://img.shields.io/badge/chat-on%20Slack-brightgreen.svg)](https://join.slack.com/t/the-asf/shared_invite/zt-1fm9561yr-uzTpjqg3jf5nxSJV5AE3KQ)\n\n## Architecture\n![Rss Architecture](docs/asset/rss_architecture.png)\nUniffle cluster consists of three components, a coordinator cluster, a shuffle server cluster and an optional remote storage (e.g., HDFS).\n\nCoordinator will collect the status of shuffle servers and assign jobs based on some strategy.\n\nShuffle server will receive the shuffle data, merge them and write to storage.\n\nDepending on different situations, Uniffle supports Memory \u0026 Local, Memory \u0026 Remote Storage(e.g., HDFS), Memory \u0026 Local \u0026 Remote Storage(recommendation for production environment).\n\n## Shuffle Process with Uniffle\n\n* Spark driver ask coordinator to get shuffle server for shuffle process\n* Spark task write shuffle data to shuffle server with following step:\n![Rss Shuffle_Write](docs/asset/rss_shuffle_write.png)\n 1. Send KV data to buffer\n 2. Flush buffer to queue when buffer is full or buffer manager is full\n 3. Thread pool get data from queue\n 4. Request memory from shuffle server first and send the shuffle data\n 5. Shuffle server cache data in memory first and flush to queue when buffer manager is full\n 6. Thread pool get data from queue\n 7. Write data to storage with index file and data file\n 8. After write data, task report all blockId to shuffle server, this step is used for data validation later\n 9. Store taskAttemptId in MapStatus to support Spark speculation\n\n* Depending on different storage types, the spark task will read shuffle data from shuffle server or remote storage or both of them.\n\n## Shuffle file format\nThe shuffle data is stored with index file and data file. Data file has all blocks for a specific partition and the index file has metadata for every block.\n\n![Rss Shuffle_Write](docs/asset/rss_data_format.png)\n\n## Supported Spark Version\nCurrently supports Spark 2.3.x, Spark 2.4.x, Spark 3.0.x, Spark 3.1.x, Spark 3.2.x, Spark 3.3.x, Spark 3.4.x, Spark 3.5.x\n\nNote: To support dynamic allocation, the patch(which is included in patch/spark folder) should be applied to Spark\n\n## Supported MapReduce Version\nCurrently supports the MapReduce framework of Hadoop 2.8.5, Hadoop 3.2.1\n\n## Building Uniffle\n\u003e note: currently Uniffle requires JDK 1.8 to build, adding later JDK support is on our roadmap.\n\nUniffle is built using [Apache Maven](https://maven.apache.org/).\nTo build it, run:\n\n    ./mvnw -DskipTests clean package\n\nTo fix code style issues, run:\n\n    ./mvnw spotless:apply -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8 -Pdashboard\n\nBuild against profile Spark 2 (2.4.6)\n\n    ./mvnw -DskipTests clean package -Pspark2\n\nBuild against profile Spark 3 (3.1.2)\n\n    ./mvnw -DskipTests clean package -Pspark3\n\nBuild against Spark 3.2.x, Except 3.2.0\n\n    ./mvnw -DskipTests clean package -Pspark3.2\n\nBuild against Spark 3.2.0\n\n    ./mvnw -DskipTests clean package -Pspark3.2.0\n\nBuild against Hadoop MapReduce 2.8.5\n\n    ./mvnw -DskipTests clean package -Pmr,hadoop2.8\n\nBuild against Hadoop MapReduce 3.2.1\n\n    ./mvnw -DskipTests clean package -Pmr,hadoop3.2\n\nBuild against Tez 0.9.1\n\n    ./mvnw -DskipTests clean package -Ptez\n\nBuild against Tez 0.9.1 and Hadoop 3.2.1\n\n    ./mvnw -DskipTests clean package -Ptez,hadoop3.2\n\nBuild with dashboard\n\n    ./mvnw -DskipTests clean package -Pdashboard\n\n\u003e note: currently Uniffle build the project against Java 8. If you want to compile it against other Java versions, you can build the code with `-Dmaven.compiler.release=${release-version}`.\n\nTo package the Uniffle, run:\n\n    ./build_distribution.sh\n\nPackage against Spark 3.2.x, Except 3.2.0, run:\n\n    ./build_distribution.sh --spark3-profile 'spark3.2'\n\nPackage against Spark 3.2.0, run:\n\n    ./build_distribution.sh --spark3-profile 'spark3.2.0'\n\nPackage will build against Hadoop 2.8.5 in default. If you want to build package against Hadoop 3.2.1, run:\n\n    ./build_distribution.sh --hadoop-profile 'hadoop3.2'\n\nPackage with hadoop jars, If you want to build package against Hadoop 3.2.1, run:\n\n    ./build_distribution.sh --hadoop-profile 'hadoop3.2' -Phadoop-dependencies-included\n\nrss-xxx.tgz will be generated for deployment\n\n## Deploy\n\nIf you have packaged tgz with hadoop jars, the env of `HADOOP_HOME` is needn't specified in `rss-env.sh`.\n\n### Deploy Coordinator\n\n1. unzip package to RSS_HOME\n2. update RSS_HOME/conf/rss-env.sh, e.g.,\n   ```\n     JAVA_HOME=\u003cjava_home\u003e\n     HADOOP_HOME=\u003chadoop home\u003e\n     COORDINATOR_XMX_SIZE=\"16g\"\n     # You can set coordinator memory size by `XMX_SIZE` too, but it affects all components.\n     # XMX_SIZE=\"16g\"\n   ```\n3. update RSS_HOME/conf/coordinator.conf, e.g.,\n   ```\n     rss.rpc.server.port 19999\n     rss.jetty.http.port 19998\n     rss.coordinator.server.heartbeat.timeout 30000\n     rss.coordinator.app.expired 60000\n     rss.coordinator.shuffle.nodes.max 5\n     # enable dynamicClientConf, and coordinator will be responsible for most of client conf\n     rss.coordinator.dynamicClientConf.enabled true\n     # config the path of client conf\n     rss.coordinator.dynamicClientConf.path \u003cRSS_HOME\u003e/conf/dynamic_client.conf\n     # config the path of excluded shuffle server\n     rss.coordinator.exclude.nodes.file.path \u003cRSS_HOME\u003e/conf/exclude_nodes\n   ```\n4. update \u003cRSS_HOME\u003e/conf/dynamic_client.conf, rss client will get default conf from coordinator e.g.,\n   ```\n    # MEMORY_LOCALFILE_HDFS is recommended for production environment\n    rss.storage.type MEMORY_LOCALFILE_HDFS\n    # multiple remote storages are supported, and client will get assignment from coordinator\n    rss.coordinator.remote.storage.path hdfs://cluster1/path,hdfs://cluster2/path\n    rss.writer.require.memory.retryMax 1200\n    rss.client.retry.max 50\n    rss.client.send.check.timeout.ms 600000\n    rss.client.read.buffer.size 14m\n   ```\n5. start Coordinator\n   ```\n    bash RSS_HOME/bin/start-coordnator.sh\n   ```\n\n### Deploy Shuffle Server\nWe recommend to use JDK 11+ if we want to have better performance when we deploy the shuffle server.\nSome benchmark tests among different JDK is as below:\n(using spark to write shuffle data with 20 executors. Single executor will total write 1G, and each time write 14M.\nShuffle Server use GRPC to transfer data)\n\n| Java version | ShuffleServer GC  | Max pause time | ThroughOutput |\n| ------------- | ------------- | ------------- | ------------- |\n| 8  | G1  | 30s | 0.3 |\n| 11  | G1  | 2.5s | 0.8 |\n| 18  | G1  | 2.5s | 0.8 |\n| 18  | ZGC  | 0.2ms | 0.99997 |\n\nDeploy Steps:\n1. unzip package to RSS_HOME\n2. update RSS_HOME/conf/rss-env.sh, e.g.,\n   ```\n     JAVA_HOME=\u003cjava_home\u003e\n     HADOOP_HOME=\u003chadoop home\u003e\n     SHUFFLE_SERVER_XMX_SIZE=\"80g\"\n     # You can set shuffle server memory size by `XMX_SIZE` too, but it affects all components.\n     # XMX_SIZE=\"80g\"\n   ```\n3. update RSS_HOME/conf/server.conf, e.g.,\n   ```\n     rss.rpc.server.port 19999\n     rss.jetty.http.port 19998\n     rss.rpc.executor.size 2000\n     # it should be configured the same as in coordinator\n     rss.storage.type MEMORY_LOCALFILE_HDFS\n     rss.coordinator.quorum \u003ccoordinatorIp1\u003e:19999,\u003ccoordinatorIp2\u003e:19999\n     # local storage path for shuffle server\n     rss.storage.basePath /data1/rssdata,/data2/rssdata....\n     # it's better to config thread num according to local disk num\n     rss.server.flush.thread.alive 5\n     rss.server.flush.localfile.threadPool.size 10\n     rss.server.flush.hadoop.threadPool.size 60\n     rss.server.buffer.capacity 40g\n     rss.server.read.buffer.capacity 20g\n     rss.server.heartbeat.interval 10000\n     rss.rpc.message.max.size 1073741824\n     rss.server.preAllocation.expired 120000\n     rss.server.commit.timeout 600000\n     rss.server.app.expired.withoutHeartbeat 120000\n     # note: the default value of rss.server.flush.cold.storage.threshold.size is 64m\n     # there will be no data written to DFS if set it as 100g even rss.storage.type=MEMORY_LOCALFILE_HDFS\n     # please set a proper value if DFS is used, e.g., 64m, 128m.\n     rss.server.flush.cold.storage.threshold.size 100g\n   ```\n4. start Shuffle Server\n   ```\n    bash RSS_HOME/bin/start-shuffle-server.sh\n   ```\n\n### Deploy Spark Client\n1. Add client jar to Spark classpath, e.g., SPARK_HOME/jars/\n\n   The jar for Spark2 is located in \u003cRSS_HOME\u003e/jars/client/spark2/rss-client-spark2-shaded-${version}.jar\n\n   The jar for Spark3 is located in \u003cRSS_HOME\u003e/jars/client/spark3/rss-client-spark3-shaded-${version}.jar\n\n2. Update Spark conf to enable Uniffle, e.g.,\n\n   ```\n   # Uniffle transmits serialized shuffle data over network, therefore a serializer that supports relocation of\n   # serialized object should be used. \n   spark.serializer org.apache.spark.serializer.KryoSerializer # this could also be in the spark-defaults.conf\n   spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager\n   spark.rss.coordinator.quorum \u003ccoordinatorIp1\u003e:19999,\u003ccoordinatorIp2\u003e:19999\n   # Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE.\n   ```\n\n### Support Spark dynamic allocation\n\nTo support spark dynamic allocation with Uniffle, spark code should be updated.\nThere are 7 patches for spark (2.3.4/2.4.6/3.0.1/3.1.2/3.2.1/3.3.1/3.4.1) in patch/spark folder for reference.\n\nAfter apply the patch and rebuild spark, add following configuration in spark conf to enable dynamic allocation:\n  ```\n  spark.shuffle.service.enabled false\n  spark.dynamicAllocation.enabled true\n  ```\nFor spark3.5 or above just add one more configuration:\n  ```\n  spark.shuffle.sort.io.plugin.class org.apache.spark.shuffle.RssShuffleDataIo\n  ```\n\n### Deploy MapReduce Client\n\n1. Add client jar to the classpath of each NodeManager, e.g., \u003cHADOOP\u003e/share/hadoop/mapreduce/\n\nThe jar for MapReduce is located in \u003cRSS_HOME\u003e/jars/client/mr/rss-client-mr-XXXXX-shaded.jar\n\n2. Update MapReduce conf to enable Uniffle, e.g.,\n\n   ```\n   -Dmapreduce.rss.coordinator.quorum=\u003ccoordinatorIp1\u003e:19999,\u003ccoordinatorIp2\u003e:19999\n   -Dyarn.app.mapreduce.am.command-opts=org.apache.hadoop.mapreduce.v2.app.RssMRAppMaster\n   -Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.RssMapOutputCollector\n   -Dmapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.RssShuffle\n   ```\nNote that the RssMRAppMaster will automatically disable slow start (i.e., `mapreduce.job.reduce.slowstart.completedmaps=1`)\nand job recovery (i.e., `yarn.app.mapreduce.am.job.recovery.enable=false`)\n\n### Deploy Tez Client\n\n1. Append client jar to pacakge which is set by 'tez.lib.uris'.\n\nIn production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'. \n\nIn development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.\n\n2. Update tez-site.xml to enable Uniffle.\n\n| Property Name              |Default| Description                  |\n|----------------------------|---|------------------------------|\n| tez.am.launch.cmd-opts     |-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster| enable remote shuffle service |\n| tez.rss.coordinator.quorum |coordinatorIp1:19999,coordinatorIp2:19999|coordinator address|\n\nNote that the RssDAGAppMaster will automatically disable slow start (i.e., `tez.shuffle-vertex-manager.min-src-fraction=1`, `tez.shuffle-vertex-manager.max-src-fraction=1`).\n\n### Deploy In Kubernetes\n\nWe have provided an operator for deploying uniffle in kubernetes environments.\n\nFor details, see the following document:\n\n[operator docs](docs/operator)\n\n## Configuration\n\nThe important configuration is listed as follows.\n\n|Role|Link|\n|---|---|\n|coordinator|[Uniffle Coordinator Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/coordinator_guide.md)|\n|shuffle server|[Uniffle Shuffle Server Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/server_guide.md)|\n|client|[Uniffle Shuffle Client Guide](https://github.com/apache/incubator-uniffle/blob/master/docs/client_guide/client_guide.md)|\n\n## Security: Hadoop kerberos authentication\nThe primary goals of the Uniffle Kerberos security are:\n1. to enable secure data access for coordinator/shuffle-servers, like dynamic conf/exclude-node files stored in secured dfs cluster\n2. to write shuffle data to kerberos secured dfs cluster for shuffle-servers.\n\nThe following security configurations are introduced.\n\n|Property Name|Default|Description|\n|---|---|---|\n|rss.security.hadoop.kerberos.enable|false|Whether enable access secured hadoop cluster|\n|rss.security.hadoop.kerberos.krb5-conf.file|-|The file path of krb5.conf. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|\n|rss.security.hadoop.kerberos.keytab.file|-|The kerberos keytab file path. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|\n|rss.security.hadoop.kerberos.principal|-|The kerberos keytab principal. And only when rss.security.hadoop.kerberos.enable is enabled, the option will be valid|\n|rss.security.hadoop.kerberos.relogin.interval.sec|60|The kerberos authentication relogin interval. unit: sec|\n|rss.security.hadoop.kerberos.proxy.user.enable|true|Whether using proxy user for job user to access secured Hadoop cluster.|\n\n* The proxy user mechanism is used to keep the data isolation in uniffle, which means the shuffle-data written by \n  shuffle-servers is owned by spark app's user. To achieve the this, the login user specified by above config should \n  be as the superuser for HDFS. For more details of related sections, \n  please see [Proxy user - Superusers Acting On Behalf Of Other Users](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html)\n\n## Benchmark\nWe provide some benchmark tests for Uniffle. For details, you can see [Uniffle 0.2.0 Benchmark](docs/benchmark.md), [Uniffle 0.9.0 Benchmark](docs/benchmark_netty_case_report.md).\n\n## LICENSE\n\nUniffle is under the Apache License Version 2.0. See the [LICENSE](https://github.com/apache/incubator-uniffle/blob/master/LICENSE) file for details.\n\n## Contributing\nFor more information about contributing issues or pull requests, see [Uniffle Contributing Guide](https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-uniffle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fincubator-uniffle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-uniffle/lists"}