{"id":19439454,"url":"https://github.com/openucx/xccl","last_synced_at":"2025-04-24T22:32:38.065Z","repository":{"id":51245470,"uuid":"239794134","full_name":"openucx/xccl","owner":"openucx","description":null,"archived":false,"fork":false,"pushed_at":"2021-05-19T08:39:47.000Z","size":772,"stargazers_count":20,"open_issues_count":6,"forks_count":14,"subscribers_count":13,"default_branch":"master","last_synced_at":"2023-08-04T13:44:42.673Z","etag":null,"topics":["collective"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openucx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-11T15:20:12.000Z","updated_at":"2023-08-04T13:44:42.673Z","dependencies_parsed_at":"2022-09-17T14:11:01.944Z","dependency_job_id":null,"html_url":"https://github.com/openucx/xccl","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openucx%2Fxccl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openucx%2Fxccl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openucx%2Fxccl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openucx%2Fxccl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openucx","download_url":"https://codeload.github.com/openucx/xccl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223969853,"owners_count":17233726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collective"],"created_at":"2024-11-10T15:23:04.346Z","updated_at":"2024-11-10T15:23:05.023Z","avatar_url":"https://github.com/openucx.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# XCCL - Mellanox Collective Communication Library\n\nThe XCCL library framework is a continuous extension of the advanced research and development on extreme-scale collective communications used by Mellanox for HPC, AI/ML application domains. The XCCL library implements \"Teams API\" concepts which is flexible and feature-rich for current and emerging programming models and runtimes. \n\n## XCCL Goals\n\n* Provides collective operations for HPC and AI/ML programming models\n* Enables hierarchial collectives (dynamic and static hiearchies)\n* Enables direct use of hardware collectives by programming model\n* Supports a variety of resource allocation models\n* Supports relaxed ordering model\n* Supports a variery of synchronization models\n* Supports repetitive collective operations (init once and invoke multiple times)\n* Support point-to-point operations in the context of group\n* Supports global memory management\n* Support multiple vendors' open and proprietary plugins\n\n### The library consists of two layers: \n\n1. **XCCL** - *teams collective communication layer*, is the lower layer and implements a subset of the *Teams API* under consideration by the **UCF Collectives WG** for the following:\n   * A UCX Team \n   * A SHARP Team\n   * A shared memory Team (proprietary)\n   * A VNC - Hardware multicast Team (proprietary)\n\n2. **XCCL** - is the upper layer and implements a light-weight, highly scalable framework for expressing hierarchical collectives in terms of the Team abstraction.\n   \n# Quick Start Guide\n\nThe SHARP teams require Mellanox's SHARP software library, and the hardware multicast team requires Mellanox's VMC software library.\n\n   ### Build and install XCCL and XCCL libraries:\n   \u003eHPCX can be downloaded from https://www.mellanox.com/products/hpc-x-toolkit\n\n``` bash\n# Line below is needed for all \"HPCX_*\" variables used in examples below\n% module load /path/to/hpcx/dir/modulefiles/hpcx-stack\n% export XCCL_DIR=$PWD/xccl\n\n% git clone https://github.com/openucx/xccl.git $XCCL_DIR\n% cd $XCCL_DIR\n% ./autogen.sh\n% ./configure --prefix=$PWD/install --with-vmc=$HPCX_VMC_DIR \\ \n  --with-ucx=$HPCX_UCX_DIR --with-sharp=$HPCX_SHARP_DIR\n% make -j install\n```\n\n   ### Build and install Open MPI :\n   \u003e OpenMPI is taken from PR https://github.com/open-mpi/ompi/pull/7409\n   \n``` bash\n% export OMPI_XCCL_DIR=$PWD/ompi-xccl\n% git clone https://github.com/open-mpi/ompi\n% cd $OMPI_XCCL_DIR\n% git fetch origin pull/7409/head\n% git submodule update --init --recursive\n% ./autogen.pl\n% ./configure --prefix=$OMPI_XCCL_DIR/install \\\n  --with-platform=contrib/platform/mellanox/optimized \\\n  --with-xccl=$XCCL_DIR/install\n% make -j install\n```\n \n   ### Run :\n\n\u003eExample shows how to run osu_allreduce benchmark (https://mvapich.cse.ohio-state.edu/benchmarks/) with XCCL support\n``` bash\n% export LD_LIBRARY_PATH=\"$XCCL_DIR/install/lib:$XCCL_DIR/install/lib/xccl\"\n% export LD_LIBRARY_PATH=\"$OMPI_XCCL_DIR/install/lib:$LD_LIBRARY_PATH\"\n% export nnodes=2 ppn=28\n\n% mpirun -np $((nnodes*ppn)) --map-by ppr:$ppn:node --bind-to core ./osu_allreduce -f\n```\n\n# Performance \n\n### One-level Allreduce: SHARP team  \n\u003eHelios Cluster: EDR 16 nodes, 1 process-per-node\n\n**OSU Allreduce**\n| msglen\t| HCOLL (SHARP) | XCCL |\t\n|:--- |:---:|---:| \n| 4 |\t2.81 | 2.21\t|\n| 8 | 2.75 | 2.04 |\n| 16 | 2.74 | 2.21 |\n| 32 | 2.78 | 2.09 |\n| 64 | 2.73 | 2.18 |\n| 128 |\t2.88 | 2.15 |\n| 256 |\t3.57 | 2.59 |\n| 512 |\t3.77 | 2.86 |\n\n\n### Two-level Allreduce: shared memory socket team, shared memory NUMA team \n\u003eSingle node POWER9 168 threads  \n\n**OSU Allreduce**\n| msglen | \thcoll | xccl |\n|:--- |:---:|---:| \n| 4 | \t4.6 | 5.6 |\t\n| 8\t| 4.53 | 5.6\t| \n| 16 | 4.65 |  5.72 |\n| 32 |\t4.66 | 5.86\t|\n| 64 |\t4.84 | 6.47\t|\n| 128 |\t5.47 | 7.26\t|\n| 256 |\t6.13 | 8.51 |\n| 512 |\t7.41 | 11.23 |\n| 1024 | 9.18 | 15.93 |\n| 2048 | 12.5 | 25.18 |\n\n\n### Three-level Broadcast: UCX team, UCX team, Hardware Multicast team (VMC) :\n\u003eHercules test bed: HDR100 110 nodes, 32 processes-per-node\n\n**OSU Bcast**\n| msglen\t| hcoll\t| xccl |\n|:--- |:---:|---:| \n| 1\t  | 5.33 | 4.53 |\n| 2\t  | 4.62 | 4.48 |\n| 4\t  | 4.51 | 4.33 |\n| 8\t  | 4.73 | 4.45 |\n| 16\t| 4.36 | 4.24 |\n| 32\t| 4.44 | 4.80 |\n| 64\t| 4.48 | 5.30 |\n| 128\t| 4.64 | 6.30 |\n| 256\t| 5.49 | 6.69 |\n| 512\t| 5.88 | 7.39 |\n| 1024| 6.45 | 8.46 |\n| 2048 | 7.94 | 9.90 |\n| 4096 | 11.30 | 13.42 |\n| 8192 | 17.09 | 19.49 |\n| 16384 | 30.41\t| 30.95 |\n| 32768\t| 38.37 |\t41.54 |\n\n\n\n\n# Publications\n\nThis framework is a continuous extension of the advanced research and development on extreme-scale collective communications published in the following scientific papers. The shared memory team is code ported from the HCOLL shared memory BCOL component. The XCCL layer is a \"distillate\" of the HCOLL framework. The HCOLL framework began its life as the **_Cheetah_** framework:\n\n1. **_Cheetah: A Framework for Scalable Hierarchical Collective Operations_**  \n Date: May 2011  \n Publication description: IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGRID)\n\n2. **_ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications_**  \n Date: May 2011      \nPublication: First Workshop on Communication Architecture for Scalable Systems (CASS) held in conjunction with the International Parallel and Distributed Processing Symposium (IPDPS)\n\n3. **_Design and Implementation of Broadcast Algorithms for Extreme-Scale Systems_**  \n Date: Sept 2011  \n Publication: IEEE Cluster 2011\n\n4. **_Analyzing the Effect of Multicore Architectures and On-host Communication Characteristics on Collective Communications_**  \n Date: Sept 2011  \n Publication: Workshop on Scheduling and Resource Management for Parallel and Distributed Systems held in conjunction with the International Conference on Parallel Processing (ICPP)\n\n5. **_Assessing the Performance and Scalability of a Novel K-Nomial Allgather on CORE-Direct Systems_**  \n Date: Aug 2012  \n Publication: 18th International Conference, Euro-Par 2012\n\n6. **_Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct_**  \n Date: Sept 2012  \n Publication: 41st International Conference on Parallel Processing, ICPP 2012\n\n7. **_Optimizing Blocking and Nonblocking Reduction Operations for Multicore Systems: Hierarchical Design and Implementation_**  \n Date: Sept 2013  \n Publication: IEEE Cluster 2013\n\n\n\n\n\n\n\n\n\n\n\n \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenucx%2Fxccl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenucx%2Fxccl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenucx%2Fxccl/lists"}