https://github.com/oneapi-src/ishmem
Intel® SHMEM - Device initiated shared memory based communication library
https://github.com/oneapi-src/ishmem
Last synced: 7 months ago
JSON representation
Intel® SHMEM - Device initiated shared memory based communication library
- Host: GitHub
- URL: https://github.com/oneapi-src/ishmem
- Owner: oneapi-src
- License: other
- Created: 2023-10-27T18:29:35.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-06-09T17:11:48.000Z (8 months ago)
- Last Synced: 2025-06-24T13:45:42.879Z (7 months ago)
- Language: C++
- Homepage: https://oneapi-src.github.io/ishmem/index.html
- Size: 4.27 MB
- Stars: 24
- Watchers: 6
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Intel® SHMEM 
[Installation](#installation) | [Usage](#usage) | [Release Notes](RELEASE_NOTES.md) | [Documentation](https://oneapi-src.github.io/ishmem/intro.html) | [How to Contribute](CONTRIBUTING.md) | [License](LICENSE)
Intel® SHMEM provides an efficient implementation of GPU-initiated communication on systems with Intel GPUs.
## Table of Contents
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Usage](#usage)
- [Launching Example Application](#launching-example-application)
- [Additional Resources](#additional-resources)
- [OpenSHMEM Specification](#openshmem-spec)
- [Specification](#ishmem-spec)
## Prerequisites
- Linux OS
- Intel® oneAPI DPC++/C++ Compiler 2024.0 or higher.
### SYCL support
Intel® oneAPI DPC++/C++ Compiler with Level Zero support.
## Installation
### Building Level Zero
For detailed information on Level Zero, refer to the [Intel® Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver repository](https://github.com/intel/compute-runtime/releases) or to the [installation guide](https://dgpu-docs.intel.com/installation-guides/index.html) for oneAPI users.
To install, download the oneAPI Level Zero from the repository.
```
git clone https://github.com/oneapi-src/level-zero.git
```
Build Level Zero following instructions below.
```
cd level-zero
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX= ..
make -j
make install
```
### The Host Back-End Library
Intel® SHMEM requires a host OpenSHMEM or MPI back-end to be used for host-sided operations support. In particular, the OpenSHMEM back-end relies on a collection of extension APIs (`shmemx_heap_create`, `shmemx_heap_preinit`, and `shmemx_heap_postinit`) to coordinate the Intel® SHMEM and OpenSHMEM heaps. We recommend [Sandia OpenSHMEM v1.5.3rc1](https://github.com/Sandia-OpenSHMEM/SOS/releases/tag/v1.5.3rc1) or newer for this purpose. A [work-in-progress branch](https://github.com/davidozog/oshmpi/tree/wip/ishmem) of [OSHMPI](https://github.com/pmodels/oshmpi.git) is also supported but is currently considered experimental. See the [Building OSHMPI](#building-oshmpi-optional-and-experimental) section before for more details.
We recommend the Intel® MPI Library as the MPI back-end option for the current version of Intel® SHMEM. See the [Building Intel® SHMEM](#building-intel-shmem) section below for more details.
### Building Sandia OpenSHMEM (SOS)
Download the SOS repo to be configured as a back-end for Intel® SHMEM.
```
git clone --recurse-submodules https://github.com/Sandia-OpenSHMEM/SOS.git SOS
```
Build SOS following instructions below. `FI_HMEM` support in the provider is required for use with Intel® SHMEM. To enable `FI_HMEM` with a supported provider, we recommend a specific set of config flags. Below are two examples for configuring and building SOS with two providers supporting `FI_HMEM`. To configure SOS with the `verbs;ofi_rxm` provider, use the following instructions:
```
cd SOS
./autogen.sh
CC=icx CXX=icpx ./configure --prefix= --with-ofi= --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-hard-polling
make -j
make install
```
To configure SOS with the HPE Slingshot provider `cxi`, please use the following instructions:
```
cd SOS
./autogen.sh
CC=icx CXX=icpx ./configure --prefix= --with-ofi= --enable-pmi-simple --enable-ofi-mr=basic --disable-ofi-inject --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-manual-progress --enable-mr-endpoint --disable-nonfetch-amo --enable-manual-progress
make -j
make install
```
To configure SOS with the `psm3` provider, please use the following instructions:
```
cd SOS
./autogen.sh
CC=icx CXX=icpx ./configure --prefix= --with-ofi= --enable-pmi-simple --enable-manual-progress --enable-ofi-hmem --disable-bounce-buffers --enable-ofi-mr=basic --enable-mr-endpoint
make -j
make install
```
Please choose an appropriate PMI configure flag based on the available PMI client library in the system. Please check for further instructions on [SOS Wiki pages](https://github.com/Sandia-OpenSHMEM/SOS/wiki). Optionally, users may also choose to add `--disable-fortran` since fortran interfaces will not be used.
### Building OSHMPI (Optional and experimental)
Intel® SHMEM has experimental support for OSHMPI when built using the Intel® MPI Library.
Here is information on how to [Get Started with Intel® MPI Library on Linux](https://www.intel.com/content/www/us/en/docs/mpi-library/get-started-guide-linux/2021-11/overview.html).
To download the OSHMPI repository:
```
git clone -b wip/ishmem --recurse-submodules https://github.com/davidozog/oshmpi.git oshmpi
```
After ensuring Intel® MPI Library is enabled (for example, by sourcing the `/opt/intel/oneapi/setvars.sh` script),
please build OSHMPI following the instructions below.
```
cd oshmpi
./autogen.sh
CC=mpiicx CXX=mpiicpx ./configure --prefix= --disable-fortran --enable-rma=direct --enable-amo=direct --enable-async-thread=yes
make -j
make install
```
### Building Intel® SHMEM
Check that the SOS build process has successfully created a `` directory with `include` and `lib` as subdirectories. Please find `shmem.h` and `shmemx.h` in `include`.
Build Intel® SHMEM with an OpenSHMEM back-end using the following instructions:
```
cd ishmem
mkdir build
cd build
CC=icx CXX=icpx cmake .. -DENABLE_OPENSHMEM=ON -DSHMEM_DIR= -DCMAKE_INSTALL_PREFIX=
make -j
```
Alternatively, Intel® SHMEM can be built by enabling an Intel® MPI Library back-end.
Here is information on how to [Get Started with Intel® MPI Library on Linux](https://www.intel.com/content/www/us/en/docs/mpi-library/get-started-guide-linux/2021-11/overview.html).
```
CC=icx CXX=icpx cmake .. -DENABLE_OPENSHMEM=OFF -DENABLE_MPI=ON -DMPI_DIR= -DCMAKE_INSTALL_PREFIX=
```
where `` is the path to the Intel® MPI Library installation.
Enabling both the OpenSHMEM and MPI back-ends is also supported. In this case,
the desired backend can be selected via the environment variable,
`ISHMEM_RUNTIME`, which can be set to either "OpenSHMEM" or "MPI".
The default value for `ISHMEM_RUNTIME` is "OpenSHMEM".
## Usage
### Launching Example Application
Validate that Intel® SHMEM was built correctly by running an example program.
1. Add the path for the back-end library to the environment, for example:
```
export LD_LIBRARY_PATH=/lib:$LD_LIBRARY_PATH
```
When enabling only the Intel® MPI Library back-end, simply source the appropriate
`setvars.sh` script. When enabling both OpenSHMEM and MPI back-ends, first
source the `setvars.sh` script, then configure the dynamic linker to load the
OpenSHMEM library (for example by prepending `/lib` to
`LD_LIBRARY_PATH`).
2. Run the example program or test on an allocated node using a process launcher:
```
ISHMEM_RUNTIME= mpiexec.hydra -n 2 -hosts ./scripts/ishmrun ./test/unit/int_get_device
```
where `` is the selected host back-end library.
- *Note:* Current supported launchers include: MPI process launchers (i.e. `mpiexec`, `mpiexec.hydra`, `mpirun`, etc.), Slurm (i.e. `srun`, `salloc`, etc.), and PBS (i.e. `qsub`).
- *Note:* Intel® SHMEM execution model requires applications to use a 1:1 mapping between PEs and GPU devices. Attempting to run an application without the ishmrun launch script may result in undefined behavior if this mapping is not maintained.
- For further details on the device selection, please see [the ONEAPI_DEVICE_SELECTOR](https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md#oneapi_device_selector).
3. Validate the application ran successfully; example output:
```
Selected device: Intel(R) Data Center GPU Max 1550
Selected vendor: Intel(R) Corporation
Selected device: Intel(R) Data Center GPU Max 1550
Selected vendor: Intel(R) Corporation
No errors
No errors
```
### Launching Example Application w/ CTest
`ctest` can be used to run Intel® SHMEM tests that are generated at compile-time. To see a list of tests available via `ctest`, run:
```
ctest -N
```
To launch a single test, execute:
```
ctest -R
```
Alternatively, all the tests in a directory (such as `test/unit/`) can be run with the following command:
```
ctest --test-dir
```
By default, a passed or failed test can be detected by the output:
```
Start 69: sync-2-gpu
1/1 Test #69: sync-2-gpu ....................... Passed 2.29 sec
100% tests passed, 0 tests failed out of 1
```
To have a test's output printed to the console, add either the `--verbose` or `--output-on-failure` flag to the `ctest` command
### Available Scheduler Wrappers for Jobs Run via CTest
The following values may be assigned to `CTEST_LAUNCHER` at configure-time (ex. `-DCTEST_LAUNCHER=mpi`) to set which scheduler will be used to run tests launched through a call to `ctest`:
- srun (default)
- Launches CTest jobs on a single node using Slurm's `srun`.
- mpi
- Uses `mpirun` to launch CTest jobs with the appropriate number of processes.
- qsub
- Launches CTest jobs on a single node using `qsub`. If this option is being used on a system where a reservation must be made (i.e. via `pbsresnode`) prior to running a test, assign the `JOB_QUEUE` environment variable to the queue associated with your reservation:
```
export JOB_QUEUE=
```
## Additional Resources
### OpenSHMEM Specification
- [OpenSHMEM](http://openshmem.org/site/)
- [Specification](http://openshmem.org/site/sites/default/site_files/OpenSHMEM-1.5.pdf)
### Intel® SHMEM Specification
- [Intel® SHMEM Specification](https://oneapi-src.github.io/ishmem/intro.html)