{"id":18711140,"url":"https://github.com/ROCm/hipMM","last_synced_at":"2025-04-12T12:30:39.430Z","repository":{"id":246454510,"uuid":"787528687","full_name":"ROCm/hipMM","owner":"ROCm","description":"HIP Memory Manager (ROCm-DS)","archived":false,"fork":false,"pushed_at":"2025-02-24T09:29:33.000Z","size":499,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"branch-23.12","last_synced_at":"2025-03-26T06:22:11.763Z","etag":null,"topics":["amd","cuda","gpu","hip","memory-management","radeon-instinct-mi-series","rocm"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ROCm.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-16T17:40:03.000Z","updated_at":"2024-12-15T20:20:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"96d07f86-9162-485f-ae08-d8d3e44f6874","html_url":"https://github.com/ROCm/hipMM","commit_stats":null,"previous_names":["rocm/hip-rmm","rocm/rmm-rocm","rocm/rmm","rocm/hipmm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2FhipMM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2FhipMM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2FhipMM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ROCm%2FhipMM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ROCm","download_url":"https://codeload.github.com/ROCm/hipMM/tar.gz/refs/heads/branch-23.12","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248560516,"owners_count":21124667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amd","cuda","gpu","hip","memory-management","radeon-instinct-mi-series","rocm"],"created_at":"2024-11-07T12:37:14.723Z","updated_at":"2025-04-12T12:30:39.421Z","avatar_url":"https://github.com/ROCm.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- MIT License\n  --\n  -- Modifications Copyright (c) 2024 Advanced Micro Devices, Inc.\n  --\n  -- Permission is hereby granted, free of charge, to any person obtaining a copy\n  -- of this software and associated documentation files (the \"Software\"), to deal\n  -- in the Software without restriction, including without limitation the rights\n  -- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n  -- copies of the Software, and to permit persons to whom the Software is\n  -- furnished to do so, subject to the following conditions:\n  --\n  -- The above copyright notice and this permission notice shall be included in all\n  -- copies or substantial portions of the Software.\n  --\n  -- THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n  -- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n  -- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n  -- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n  -- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n  -- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n  -- SOFTWARE.\n--\u003e\n\n# HIP Memory Manager for AMD GPUs (hipMM)\n\n\u003e [!CAUTION]\n\u003e This release is an *early-access* software technology preview. Running production workloads is *not* recommended.\n\n\u003e [!NOTE]\n\u003e This repository will be eventually moved to the [ROCm-DS](https://github.com/rocm-ds) Github organization.\n\n\u003e [!NOTE]\n\u003e This ROCm\u0026trade; port is derived from the NVIDIA RAPIDS\u0026reg; RMM project. It aims to\nfollow the latter's directory structure, file naming and API naming as closely as possible to minimize porting friction for users that are interested in using both projects.\n\n## RMM Resources\n\n- [RMM Reference Documentation](https://docs.rapids.ai/api/rmm/stable/): Python API reference, tutorials, and topic guides.\n- [librmm Reference Documentation](https://docs.rapids.ai/api/librmm/stable/): C/C++ library API reference.\n- [Getting Started](https://rapids.ai/start.html): Instructions for installing RMM.\n- [RAPIDS Community](https://rapids.ai/community.html): Get help, contribute, and collaborate.\n- [GitHub repository](https://github.com/rapidsai/rmm): Download the RMM source code.\n- [Issue tracker](https://github.com/rapidsai/rmm/issues): Report issues or request features.\n\n## Overview\n\nAchieving optimal performance in GPU-centric workflows frequently requires customizing how host and\ndevice memory are allocated. For example, using \"pinned\" host memory for asynchronous\nhost \u003c-\u003e device memory transfers, or using a device memory pool sub-allocator to reduce the cost of\ndynamic device memory allocation.\n\nThe goal of the HIP Memory Manager (hipMM) is to provide:\n- A common interface that allows customizing [device](#device_memory_resource) and\n  [host](#host_memory_resource) memory allocation\n- A collection of [implementations](#available-resources) of the interface\n- A collection of [data structures](#device-data-structures) that use the interface for memory allocation\n\nFor information on the interface hipMM provides and how to use hipMM in your C++ code, see\n[below](#using-rmm-in-c).\n\nFor a walkthrough about the design of the HIP Memory Manager, read [Fast, Flexible Allocation for NVIDIA with RAPIDS Memory Manager](https://developer.nvidia.com/blog/fast-flexible-allocation-for-cuda-with-rapids-memory-manager/) on the NVIDIA Developer Blog.\n\n## Installation\n\n\u003e [!NOTE]\n\u003e We support only AMD GPUs. Use the NVIDIA RAPIDS package for NVIDIA GPUs.\n\n\u003e [!NOTE]\n\u003e Currently, it is not possible to install hipMM via `conda`.\n\n\u003c!-- ### Conda\n\nhipMM can be installed with Conda ([miniconda](https://conda.io/miniconda.html), or the full\n[Anaconda distribution](https://www.anaconda.com/download)) from the `rapidsai` channel:\n\n```bash\n# NOTE: Conda installation not supported for hipMM for AMD GPUs.\n# conda install -c rapidsai -c conda-forge -c nvidia rmm cuda-version=11.8\n```\n\nWe also provide [nightly Conda packages](https://anaconda.org/rapidsai-nightly) built from the HEAD\nof our latest development branch.\n\nNote: hipMM is supported only on Linux, and only tested with Python versions 3.9 and 3.10.\n\n\nNote: The hipMM package from Conda requires building with GCC 9 or later. Otherwise, your application may fail to build.\n\nSee the [Get RAPIDS version picker](https://rapids.ai/start.html) for more OS and version info. --\u003e\n\n## Building from Source\n\n### Get hipMM Dependencies\n\nCompiler requirements:\n\n* `gcc`                    version 9.3+\n* ROCm HIP SDK compilers   version 5.6.0+, recommended is 6.3.0+\n* `cmake`                  version 3.26.4+\n\nGPU requirements:\n\n* ROCm HIP SDK 5.6.0+, recommended is 6.3.0+\n\nPython requirements:\n* `scikit-build`\n* `hip-python`\n* `hip-python-as-cuda`\n* `cython`\n\nFor more details, see [pyproject.toml](python/pyproject.toml)\n\n\n### Script to build hipMM from source\n\nTo install hipMM from source, ensure the dependencies are met and follow the steps below:\n\n- Clone the repository and submodules\n```bash\n$ git clone --recurse-submodules https://github.com/ROCM/hipMM.git\n$ cd rmm\n```\n\n- Create the conda development environment `rmm_dev`\n```bash\n# create the conda environment (assuming in base `rmm` directory)\n$ conda env create --name rmm_dev --file conda/environments/all_rocm_arch-x86_64.yaml\n# activate the environment\n$ conda activate rmm_dev\n```\n\n- Install ROCm dependencies that are not yet distributed via a conda channel. We install HIP Python and the optional Numba HIP dependency via the Github-distributed `numba-hip` package. We select dependencies of Numba HIP that agree with our ROCm installation by providing a parameter `rocm-${ROCM_MAJOR}-${ROCM-MINOR}-${ROCM-PATCH}`\n(example: `rocm-6-1-2`) in square brackets:\n\n\u003e [!IMPORTANT]\n\u003e Some hipMM dependencies are currently distributed via: https://test.pypi.org/simple\n\u003e We need to specify 'https://test.pypi.org/simple' as additional global extra index URL.\n\u003e To append the URL and not overwrite what else is specified already, we combine `pip\n\u003e config set` and `pip config get` as shown below. We further restore the original URLs.\n\u003e (Note that specifying the `--extra-index-url` command line option does not have\n\u003e the same effect.)\n\n```bash\n(rmm_dev) $ pip install --upgrade pip\n(rmm_dev) $ previous_urls=$(pip config get global.extra-index-url)\n(rmm_dev) $ pip config set global.extra-index-url \"${previous_urls} https://test.pypi.org/simple\"\n(rmm_dev) $ pip install numba-hip[rocm-${ROCM_MAJOR}-${ROCM-MINOR}-${ROCM-PATCH}]@git+https://github.com/rocm/numba-hip.git\n# example: pip install numba-hip[rocm-6-1-2]@git+https://github.com/rocm/numba-hip.git\n(rmm_dev) $ pip config set global.extra-index-url \"${previous_urls}\" # restore urls\n```\n\n\u003e [!IMPORTANT]\n\u003e When compiling for AMD GPUs, we always need to set the environment variable `CXX` before building so that the Cython build process uses a HIP C++ compiler.\n\u003e\n\u003e Example:\n\u003e\n\u003e `(rmm_dev) $ export CXX=hipcc`\n\u003e\n\u003e We further need to provide the location of the ROCm CMake scripts to CMake via the `CMAKE_PREFIX_PATH` CMake or environment variable. We append via the `:` char to not modify configurations performeed by the active Conda environment.\n\u003e\n\u003e Example:\n\u003e\n\u003e `(rmm_dev) $ export CMAKE_PREFIX_PATH=\"${CMAKE_PREFIX_PATH}:/opt/rocm/lib/cmake\"`\n\u003e\n\n- Build and install `librmm` using cmake \u0026 make.\n\n```bash\n\n(rmm_dev) $ export CXX=\"hipcc\"                                # Cython CXX compiler, adjust according to your setup.\n(rmm_dev) $ export CMAKE_PREFIX_PATH=\"${CMAKE_PREFIX_PATH}:/opt/rocm/lib/cmake\" # ROCm CMake packages\n(rmm_dev) $ mkdir build                                       # make a build directory\n(rmm_dev) $ cd build                                          # enter the build directory\n(rmm_dev) $ cmake .. -DCMAKE_INSTALL_PREFIX=/install/path     # configure cmake ... use $CONDA_PREFIX if you're using Anaconda\n(rmm_dev) $ make -j                                           # compile the library librmm.so ... '-j' will start a parallel job using the number of physical cores available on your system\n(rmm_dev) $ make install                                      # install the library librmm.so to '/install/path'\n```\n\n- Building and installing `librmm` and `rmm` using build.sh. Build.sh creates build dir at root of\n  git repository.\n\n```bash\n\n(rmm_dev) $ export CXX=\"hipcc\"                                # Cython CXX compiler, adjust according to your setup.\n(rmm_dev) $ export CMAKE_PREFIX_PATH=\"${CMAKE_PREFIX_PATH}:/opt/rocm/lib/cmake\" # ROCm CMake packages\n(rmm_dev) $ ./build.sh -h                                     # Display help and exit\n(rmm_dev) $ ./build.sh -n librmm                              # Build librmm without installing\n(rmm_dev) $ ./build.sh -n rmm                                 # Build rmm without installing\n(rmm_dev) $ ./build.sh -n librmm rmm                          # Build librmm and rmm without installing\n(rmm_dev) $ ./build.sh librmm rmm                             # Build and install librmm and rmm\n```\n\n\u003e [!Note]\n\u003e Before rebuilding, it is recommended to remove previous build files.\n\u003e When you are using the `./build.sh` script, this can be accomplished\n\u003e by additionally specifying `clean` (example: `./build.sh clean rmm`).\n\n- To run tests (Optional):\n```bash\n(rmm_dev) $ cd build (if you are not already in build directory)\n$ make test\n```\n\n- Build, install, and test the `rmm` python package, in the `python` folder:\n```bash\n(rmm_dev) $ export CXX=\"hipcc\" # Cython CXX compiler, adjust according to your setup.\n(rmm_dev) $ export CMAKE_PREFIX_PATH=\"${CMAKE_PREFIX_PATH}:/opt/rocm/lib/cmake\" # ROCm CMake packages\n(rmm_dev) $ python setup.py build_ext --inplace\n(rmm_dev) $ python setup.py install\n(rmm_dev) $ pytest -v\n```\n\n- Build the `rmm` python package and create a binary wheel, in the `python` folder:\n```bash\n(rmm_dev) $ export CXX=\"hipcc\" # Cython CXX compiler, adjust according to your setup.\n(rmm_dev) $ export CMAKE_PREFIX_PATH=\"${CMAKE_PREFIX_PATH}:/opt/rocm/lib/cmake\" # ROCm CMake packages\n(rmm_dev) $ python3 setup.py bdist_wheel\n```\n\nDone! You are ready to develop for the hipMM OSS project.\n\n### Installing a hipMM Python wheel\n\nWhen you install the hipMM-ROCm Python wheel, you can again specify the ROCm version of the dependencies via the optional dependency key `rocm-${ROCM_MAJOR}_${ROCM_MINOR}-${ROCM-PATCH}`. Again, you need to specify an extra `pip` index URL to make it possible for `pip` to find some dependencies.\n\n```bash\n$ previous_urls=$(pip config get global.extra-index-url)\n$ pip config set global.extra-index-url \"${previous_urls} https://test.pypi.org/simple\"\n\n$ pip install ${path_to_wheel}.whl[rocm-${ROCM_MAJOR}_${ROCM_MINOR}-${ROCM-PATCH}]\n# example: pip install ${path_to_wheel}.whl[rocm-6-1-2]\n```\n\n\u003e [!IMPORTANT]\n\u003e Each hipMM-ROCm wheel has been built against a particular ROCm version.\n\u003e The ROCm dependency key helps you to install hipMM dependencies for this\n\u003e particular ROCm version. Using the wheel with an incompatible\n\u003e ROCm installation or specifying dependencies that are not compatible\n\u003e with the ROCm installation assumed by the hipMM wheel,\n\u003e will likely result in issues.\n\n### Caching third-party dependencies\n\nhipMM uses [CPM.cmake](https://github.com/TheLartians/CPM.cmake) to\nhandle third-party dependencies like spdlog, Thrust, GoogleTest,\nGoogleBenchmark. In general you won't have to worry about it. If CMake\nfinds an appropriate version on your system, it uses it (you can\nhelp it along by setting `CMAKE_PREFIX_PATH` to point to the\ninstalled location). Otherwise those dependencies will be downloaded as\npart of the build.\n\nIf you frequently start new builds from scratch, consider setting the\nenvironment variable `CPM_SOURCE_CACHE` to an external download\ndirectory to avoid repeated downloads of the third-party dependencies.\n\n## Using hipMM in a downstream CMake project\n\nThe installed hipMM library provides a set of config files that makes it easy to\nintegrate hipMM into your own CMake project. In your `CMakeLists.txt`, just add\n\n```cmake\nfind_package(rmm [VERSION])\n# ...\ntarget_link_libraries(\u003cyour-target\u003e (PRIVATE|PUBLIC) rmm::rmm)\n```\n\nSince hipMM is a header-only library, this does not actually link hipMM,\nbut it makes the headers available and pulls in transitive dependencies.\nIf hipMM is not installed in a default location, use\n`CMAKE_PREFIX_PATH` or `rmm_ROOT` to point to its location.\n\nOne of hipMM's dependencies is the Thrust library, so the above\nautomatically pulls in `Thrust` by means of a dependency on the\n`rmm::Thrust` target. By default it uses the standard configuration of\nThrust. If you want to customize it, you can set the variables\n`THRUST_HOST_SYSTEM` and `THRUST_DEVICE_SYSTEM`; see\n[Thrust's CMake documentation](https://github.com/NVIDIA/thrust/blob/main/thrust/cmake/README.md).\n\n# Using hipMM in C++\n\nThe first goal of hipMM is to provide a common interface for device and host memory allocation.\nThis allows both _users_ and _implementers_ of custom allocation logic to program to a single\ninterface.\n\nTo this end, hipMM defines two abstract interface classes:\n- [`rmm::mr::device_memory_resource`](#device_memory_resource) for device memory allocation\n- [`rmm::mr::host_memory_resource`](#host_memory_resource) for host memory allocation\n\nThese classes are based on the\n[`std::pmr::memory_resource`](https://en.cppreference.com/w/cpp/memory/memory_resource) interface\nclass introduced in C++17 for polymorphic memory allocation.\n\n## `device_memory_resource`\n\n`rmm::mr::device_memory_resource` is the base class that defines the interface for allocating and\nfreeing device memory.\n\nIt has two key functions:\n\n1. `void* device_memory_resource::allocate(std::size_t bytes, cuda_stream_view s)`\n   - Returns a pointer to an allocation of at least `bytes` bytes.\n\n2. `void device_memory_resource::deallocate(void* p, std::size_t bytes, cuda_stream_view s)`\n   - Reclaims a previous allocation of size `bytes` pointed to by `p`.\n   - `p` *must* have been returned by a previous call to `allocate(bytes)`, otherwise behavior is\n     undefined\n\nIt is up to a derived class to provide implementations of these functions. See\n[available resources](#available-resources) for example `device_memory_resource` derived classes.\n\nUnlike `std::pmr::memory_resource`, `rmm::mr::device_memory_resource` does not allow specifying an\nalignment argument. All allocations are required to be aligned to at least 256B. Furthermore,\n`device_memory_resource` adds an additional `cuda_stream_view` argument to allow specifying the stream\non which to perform the (de)allocation.\n\n## `cuda_stream_view` and `cuda_stream`\n\n`rmm::cuda_stream_view` is a simple non-owning wrapper around a `hipStream_t`. This wrapper's\npurpose is to provide strong type safety for stream types. (`hipStream_t` is an alias for a pointer,\nwhich can lead to ambiguity in APIs when it is assigned `0`.)  All hipMM stream-ordered APIs take a\n`rmm::cuda_stream_view` argument.\n\n`rmm::cuda_stream` is a simple owning wrapper around a `hipStream_t`. This class provides\nRAII semantics (constructor creates the stream, destructor destroys it). An `rmm::cuda_stream`\ncan never represent the default stream or per-thread default stream; it only ever represents\na single non-default stream. `rmm::cuda_stream` cannot be copied, but can be moved.\n\n## `cuda_stream_pool`\n\n`rmm::cuda_stream_pool` provides fast access to a pool of streams. This class can be used to\ncreate a set of `cuda_stream` objects whose lifetime is equal to the `cuda_stream_pool`. Using the\nstream pool can be faster than creating the streams on the fly. The size of the pool is configurable.\nDepending on this size, multiple calls to `cuda_stream_pool::get_stream()` may return instances of\n`rmm::cuda_stream_view` that represent identical streams.\n\n### Thread Safety\n\nAll current device memory resources are thread safe unless documented otherwise. More specifically,\ncalls to memory resource `allocate()` and `deallocate()` methods are safe with respect to calls to\neither of these functions from other threads. They are _not_ thread safe with respect to\nconstruction and destruction of the memory resource object.\n\nNote that a class `thread_safe_resource_adapter` is provided which can be used to adapt a memory\nresource that is not thread safe to be thread safe (as described above). This adapter is not needed\nwith any current hipMM device memory resources.\n\n### Stream-ordered Memory Allocation\n\n`rmm::mr::device_memory_resource` is a base class that provides stream-ordered memory allocation.\nThis allows optimizations such as re-using memory deallocated on the same stream without the\noverhead of synchronization.\n\nA call to `device_memory_resource::allocate(bytes, stream_a)` returns a pointer that is valid to use\non `stream_a`. Using the memory on a different stream (say `stream_b`) is Undefined Behavior unless\nthe two streams are first synchronized, for example by using `hipStreamSynchronize(stream_a)` or by\nrecording a event on `stream_a` and then calling `hipStreamWaitEvent(stream_b, event)`.\n\nThe stream specified to `device_memory_resource::deallocate` should be a stream on which it is valid\nto use the deallocated memory immediately for another allocation. Typically this is the stream\non which the allocation was *last* used before the call to `deallocate`. The passed stream may be\nused internally by a `device_memory_resource` for managing available memory with minimal\nsynchronization, and it may also be synchronized at a later time, for example using a call to\n`hipStreamSynchronize()`.\n\nFor this reason, it is Undefined Behavior to destroy a stream that is passed to\n`device_memory_resource::deallocate`. If the stream on which the allocation was last used has been\ndestroyed before calling `deallocate` or it is known that it will be destroyed, it is likely better\nto synchronize the stream (before destroying it) and then pass a different stream to `deallocate`\n(e.g. the default stream).\n\nNote that device memory data structures such as `rmm::device_buffer` and `rmm::device_uvector`\nfollow these stream-ordered memory allocation semantics and rules.\n\nFor further information about stream-ordered memory allocation semantics, read\n[Using the NVIDIA Stream-Ordered Memory\nAllocator](https://developer.nvidia.com/blog/using-cuda-stream-ordered-memory-allocator-part-1/)\non the NVIDIA Developer Blog.\n\n### Available Resources\n\nhipMM provides several `device_memory_resource` derived classes to satisfy various user requirements.\nFor more detailed information about these resources, see their respective documentation.\n\n#### `cuda_memory_resource`\n\nAllocates and frees device memory using `hipMalloc` and `hipFree`.\n\n#### `managed_memory_resource`\n\nAllocates and frees device memory using `hipMallocManaged` and `hipFree`.\n\nNote that `managed_memory_resource` cannot be used with NVIDIA Virtual GPU Software (vGPU, for use\nwith virtual machines or hypervisors) because [NVIDIA Unified Memory is not supported by\nNVIDIA vGPU](https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu).\n\n#### `pool_memory_resource`\n\nA coalescing, best-fit pool sub-allocator.\n\n#### `fixed_size_memory_resource`\n\nA memory resource that can only allocate a single fixed size. Average allocation and deallocation\ncost is constant.\n\n#### `binning_memory_resource`\n\nConfigurable to use multiple upstream memory resources for allocations that fall within different\nbin sizes. Often configured with multiple bins backed by `fixed_size_memory_resource`s and a single\n`pool_memory_resource` for allocations larger than the largest bin size.\n\n### Default Resources and Per-device Resources\n\nhipMM users commonly need to configure a `device_memory_resource` object to use for all allocations\nwhere another resource has not explicitly been provided. A common example is configuring a\n`pool_memory_resource` to use for all allocations to get fast dynamic allocation.\n\nTo enable this use case, hipMM provides the concept of a \"default\" `device_memory_resource`. This\nresource is used when another is not explicitly provided.\n\nAccessing and modifying the default resource is done through two functions:\n- `device_memory_resource* get_current_device_resource()`\n   - Returns a pointer to the default resource for the current device.\n   - The initial default memory resource is an instance of `cuda_memory_resource`.\n   - This function is thread safe with respect to concurrent calls to it and\n     `set_current_device_resource()`.\n   - For more explicit control, you can use `get_per_device_resource()`, which takes a device ID.\n\n- `device_memory_resource* set_current_device_resource(device_memory_resource* new_mr)`\n   - Updates the default memory resource pointer for the current device to `new_mr`\n   - Returns the previous default resource pointer\n   - If `new_mr` is `nullptr`, then resets the default resource to `cuda_memory_resource`\n   - This function is thread safe with respect to concurrent calls to it and\n     `get_current_device_resource()`\n   - For more explicit control, you can use `set_per_device_resource()`, which takes a device ID.\n\n#### Example\n\n```c++\nrmm::mr::cuda_memory_resource cuda_mr;\n// Construct a resource that uses a coalescing best-fit pool allocator\nrmm::mr::pool_memory_resource\u003crmm::mr::cuda_memory_resource\u003e pool_mr{\u0026cuda_mr};\nrmm::mr::set_current_device_resource(\u0026pool_mr); // Updates the current device resource pointer to `pool_mr`\nrmm::mr::device_memory_resource* mr = rmm::mr::get_current_device_resource(); // Points to `pool_mr`\n```\n\n#### Multiple Devices\n\nA `device_memory_resource` should only be used when the active device is the same device\nthat was active when the `device_memory_resource` was created. Otherwise behavior is undefined.\n\nIf a `device_memory_resource` is used with a stream associated with a different device than the\ndevice for which the memory resource was created, behavior is undefined.\n\nCreating a `device_memory_resource` for each device requires care to set the current device before\ncreating each resource, and to maintain the lifetime of the resources as long as they are set as\nper-device resources. Here is an example loop that creates `unique_ptr`s to `pool_memory_resource`\nobjects for each device and sets them as the per-device resource for that device.\n\n```c++\nstd::vector\u003cunique_ptr\u003cpool_memory_resource\u003e\u003e per_device_pools;\nfor(int i = 0; i \u003c N; ++i) {\n  hipSetDevice(i); // set device i before creating MR\n  // Use a vector of unique_ptr to maintain the lifetime of the MRs\n  per_device_pools.push_back(std::make_unique\u003cpool_memory_resource\u003e());\n  // Set the per-device resource for device i\n  set_per_device_resource(cuda_device_id{i}, \u0026per_device_pools.back());\n}\n```\n\nNote that the device that is current when creating a `device_memory_resource` must also be\ncurrent any time that `device_memory_resource` is used to deallocate memory, including in a\ndestructor. This affects RAII classes like `rmm::device_buffer` and `rmm::device_uvector`. Here's an\n(incorrect) example that assumes the above example loop has been run to create a\n`pool_memory_resource` for each device. A correct example adds a call to `hipSetDevice(0)` on the\nline of the error comment.\n\n```c++\n{\n  RMM_CUDA_TRY(hipSetDevice(0));\n  rmm::device_buffer buf_a(16);\n\n  {\n    RMM_CUDA_TRY(hipSetDevice(1));\n    rmm::device_buffer buf_b(16);\n  }\n\n  // Error: when buf_a is destroyed, the current device must be 0, but it is 1\n}\n```\n\n### Allocators\n\nC++ interfaces commonly allow customizable memory allocation through an [`Allocator`](https://en.cppreference.com/w/cpp/named_req/Allocator) object.\nhipMM provides several `Allocator` and `Allocator`-like classes.\n\n#### `polymorphic_allocator`\n\nA [stream-ordered](#stream-ordered-memory-allocation) allocator similar to [`std::pmr::polymorphic_allocator`](https://en.cppreference.com/w/cpp/memory/polymorphic_allocator).\nUnlike the standard C++ `Allocator` interface, the `allocate` and `deallocate` functions take a `cuda_stream_view` indicating the stream on which the (de)allocation occurs.\n\n#### `stream_allocator_adaptor`\n\n`stream_allocator_adaptor` can be used to adapt a stream-ordered allocator to present a standard `Allocator` interface to consumers that may not be designed to work with a stream-ordered interface.\n\nExample:\n```c++\nrmm::cuda_stream stream;\nrmm::mr::polymorphic_allocator\u003cint\u003e stream_alloc;\n\n// Constructs an adaptor that forwards all (de)allocations to `stream_alloc` on `stream`.\nauto adapted = rmm::mr::make_stream_allocator_adaptor(stream_alloc, stream);\n\n// Allocates 100 bytes using `stream_alloc` on `stream`\nauto p = adapted.allocate(100);\n...\n// Deallocates using `stream_alloc` on `stream`\nadapted.deallocate(p,100);\n```\n\n#### `thrust_allocator`\n\n`thrust_allocator` is a device memory allocator that uses the strongly typed `thrust::device_ptr`, making it usable with containers like `thrust::device_vector`.\n\nSee [below](#using-rmm-with-thrust) for more information on using hipMM with Thrust.\n\n## Device Data Structures\n\n### `device_buffer`\n\nAn untyped, uninitialized RAII class for stream ordered device memory allocation.\n\n#### Example\n\n```c++\ncuda_stream_view s{...};\n// Allocates at least 100 bytes on stream `s` using the *default* resource\nrmm::device_buffer b{100,s};\nvoid* p = b.data();                   // Raw, untyped pointer to underlying device memory\n\nkernel\u003c\u003c\u003c..., s.value()\u003e\u003e\u003e(b.data()); // `b` is only safe to use on `s`\n\nrmm::mr::device_memory_resource * mr = new my_custom_resource{...};\n// Allocates at least 100 bytes on stream `s` using the resource `mr`\nrmm::device_buffer b2{100, s, mr};\n```\n\n### `device_uvector\u003cT\u003e`\nA typed, uninitialized RAII class for allocation of a contiguous set of elements in device memory.\nSimilar to a `thrust::device_vector`, but as an optimization, does not default initialize the\ncontained elements. This optimization restricts the types `T` to trivially copyable types.\n\n#### Example\n\n```c++\ncuda_stream_view s{...};\n// Allocates uninitialized storage for 100 `int32_t` elements on stream `s` using the\n// default resource\nrmm::device_uvector\u003cint32_t\u003e v(100, s);\n// Initializes the elements to 0\nthrust::uninitialized_fill(thrust::cuda::par.on(s.value()), v.begin(), v.end(), int32_t{0});\n\nrmm::mr::device_memory_resource * mr = new my_custom_resource{...};\n// Allocates uninitialized storage for 100 `int32_t` elements on stream `s` using the resource `mr`\nrmm::device_uvector\u003cint32_t\u003e v2{100, s, mr};\n```\n\n### `device_scalar`\nA typed, RAII class for allocation of a single element in device memory.\nThis is similar to a `device_uvector` with a single element, but provides convenience functions like\nmodifying the value in device memory from the host, or retrieving the value from device to host.\n\n#### Example\n```c++\ncuda_stream_view s{...};\n// Allocates uninitialized storage for a single `int32_t` in device memory\nrmm::device_scalar\u003cint32_t\u003e a{s};\na.set_value(42, s); // Updates the value in device memory to `42` on stream `s`\n\nkernel\u003c\u003c\u003c...,s.value()\u003e\u003e\u003e(a.data()); // Pass raw pointer to underlying element in device memory\n\nint32_t v = a.value(s); // Retrieves the value from device to host on stream `s`\n```\n\n## `host_memory_resource`\n\n`rmm::mr::host_memory_resource` is the base class that defines the interface for allocating and\nfreeing host memory.\n\nSimilar to `device_memory_resource`, it has two key functions for (de)allocation:\n\n1. `void* host_memory_resource::allocate(std::size_t bytes, std::size_t alignment)`\n   - Returns a pointer to an allocation of at least `bytes` bytes aligned to the specified\n     `alignment`\n\n2. `void host_memory_resource::deallocate(void* p, std::size_t bytes, std::size_t alignment)`\n   - Reclaims a previous allocation of size `bytes` pointed to by `p`.\n\n\nUnlike `device_memory_resource`, the `host_memory_resource` interface and behavior is identical to\n`std::pmr::memory_resource`.\n\n### Available Resources\n\n#### `new_delete_resource`\n\nUses the global `operator new` and `operator delete` to allocate host memory.\n\n#### `pinned_memory_resource`\n\nAllocates \"pinned\" host memory using `cuda(Malloc/Free)Host`.\n\n## Host Data Structures\n\nhipMM does not currently provide any data structures that interface with `host_memory_resource`.\nIn the future, hipMM will provide a similar host-side structure like `device_buffer` and an allocator\nthat can be used with STL containers.\n\n## Using hipMM with Thrust\n\nROCm-DS and other libraries make heavy use of Thrust. Thrust uses device memory in two\nsituations:\n\n 1. As the backing store for `thrust::device_vector`, and\n 2. As temporary storage inside some algorithms, such as `thrust::sort`.\n\nhipMM provides `rmm::mr::thrust_allocator` as a conforming Thrust allocator that uses\n`device_memory_resource`s.\n\n### Thrust Algorithms\n\nTo instruct a Thrust algorithm to use `rmm::mr::thrust_allocator` to allocate temporary storage, you\ncan use the custom Thrust device execution policy: `rmm::exec_policy(stream)`.\n\n```c++\nthrust::sort(rmm::exec_policy(stream, ...);\n```\n\nThe first `stream` argument is the `stream` to use for `rmm::mr::thrust_allocator`.\nThe second `stream` argument is what should be used to execute the Thrust algorithm.\nThese two arguments must be identical.\n\n## Logging\n\nhipMM includes two forms of logging. Memory event logging and debug logging.\n\n### Memory Event Logging and `logging_resource_adaptor`\n\nMemory event logging writes details of every allocation or deallocation to a CSV (comma-separated\nvalue) file. In C++, Memory Event Logging is enabled by using the `logging_resource_adaptor` as a\nwrapper around any other `device_memory_resource` object.\n\nEach row in the log represents either an allocation or a deallocation. The columns of the file are\n\"Thread, Time, Action, Pointer, Size, Stream\".\n\nThe CSV output files of the `logging_resource_adaptor` can be used as input to `REPLAY_BENCHMARK`,\nwhich is available when building hipMM from source, in the `gbenchmarks` folder in the build directory.\nThis log replayer can be useful for profiling and debugging allocator issues.\n\nThe following C++ example creates a logging version of a `cuda_memory_resource` that outputs the log\nto the file \"logs/test1.csv\".\n\n```c++\nstd::string filename{\"logs/test1.csv\"};\nrmm::mr::cuda_memory_resource upstream;\nrmm::mr::logging_resource_adaptor\u003crmm::mr::cuda_memory_resource\u003e log_mr{\u0026upstream, filename};\n```\n\nIf a file name is not specified, the environment variable `RMM_LOG_FILE` is queried for the file\nname. If `RMM_LOG_FILE` is not set, then an exception is thrown by the `logging_resource_adaptor`\nconstructor.\n\nIn Python, memory event logging is enabled when the `logging` parameter of `rmm.reinitialize()` is\nset to `True`. The log file name can be set using the `log_file_name` parameter. See\n`help(rmm.reinitialize)` for full details.\n\n### Debug Logging\n\nhipMM includes a debug logger which can be enabled to log trace and debug information to a file. This\ninformation can show when errors occur, when additional memory is allocated from upstream resources,\netc. The default log file is `rmm_log.txt` in the current working directory, but the environment\nvariable `RMM_DEBUG_LOG_FILE` can be set to specify the path and file name.\n\nThere is a CMake configuration variable `RMM_LOGGING_LEVEL`, which can be set to enable compilation\nof more detailed logging. The default is `INFO`. Available levels are `TRACE`, `DEBUG`, `INFO`,\n`WARN`, `ERROR`, `CRITICAL` and `OFF`.\n\nThe log relies on the [spdlog](https://github.com/gabime/spdlog.git) library.\n\nNote that to see logging below the `INFO` level, the application must also set the logging level at\nrun time. C++ applications must must call `rmm::logger().set_level()`, for example to enable all\nlevels of logging down to `TRACE`, call `rmm::logger().set_level(spdlog::level::trace)` (and compile\nlibrmm with `-DRMM_LOGGING_LEVEL=TRACE`). Python applications must call `rmm.set_logging_level()`,\nfor example to enable all levels of logging down to `TRACE`, call `rmm.set_logging_level(\"trace\")`\n(and compile the hipMM Python module with `-DRMM_LOGGING_LEVEL=TRACE`).\n\nNote that debug logging is different from the CSV memory allocation logging provided by\n`rmm::mr::logging_resource_adapter`. The latter is for logging a history of allocation /\ndeallocation actions which can be useful for replay with hipMM's replay benchmark.\n\n## hipMM and Memory Bounds Checking\n\nMemory allocations taken from a memory resource that allocates a pool of memory (such as\n`pool_memory_resource` and `arena_memory_resource`) are part of the same low-level memory\nallocation. Therefore, out-of-bounds or misaligned accesses to these allocations are not likely to\nbe detected by tools such as\n[Compute Sanitizer](https://docs.nvidia.com/cuda/compute-sanitizer/index.html) memcheck.\n\nExceptions to this are `cuda_memory_resource`, which wraps `hipMalloc`, and\n`cuda_async_memory_resource`, which uses `hipMallocAsync` with the device runtime's built-in memory pool\nfunctionality (11.2 or later required). Illegal memory accesses to memory allocated by these\nresources are detectable with Compute Sanitizer Memcheck.\n\nIt may be possible in the future to add support for memory bounds checking with other memory\nresources using NVTX APIs.\n\n## Using hipMM in Python Code\n\nThere are two ways to use hipMM in Python code:\n\n1. Using the `rmm.DeviceBuffer` API to explicitly create and manage\n   device memory allocations\n2. Transparently via external libraries such as CuPy and Numba\n\nhipMM provides a `MemoryResource` abstraction to control _how_ device\nmemory is allocated in both the above uses.\n\n### DeviceBuffers\n\nA DeviceBuffer represents an **untyped, uninitialized device memory\nallocation**.  DeviceBuffers can be created by providing the\nsize of the allocation in bytes:\n\n```python\n\u003e\u003e\u003e import rmm\n\u003e\u003e\u003e buf = rmm.DeviceBuffer(size=100)\n```\n\nThe size of the allocation and the memory address associated with it\ncan be accessed via the `.size` and `.ptr` attributes respectively:\n\n```python\n\u003e\u003e\u003e buf.size\n100\n\u003e\u003e\u003e buf.ptr\n140202544726016\n```\n\nDeviceBuffers can also be created by copying data from host memory:\n\n```python\n\u003e\u003e\u003e import rmm\n\u003e\u003e\u003e import numpy as np\n\u003e\u003e\u003e a = np.array([1, 2, 3], dtype='float64')\n\u003e\u003e\u003e buf = rmm.DeviceBuffer.to_device(a.tobytes())\n\u003e\u003e\u003e buf.size\n24\n```\n\nConversely, the data underlying a DeviceBuffer can be copied to the\nhost:\n\n```python\n\u003e\u003e\u003e np.frombuffer(buf.tobytes())\narray([1., 2., 3.])\n```\n\n### MemoryResource objects\n\n`MemoryResource` objects are used to configure how device memory allocations are made by\nhipMM.\n\nBy default if a `MemoryResource` is not set explicitly, hipMM uses the `CudaMemoryResource`, which\nuses `hipMalloc` for allocating device memory.\n\n`rmm.reinitialize()` provides an easy way to initialize hipMM with specific memory resource options\nacross multiple devices. See `help(rmm.reinitialize)` for full details.\n\nFor lower-level control, the `rmm.mr.set_current_device_resource()` function can be\nused to set a different MemoryResource for the current device.  For\nexample, enabling the `ManagedMemoryResource` tells hipMM to use\n`hipMallocManaged` instead of `hipMalloc` for allocating memory:\n\n```python\n\u003e\u003e\u003e import rmm\n\u003e\u003e\u003e rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())\n```\n\n\u003e :warning: The default resource must be set for any device **before**\n\u003e allocating any device memory on that device.  Setting or changing the\n\u003e resource after device allocations have been made can lead to unexpected\n\u003e behaviour or crashes. See [Multiple Devices](#multiple-devices)\n\nAs another example, `PoolMemoryResource` allows you to allocate a\nlarge \"pool\" of device memory up-front. Subsequent allocations will\ndraw from this pool of already allocated memory.  The example\nbelow shows how to construct a PoolMemoryResource with an initial size\nof 1 GiB and a maximum size of 4 GiB. The pool uses\n`CudaMemoryResource` as its underlying (\"upstream\") memory resource:\n\n```python\n\u003e\u003e\u003e import rmm\n\u003e\u003e\u003e pool = rmm.mr.PoolMemoryResource(\n...     rmm.mr.CudaMemoryResource(),\n...     initial_pool_size=2**30,\n...     maximum_pool_size=2**32\n... )\n\u003e\u003e\u003e rmm.mr.set_current_device_resource(pool)\n```\nOther MemoryResources include:\n\n* `FixedSizeMemoryResource` for allocating fixed blocks of memory\n* `BinningMemoryResource` for allocating blocks within specified \"bin\" sizes from different memory\nresources\n\nMemoryResources are highly configurable and can be composed together in different ways.\nSee `help(rmm.mr)` for more information.\n\n## Using hipMM with third-party libraries\n\n\u003e [!WARNING]\n\u003e The contents of this section have not been tested explicitly with **hipMM**.\n\n### Using hipMM with CuPy\n\nYou can configure [CuPy](https://cupy.dev/) to use hipMM for memory\nallocations by setting the CuPy allocator to\n`rmm_cupy_allocator`:\n\n```python\n\u003e\u003e\u003e from rmm.allocators.cupy import rmm_cupy_allocator\n\u003e\u003e\u003e import cupy\n\u003e\u003e\u003e cupy.cuda.set_allocator(rmm_cupy_allocator)\n```\n\n\n**Note:** This only configures CuPy to use the current hipMM resource for allocations.\nIt does not initialize nor change the current resource, e.g., enabling a memory pool.\nSee [here](#memoryresource-objects) for more information on changing the current memory resource.\n\n### Using hipMM with Numba\n\nYou can configure Numba to use hipMM for memory allocations using the\nNumba [EMM Plugin](https://numba.readthedocs.io/en/stable/cuda/external-memory.html#setting-emm-plugin).\n\nThis can be done in two ways:\n\n1. Setting the environment variable `NUMBA_CUDA_MEMORY_MANAGER`:\n\n  ```python\n  $ NUMBA_CUDA_MEMORY_MANAGER=rmm.allocators.numba python (args)\n  ```\n\n2. Using the `set_memory_manager()` function provided by Numba:\n\n  ```python\n  \u003e\u003e\u003e from numba import cuda\n  \u003e\u003e\u003e from rmm.allocators.numba import RMMNumbaManager\n  \u003e\u003e\u003e cuda.set_memory_manager(RMMNumbaManager)\n  ```\n\n**Note:** This only configures Numba to use the current hipMM resource for allocations.\nIt does not initialize nor change the current resource, e.g., enabling a memory pool.\nSee [here](#memoryresource-objects) for more information on changing the current memory resource.\n\n### Using hipMM with PyTorch\n\n[PyTorch](https://pytorch.org/docs/stable/notes/cuda.html) can use hipMM\nfor memory allocation.  For example, to configure PyTorch to use an\nhipMM-managed pool:\n\n```python\nimport rmm\nfrom rmm.allocators.torch import rmm_torch_allocator\nimport torch\n\nrmm.reinitialize(pool_allocator=True)\ntorch.cuda.memory.change_current_allocator(rmm_torch_allocator)\n```\n\nPyTorch and hipMM will now share the same memory pool.\n\nYou can, of course, use a custom memory resource with PyTorch as well:\n\n```python\nimport rmm\nfrom rmm.allocators.torch import rmm_torch_allocator\nimport torch\n\n# note that you can configure PyTorch to use hipMM either before or\n# after changing hipMM's memory resource.  PyTorch will use whatever\n# memory resource is configured to be the \"current\" memory resource at\n# the time of allocation.\ntorch.cuda.change_current_allocator(rmm_torch_allocator)\n\n# configure hipMM to use a managed memory resource, wrapped with a\n# statistics resource adaptor that can report information about the\n# amount of memory allocated:\nmr = rmm.mr.StatisticsResourceAdaptor(rmm.mr.ManagedMemoryResource())\nrmm.mr.set_current_device_resource(mr)\n\nx = torch.tensor([1, 2]).cuda()\n\n# the memory resource reports information about PyTorch allocations:\nmr.allocation_counts\nOut[6]:\n{'current_bytes': 16,\n 'current_count': 1,\n 'peak_bytes': 16,\n 'peak_count': 1,\n 'total_bytes': 16,\n 'total_count': 1}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FROCm%2FhipMM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FROCm%2FhipMM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FROCm%2FhipMM/lists"}