### device_buffer usage examples Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Demonstrates various ways to allocate, copy, move, and resize device_buffer objects. ```cpp // Allocates at least 100 bytes of device memory using the default memory // resource and default stream. device_buffer buff(100); // Allocates at least 100 bytes using the custom memory resource and // specified stream custom_memory_resource mr; auto stream = cuda::stream_ref{cudaStream_t{cudaStreamDefault}}; device_buffer custom_buff(100, stream, &mr); // Deep copies `buff` into a new device buffer using the specified stream device_buffer buff_copy(buff, stream); // Moves the memory in `from_buff` to `to_buff`. Deallocates previously allocated // to_buff memory on `to_buff.stream()`. device_buffer to_buff(std::move(from_buff)); // Deep copies `buff` into a new device buffer using the specified stream device_buffer buff_copy(buff, stream); // Shallow copies `buff` into a new device_buffer, `buff` is now empty device_buffer buff_move(std::move(buff)); // Default construction. Buffer is empty device_buffer buff_default{}; // If the requested size is larger than the current size, resizes allocation to the new size and // deep copies any previous contents. Otherwise, simply updates the value of `size()` to the // newly requested size without any allocations or copies. Uses the specified stream. buff_default.resize(100, stream); ``` -------------------------------- ### Set debug logging level Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Example of setting the debug logging level to WARN. ```python >>> import rmm >>> rmm.set_logging_level(rmm.logging_level.WARN) # set logging level to warn ``` -------------------------------- ### Adapting a stream-ordered allocator Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Example showing how to use stream_allocator_adaptor to wrap a stream-ordered allocator for use with standard allocator interfaces. ```cpp my_stream_ordered_allocator a{...}; cuda::stream_ref s = // create stream; auto adapted = stream_allocator_adaptor(a, s); // Allocates storage for `n` int's on stream `s` int * p = std::allocator_traits::allocate(adapted, n); ``` -------------------------------- ### rmm.DeviceBuffer.nbytes Source: https://docs.nvidia.com/rmm/latest/python/rmm Gets the size of the buffer in bytes. ```APIDOC ## rmm.DeviceBuffer.nbytes ### Description Gets the size of the buffer in bytes. ``` -------------------------------- ### rmm.DeviceBuffer.nbytes Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Gets the size of the buffer in bytes. ```APIDOC ## rmm.DeviceBuffer.nbytes ### Description Gets the size of the buffer in bytes. ### Returns - **int** - The size of the buffer in bytes. ``` -------------------------------- ### Implementing custom memory resources Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Demonstrates the minimal migration for custom memory resources to comply with the CCCL resource concept, requiring both async and synchronous interfaces. ```cpp // 26.04 class my_resource : public rmm::mr::device_memory_resource { void* do_allocate(std::size_t bytes, rmm::cuda_stream_view stream) override { return upstream_->allocate(bytes, stream); } void do_deallocate(void* p, std::size_t bytes, rmm::cuda_stream_view stream) override { upstream_->deallocate(p, bytes, stream); } bool do_is_equal(device_memory_resource const& other) const noexcept override { return this == &other; } rmm::mr::device_memory_resource* upstream_; }; // 26.06 class my_resource { public: explicit my_resource(cuda::mr::any_resource upstream) : upstream_(std::move(upstream)) {} // Async (stream-ordered) interface void* allocate(cuda::stream_ref stream, std::size_t bytes, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT) { return upstream_.allocate(stream, bytes, alignment); } void deallocate(cuda::stream_ref stream, void* p, std::size_t bytes, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT) noexcept { upstream_.deallocate(stream, p, bytes, alignment); } // Synchronous interface — required by the CCCL resource concept void* allocate_sync(std::size_t bytes, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT) { return upstream_.allocate_sync(bytes, alignment); } void deallocate_sync(void* p, std::size_t bytes, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT) noexcept { upstream_.deallocate_sync(p, bytes, alignment); } bool operator==(my_resource const& other) const noexcept { return this == &other; } bool operator!=(my_resource const& other) const noexcept { return !(*this == other); } constexpr friend void get_property(my_resource const&, cuda::mr::device_accessible) noexcept {} private: cuda::mr::any_resource upstream_; }; static_assert(cuda::mr::resource_with); ``` -------------------------------- ### Copy host data to device with copy_host_to_ptr Source: https://docs.nvidia.com/rmm/latest/python/pylibrmm/index.html Shows how to copy host data into a device buffer pointer. ```python >>> import rmm >>> db = rmm.DeviceBuffer(size=10) >>> hb = b"abc" >>> rmm.pylibrmm.device_buffer.copy_host_to_ptr(hb, db.ptr) >>> hb = db.copy_to_host() >>> print(hb) array([97, 98, 99, 0, 0, 0, 0, 0, 0, 0], dtype=uint8) ``` -------------------------------- ### cuda_stream::value() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams Gets the value of the wrapped CUDA stream. ```APIDOC ## cuda_stream::value() ### Description Gets the value of the wrapped CUDA stream. ### Returns - **cudaStream_t** - The wrapped CUDA stream. ``` -------------------------------- ### rmm::cuda_stream::value() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams/index.html Get the value of the wrapped CUDA stream. ```APIDOC ## value() ### Description Get the value of the wrapped CUDA stream. ### Returns - cudaStream_t - The wrapped CUDA stream. ### Example ```cpp cudaStream_t raw = stream.value(); ``` ``` -------------------------------- ### Allocating and initializing a device_uvector Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Demonstrates how to allocate uninitialized device memory using a custom resource and stream, followed by an explicit initialization using thrust::uninitialized_fill. ```cpp auto mr = new my_custom_resource(); auto s = cuda::stream_ref{cudaStream_t{cudaStreamDefault}}; // Allocates *uninitialized* device memory on stream `s` sufficient for 100 ints using the // supplied resource `mr` rmm::device_uvector uv(100, s, mr); // Initializes all elements to 0 on stream `s` thrust::uninitialized_fill(thrust::cuda::par.on(s), uv.begin(), uv.end(), 0); ``` -------------------------------- ### Get the current logging level Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Retrieves the current debug logging level. ```python >>> import rmm >>> rmm.get_logging_level() # get current logging level ``` -------------------------------- ### Implement a CallbackMemoryResource Source: https://docs.nvidia.com/rmm/latest/python/mr Demonstrates how to define custom allocation and deallocation functions and use them to set the current device memory resource. ```python >>> import rmm >>> base_mr = rmm.mr.CudaMemoryResource() >>> def allocate_func(size, stream): ... print(f"Allocating {size} bytes") ... return base_mr.allocate(size, stream) ... >>> def deallocate_func(ptr, size, stream): ... print(f"Deallocating {size} bytes") ... return base_mr.deallocate(ptr, size, stream) ... >>> rmm.mr.set_current_device_resource( rmm.mr.CallbackMemoryResource(allocate_func, deallocate_func) ) >>> dbuf = rmm.DeviceBuffer(size=256) Allocating 256 bytes >>> del dbuf Deallocating 256 bytes ``` -------------------------------- ### Migrating owning_wrapper to value-based storage Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Shows the transition from using rmm::mr::owning_wrapper to storing resources by value with internal shared ownership. ```cpp // 26.04 auto cuda_mr = std::make_shared(); auto pool = rmm::mr::make_owning_wrapper( cuda_mr, pool_size); // 26.06 — resources are value types with shared ownership rmm::mr::cuda_memory_resource cuda_mr; rmm::mr::pool_memory_resource pool{cuda_mr, pool_size}; // pool is stored by value; its state is kept alive by internal shared ownership ``` -------------------------------- ### get_per_device_resource_ref Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Get the device_async_resource_ref for the specified device. Returns a device_async_resource_ref for the specified device, initially referencing a cuda_memory_resource. ```APIDOC ## Function: get_per_device_resource_ref ### Description Get the device_async_resource_ref for the specified device. Returns a device_async_resource_ref for the specified device. The initial resource_ref references a cuda_memory_resource. ### Signature inline device_async_resource_ref get_per_device_resource_ref(_cuda_device_id device_id_); ### Parameters - **device_id** (_cuda_device_id) - The id of the target device. device_id.value() must be in the range [0, cudaGetDeviceCount()), otherwise behavior is undefined. ### Returns The current device_async_resource_ref for device device_id. ### Thread Safety This function is thread-safe with respect to concurrent calls to set_per_device_resource, get_per_device_resource_ref, get_current_device_resource_ref, set_current_device_resource, reset_per_device_resource, and reset_current_device_resource. Concurrent calls to any of these functions will result in a valid state, but the order of execution is undefined. ### Notes The returned device_async_resource_ref should only be used when CUDA device device_id is the current device (e.g. set using cudaSetDevice()). The behavior of a device_async_resource_ref is undefined if used while the active CUDA device is a different device from the one that was active when the memory resource was created. ``` -------------------------------- ### Enable statistics tracking with context manager and globally Source: https://docs.nvidia.com/rmm/latest/user_guide/guide/index.html Shows how to enable RMM statistics tracking using the statistics() context manager or enable_statistics() globally, and demonstrates that the current device resource becomes a StatisticsResourceAdaptor. ```python >>> import rmm >>> import rmm.statistics >>> # We start with the default CUDA memory resource >>> rmm.mr.get_current_device_resource() >>> # When using statistics, we get a StatisticsResourceAdaptor with the context >>> with rmm.statistics.statistics(): ... rmm.mr.get_current_device_resource() >>> # We can also enable statistics globally >>> rmm.statistics.enable_statistics() >>> print(rmm.mr.get_current_device_resource()) ``` -------------------------------- ### Create DeviceBuffer from host memory Source: https://docs.nvidia.com/rmm/latest/user_guide/guide/index.html Initialize a DeviceBuffer by copying data from a host NumPy array. ```python >>> import rmm >>> import numpy as np >>> a = np.array([1, 2, 3], dtype='float64') >>> buf = rmm.DeviceBuffer.to_device(a.view("uint8")) # to_device expects an unsigned 8-bit dtype >>> buf.size 24 ``` -------------------------------- ### cuda_stream_pool::get_pool_size() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams Gets the number of streams in the pool. This function is thread safe with respect to other calls to the same function. ```APIDOC ## cuda_stream_pool::get_pool_size() ### Description Gets the number of streams in the pool. This function is thread safe with respect to other calls to the same function. ### Returns - **std::size_t** - The number of streams in the pool. ``` -------------------------------- ### Construct a DeviceBuffer Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Initialize a new DeviceBuffer with a specified size. ```python >>> import rmm >>> db = rmm.DeviceBuffer(size=5) ``` -------------------------------- ### Set flush level for debug logger Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Example of setting the flush level for the debug logger to WARN. ```python >>> import rmm >>> rmm.flush_on(rmm.logging_level.WARN) # set flush level to warn ``` -------------------------------- ### Copy device data to host with copy_ptr_to_host Source: https://docs.nvidia.com/rmm/latest/python/pylibrmm/index.html Demonstrates copying data from a device pointer into a host-side buffer. ```python >>> import rmm >>> db = rmm.DeviceBuffer.to_device(b"abc") >>> hb = bytearray(db.nbytes) >>> rmm.pylibrmm.device_buffer.copy_ptr_to_host(db.ptr, hb) >>> print(hb) bytearray(b'abc') ``` -------------------------------- ### rmm.mr.get_per_device_resource(device) Source: https://docs.nvidia.com/rmm/latest/python/mr Gets the default memory resource for the specified device. The returned resource should only be used while that device is active. ```APIDOC ## rmm.mr.get_per_device_resource(device) ### Description Get the default memory resource for the specified device. ### Method Function call ### Endpoint rmm.mr.get_per_device_resource(device) ### Parameters #### Path Parameters - **device** (int) - Required - The ID of the device for which to get the memory resource. ### Returns - **DeviceMemoryResource** - The memory resource for the specified device. ### Notes If the returned memory resource is used when a different device is the active CUDA device, behavior is undefined. ``` -------------------------------- ### Pass Upstream Resources by Value Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Constructors now accept cuda::mr::any_resource instead of raw pointers. Resources can be passed directly. ```cpp // 26.04 - pass a device_memory_resource pointer rmm::mr::cuda_memory_resource cuda_mr; rmm::mr::pool_memory_resource pool{&cuda_mr, pool_size}; // 26.06 — pass the resource directly rmm::mr::cuda_memory_resource cuda_mr; rmm::mr::pool_memory_resource pool{cuda_mr, pool_size}; ``` -------------------------------- ### rmm::cuda_stream_pool::get_pool_size() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams/index.html Get the number of streams in the pool. This function is thread safe with respect to other calls to the same function. ```APIDOC ## get_pool_size() ### Description Get the number of streams in the pool. This function is thread safe with respect to other calls to the same function. ### Returns - std::size_t - The number of streams in the pool. ### Example ```cpp std::size_t size = pool.get_pool_size(); ``` ``` -------------------------------- ### Create a DeviceBuffer Source: https://docs.nvidia.com/rmm/latest/user_guide/guide/index.html Create an untyped, uninitialized device memory allocation by specifying the size in bytes. ```python >>> import rmm >>> buf = rmm.DeviceBuffer(size=100) ``` -------------------------------- ### cuda_stream_pool::get_stream() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams Gets a cuda::stream_ref of a stream in the pool. This function is thread safe with respect to other calls to the same function. ```APIDOC ## cuda_stream_pool::get_stream() ### Description Gets a cuda::stream_ref of a stream in the pool. This function is thread safe with respect to other calls to the same function. ### Returns - **cuda::stream_ref** ``` -------------------------------- ### Update Per-Device Resource Management Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Comparison of per-device resource management between 26.04 and 26.06, showing the shift from raw pointers to owning any_resource types and the use of RAII-style restoration. ```cpp // 26.04 rmm::mr::pool_memory_resource pool{&cuda_mr, size}; rmm::mr::set_current_device_resource(&pool); auto* mr = rmm::mr::get_current_device_resource(); // 26.06 rmm::mr::pool_memory_resource pool{cuda_mr, size}; auto prev = rmm::mr::set_current_device_resource(pool); auto ref = rmm::mr::get_current_device_resource_ref(); // When done, restore: rmm::mr::set_current_device_resource(std::move(prev)); // Or reset to default: rmm::mr::reset_current_device_resource(); ``` -------------------------------- ### RMMNumbaManager.get_ipc_handle(memory: MemoryPointer) -> IpcHandle Source: https://docs.nvidia.com/rmm/latest/python/allocators/index.html Get an IPC handle for the MemoryPointer memory with offset modified by the RMM memory pool. ```APIDOC ## RMMNumbaManager.get_ipc_handle(memory: MemoryPointer) -> IpcHandle ### Description Get an IPC handle for the MemoryPointer memory with offset modified by the RMM memory pool. ### Parameters - **memory** (MemoryPointer) - Required - The memory pointer for which to get an IPC handle. ### Returns - **IpcHandle** - The IPC handle for the memory. ``` -------------------------------- ### device_uvector constructor (size, stream, mr) Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Constructs a new device_uvector with sufficient uninitialized storage for the specified number of elements. The elements are uninitialized, and reading them before initialization is undefined behavior. ```APIDOC ## Constructor: device_uvector(size_type size, cuda::stream_ref stream, cuda::mr::any_resource mr = mr::get_current_device_resource_ref()) ### Description Constructs a new device_uvector with sufficient uninitialized storage for `size` elements. Elements are uninitialized; reading an element before it is initialized results in undefined behavior. ### Parameters - **size** (size_type) - Required - The number of elements to allocate storage for. - **stream** (cuda::stream_ref) - Required - The stream on which to perform the allocation. - **mr** (cuda::mr::any_resource) - Optional - The resource used to allocate the device storage. Defaults to the current device resource. ### Throws - **rmm::bad_alloc** - If the provided memory resource cannot allocate with alignment to satisfy the alignment requirements of the value type. - **rmm::invalid_argument** - If the requested size overflows the maximum allocation size. ### Example ```cpp auto mr = new my_custom_resource(); auto s = cuda::stream_ref{cudaStream_t{cudaStreamDefault}}; rmm::device_uvector uv(100, s, mr); ``` ``` -------------------------------- ### Create a DeviceBuffer with to_device Source: https://docs.nvidia.com/rmm/latest/python/pylibrmm/index.html Creates a new DeviceBuffer containing a copy of the provided host data. ```python >>> import rmm >>> db = rmm.pylibrmm.device_buffer.to_device(b"abc") >>> print(bytes(db)) b'abc' ``` -------------------------------- ### get_current_device_resource_ref Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Gets the device_async_resource_ref for the current device. Returns the resource reference set for the current device, initially referencing a cuda_memory_resource. ```APIDOC ## get_current_device_resource_ref ### Description Gets the device_async_resource_ref for the current device. Returns the device_async_resource_ref set for the current device. The initial resource_ref references a cuda_memory_resource. The current device is the device returned by cudaGetDevice. This function is thread-safe with respect to concurrent calls to set_per_device_resource, get_per_device_resource_ref, get_current_device_resource_ref, set_current_device_resource, reset_per_device_resource, and reset_current_device_resource. Concurrent calls to any of these functions will result in a valid state, but the order of execution is undefined. ### Method N/A (C++ inline function) ### Signature inline device_async_resource_ref get_current_device_resource_ref() ### Parameters None ### Returns device_async_resource_ref active for the current device ### Notes The returned device_async_resource_ref should only be used with the current CUDA device. Changing the current device (e.g. using cudaSetDevice()) and then using the returned resource_ref can result in undefined behavior. The behavior of a device_async_resource_ref is undefined if used while the active CUDA device is a different device from the one that was active when the memory resource was created. ``` -------------------------------- ### rmm::cuda_stream_pool::get_stream() Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams/index.html Get a cuda::stream_ref of a stream in the pool. This function is thread safe with respect to other calls to the same function. ```APIDOC ## get_stream() ### Description Get a cuda::stream_ref of a stream in the pool. This function is thread safe with respect to other calls to the same function. ### Returns - cuda::stream_ref ### Example ```cpp auto stream_ref = pool.get_stream(); ``` ``` -------------------------------- ### bad_alloc::what() Source: https://docs.nvidia.com/rmm/latest/cpp/errors Returns the explanatory string for the bad_alloc exception. ```APIDOC ## bad_alloc::what() ### Description Returns the explanatory string for the bad_alloc exception. ### Returns - **const char*** - The explanatory string ``` -------------------------------- ### void* data() noexcept Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Returns a pointer to the device memory allocation. ```APIDOC ## void* data() noexcept ### Description Pointer to the device memory allocation. ### Returns Pointer to the device memory allocation. ``` -------------------------------- ### BinningMemoryResource Source: https://docs.nvidia.com/rmm/latest/python/mr A memory resource that bins allocations by size. It inherits from UpstreamResourceAdaptor and provides methods to add bins, allocate, deallocate, and get the upstream resource. ```APIDOC ## BinningMemoryResource ### Description A memory resource that bins allocations by size. It inherits from UpstreamResourceAdaptor and provides methods to add bins, allocate, deallocate, and get the upstream resource. ### Constructor `rmm.mr.BinningMemoryResource(upstream_mr, min_size_exponent=-1, max_size_exponent=-1)` #### Parameters - **upstream_mr** (DeviceMemoryResource) - Required - The upstream memory resource. - **min_size_exponent** (int8_t) - Optional - Minimum size exponent. - **max_size_exponent** (int8_t) - Optional - Maximum size exponent. ### Attributes - **bin_mrs** (list) - The list of binned memory resources. - **upstream_mr** - The upstream memory resource. ### Methods #### add_bin ##### Description Adds a bin of the specified maximum allocation size to this memory resource. If specified, uses bin_resource for allocation for this bin. If not specified, creates and uses a FixedSizeMemoryResource for allocation for this bin. Allocations smaller than allocation_size and larger than the next smaller bin size will use this fixed-size memory resource. ##### Parameters - **allocation_size** (size_t) - Required - The maximum allocation size in bytes for the created bin. - **bin_resource** (DeviceMemoryResource) - Optional - The resource to use for this bin. #### allocate ##### Description Allocate `nbytes` bytes of memory. ##### Parameters - **nbytes** (size_t) - Required - The size of the allocation in bytes. - **stream** (Stream) - Optional - Optional stream for the allocation. ##### Raises - **MemoryError** - If allocation fails. ##### Notes - Allocations are always requested with 256-byte alignment. - On integrated memory systems, attempting to allocate more memory than available can cause the process to be killed by the operating system instead of raising a catchable `MemoryError`. #### deallocate ##### Description Deallocate memory pointed to by `ptr` of size `nbytes`. ##### Parameters - **ptr** (uintptr_t) - Required - Pointer to be deallocated. - **nbytes** (size_t) - Required - Size of the allocation in bytes. - **stream** (Stream) - Optional - Optional stream for the deallocation. #### get_upstream ##### Description Returns the upstream memory resource. ##### Returns - **DeviceMemoryResource** - The upstream memory resource. ``` -------------------------------- ### rmm.DeviceBuffer.copy_from_host() Source: https://docs.nvidia.com/rmm/latest/python/rmm Copies from a buffer on host to self. ```APIDOC ## rmm.DeviceBuffer.copy_from_host(ary, stream=DEFAULT_STREAM) ### Description Copy from a buffer on host to self. ### Parameters - **ary** (bytes-like) - Required - Buffer to copy from. - **stream** (Stream) - Optional - CUDA stream to use for copying, defaults to the default stream. ### Notes If stream is the default stream, it is synchronized after the copy. If a non-default stream is provided, the copy is asynchronous. The caller must keep ary alive and unmodified until the stream is synchronized. ### Example ```python >>> import rmm >>> db = rmm.DeviceBuffer(size=10) >>> hb = b"abcdef" >>> db.copy_from_host(hb) >>> hb = db.copy_to_host() >>> print(hb) array([97, 98, 99, 0, 0, 0, 0, 0, 0, 0], dtype=uint8) ``` ``` -------------------------------- ### void* data() Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers Returns a pointer to the device memory allocation. ```APIDOC ## void* data() ### Description Returns a pointer to the device memory allocation. ### Returns Pointer to the device memory allocation. ``` -------------------------------- ### StatisticsResourceAdaptor.allocation_counts Source: https://docs.nvidia.com/rmm/latest/python/mr Gets the current, peak, and total allocated bytes and number of allocations. Returns a dictionary with keys: current_bytes, current_count, peak_bytes, peak_count, total_bytes, total_count. ```APIDOC ## allocation_counts ### Description Gets the current, peak, and total allocated bytes and number of allocations. The dictionary keys are `current_bytes`, `current_count`, `peak_bytes`, `peak_count`, `total_bytes`, and `total_count`. ### Returns - **dict** - Dictionary containing allocation counts and bytes. ``` -------------------------------- ### begin() noexcept Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers Returns an iterator to the first element. If the vector is empty, begin() equals end(). ```APIDOC ## begin() noexcept ### Description Returns an iterator to the first element. If the vector is empty, then `begin() == end()`. ### Returns - **iterator** - Iterator to the first element. ``` -------------------------------- ### cuda_async_memory_resource Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Memory resource that uses cudaMallocAsync and cudaFreeAsync for allocation and deallocation. Provides constructors, assignment operators, and a method to get the underlying CUDA pool handle. ```APIDOC ## cuda_async_memory_resource ### Description Memory resource that uses `cudaMallocAsync`/`cudaFreeAsync` for allocation/deallocation. ### Constructors #### cuda_async_memory_resource() Default constructor. #### cuda_async_memory_resource(cuda_async_memory_resource const&) Default copy constructor. #### cuda_async_memory_resource(cuda_async_memory_resource &&) Default move constructor. ### Assignment Operators #### operator=(cuda_async_memory_resource const&) Default copy assignment operator. **Returns:** - `cuda_async_memory_resource&` - Reference to the assigned object #### operator=(cuda_async_memory_resource &&) Default move assignment operator. **Returns:** - `cuda_async_memory_resource&` - Reference to the assigned object ### Destructor #### ~cuda_async_memory_resource() = default ### Friend Functions #### get_property(cuda_async_memory_resource const&, cuda::mr::device_accessible) Enables the `cuda::mr::device_accessible` property. ### Methods #### pool_handle() const noexcept Returns the underlying native handle to the CUDA pool. **Returns:** - `cudaMemPool_t` - Handle to the underlying CUDA pool ``` -------------------------------- ### cuda_stream_pool::get_stream(stream_id) Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams Gets a cuda::stream_ref of the stream associated with stream_id. Equivalent values of stream_id return a stream_ref to the same underlying stream. This function is thread safe with respect to other calls to the same function. ```APIDOC ## cuda_stream_pool::get_stream(stream_id) ### Description Gets a cuda::stream_ref of the stream associated with stream_id. Equivalent values of stream_id return a stream_ref to the same underlying stream. This function is thread safe with respect to other calls to the same function. ### Parameters - **stream_id** (std::size_t) - Required - Unique identifier for the desired stream. Note: stream_id is wrapped around the pool size, therefore any size_t value is allowed. ### Returns - **cuda::stream_ref** ``` -------------------------------- ### rmm.DeviceBuffer.copy_to_host() Source: https://docs.nvidia.com/rmm/latest/python/rmm Copies from a DeviceBuffer to a buffer on host. ```APIDOC ## rmm.DeviceBuffer.copy_to_host(ary=None, stream=DEFAULT_STREAM) ### Description Copy from a DeviceBuffer to a buffer on host. ### Parameters - **ary** (bytes-like) - Optional - Buffer to write into. - **stream** (Stream) - Optional - CUDA stream to use for copying, defaults to the default stream. ### Example ```python >>> import rmm >>> db = rmm.DeviceBuffer.to_device(b"abc") >>> hb = bytearray(db.nbytes) >>> db.copy_to_host(hb) >>> print(hb) bytearray(b'abc') >>> hb = db.copy_to_host() >>> print(hb) bytearray(b'abc') ``` ``` -------------------------------- ### rmm::cuda_stream_pool::get_stream(stream_id) Source: https://docs.nvidia.com/rmm/latest/cpp/cuda_streams/index.html Get a cuda::stream_ref of the stream associated with stream_id. Equivalent values of stream_id return a stream_ref to the same underlying stream. This function is thread safe with respect to other calls to the same function. ```APIDOC ## get_stream(stream_id) ### Description Get a cuda::stream_ref of the stream associated with stream_id. Equivalent values of stream_id return a stream_ref to the same underlying stream. This function is thread safe with respect to other calls to the same function. ### Parameters - **stream_id** (std::size_t) - Required - Unique identifier for the desired stream. Note: stream_id is wrapped around the pool size, therefore any size_t value is allowed. ### Returns - cuda::stream_ref ### Example ```cpp auto stream_ref = pool.get_stream(0); ``` ``` -------------------------------- ### rmm.pylibrmm.device_buffer.to_device() Source: https://docs.nvidia.com/rmm/latest/python/pylibrmm Creates a new DeviceBuffer with a copy of the host data. ```APIDOC ## rmm.pylibrmm.device_buffer.to_device() ### Description Return a new `DeviceBuffer` with a copy of the data. ### Method `to_device(const unsigned char[::1] b, Stream stream=DEFAULT_STREAM) -> DeviceBuffer` ### Parameters - **b** (buffer) - Required - Host data to copy to device. - **stream** (Stream) - Optional - CUDA stream to use for copying. Defaults to the default stream. ### Returns - **DeviceBuffer** - Device buffer containing a copy of the host data. ### Notes If `stream` is the default stream, it is synchronized after the copy. If a non-default `stream` is provided, the copy is asynchronous. The caller must keep `b` alive and unmodified until the stream is synchronized. ### Example ``` >>> import rmm >>> db = rmm.pylibrmm.device_buffer.to_device(b"abc") >>> print(bytes(db)) b'abc' ``` ``` -------------------------------- ### Get log filenames for each device Source: https://docs.nvidia.com/rmm/latest/python/mr Shows how to use rmm.get_log_filenames() after reinitializing RMM with logging enabled for devices 0 and 1. The output is a dictionary mapping device IDs to log file paths, with the suffix .dev{id} added to the log file name. ```python >>> import rmm >>> rmm.reinitialize(devices=[0, 1], logging=True, log_file_name="rmm.log") >>> rmm.get_log_filenames() {0: '/home/user/workspace/rapids/rmm/python/rmm.dev0.log', 1: '/home/user/workspace/rapids/rmm/python/rmm.dev1.log'} ``` -------------------------------- ### Nested memory profiling with rmm.statistics.profiler Source: https://docs.nvidia.com/rmm/latest/user_guide/guide/index.html Use rmm.statistics.profiler as a context manager to profile nested code blocks. The example creates two DeviceBuffers inside outer and inner profiler contexts, then prints the report from rmm.statistics.default_profiler_records.report(). The report shows ncalls, memory_peak, and memory_total for each profiled block. ```python >>> with rmm.statistics.profiler(name="outer"): ... buf1 = rmm.DeviceBuffer(size=10) ... with rmm.statistics.profiler(name="inner"): ... buf2 = rmm.DeviceBuffer(size=10) >>> print(rmm.statistics.default_profiler_records.report()) Memory Profiling ================ Legends: ncalls - number of times the function or code block was called memory_peak - peak memory allocated in function or code block (in bytes) memory_total - total memory allocated in function or code block (in bytes) Ordered by: memory_peak ncalls memory_peak memory_total filename:lineno(function) 1 1,008 1,008 :1(f) 1 32 32 my code block 1 32 32 outer 1 16 16 inner ``` -------------------------------- ### host_device_async_resource_ref Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources Alias for an asynchronous resource reference with both host and device accessibility. ```APIDOC ## host_device_async_resource_ref ### Description Alias for a `cuda::mr::resource_ref` with the properties `cuda::mr::host_accessible` and `cuda::mr::device_accessible`. ### Type using host_device_async_resource_ref = cuda::mr::resource_ref ``` -------------------------------- ### Making custom resources copyable for any_resource Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Illustrates how to wrap non-copyable members in std::shared_ptr to satisfy the copyability requirement of cuda::mr::any_resource. ```cpp // Won't work — shared_mutex is not copyable class my_tracking_resource { std::shared_mutex mutex_; // NOT copyable std::unordered_set allocations_; // copyable, but expensive // ... }; // Fix: wrap shared state in shared_ptr class my_tracking_resource { struct shared_state { std::shared_mutex mutex; std::unordered_set allocations; }; std::shared_ptr state_{std::make_shared()}; // Now the resource is copyable; copies share the tracking state // ... }; ``` -------------------------------- ### device_buffer(void const* source_data, std::size_t size, cuda::stream_ref stream, cuda::mr::any_resource mr = mr::get_current_device_resource_ref()) Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Constructs a new device buffer by copying from a raw pointer to an existing host or device memory allocation. The copy is performed on the specified stream, and the caller is responsible for synchronization. The buffer is guaranteed to have an alignment of at least rmm::CUDA_ALLOCATION_ALIGNMENT. ```APIDOC ## device_buffer(void const* source_data, std::size_t size, cuda::stream_ref stream, cuda::mr::any_resource mr = mr::get_current_device_resource_ref()) ### Description Constructs a new device buffer by copying from a raw pointer to an existing host or device memory allocation. The copy is performed on the specified stream, and the caller is responsible for synchronization. The buffer is guaranteed to have an alignment of at least rmm::CUDA_ALLOCATION_ALIGNMENT. ### Parameters - **source_data** (void const*) - Required - Pointer to the host or device memory to copy from. - **size** (std::size_t) - Required - Size in bytes to copy. - **stream** (cuda::stream_ref) - Required - CUDA stream on which memory may be allocated if the memory resource supports streams. - **mr** (cuda::mr::any_resource) - Optional - Memory resource to use for the device memory allocation. Defaults to mr::get_current_device_resource_ref(). ### Throws - rmm::bad_alloc - If creating the new allocation fails. - rmm::logic_error - If source_data is null, and size != 0. - rmm::cuda_error - If copying from the device memory fails. ``` -------------------------------- ### Copy a DeviceBuffer Source: https://docs.nvidia.com/rmm/latest/python/rmm/index.html Create a deep copy of an existing DeviceBuffer. ```python >>> import rmm >>> db = rmm.DeviceBuffer.to_device(b"abc") >>> db_copy = db.copy() >>> db.copy_to_host() array([97, 98, 99], dtype=uint8) >>> db_copy.copy_to_host() array([97, 98, 99], dtype=uint8) >>> assert db is not db_copy >>> assert db.ptr != db_copy.ptr ``` -------------------------------- ### rmm::bad_alloc::what() Source: https://docs.nvidia.com/rmm/latest/cpp/errors/index.html Returns the explanatory string for the bad_alloc exception. ```APIDOC ## what() ### Description The explanatory string. ### Returns The explanatory string ``` -------------------------------- ### ProfilerContext.__call__ Source: https://docs.nvidia.com/rmm/latest/python/statistics/index.html Use the profiler as a decorator. Wraps a function to profile its memory usage. ```APIDOC ## ProfilerContext.__call__(func) ### Description Use the profiler as a decorator. Wraps the given function to profile memory allocations. ### Method Method call ### Endpoint ProfilerContext.__call__(func) ### Parameters - **func** (Callable) - Required - The function to profile. ### Returns The wrapped function. ``` -------------------------------- ### using host_device_async_resource_ref Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Alias for a cuda::mr::resource_ref with the properties cuda::mr::host_accessible and cuda::mr::device_accessible. ```APIDOC ## Type Alias: host_device_async_resource_ref ### Description Alias for a cuda::mr::resource_ref with the properties cuda::mr::host_accessible and cuda::mr::device_accessible. ### Signature using host_device_async_resource_ref = cuda::mr::resource_ref; ``` -------------------------------- ### Implement Custom Resources with CCCL Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html Replace inheritance from device_memory_resource with direct implementation of the CCCL resource concept. ```cpp // 26.04 #include class my_resource : public rmm::mr::device_memory_resource { void* do_allocate(std::size_t bytes, rmm::cuda_stream_view stream) override { ... } void do_deallocate(void* ptr, std::size_t bytes, rmm::cuda_stream_view stream) override { ... } }; // 26.06 #include class my_resource { public: void* allocate(cuda::stream_ref stream, std::size_t bytes, std::size_t alignment = 256) { ... } void deallocate(cuda::stream_ref stream, void* ptr, std::size_t bytes, std::size_t alignment = 256) noexcept { ... } bool operator==(my_resource const&) const noexcept { ... } bool operator!=(my_resource const& other) const noexcept { return !(*this == other); } // Declare properties via ADL friend: constexpr friend void get_property(my_resource const&, cuda::mr::device_accessible) noexcept {} }; static_assert(cuda::mr::resource_with); ``` -------------------------------- ### Configure PoolMemoryResource with CudaMemoryResource Source: https://docs.nvidia.com/rmm/latest/user_guide/guide/index.html Create a memory pool using CudaMemoryResource as the upstream resource. ```python >>> import rmm >>> pool = rmm.mr.PoolMemoryResource( ... rmm.mr.CudaMemoryResource(), ... initial_pool_size="1GiB", # equivalent to initial_pool_size=2**30 ... maximum_pool_size="4GiB" ... ) >>> rmm.mr.set_current_device_resource(pool) ``` -------------------------------- ### Define Custom Cython Resource Wrappers Source: https://docs.nvidia.com/rmm/latest/migration_guide_2606/index.html For custom shared_resource-based types, define an inline C++ helper that returns an optional device_async_resource_ref. ```cython # my_project/my_resource.pxd from libcpp.optional cimport optional from rmm.librmm.memory_resource cimport ( any_resource, device_accessible, device_async_resource_ref, ) cdef extern from "" nogil: cdef cppclass cpp_MyResource "my_project::MyResource": cpp_MyResource(any_resource[device_accessible] upstream) except + # ... cdef extern from *: """ #include #include #include std::optional make_my_resource_ref(my_project::MyResource& r) { return std::optional( rmm::device_async_resource_ref(r)); } """ optional[device_async_resource_ref] make_my_resource_ref( cpp_MyResource&) except + ``` ```cython # my_project/my_resource.pyx self.c_obj.reset(new cpp_MyResource(upstream_mr.get_mr())) self.c_ref = make_my_resource_ref(deref(self.c_obj)) ``` -------------------------------- ### front_element Source: https://docs.nvidia.com/rmm/latest/cpp/data_containers/index.html Returns the first element from device memory. Synchronizes the stream. ```APIDOC ## front_element ### Description Returns the first element. ### Method N/A (C++ member function) ### Parameters #### Parameters - **stream** (_cuda::stream_ref) - Required - The stream on which to perform the copy ### Throws - **rmm::out_of_range** - exception if the vector is empty. ### Returns The value of the first element ``` -------------------------------- ### host_device_resource_ref Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources Alias for a synchronous resource reference with both host and device accessibility. ```APIDOC ## host_device_resource_ref ### Description Alias for a `cuda::mr::synchronous_resource_ref` with the properties `cuda::mr::host_accessible` and `cuda::mr::device_accessible`. ### Type using host_device_resource_ref = cuda::mr::synchronous_resource_ref ``` -------------------------------- ### system_memory_resource constructor Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resources/index.html Default constructor for system_memory_resource. ```APIDOC ## system_memory_resource() ### Description Default constructor. ``` -------------------------------- ### tracking_resource_adaptor::get_outstanding_allocations_str Source: https://docs.nvidia.com/rmm/latest/cpp/memory_resources/memory_resource_adaptors/index.html Returns a string describing all outstanding allocations. ```APIDOC ## tracking_resource_adaptor::get_outstanding_allocations_str ### Description Gets a string describing all outstanding allocations (pointer, size, optional stack). ### Method std::string get_outstanding_allocations_str() const ### Returns std::string describing outstanding allocations ``` -------------------------------- ### rmm.DeviceBuffer.copy_from_device() Source: https://docs.nvidia.com/rmm/latest/python/rmm Copies from a buffer on device to self. ```APIDOC ## rmm.DeviceBuffer.copy_from_device(cuda_ary, stream=DEFAULT_STREAM) ### Description Copy from a buffer on host to self. ### Parameters - **cuda_ary** (object) - Required - Object to copy from that has __cuda_array_interface__. - **stream** (Stream) - Optional - CUDA stream to use for copying, defaults to the default stream. ### Example ```python >>> import rmm >>> db = rmm.DeviceBuffer(size=5) >>> db2 = rmm.DeviceBuffer.to_device(b"abc") >>> db.copy_from_device(db2) >>> hb = db.copy_to_host() >>> print(hb) array([97, 98, 99, 0, 0], dtype=uint8) ``` ```