### Start Dask CUDA Worker with GPU UUID Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/worker_count.rst This example demonstrates starting a dask cuda worker using a specific GPU UUID provided via the CUDA_VISIBLE_DEVICES environment variable. ```bash $ CUDA_VISIBLE_DEVICES="GPU-dae76d0e-3414-958a-8f3e-fc6682b36f31" \ > dask cuda worker 127.0.0.1:8786 ``` -------------------------------- ### Install Dask-CUDA from Source Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA after cloning the repository and navigating into the source directory. This command installs the package using pip. ```bash python -m pip install . ``` -------------------------------- ### Start Dask Scheduler Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/quickstart.rst Starts a Dask scheduler process. This is a prerequisite for connecting Dask-CUDA workers from the command line. ```bash $ dask scheduler distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:8786 ``` -------------------------------- ### Start Dask CUDA Workers with UCX and Manual Transport Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Starts Dask CUDA workers connected to a UCX scheduler, enabling various transports (TCP, NVLink, InfiniBand, RDMACM) and configuring an RMM pool size. This is for manual worker setup. ```bash $ dask cuda worker ucx://:8786 \ > --enable-tcp-over-ucx \ > --enable-nvlink \ > --enable-infiniband \ > --enable-rdmacm \ > --rmm-pool-size="1GB" ``` -------------------------------- ### Initialize Local CUDA Cluster and Client Source: https://github.com/rapidsai/dask-cuda/blob/main/README.md This snippet shows how to set up a local Dask CUDA cluster and connect a client to it. Ensure dask-cuda and dask are installed. ```python from dask_cuda import LocalCUDACluster from dask.distributed import Client cluster = LocalCUDACluster() client = Client(cluster) ``` -------------------------------- ### Install Dask-CUDA with Conda and UCXX Support Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA and the distributed-ucxx package for accelerated communications with UCX using conda. This command assumes prerequisites are met. ```bash conda install -c rapidsai -c conda-forge dask-cuda cuda-version=13.2 distributed-ucxx ``` -------------------------------- ### Start Dask Scheduler and CUDA Worker with Environment Variable Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/worker_count.rst When using 'dask cuda worker', CUDA_VISIBLE_DEVICES must be set as an environment variable. This example shows starting a scheduler and then a CUDA worker restricted to GPUs 0 and 1. ```bash $ dask scheduler distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:8786 $ CUDA_VISIBLE_DEVICES=0,1 dask cuda worker 127.0.0.1:8786 ``` -------------------------------- ### Install Dask-CUDA and UCXX with Pip Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA and the distributed-ucxx package for accelerated communications using pip. This command is for environments where Conda is not used. ```bash python -m pip install dask-cuda distributed-ucxx ``` -------------------------------- ### Dask Scheduler with Automatic UCX Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Start a Dask scheduler using UCX with automatic configuration. Ensure a CUDA context is created on the scheduler before UCX initialization by setting the environment variable DASK_DISTRIBUTED_UCXX__CREATE_CUDA_CONTEXT=True. ```bash $ DASK_DISTRIBUTED_UCXX__CREATE_CUDA_CONTEXT=True \ > DASK_DISTRIBUTED_UCXX__RMM__POOL_SIZE=1GB \ > dask scheduler --protocol ucx --interface ib0 ``` -------------------------------- ### Start Dask CUDA Worker Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/quickstart.rst Starts a Dask-CUDA worker and connects it to a running Dask scheduler. Replace '127.0.0.1:8786' with the scheduler's address if it's different. ```bash $ dask cuda worker 127.0.0.1:8786 ``` -------------------------------- ### Clone Dask-CUDA Repository Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Clone the Dask-CUDA source code repository from GitHub. This is the first step for installing from source. ```bash git clone https://github.com/rapidsai/dask-cuda.git cd dask-cuda ``` -------------------------------- ### Install Dask-CUDA and UCXX with Pip for CUDA 12 Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA and distributed-ucxx with CUDA 12 specific extras using pip. This ensures compatibility and optimal performance for CUDA 12 environments. ```bash python -m pip install 'dask-cuda[cu12]' 'distributed-ucxx[cu12]' ``` -------------------------------- ### Start Dask Scheduler with UCX and Manual Transport Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Launches a Dask scheduler configured to use UCX with specific transports (TCP, NVLink, InfiniBand, RDMACM) enabled and sets an RMM pool size. This is for manual configuration scenarios. ```bash $ DASK_DISTRIBUTED_UCXX__CUDA_COPY=True \ > DASK_DISTRIBUTED_UCXX__TCP=True \ > DASK_DISTRIBUTED_UCXX__NVLINK=True \ > DASK_DISTRIBUTED_UCXX__INFINIBAND=True \ > DASK_DISTRIBUTED_UCXX__RDMACM=True \ > DASK_DISTRIBUTED_UCXX__RMM__POOL_SIZE=1GB \ > dask scheduler --protocol ucx --interface ib0 ``` -------------------------------- ### Install Dask-CUDA and UCXX with Pip for CUDA 13 Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA and distributed-ucxx with CUDA 13 specific extras using pip. This ensures compatibility and optimal performance for CUDA 13 environments. ```bash python -m pip install 'dask-cuda[cu13]' 'distributed-ucxx[cu13]' ``` -------------------------------- ### Install Dask-CUDA with Pip and CUDA 12 Extras Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA with optional dependencies for CUDA 12 using pip. This ensures a consistent set of dependencies for the specified CUDA version. ```bash python -m pip install 'dask-cuda[cu12]' ``` -------------------------------- ### Install Dask-CUDA with Pip and CUDA 13 Extras Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install Dask-CUDA with optional dependencies for CUDA 13 using pip. This ensures a consistent set of dependencies for the specified CUDA version. ```bash python -m pip install 'dask-cuda[cu13]' ``` -------------------------------- ### Install Dask-CUDA with Pip Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install the latest version of Dask-CUDA using pip. This is suitable for environments outside of Conda. Ensure CUDA Toolkit and NVIDIA drivers are installed separately. ```bash python -m pip install dask-cuda ``` -------------------------------- ### Install Dask-CUDA with Conda Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/install.rst Install the latest version of Dask-CUDA along with a specific CUDA Toolkit version using conda. Ensure NVIDIA drivers and a compatible CUDA Toolkit are installed beforehand. ```bash conda install -c rapidsai -c conda-forge dask-cuda cuda-version=13.2 ``` -------------------------------- ### Dask CUDA Worker with Automatic UCX Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Start Dask CUDA workers with automatic UCX configuration and an RMM pool. The interface='ib0' is specified for RDMACM with InfiniBand. UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda is recommended for optimal performance with InfiniBand. ```bash $ UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda > dask cuda worker ucx://:8786 \ > --rmm-pool-size="14GB" \ > --interface="ib0" ``` -------------------------------- ### dask_cuda.cli.worker Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/api.rst Command-line interface for starting a Dask CUDA worker. This is typically used to launch worker processes that can connect to a Dask scheduler and utilize GPUs. ```APIDOC ## dask_cuda.cli.worker ### Description Command-line interface for starting a Dask CUDA worker. This is typically used to launch worker processes that can connect to a Dask scheduler and utilize GPUs. ### Command `dask cuda worker` ### Usage (Usage details would typically be listed here if available in the source.) ``` -------------------------------- ### Restrict Activity to Specific GPUs with LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/worker_count.rst Use the CUDA_VISIBLE_DEVICES keyword argument to restrict activity to specific GPUs when initializing LocalCUDACluster. This example restricts activity to the first two indexed GPUs (0 and 1). ```python from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="0,1") ``` -------------------------------- ### Initialize Dask Client with UCX via Environment Variables Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Connects a Dask client to a scheduler using UCX, enabling CUDA context creation via environment variables. This is an alternative to using the `with dask.config.set` statement. ```python import os os.environ["UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES"] = "cuda" os.environ["DASK_DISTRIBUTED_UCXX__CREATE_CUDA_CONTEXT"] = "True" from dask.distributed import Client client = Client("ucx://:8786") ``` -------------------------------- ### New Schema UCX Configuration (Environment Variables) Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Set UCX transport options, RMM pool size, and logging level using environment variables with the new schema. Note the double underscore for nested keys. ```bash export DASK_DISTRIBUTED_UCXX__TCP=true export DASK_DISTRIBUTED_UCXX__NVLINK=true export DASK_DISTRIBUTED_UCXX__RMM__POOL_SIZE=1GB ``` -------------------------------- ### Initialize Dask Client with UCX and CUDA Context Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Connects a Dask client to a scheduler using UCX, ensuring a CUDA context is created for GPU operations. This method uses a context manager for configuration. ```python import os os.environ["UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES"] = "cuda" import dask from dask.distributed import Client with dask.config.set({"distributed-ucxx.create_cuda_context": True}): client = Client("ucx://:8786") ``` -------------------------------- ### Initialize Dask Client with UCX and Manual Transport Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Initializes the Dask client for UCX communication, enabling specific transports (TCP, NVLink, InfiniBand, RDMACM) using the `dask_cuda.initialize` utility. This prepares the client to connect to a manually configured UCX cluster. ```python from dask.distributed import Client from dask_cuda.initialize import initialize initialize( enable_tcp_over_ucx=True, enable_nvlink=True, enable_infiniband=True, enable_rdmacm=True, ) client = Client("ucx://:8786") ``` -------------------------------- ### Configure Dask-CUDA UCX with Forkserver Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst This snippet shows how to ensure the forkserver is running and configure Dask to use it for multiprocessing. This is crucial for stability when using a large number of workers with Dask-CUDA's UCX integration. ```python import dask if __name__ == "__main__": import multiprocessing.forkserver as f f.ensure_running() with dask.config.set( {"distributed.worker.multiprocessing-method": "forkserver"} ): run_analysis(...) ``` -------------------------------- ### dask_cuda.initialize.initialize Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/api.rst Initializes the Dask client for use with CUDA. This function sets up the necessary environment for Dask to leverage GPUs for computation. ```APIDOC ## dask_cuda.initialize.initialize ### Description Initializes the Dask client for use with CUDA. This function sets up the necessary environment for Dask to leverage GPUs for computation. ### Function `dask_cuda.initialize()` ### Parameters (Parameters would typically be listed here if available in the source.) ### Returns (Return value details would typically be listed here if available in the source.) ``` -------------------------------- ### Print GPU CPU Affinity Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/troubleshooting.rst This script helps diagnose CPU affinity issues by querying and printing the CPU affinity for each GPU using the pynvml library. It requires the pynvml and multiprocessing libraries. ```python # print_affinity.py import math from multiprocessing import cpu_count import pynvml pynvml.nvmlInit() for i in range(pynvml.nvmlDeviceGetCount()): handle = pynvml.nvmlDeviceGetHandleByIndex(i) cpu_affinity = pynvml.nvmlDeviceGetCpuAffinity(handle, math.ceil(cpu_count() / 64)) print(f"GPU {i}: list(cpu_affinity)") ``` -------------------------------- ### Running Local CUDA Merge with UCX Protocol Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/best-practices.rst This command initiates a local CUDA merge operation using the UCX protocol with automatic transport selection. It specifies the devices to use, the protocol, the number of rows, and the RMM pool size. ```bash python local_cudf_merge.py -d 0,1 -p ucx -c 50_000_000 --rmm-pool-size 30GB ``` -------------------------------- ### Running Local CUDA Merge with TCP Protocol Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/best-practices.rst This command initiates a local CUDA merge operation using the TCP protocol. It specifies the devices to use, the protocol, the number of rows, and the RMM pool size. ```bash python local_cudf_merge.py -d 0,1 -p tcp -c 50_000_000 --rmm-pool-size 30GB ``` -------------------------------- ### Configure LocalCUDACluster with RMM Pool Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/best-practices.rst Set up a LocalCUDACluster with a specified RMM pool size and UCX protocol for efficient GPU memory management and accelerated networking. This is recommended for performance-critical workflows. ```python from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="0,1", protocol="ucx", rmm_pool_size="30GB") ``` -------------------------------- ### Legacy Schema UCX Configuration (Environment Variables) Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Set UCX transport options, RMM pool size, and logging level using environment variables with the legacy schema. This schema may be removed in future versions. ```bash export DASK_DISTRIBUTED__COMM__UCX__TCP=true export DASK_DISTRIBUTED__COMM__UCX__NVLINK=true export DASK_DISTRIBUTED__RMM__POOL_SIZE=1GB ``` -------------------------------- ### Connect Client to Command-Line Cluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/quickstart.rst Connects a Dask.distributed Client to a Dask cluster managed via the command line. Ensure the address '127.0.0.1:8786' matches your scheduler's address. ```python from dask.distributed import Client client = Client("127.0.0.1:8786") ``` -------------------------------- ### New Schema UCX Configuration (YAML) Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Configure UCX transports, RMM pool size, and environment logging using the new YAML schema. This is useful for benchmarking specific transports. ```yaml distributed-ucxx: tcp: true nvlink: true infiniband: false cuda-copy: true create-cuda-context: true multi-buffer: false environment: log-level: "info" rmm: pool-size: "1GB" ``` -------------------------------- ### dask_cuda.cli.config Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/api.rst Command-line interface for configuring Dask CUDA. This tool allows users to manage and inspect the configuration settings for Dask CUDA. ```APIDOC ## dask_cuda.cli.config ### Description Command-line interface for configuring Dask CUDA. This tool allows users to manage and inspect the configuration settings for Dask CUDA. ### Command `dask cuda config` ### Usage (Usage details would typically be listed here if available in the source.) ``` -------------------------------- ### Legacy UCX Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Shows the older configuration keys for UCX transports and RMM pool size in Dask. This configuration is still valid for convenience but may be removed in the future. ```python import dask dask.config.set({ "distributed.comm.ucx.tcp": True, "distributed.comm.ucx.nvlink": True, "distributed.rmm.pool-size": "1GB" }) ``` -------------------------------- ### Enable cuDF Spilling with LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Enable native cuDF spilling when deploying a LocalCUDACluster. Requires importing Client and LocalCUDACluster. ```python >>> from distributed import Client >>> from dask_cuda import LocalCUDACluster >>> cluster = LocalCUDACluster(n_workers=10, enable_cudf_spill=True) >>> client = Client(cluster) ``` -------------------------------- ### Configure UCX Transports and RMM Pool Size Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Sets Dask configuration to enable TCP and NVLink transports for UCX, and specifies an RMM pool size. This is useful for manual transport selection and resource management. ```python import dask dask.config.set({ "distributed-ucxx.tcp": True, "distributed-ucxx.nvlink": True, "distributed-ucxx.rmm.pool-size": "1GB" }) ``` -------------------------------- ### LocalCUDACluster with Automatic UCX Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Connect a Dask client to a cluster with automatically configured UCX and an RMM pool. This method is recommended for routine deployments and requires Dask-CUDA 22.02+ and UCX 1.11.1+. ```python from dask.distributed import Client from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster( protocol="ucx", interface="ib0", rmm_pool_size="1GB" ) client = Client(cluster) ``` -------------------------------- ### LocalCUDACluster with Manual UCX Transport Configuration Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/ucx.rst Configure LocalCUDACluster with explicit UCX transport flags for benchmarking specific transports. This allows enabling all supported transports manually. ```python from dask.distributed import Client from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster( protocol="ucx", interface="ib0", enable_tcp_over_ucx=True, enable_nvlink=True, enable_infiniband=True, enable_rdmacm=True, rmm_pool_size="1GB" ) ``` -------------------------------- ### Configure Dask-CUDA Device Memory Limit Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Set the device memory limit for Dask-CUDA spilling. Accepts bytes, string memory size, or a percentage. ```python from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster(device_memory_limit=50000) # spilling after 50000 bytes cluster = LocalCUDACluster(device_memory_limit="5GB") # spilling after 5 GB cluster = LocalCUDACluster(device_memory_limit=0.3) # spilling after 30% memory utilization ``` -------------------------------- ### Configure Dask CUDA Worker Device Memory Limit Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Control Dask CUDA worker spilling using the --device-memory-limit command-line argument. ```bash $ dask cuda worker --device-memory-limit 50000 $ dask cuda worker --device-memory-limit 5GB $ dask cuda worker --device-memory-limit 0.3 $ dask cuda worker --device-memory-limit 0 ``` -------------------------------- ### Legacy Schema UCX Configuration (YAML) Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/ucx.rst Configure UCX transports, RMM pool size, and environment logging using the legacy YAML schema. This schema may be removed in future versions. ```yaml distributed: comm: ucx: tcp: true nvlink: true infiniband: false cuda-copy: true create-cuda-context: true multi-buffer: false environment: log-level: "info" rmm: pool-size: "1GB" ``` -------------------------------- ### Enable cuDF Spilling Statistics with Dask CUDA Worker Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Enable cuDF spilling statistics for Dask CUDA workers using --enable-cudf-spill and --cudf-spill-stats 1. ```bash $ dask cuda worker --enable-cudf-spill --cudf-spill-stats 1 ``` -------------------------------- ### Enable cuDF Spilling with Dask CUDA Worker Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Enable native cuDF spilling for Dask CUDA workers using the --enable-cudf-spill command-line argument. ```bash $ dask cuda worker --enable-cudf-spill ``` -------------------------------- ### Initialize Dask-CUDA LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/best-practices.rst Configure the number of workers or specify visible GPUs for the LocalCUDACluster. This is useful for setting up distributed computing environments on multi-GPU machines. ```python from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster(n_workers=2) # will use GPUs 0,1 cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="3,4") # will use GPUs 3,4 ``` -------------------------------- ### Set CUDA Device Order for Correct Indexing Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/troubleshooting.rst Set the CUDA_DEVICE_ORDER environment variable to PCI_BUS_ID to ensure correct GPU indexing based on PCI Bus ID, especially in systems with complex GPU configurations like DGX Station A100. ```bash $ CUDA_DEVICE_ORDER=PCI_BUS_ID python $ CUDA_DEVICE_ORDER=PCI_BUS_ID ipython $ CUDA_DEVICE_ORDER=PCI_BUS_ID jupyter lab $ CUDA_DEVICE_ORDER=PCI_BUS_ID dask-cuda-worker ... ``` -------------------------------- ### Enable cuDF Spilling Statistics with LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Enable cuDF spilling statistics collection when deploying a LocalCUDACluster by setting cudf_spill_stats to 1. ```python >>> cluster = LocalCUDACluster(n_workers=10, enable_cudf_spill=True, cudf_spill_stats=1) ``` -------------------------------- ### Print cuDF Spill Statistics Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Submit a task to a Dask client to print the global spill statistics from cuDF. ```python def spill_info(): from cudf.core.buffer.spill_manager import get_global_manager print(get_global_manager().statistics) client.submit(spill_info) ``` -------------------------------- ### Select GPUs by UUID with LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/worker_count.rst LocalCUDACluster can accept GPU UUIDs for the CUDA_VISIBLE_DEVICES argument, allowing for selection of specific GPUs by their unique identifiers. First, acquire UUIDs using nvidia-smi -L. ```python cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="GPU-dae76d0e-3414-958a-8f3e-fc6682b36f31") ``` -------------------------------- ### Manually Specify CUDA Visible Devices Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/troubleshooting.rst When using LocalCUDACluster, explicitly define the CUDA_VISIBLE_DEVICES list to match the desired GPU PCI Bus ID ordering, particularly when the default nvidia-smi indexing may not align. ```python from dask_cuda import LocalCUDACluster cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES=[0, 1, 2, 4]) ``` -------------------------------- ### Disable Dask-CUDA Device Memory Spilling Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/spilling.rst Disable Dask-CUDA's device memory spilling by setting the limit to 0. ```python cluster = LocalCUDACluster(device_memory_limit=0) # spilling disabled ``` -------------------------------- ### dask_cuda.LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/api.rst Manages a local cluster of CUDA-enabled Dask workers. This class provides methods for creating, managing, and interacting with a Dask cluster that utilizes GPUs. ```APIDOC ## dask_cuda.LocalCUDACluster ### Description Manages a local cluster of CUDA-enabled Dask workers. This class provides methods for creating, managing, and interacting with a Dask cluster that utilizes GPUs. ### Class `dask_cuda.LocalCUDACluster` ### Members (Members are documented via autoclass directive in the source, details would typically be listed here if available in the source.) ``` -------------------------------- ### Set Number of Workers with LocalCUDACluster Source: https://github.com/rapidsai/dask-cuda/blob/main/docs/source/examples/worker_count.rst The n_workers argument in LocalCUDACluster restricts activity to the first N GPUs listed in CUDA_VISIBLE_DEVICES. It can be used independently or with CUDA_VISIBLE_DEVICES. ```python cluster = LocalCUDACluster(n_workers=2) # will use GPUs 0,1 cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES="3,4,5", n_workers=2) # will use GPUs 3,4 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.