### Install Example Parquet File Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/parquet_io/CMakeLists.txt Installs a sample 'example.parquet' file to the installation directory for examples. ```cmake # Install the example.parquet file install(FILES ${CMAKE_CURRENT_LIST_DIR}/example.parquet DESTINATION bin/examples/libcudf/parquet_io) ``` -------------------------------- ### Install Example Parquet File Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/hybrid_scan_io/CMakeLists.txt Installs the example.parquet file to the specified destination directory for use with the hybrid scan IO examples. ```cmake install(FILES ${CMAKE_CURRENT_LIST_DIR}/example.parquet DESTINATION bin/examples/libcudf/hybrid_scan_io ) ``` -------------------------------- ### Install Data File Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/strings/CMakeLists.txt Installs a CSV data file required by one of the strings examples. ```cmake install(FILES ${CMAKE_CURRENT_LIST_DIR}/names.csv DESTINATION bin/examples/libcudf/strings) ``` -------------------------------- ### Install Info CSV Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/string_transforms/CMakeLists.txt Installs an info.csv file to the specified destination directory, likely containing metadata or configuration for the string transformer examples. ```cmake install(FILES ${CMAKE_CURRENT_LIST_DIR}/info.csv DESTINATION bin/examples/libcudf/string_transformers ) ``` -------------------------------- ### Add Strings Example Executables Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/strings/CMakeLists.txt Calls the add_strings_example function to build and install specific strings example executables. ```cmake add_strings_example(libcudf_apis libcudf_apis.cpp) add_strings_example(custom_with_malloc custom_with_malloc.cu) add_strings_example(custom_prealloc custom_prealloc.cu) add_strings_example(custom_optimized custom_optimized.cu) ``` -------------------------------- ### Get Build Script Help Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Display help information for the build.sh script to see all available options and libraries that can be installed. ```bash ./build.sh --help ``` -------------------------------- ### Quick Start with RayEngine Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/index.md Demonstrates using the RayEngine for GPU-accelerated Polars queries. This example scans Parquet files, filters data, groups by customer ID, aggregates amounts, and collects the result using the specified engine. ```python import polars as pl from cudf_polars.engine.ray import RayEngine query = ( pl.scan_parquet("/data/dataset/*.parquet") .filter(pl.col("amount") > 100) .group_by("customer_id") .agg(pl.col("amount").sum()) ) with RayEngine() as engine: result = query.collect(engine=engine) ``` -------------------------------- ### Add Parquet I/O Example Executables Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/parquet_io/CMakeLists.txt Adds the main Parquet I/O example and a multithreaded version using the defined function. These executables are built and installed. ```cmake add_parquet_io_example(parquet_io parquet_io.cpp) add_parquet_io_example(parquet_io_multithreaded parquet_io_multithreaded.cpp) ``` -------------------------------- ### Compile and Execute libcudf String Transforms Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/string_transforms/README.md This bash script configures, builds, and executes the libcudf string transforms C++ example. Ensure you have the necessary build tools and libcudf installed. ```bash # Configure project cmake -S . -B build/ # Build cmake --build build/ --parallel $PARALLEL_LEVEL # Execute build/output info.csv output.csv 100000 ``` -------------------------------- ### Install cudf Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_pandas/benchmarks.md Installs the cudf library for GPU acceleration. ```bash pip install cudf-cu12 ``` -------------------------------- ### Compile and Execute libcudf 1BRC Examples Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/billion_rows/README.md Shows the commands to configure, build, and execute the three different C++ examples for the 1 billion row challenge using libcudf. ```bash # Configure project cmake -S . -B build/ # Build cmake --build build/ --parallel $PARALLEL_LEVEL # Execute build/brc input.txt # Execute in chunked mode with 25 chunks (default) build/brc_chunks input.txt 25 # Execute in pipeline mode with 25 chunks and 2 threads (defaults) build/brc_pipeline input.txt 25 2 ``` -------------------------------- ### Compile and Execute libcudf C++ Examples Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/strings/README.md Instructions for configuring, building, and executing the libcudf C++ string column examples using CMake. Ensure the PARALLEL_LEVEL environment variable is set for build parallelism. ```bash # Configure project cmake -S . -B build/ # Build cmake --build build/ --parallel $PARALLEL_LEVEL # Execute build/libcudf_apis names.csv --OR-- build/custom_with_malloc names.csv --OR-- build/custom_prealloc names.csv --OR-- build/custom_optimized names.csv ``` -------------------------------- ### Install pre-commit using Pip Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Install the pre-commit tool using pip. This tool automates code linting and formatting checks. ```bash pip install pre-commit ``` -------------------------------- ### Add String Transforms Examples Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/string_transforms/CMakeLists.txt Adds several string transform examples to the build system using the defined function. Each example is compiled from a specified C++ source file. ```cmake add_string_transforms_example(compute_checksum_jit compute_checksum_jit.cpp) add_string_transforms_example(extract_email_jit extract_email_jit.cpp) add_string_transforms_example(extract_email_precompiled extract_email_precompiled.cpp) add_string_transforms_example(format_phone_jit format_phone_jit.cpp) add_string_transforms_example(format_phone_precompiled format_phone_precompiled.cpp) add_string_transforms_example(localize_phone_jit localize_phone_jit.cpp) add_string_transforms_example(localize_phone_precompiled localize_phone_precompiled.cpp) ``` -------------------------------- ### Verify Nsight Systems Installation Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/PROFILING.md Check if the NVIDIA Nsight Systems command-line tool is installed and accessible. ```bash nsys --version ``` -------------------------------- ### Pseudocode Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Use the @code{.pseudo} tag to include pseudocode examples for clarity when actual code is not necessary. ```pseudo s = int column of [ 1, 2, null, 4 ] r = fill( s, [1, 2], 0 ) r is now [ 1, 0, 0, 4 ] ``` -------------------------------- ### Add Parquet I/O Example Executable Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/parquet_io/CMakeLists.txt A CMake function to add a Parquet I/O example executable. It links against cudf, nvtx3, and the utility library, and installs the executable to a specific directory. ```cmake # Creates a parquet I/O example executable and installs it. # # Arguments: target_name - Name of the CMake target and resulting executable source_file - Source # file to compile for the example function(add_parquet_io_example target_name source_file) add_executable(${target_name} ${source_file}) target_link_libraries( ${target_name} PRIVATE cudf::cudf $ $ ) target_compile_features(${target_name} PRIVATE cxx_std_20) install(TARGETS ${target_name} DESTINATION bin/examples/libcudf/parquet_io) endfunction() ``` -------------------------------- ### Free Function Documentation with @code and @throw Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Documents a free function, including a brief and detailed description, an inline code example, exception specification, template parameters, and parameter descriptions. ```cpp /** * @brief Short, one line description of this free function * * @ingroup optional_predefined_group_id * * A detailed description must start after a blank line. * * @code * template * struct myfunctor { * bool operator()(T input) { return input % 2 > 0; } * }; * free_function(myfunctor{},12); * @endcode * * @throw cudf::logic_error if `input_argument` is negative or zero * * @tparam functor_type The type of the functor * @tparam input_type The datatype of the input argument * * @param[in] functor The functor to be called on the input argument * @param[in] input_argument The input argument passed into the functor * @return The result of calling the functor on the input argument */ template bool free_function(functor_type functor, input_type input_argument) { CUDF_EXPECTS( input_argument > 0, "input_argument must be positive"); return functor(input_argument); } ``` -------------------------------- ### CMakeLists.txt Configuration for Nested Types Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/nested_types/CMakeLists.txt This CMakeLists.txt file configures the build for the nested types example. It sets up CUDA architecture, fetches dependencies, and defines the build target. ```cmake # cmake-format: off # SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION. # SPDX-License-Identifier: Apache-2.0 # cmake-format: on cmake_minimum_required(VERSION 4.0 FATAL_ERROR) include(../set_cuda_architecture.cmake) # initialize cuda architecture rapids_cuda_init_architectures(nested_types) project( nested_types VERSION 0.0.1 LANGUAGES CXX CUDA ) include(../fetch_dependencies.cmake) include(rapids-cmake) rapids_cmake_build_type("Release") # For now, disable CMake's automatic module scanning for C++ files. There is an sccache bug in the # version RAPIDS uses in CI that causes it to handle the resulting -M* flags incorrectly with # gcc>=14. We can remove this once we upgrade to a newer sccache version. set(CMAKE_CXX_SCAN_FOR_MODULES OFF) # Configure your project here add_executable(deduplication deduplication.cpp) target_link_libraries(deduplication PRIVATE cudf::cudf) target_compile_features(deduplication PRIVATE cxx_std_20) install(TARGETS deduplication DESTINATION bin/examples/libcudf/nested_types) install(FILES ${CMAKE_CURRENT_LIST_DIR}/example.json DESTINATION bin/examples/libcudf/nested_types) ``` -------------------------------- ### Single-GPU Setup with SPMDEngine Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/spmd_engine.md Demonstrates how to use SPMDEngine for a single GPU. The engine is created, and a Polars query is executed and collected using the engine. ```python import polars as pl from cudf_polars.engine.spmd import SPMDEngine with SPMDEngine() as engine: result = ( pl.scan_parquet("/data/dataset/*.parquet") .filter(pl.col("amount") > 100) .group_by("customer_id") .agg(pl.col("amount").sum()) .collect(engine=engine) ) ``` -------------------------------- ### CUDf Parquet Inspect CMakeLists.txt Configuration Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/parquet_inspect/CMakeLists.txt Configures the build for the parquet_inspect example. It sets up CUDA architecture, includes necessary dependencies, defines the library and executable targets, and specifies installation paths. ```cmake # cmake-format: off # SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION. # SPDX-License-Identifier: Apache-2.0 # cmake-format: on cmake_minimum_required(VERSION 4.0 FATAL_ERROR) include(../set_cuda_architecture.cmake) # initialize cuda architecture rapids_cuda_init_architectures(parquet_inspect) project( parquet_inspect VERSION 0.0.1 LANGUAGES CXX CUDA ) include(../fetch_dependencies.cmake) include(rapids-cmake) rapids_cmake_build_type("Release") # For now, disable CMake's automatic module scanning for C++ files. There is an sccache bug in the # version RAPIDS uses in CI that causes it to handle the resulting -M* flags incorrectly with # gcc>=14. We can remove this once we upgrade to a newer sccache version. set(CMAKE_CXX_SCAN_FOR_MODULES OFF) add_library(parquet_inspect_utils OBJECT parquet_inspect_utils.cpp) target_compile_features(parquet_inspect_utils PRIVATE cxx_std_20) target_link_libraries(parquet_inspect_utils PRIVATE cudf::cudf) # Build and install parquet_inspect add_executable(parquet_inspect parquet_inspect.cpp) target_link_libraries( parquet_inspect PRIVATE cudf::cudf $ $ ) target_compile_features(parquet_inspect PRIVATE cxx_std_20) install(TARGETS parquet_inspect DESTINATION bin/examples/libcudf/parquet_inspect) # Install the example.parquet file install(FILES ${CMAKE_CURRENT_LIST_DIR}/example.parquet DESTINATION bin/examples/libcudf/parquet_inspect ) ``` -------------------------------- ### Member Function Documentation with @code Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Documents a member function using Doxygen tags, including a detailed description, an inline code example, parameter descriptions, and return value. ```cpp /** * @brief Short, one line description of the member function * * A more detailed description of what this function does and what * its logic does. * * @code * example_class inst; * inst.set_my_int(5); * int output = inst.complicated_function(1,dptr,fptr); * @endcode * * @param[in] first This parameter is an input parameter to the function * @param[in,out] second This parameter is used both as an input and output * @param[out] third This parameter is an output of the function * * @return The result of the complex function */ T complicated_function(int first, double* second, float* third) { // Do not use doxygen-style block comments // for code logic documentation. } ``` -------------------------------- ### Install cuDF Targets Source: https://github.com/rapidsai/cudf/blob/main/cpp/CMakeLists.txt Configures installation paths for cuDF libraries and headers, including version configuration and export sets for components. ```cmake rapids_cmake_install_lib_dir(lib_dir) include(CPack) include(GNUInstallDirs) set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME cudf) install( TARGETS cudf DESTINATION ${lib_dir} EXPORT cudf-exports ) install(FILES ${CUDF_BINARY_DIR}/include/cudf/version_config.hpp DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/cudf ) set(_components_export_string) if(TARGET cudftestutil) install( TARGETS cudftest_default_stream cudftestutil cudftestutil_impl DESTINATION ${lib_dir} EXPORT cudf-testing-exports ) set(_components_export_string COMPONENTS testing COMPONENTS_EXPORT_SET cudf-testing-exports) endif() install(DIRECTORY ${CUDF_SOURCE_DIR}/include/cudf ${CUDF_SOURCE_DIR}/include/cudf_test ${CUDF_SOURCE_DIR}/include/nvtext DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} ) # In standalone DSO mode, rmm and rapids_logger are absorbed into libcudf via whole-archive. ``` -------------------------------- ### Start Docker Container with GPU Access Source: https://github.com/rapidsai/cudf/blob/main/java/ci/README.md Starts a Docker container from the previously built cuDF image, enabling GPU access for the build process. ```bash nvidia-docker run -it cudf-build:12.9.1-devel-rocky8 bash ``` -------------------------------- ### Install cuDF Headers Source: https://github.com/rapidsai/cudf/blob/main/cpp/CMakeLists.txt Installs cuDF, rmm, rapids_logger, and nvtx3 headers. This is typically done when CUDF_INSTALL_LIBRARY_DEPS is not set. ```cmake if(NOT CUDF_INSTALL_LIBRARY_DEPS) if(rmm_SOURCE_DIR) install(DIRECTORY ${rmm_SOURCE_DIR}/cpp/include/rmm DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) install(DIRECTORY ${rmm_BINARY_DIR}/include/rmm DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) endif() if(rapids_logger_SOURCE_DIR) install(DIRECTORY ${rapids_logger_SOURCE_DIR}/include/rapids_logger DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} ) endif() if(nvtx3_SOURCE_DIR) install(DIRECTORY ${nvtx3_SOURCE_DIR}/include/nvtx3 DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) endif() endif() ``` -------------------------------- ### Install cuStreamz (Nightly) Source: https://github.com/rapidsai/cudf/blob/main/python/custreamz/README.md Installs the cuStreamz library and its Kafka integration using conda from the rapidsai-nightly channel for the latest development version. ```bash conda install -c rapidsai-nightly cudf_kafka custreamz ``` -------------------------------- ### Install pre-commit hooks to run on commit Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Set up pre-commit hooks to automatically execute before each git commit. This ensures code quality is maintained consistently. ```bash pre-commit install ``` -------------------------------- ### Output Memory Allocation Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DEVELOPER_GUIDE.md APIs returning allocated memory must accept a `rmm::device_async_resource_ref`. This example shows how a function allocating output memory takes a memory resource reference. ```c++ std::unique_ptr returns_output_memory( ..., rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()); void does_not_allocate_output_memory(...); ``` -------------------------------- ### SPMDEngine with Default Options Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md Initialize an SPMDEngine using default configurations. This is a simpler way to start distributed processing when custom options are not required. ```python with SPMDEngine() as engine: result = pl.scan_parquet(...).collect(engine=engine) ``` -------------------------------- ### Dask Expressions with Query Planning Source: https://github.com/rapidsai/cudf/blob/main/docs/dask_cudf/source/index.rst Example demonstrating Dask Expressions with automatic query planning enabled. This code will benefit from predicate pushdown when computed. ```python import dask.dataframe as dd df = dd.read_parquet("/my/parquet/dataset/") result = df.sort_values('B')['A'] ``` -------------------------------- ### Valid Query Symmetry Example Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/spmd_engine.md An example demonstrating a correct query structure where all ranks execute the same sequence of operations, ensuring query symmetry and avoiding deadlocks. ```python # OK: every rank runs the same query in the same order. with SPMDEngine() as engine: result = ( pl.scan_parquet("/data/*.parquet") .group_by("customer_id") .agg(pl.col("amount").sum()) .collect(engine=engine) ) ``` -------------------------------- ### Fixture with Simple Parametrization Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/developer_guide/testing.md Use fixtures for complex initialization when the parametrization itself is simple. This example shows a fixture parametrized by strings 'a' and 'b'. ```python @pytest.fixture(params=["a", "b"]) def foo(request): if request.param == "a": # Some complex initialization elif request.param == "b": # Some other complex initialization ``` -------------------------------- ### Install pre-commit using Conda Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Install the pre-commit tool using the conda package manager. This tool helps maintain code consistency by running linters and formatters before committing. ```bash conda install -c conda-forge pre-commit ``` -------------------------------- ### Add Billion Rows Example Executables Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/billion_rows/CMakeLists.txt Adds three billion rows example executables: 'brc', 'brc_chunks', and 'brc_pipeline', using their respective C++ source files. ```cmake add_billion_rows_example(brc brc.cpp) add_billion_rows_example(brc_chunks brc_chunks.cpp) add_billion_rows_example(brc_pipeline brc_pipeline.cpp) ``` -------------------------------- ### Import Libraries and Print cuDF Version Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/performance-comparisons/performance-comparisons.ipynb Imports necessary libraries for performance comparison and prints the installed cuDF version. Ensure cuDF is installed to run this code. ```python import timeit import cudf import matplotlib.pyplot as plt import numpy as np import pandas as pd print(f"{cudf.__version__=}") ``` -------------------------------- ### Dask cuDF Quick-start Example Source: https://github.com/rapidsai/cudf/blob/main/python/dask_cudf/README.md This example demonstrates a common Dask cuDF use case for single-node multi-GPU data processing. It sets up a GPU-aware Dask cluster, registers 'cudf' as the dataframe backend, reads data, performs a groupby aggregation, and computes the result. Ensure CUDA_VISIBLE_DEVICES is set appropriately for your available GPUs. ```python import dask import dask.dataframe as dd from dask_cuda import LocalCUDACluster from distributed import Client if __name__ == "__main__": # Define a GPU-aware cluster to leverage multiple GPUs client = Client( LocalCUDACluster( CUDA_VISIBLE_DEVICES="0,1", # Use two workers (on devices 0 and 1) rmm_pool_size=0.9, # Use 90% of GPU memory as a pool for faster allocations enable_cudf_spill=True, # Improve device memory stability local_directory="/fast/scratch/", # Use fast local storage for spilling ) ) # Set the default dataframe backend to "cudf" dask.config.set({"dataframe.backend": "cudf"}) # Create your DataFrame collection from on-disk # or in-memory data df = dd.read_parquet("/my/parquet/dataset/") # Use cudf-like syntax to transform and/or query your data query = df.groupby('item')['price'].mean() # Compute, persist, or write out the result query.head() ``` -------------------------------- ### Add Hybrid Scan Example Executable Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/hybrid_scan_io/CMakeLists.txt A CMake function to add an executable for a hybrid scan example. It links necessary libraries and installs the executable to a specific directory. ```cmake function(add_hybrid_scan_example target_name source_file) add_executable(${target_name} ${source_file}) target_link_libraries( ${target_name} PRIVATE cudf::cudf $ $ ) target_compile_features(${target_name} PRIVATE cxx_std_20) install(TARGETS ${target_name} DESTINATION bin/examples/libcudf/hybrid_scan_io) endfunction() ``` -------------------------------- ### Configure Streaming Engine with StreamingOptions Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/options.md Instantiate StreamingOptions with specific parameters and use it to construct a RayEngine for processing data. This example demonstrates setting thread count, fallback mode, and spill device limit. ```python import polars as pl from cudf_polars.engine.options import StreamingOptions from cudf_polars.engine.ray import RayEngine opts = StreamingOptions( num_streaming_threads=8, fallback_mode="silent", spill_device_limit="70%", ) with RayEngine.from_options(opts) as engine: result = ( pl.scan_parquet("/data/*.parquet") .filter(pl.col("amount") > 100) .group_by("customer_id") .agg(pl.col("amount").sum()) .collect(engine=engine) ) ``` -------------------------------- ### Define Strings Example Target Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/strings/CMakeLists.txt Defines a CMake function to create an executable for a strings example. It sets C++ standard, CUDA flags, links against CUDF and NVTX, and installs the executable. ```cmake function(add_strings_example target_name source_file) add_executable(${target_name} ${source_file}) target_compile_features(${target_name} PRIVATE cxx_std_20) target_compile_options(${target_name} PRIVATE "<$:${CUDF_CUDA_FLAGS}>") target_link_libraries(${target_name} PRIVATE cudf::cudf $) install(TARGETS ${target_name} DESTINATION bin/examples/libcudf/strings) endfunction() ``` -------------------------------- ### Define String Transforms Example Function Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/string_transforms/CMakeLists.txt Defines a CMake function to add and configure an executable for string transform examples. It sets C++20 standard, CUDA flags, links against cudf and nvtx, and installs the executable. ```cmake function(add_string_transforms_example target_name source_file) add_executable(${target_name} ${source_file}) target_compile_features(${target_name} PRIVATE cxx_std_20) target_compile_options(${target_name} PRIVATE "<$:${CUDF_CUDA_FLAGS}>") target_link_libraries(${target_name} PRIVATE cudf::cudf $) install(TARGETS ${target_name} DESTINATION bin/examples/libcudf/string_transformers) endfunction() ``` -------------------------------- ### Construct and Use RayEngine with Default Options Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md This snippet shows the simplest way to construct and use RayEngine with its default settings. It automatically initializes Ray if not already running and sets up the necessary infrastructure for distributed processing. ```python with RayEngine() as engine: result = pl.scan_parquet(...).collect(engine=engine) ``` -------------------------------- ### Add Billion Rows Example Function Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/billion_rows/CMakeLists.txt Defines a CMake function to add an executable for a billion rows example. It compiles the source file, links necessary libraries (cuDF, NVTX, and the groupby_results object library), sets C++ standard to C++20, and installs the executable to a specific directory. ```cmake # Creates a billion rows example executable and installs it. # # Arguments: target_name - Name of the CMake target and resulting executable source_file - Source # file to compile for the example function(add_billion_rows_example target_name source_file) add_executable(${target_name} ${source_file}) target_link_libraries( ${target_name} PRIVATE cudf::cudf $ $ ) target_compile_features(${target_name} PRIVATE cxx_std_20) install(TARGETS ${target_name} DESTINATION bin/examples/libcudf/billion_rows) endfunction() ``` -------------------------------- ### Configuring RayEngine with Options Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md Use RayEngine.from_options() with a StreamingOptions object for unified configuration, or pass raw dictionaries to the __init__ for fine-grained control. ```python from rapidsmpf.config import Options with RayEngine( rapidsmpf_options=Options(num_streaming_threads=8), executor_options={"num_py_executors": 2}, executor_options={"max_rows_per_partition": 500_000}, engine_options={"raise_on_fail": True}, ray_init_options={"num_cpus": 4}, ) as engine: ... ``` -------------------------------- ### Ray Lifecycle Management with Existing Ray Cluster Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md This example demonstrates how to use RayEngine when Ray is already initialized. It attaches to the existing cluster and avoids calling ray.shutdown() on exit, ensuring proper resource management. ```python import ray import polars as pl from cudf_polars.engine.ray import RayEngine ray.init(address="auto") try: with RayEngine() as engine: result = pl.scan_parquet(...).collect(engine=engine) finally: ray.shutdown() ``` -------------------------------- ### View Benchmark Options Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/BENCHMARKING.md Display the available command-line options for a specific NVBench benchmark executable. ```bash ./cpp/build/latest/benchmarks/_NVBENCH --help ``` -------------------------------- ### File and Class Documentation Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Demonstrates Doxygen block comment styles for file and class documentation, including brief and detailed descriptions, template parameters, and member documentation. ```cpp /** * @file source_file.cpp * @brief Description of source file contents * * Longer description of the source file contents. */ /** * @brief One line description of the class * * @ingroup optional_predefined_group_id * * Longer, more detailed description of the class. * * @tparam T Short description of each template parameter * @tparam U Short description of each template parameter */ template class example_class { ``` -------------------------------- ### Basic Typed Test Setup in C++ Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/TESTING.md Demonstrates the fundamental structure for setting up a typed test fixture and test suite in C++ using libcudf's custom type list. ```c++ // Fixture must be a template template class TypedTestFixture : cudf::test::BaseFixture {...}; using TestTypes = cudf::test:Types; // Notice custom cudf type list type TYPED_TEST_SUITE(TypedTestFixture, TestTypes); TYPED_TEST(TypedTestFixture, FirstTest){ // Access the current type using `TypeParam` using T = TypeParam; } ``` -------------------------------- ### Install Relocatable Tests Source: https://github.com/rapidsai/cudf/blob/main/cpp/libcudf_kafka/tests/CMakeLists.txt Installs the test executables in a relocatable manner to the specified destination within the testing component set. This ensures tests can be found and run after installation. ```cmake rapids_test_install_relocatable(INSTALL_COMPONENT_SET testing DESTINATION bin/gtests/libcudf_kafka) ``` -------------------------------- ### Install cudf_polars Package in Editable Mode Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/developer_docs.md After building cudf, navigate to the cudf_polars directory and install the package in editable mode. This command installs the package without building isolated dependencies. ```sh cd cudf/python/cudf_polars pip install --no-build-isolation --no-deps -e . ``` -------------------------------- ### Numpy Style Docstring Example Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/developer_guide/documentation.md This example demonstrates the NumPy docstring format used for cuDF, including sections for parameters, returns, raises, and examples. Ensure docstrings are validated using ruff pydocstyle rules for consistency. ```python class A: """Brief description of A. Longer description of A. Parameters ---------- x : int Description of x, the first constructor parameter. """ def __init__(self, x: int): pass def foo(self, bar: str): """Short description of foo. Longer description of foo. Parameters ---------- bar : str Description of bar. Returns ------- float Description of the return value of foo. Raises ------ ValueError Explanation of when a ValueError is raised. In this case, a ValueError is raised if bar is "fail". Examples -------- The examples section is _strongly_ encouraged. Where appropriate, it may mimic the examples for the corresponding pandas API. >>> a = A() >>> a.foo('baz') 0.0 >>> a.foo('fail') ... ValueError: Failed! """ if bar == "fail": raise ValueError("Failed!") return 0.0 ``` -------------------------------- ### Configuring SPMDEngine with StreamingOptions Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/spmd_engine.md Illustrates how to create a custom StreamingOptions object and use it to initialize SPMDEngine via `from_options()`. This allows for fine-tuning parameters like the number of streaming threads. ```python import polars as pl from cudf_polars.engine.options import StreamingOptions from cudf_polars.engine.spmd import SPMDEngine opts = StreamingOptions(num_streaming_threads=8, fallback_mode="silent") with SPMDEngine.from_options(opts) as engine: result = pl.scan_parquet("/data/dataset/*.parquet").collect(engine=engine) ``` -------------------------------- ### Install cudf-polars with Conda Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/index.md Installs the cudf-polars package using conda, including dependencies from rapidsai, conda-forge, and nvidia channels. ```bash conda install -c rapidsai -c conda-forge -c nvidia cudf-polars ``` -------------------------------- ### Install cuDF Testing Targets Source: https://github.com/rapidsai/cudf/blob/main/cpp/CMakeLists.txt Installs cuDF testing targets if CUDF_BUILD_STREAMS_TEST_UTIL is enabled. These targets are placed in the testing component. ```cmake if(CUDF_BUILD_STREAMS_TEST_UTIL) install( TARGETS cudf_identify_stream_usage_mode_cudf DESTINATION ${lib_dir} COMPONENT testing ) install( TARGETS cudf_identify_stream_usage_mode_testing DESTINATION ${lib_dir} COMPONENT testing ) endif() ``` -------------------------------- ### Enum Class Documentation Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Demonstrates Doxygen documentation for an enum class, including a brief and detailed description, and inline documentation for enum members. ```cpp /** * @brief Short, one line description * * @ingroup optional_predefined_group_id * * Optional, longer description. */ enum class example_enum { first_enum, ///< Description of the first enum second_enum, ///< Description of the second enum third_enum ///< Description of the third enum }; ``` -------------------------------- ### Install cuStreamz (Release) Source: https://github.com/rapidsai/cudf/blob/main/python/custreamz/README.md Installs the cuStreamz library and its Kafka integration using conda from the rapidsai channel for a stable release version. ```bash conda install -c rapidsai cudf_kafka custreamz ``` -------------------------------- ### Apply Function to Rolling Window Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/guide-to-udfs.ipynb Applies the defined 'example_func' to the rolling window object. Note that the first two values are null due to min_periods. ```python rolling.apply(example_func) ``` -------------------------------- ### Install cudf-polars with Pip Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/index.md Installs the cudf-polars package using pip. Specify the CUDA version (e.g., -cu13 for CUDA 13). ```bash pip install cudf-polars-cu13 ``` -------------------------------- ### List Available Benchmarks Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/BENCHMARKING.md List all compiled NVBench benchmark executables located in the benchmarks directory. ```bash ls cpp/build/latest/benchmarks/*_NVBENCH ``` -------------------------------- ### Install cuDF Libraries with pip Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/README.md Installs stable releases of cuDF libraries with pip, matching the CUDA version. Ensure your CUDA version is compatible. ```bash # CUDA 13 pip install libcudf-cu13 pip install pylibcudf-cu13 pip install cudf-cu13 pip install cudf-polars-cu13 pip install dask-cudf-cu13 # CUDA 12 pip install libcudf-cu12 pip install pylibcudf-cu12 pip install cudf-cu12 pip install cudf-polars-cu12 pip install dask-cudf-cu12 ``` -------------------------------- ### C++ Inline Code Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DOCUMENTATION.md Use the @code and @endcode tags for inline C++ code examples. Doxygen provides syntax highlighting by default. ```cpp auto result = cudf::make_column( ); ``` -------------------------------- ### Construct and Use RayEngine with Streaming Options Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md This snippet demonstrates how to initialize RayEngine with custom streaming options and use it to collect a Polars DataFrame from a Parquet scan. It ensures Ray is running and sets up a UCXX communicator. ```python import polars as pl from cudf_polars.engine.options import StreamingOptions from cudf_polars.engine.ray import RayEngine opts = StreamingOptions(num_streaming_threads=8, fallback_mode="silent") with RayEngine.from_options(opts) as engine: result = ( pl.scan_parquet("/data/dataset/*.parquet") .filter(pl.col("amount") > 100) .group_by("customer_id") .agg(pl.col("amount").sum()) .collect(engine=engine) ) print(result) ``` -------------------------------- ### Configure Hardware Binding via CLI Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md Enable failure reporting for hardware binding via the command-line interface by passing a JSON string for the --hardware-binding argument. ```bash python my_script.py --hardware-binding '{"raise_on_fail": true}' ``` -------------------------------- ### DaskEngine with Existing Dask Client Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/dask_engine.md Illustrates attaching DaskEngine to an already running Dask scheduler by providing an existing distributed.Client instance. ```python from distributed import Client import polars as pl from cudf_polars.engine.dask import DaskEngine with Client("scheduler-address:8786") as dc: with DaskEngine(dask_client=dc) as engine: result = pl.scan_parquet("/data/*.parquet").collect(engine=engine) ``` -------------------------------- ### Get Value Counts with cuDF Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/10min.ipynb Demonstrates how to get the top 5 value counts for a column in a cuDF DataFrame. This is useful for understanding data distribution. ```python ddf.a.value_counts().head() ``` -------------------------------- ### Install cuDF stable release via conda Source: https://github.com/rapidsai/cudf/blob/main/README.md Installs the stable release of libcudf, pylibcudf, cudf, cudf-polars, and dask-cudf using the rapidsai conda channel. ```bash conda install -c rapidsai libcudf conda install -c rapidsai pylibcudf conda install -c rapidsai cudf conda install -c rapidsai cudf-polars conda install -c rapidsai dask-cudf ``` -------------------------------- ### Configure StreamingOptions for Multi-GPU Polars Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md Instantiate StreamingOptions to configure parameters for Ray, Dask, and SPMD engines. This example sets the number of streaming threads, log level, fallback mode, and spill device limit. ```python from cudf_polars.engine.options import StreamingOptions opts = StreamingOptions( num_streaming_threads=8, log="DEBUG", fallback_mode="silent", spill_device_limit="70%", ) ``` -------------------------------- ### Install cuDF for CUDA 12 via pip Source: https://github.com/rapidsai/cudf/blob/main/README.md Installs the stable release of libcudf, pylibcudf, cudf, cudf-polars, and dask-cudf for CUDA 12 using pip. ```bash pip install libcudf-cu12 pip install pylibcudf-cu12 pip install cudf-cu12 pip install cudf-polars-cu12 pip install dask-cudf-cu12 ``` -------------------------------- ### Configure In-Memory GPU Engine with Options Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/in_memory_engine.md Instantiate the in-memory GPU engine with specific options, such as parquet read options. Keyword arguments are passed directly to the GPUEngine constructor. ```python import polars as pl engine = pl.GPUEngine(executor="in-memory", parquet_options={"chunked": True}) ``` -------------------------------- ### Install cuDF for CUDA 13 via pip Source: https://github.com/rapidsai/cudf/blob/main/README.md Installs the stable release of libcudf, pylibcudf, cudf, cudf-polars, and dask-cudf for CUDA 13 using pip. ```bash pip install libcudf-cu13 pip install pylibcudf-cu13 pip install cudf-cu13 pip install cudf-polars-cu13 pip install dask-cudf-cu13 ``` -------------------------------- ### Documenting Class Attributes and Methods with numpydoc Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf/developer_guide/documentation.md Example of how to explicitly list attributes and methods for a class when only a subset of APIs should be documented. Use 'None' for empty sections. ```rst Attributes ---------- codes categories Methods ------- None ``` -------------------------------- ### Activate conda environment and install cuDF Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Activate the 'cudf_dev' conda environment and install a recent version of cuDF from the specified channels. This is a prerequisite for building and viewing documentation locally. ```bash conda activate cudf_dev conda install cudf -c rapidsai-nightly -c conda-forge ``` -------------------------------- ### Enable Benchmark Building with CMake Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/BENCHMARKING.md Pass the BUILD_BENCHMARKS=ON flag to CMake to include benchmarks in the libcudf build process. ```bash cmake -DBUILD_BENCHMARKS=ON cpp/build/latest ``` -------------------------------- ### DaskEngine with Custom Streaming Options Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/dask_engine.md Shows how to configure DaskEngine with custom streaming options, such as the number of streaming threads and fallback mode. ```python import polars as pl from cudf_polars.engine.options import StreamingOptions from cudf_polars.engine.dask import DaskEngine opts = StreamingOptions(num_streaming_threads=8, fallback_mode="silent") with DaskEngine.from_options(opts) as engine: result = pl.scan_parquet("/data/dataset/*.parquet").collect(engine=engine) ``` -------------------------------- ### Invalid Query Symmetry Example (Deadlock) Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/spmd_engine.md This example shows a query structure that violates query symmetry by conditionally executing a collective operation on only one rank, leading to a deadlock. ```python # DEADLOCKS: rank 0 issues a group_by collective the other ranks never see. with SPMDEngine() as engine: df = pl.scan_parquet("/data/*.parquet") if engine.rank == 0: # don't do this df = df.group_by("customer_id").agg(pl.col("amount").sum()) result = df.collect(engine=engine) ``` -------------------------------- ### Writing Results to Parquet Source: https://github.com/rapidsai/cudf/blob/main/skills/accelerated-computing-cudf/references/dask-cudf-patterns.md Recommended for partitioned output. Use `ddf.to_parquet()` for efficient storage. ```python # Parquet (recommended — partitioned output) ddf.to_parquet("output/", write_index=False) ``` -------------------------------- ### Setup LocalCUDACluster Source: https://github.com/rapidsai/cudf/blob/main/skills/accelerated-computing-cudf/references/dask-cudf-patterns.md Configure a `LocalCUDACluster` for dask-cuDF, ensuring GPU affinity and enabling features like the dashboard and spill configuration. Always use `LocalCUDACluster` even for single-GPU setups. ```python from dask_cuda import LocalCUDACluster from dask.distributed import Client import dask dask.config.set({"dataframe.backend": "cudf"}) # Standard setup — one worker per GPU cluster = LocalCUDACluster( enable_cudf_spill=True, # cuDF-native spill; preferred over device_memory_limit rmm_pool_size=0.8, # leave headroom for non-RMM allocations ) client = Client(cluster) # With UCX automatic transport selection for communication-heavy workloads cluster = LocalCUDACluster( enable_cudf_spill=True, rmm_pool_size=0.8, protocol="ucx", ) ``` -------------------------------- ### SPMDEngine with StreamingOptions Source: https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/docs/cudf-polars-mp.md Construct an SPMDEngine with custom streaming options for distributed query execution. This example demonstrates filtering, grouping, and aggregating data from Parquet files, then collecting the results using the SPMD engine and gathering all distributed fragments. ```python import polars as pl from cudf_polars.streaming.collectives.common import reserve_op_id from cudf_polars.engine.options import StreamingOptions from cudf_polars.engine.spmd import ( SPMDEngine, allgather_polars_dataframe, ) opts = StreamingOptions(num_streaming_threads=8, fallback_mode="silent") with SPMDEngine.from_options(opts) as engine: result = ( pl.scan_parquet("/data/dataset/*.parquet") .filter(pl.col("amount") > 100) .group_by("customer_id") .agg(pl.col("amount").sum()) .collect(engine=engine) ) with reserve_op_id() as op_id: full = allgather_polars_dataframe( engine=engine, local_df=result, op_id=op_id, ) ``` -------------------------------- ### Configure NVBench for quantiles benchmark Source: https://github.com/rapidsai/cudf/blob/main/cpp/benchmarks/CMakeLists.txt Configures the NVBench for the quantiles benchmark, specifying the main C++ source file. ```cmake ConfigureNVBench(QUANTILES_NVBENCH quantiles/quantiles.cpp) ``` -------------------------------- ### Install Polars Development Dependencies with uv Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/developer_docs.md After cloning the polars repository, install its development dependencies using uv. Ensure your cudf environment (conda or pip) is active before running this command. ```sh # cudf environment (conda or pip) is active pip install --upgrade uv uv pip install --upgrade -r py-polars/requirements-dev.txt ``` -------------------------------- ### Configure Streaming Executor with Target Partition Size Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/developer_docs.md Sets up a streaming executor with a specific target partition size for optimizing file splitting or fusion. Tune this value via executor_options or environment variables. ```python engine = pl.GPUEngine( executor="streaming", executor_options={\"target_partition_size\": 2_000_000_000}, ) ``` -------------------------------- ### Build pandas environment Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_pandas/benchmarks.md Creates a virtual environment for pandas. ```bash virtualenv pandas/py-pandas ``` -------------------------------- ### Manage cudf Dependencies for Testing Source: https://github.com/rapidsai/cudf/blob/main/cpp/tests/CMakeLists.txt Configures dependency installation and exclusion for testing. It sets flags to control the installation of Arrow libraries and excludes dependencies from the 'all' target when only test executables are needed. ```cmake # No need to install Arrow libs when only the final test executables are shipped. set(CUDF_INSTALL_LIBRARY_DEPS OFF) set(CUDF_EXCLUDE_DEPS_FROM_ALL ON) set(CUDF_EXCLUDE_DEPS_FROM_ALL_FLAG EXCLUDE_FROM_ALL) set(CUDF_ENABLE_ARROW_COMPUTE ON) include(../cmake/thirdparty/get_arrow.cmake) ``` -------------------------------- ### Temporary Memory Allocation Example Source: https://github.com/rapidsai/cudf/blob/main/cpp/doxygen/developer_guide/DEVELOPER_GUIDE.md Temporary memory allocations within an API should use the default resource from `cudf::get_current_device_resource_ref()`. This example differentiates between returned buffer allocation and temporary buffer allocation. ```c++ rmm::device_buffer some_function( ..., rmm::device_async_resource_ref mr = cudf::get_current_device_resource_ref()) { rmm::device_buffer returned_buffer(..., mr); // Returned buffer uses the passed in MR ... rmm::device_buffer temporary_buffer(...); // Temporary buffer uses default MR ... return returned_buffer; } ``` -------------------------------- ### Compile and Execute libcudf Application Source: https://github.com/rapidsai/cudf/blob/main/cpp/examples/basic/README.md Steps to configure, build, and execute a basic libcudf C++ application using CMake. Ensure PARALLEL_LEVEL is set appropriately for build speed. ```bash # Configure project cmake -S . -B build/ # Build cmake --build build/ --parallel $PARALLEL_LEVEL # Execute build/basic_example ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/rapidsai/cudf/blob/main/CONTRIBUTING.md Create a conda development environment using the provided YAML file and activate it. This is the recommended method for managing dependencies. ```bash # create the conda environment (assuming in base `cudf` directory) conda env create --name cudf_dev --file conda/environments/all_cuda-132_arch-$(uname -m).yaml # activate the environment conda activate cudf_dev ``` -------------------------------- ### Switching from Default Singleton to Explicit Engine Source: https://github.com/rapidsai/cudf/blob/main/docs/cudf/source/cudf_polars/default_singleton_engine.md To switch from the default singleton engine to an explicit engine (e.g., SPMD), first shut down the singleton using `DefaultSingletonEngine.shutdown()`, then construct the desired explicit engine. ```python DefaultSingletonEngine.shutdown() explicit_engine = SPMDEngine.from_options(opts) ```