### Build and install Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Compile the source code and install the library to the ROCm directory. ```bash make make install ``` -------------------------------- ### Run CG solver Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Execute the CG solver client to verify the installation. ```bash .\cg.exe gr_30_30.mtx ``` -------------------------------- ### Build rocALUTION from source Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Use the rmake.py script to build the library with various installation options. ```python python3 rmake.py ``` ```python python3 rmake.py -i ``` ```python python3 rmake.py -ci ``` -------------------------------- ### Synchronous Data Transfer Example Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-accelerators.html Demonstrates moving matrix and vector data between host and accelerator backends for computation. ```cpp LocalMatrix mat; LocalVector vec1, vec2; // Perform matrix vector multiplication on the host mat.Apply(vec1, &vec2); // Move data to the accelerator mat.MoveToAccelerator(); vec1.MoveToAccelerator(); vec2.MoveToAccelerator(); // Perform matrix vector multiplication on the accelerator mat.Apply(vec1, &vec2); // Move data to the host mat.MoveToHost(); vec1.MoveToHost(); vec2.MoveToHost(); ``` -------------------------------- ### Run CG solver client Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Execute the Conjugate Gradient solver using the extracted matrix file to verify the installation. ```bash ./cg gr_30_30.mtx ``` -------------------------------- ### Verify HIP SDK version Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Check the installed version of the HIP SDK to ensure compatibility. ```bash hipcc --version ``` -------------------------------- ### Configure CMake with rocALUTION path Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/include-rocALUTION.html Set the CMAKE_PREFIX_PATH to the location of the AMD HIP SDK installation. ```cmake -DCMAKE_PREFIX_PATH="C:\Program Files\AMD\ROCm\5.5" ``` -------------------------------- ### Create build directory Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Prepare the build environment by creating and entering a dedicated build directory. ```bash mkdir build cd build ``` -------------------------------- ### Navigate to client directory Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Change to the staging directory containing the built client applications for either Release or Debug builds. ```bash cd rocALUTION\build\release\clients\staging ``` ```bash cd rocALUTION\build\debug\clients\staging ``` -------------------------------- ### Building on Host and Solving on the Accelerator Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html This approach builds the solver on the host and then moves the solver, matrix, and vectors to the accelerator for the solving phase. ```cpp // CG solver CG, LocalVector, ValueType> ls; // Multi-Colored ILU preconditioner MultiColoredILU, LocalVector, ValueType> p; // Set mat to be the operator ls.SetOperator(mat); // Set p as the preconditioner of ls ls.SetPreconditioner(p); // Build the solver and preconditioner on the host ls.Build(); // Move matrix and vectors to the accelerator mat.MoveToAccelerator(); rhs.MoveToAccelerator(); x.MoveToAccelerator(); // Move linear solver to the accelerator ls.MoveToAccelerator(); // Compute the solution on the accelerator ls.Solve(rhs, &x); ``` -------------------------------- ### Initialize rocALUTION with MPI Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-api-basics.html Demonstrates initializing the library with MPI rank and accelerator mapping. Ensure MPI is initialized before calling init_rocalution. ```cpp #include #include using namespace rocalution; int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); MPI_Comm comm = MPI_COMM_WORLD; int num_processes; int rank; MPI_Comm_size(comm, &num_processes); MPI_Comm_rank(comm, &rank); int nacc_per_node = 2; init_rocalution(rank, nacc_per_node); // ... do some work stop_rocalution(); return 0; } ``` -------------------------------- ### Download test matrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Retrieve a sample matrix in Matrix Market format for testing the solver. ```bash wget https://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/laplace/gr_30_30.mtx.gz ``` -------------------------------- ### Building and Solving on the Accelerator Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html This approach builds the solver and preconditioner directly on the accelerator after moving the matrix and vectors. ```cpp // CG solver CG, LocalVector, ValueType> ls; // Multi-Colored ILU preconditioner MultiColoredILU, LocalVector, ValueType> p; // Move matrix and vectors to the accelerator mat.MoveToAccelerator(); rhs.MoveToAccelerator(); x.MoveToAccelerator(); // Set mat to be the operator ls.SetOperator(mat); // Set p as the preconditioner of ls ls.SetPreconditioner(p); // Build the solver and preconditioner on the accelerator ls.Build(); // Compute the solution on the accelerator ls.Solve(rhs, &x); ``` -------------------------------- ### Configure Block-Preconditioner Settings Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Methods for setting solver parameters, modes, and external matrices for the Block-Preconditioner. ```cpp void rocalution::BlockPreconditioner::Set(int n, const int *size, Solver **D_solver) ``` ```cpp void rocalution::BlockPreconditioner::SetDiagonalSolver(void) ``` ```cpp void rocalution::BlockPreconditioner::SetLSolver(void) ``` ```cpp void rocalution::BlockPreconditioner::SetExternalLastMatrix(const OperatorType &mat) ``` ```cpp virtual void rocalution::BlockPreconditioner::SetPermutation(const LocalVector &perm) ``` -------------------------------- ### rocalution::init_rocalution Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-api-basics.html Initializes the rocALUTION platform, defining the backend descriptor and hardware specifications. ```APIDOC ## int rocalution::init_rocalution(int rank = -1, int dev_per_node = 1) ### Description Initializes the rocALUTION platform and allocates hardware resources. This function must be called before any other library operations. ### Parameters - **rank** (int) - Optional - Specifies the MPI rank in a multi-node environment. - **dev_per_node** (int) - Optional - Specifies the number of accelerator devices per node in a multi-GPU environment. ``` -------------------------------- ### rocalution::MultiColoredILU::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Initializes the multi-colored ILU preconditioner with specific parameters. ```APIDOC ## void rocalution::MultiColoredILU::Set(int p) ## void rocalution::MultiColoredILU::Set(int p, int q, bool level = true) ### Description Initializes a multi-colored ILU(p, q) preconditioner. The single parameter version initializes an ILU(p, p+1) preconditioner. ### Parameters - **p** (int) - Required - The p parameter for ILU. - **q** (int) - Optional - The q parameter for ILU. - **level** (bool) - Optional - If true, performs factorization with levels; if false, performs factorization only on the power(q)-pattern (default is true). ``` -------------------------------- ### rocalution::Solver::Build Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Allocates data and prepares the solver structure. ```APIDOC ## virtual void rocalution::Solver::Build(void) ### Description Builds the solver, performing necessary data allocation, structure setup, and initial numerical computation. The operator must be set via SetOperator() before calling this method. ``` -------------------------------- ### rocalution::Solver::MoveToHost / MoveToAccelerator Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Manages data placement between host and accelerator. ```APIDOC ## virtual void rocalution::Solver::MoveToHost(void) ## virtual void rocalution::Solver::MoveToAccelerator(void) ### Description These methods offload the solver, including all preconditioners and sub-solvers, to either the host or the accelerator. ``` -------------------------------- ### Configuration Functions Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-api-basics.html Functions for configuring the runtime environment, including thread management, device selection, and backend information. ```APIDOC ## Configuration Functions - **set_device_rocalution(int id)**: Selects a specific accelerator device. Must be called before init_rocalution(). - **set_omp_threads_rocalution(int n)**: Sets the number of OpenMP threads. Must be called after init_rocalution(). - **set_omp_threshold_rocalution(int size)**: Modifies the OpenMP threshold size for host backend performance optimization. - **set_omp_affinity_rocalution(bool enable)**: Enables or disables thread affinity. Must be called before initialization. - **disable_accelerator_rocalution()**: Disables the use of accelerators. - **info_rocalution()**: Prints detailed information about the current backend and available accelerators. ``` -------------------------------- ### Convert and Apply Matrix Formats Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Demonstrates converting a matrix to specific storage formats and performing matrix-vector multiplication. ```cpp // Convert mat to CSR storage format mat.ConvertToCSR(); // Perform a matrix-vector multiplication y = mat * x in CSR format mat.Apply(x, &y); // Convert mat to ELL storage format mat.ConvertToELL(); // Perform a matrix-vector multiplication y = mat * x in ELL format mat.Apply(x, &y); ``` ```cpp // Convert mat to CSR storage format mat.ConvertTo(CSR); // Perform a matrix-vector multiplication y = mat * x in CSR format mat.Apply(x, &y); // Convert mat to ELL storage format mat.ConvertTo(ELL); // Perform a matrix-vector multiplication y = mat * x in ELL format mat.Apply(x, &y); ``` -------------------------------- ### Download and decompress test matrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Retrieve and extract a sample matrix for testing the solver. ```bash curl -LO https://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/laplace/gr_30_30.mtx.gz ``` ```bash gunzip gr_30_30.mtx.gz ``` -------------------------------- ### Configure build with CMake Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Generate build files using CMake, specifying the ROCm path and enabling optional features like MPI. ```bash cmake .. -DSUPPORT_MPI=ON -DROCM_PATH=/opt/rocm/ ``` -------------------------------- ### Print Object Information Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Use the Info() method to display properties and backend data for rocALUTION objects. ```cpp mat.Info(); vec.Info(); ``` -------------------------------- ### rocalution::DiagJacobiSaddlePointPrecond::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Initializes the solver for the K and S components of the saddle-point problem. ```APIDOC ## void rocalution::DiagJacobiSaddlePointPrecond::Set(Solver &K_Solver, Solver &S_Solver) ### Description Initializes the preconditioner with external solvers for the K and S matrices. ### Parameters - **K_Solver** (Solver&) - Required - Solver for the K matrix. - **S_Solver** (Solver&) - Required - Solver for the S matrix. ``` -------------------------------- ### Registering a new solver in rocalution.hpp Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Include the new solver header file in the main rocalution.hpp to make it visible to the library. ```cpp ... #include "solvers/krylov/cg.hpp" #include "solvers/krylov/mysolver.hpp" #include "solvers/krylov/cr.hpp" ... ``` -------------------------------- ### Implement HIP-based algorithm for CSR Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Provide the HIP implementation in src/base/hip/hip_matrix_csr.cpp, using hipLaunchKernelGGL to execute the kernel on the accelerator. ```cpp template bool HIPAcceleratorMatrixCSR::MyNewFunctionality(void) { // Place some asserts to verify sanity of input data // Our algorithm works only for squared matrices assert(this->nrow_ == this->ncol_); assert(this->nnz_ > 0); // Enqueue the HIP kernel hipLaunchKernelGGL((kernel_csr_mynewfunctionality), dim3((this->nrow_ - 1) / this->local_backend_.HIP_block_size + 1), dim3(this->local_backend_.HIP_block_size), 0, 0, this->mat_.row_offset, this->mat_.col, this->mat_.val); // Check for HIP execution error before successfully returning CHECK_HIP_ERROR(__FILE__, __LINE__); return true; } ``` -------------------------------- ### PairwiseAMG Configuration Methods Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Configuration methods for the Pairwise Aggregation AMG solver. ```APIDOC ## void rocalution::PairwiseAMG::SetBeta(ValueType beta) ### Description Sets the beta parameter for pairwise aggregation. ## void rocalution::PairwiseAMG::SetOrdering(unsigned int ordering) ### Description Sets the re-ordering strategy for aggregation. ## void rocalution::PairwiseAMG::SetCoarseningFactor(double factor) ### Description Sets the target coarsening factor for the solver. ``` -------------------------------- ### Allocate and Clear LocalMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Demonstrates allocating local matrices using CSR and COO formats, followed by clearing the memory. ```cpp LocalMatrix mat; mat.AllocateCSR("my CSR matrix", 456, 100, 100); mat.Clear(); mat.AllocateCOO("my COO matrix", 200, 100, 100); mat.Clear(); ``` -------------------------------- ### rocalution::BaseRocalution::MoveToHost Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-accelerators.html Moves the object to the host backend. ```APIDOC ## virtual void rocalution::BaseRocalution::MoveToHost(void) ### Description Move the object to the host backend. ``` -------------------------------- ### Implement default BaseMatrix functionality Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Provide a default implementation that returns false to indicate that the functionality is not supported by default. ```cpp template bool MyNewFunctionality(void) { return false; } ``` -------------------------------- ### MultiElimination::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Initializes the Multi-Elimination ILU preconditioner recursively. ```APIDOC ## void rocalution::MultiElimination::Set(Solver &AA_Solver, int level, double drop_off = 0.0) ### Description Initializes the ME-ILU preconditioner recursively. The AA_Solver defines the solver used for the last block, while level specifies the recursion depth and drop_off defines the tolerance. ### Parameters - **AA_Solver** (Solver&) - Required - The solver to be used for the last block. - **level** (int) - Required - The depth of the recursion. - **drop_off** (double) - Optional - The drop-off tolerance for the factorization. ``` -------------------------------- ### rocalution::LocalMatrix::SetDataPtr Methods Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Methods to initialize LocalMatrix objects on the host with externally allocated data for various formats. ```APIDOC ## rocalution::LocalMatrix::SetDataPtr[Format] ### Description Initializes a LocalMatrix on the host with externally allocated data. Supported formats include COO, CSR, MCSR, ELL, DIA, and DENSE. The original pointers passed to these methods will be set to NULL. ### Methods - **SetDataPtrCOO** - **SetDataPtrCSR** - **SetDataPtrMCSR** - **SetDataPtrELL** - **SetDataPtrDIA** - **SetDataPtrDENSE** ### Parameters - **Pointers** (Various**) - Required - Pointers to the matrix data structures (e.g., row, col, val). - **name** (std::string) - Required - Name of the matrix. - **nnz** (int64_t) - Required - Number of non-zero elements (not applicable to DENSE). - **nrow** (int64_t) - Required - Number of rows. - **ncol** (int64_t) - Required - Number of columns. ``` -------------------------------- ### Implement host-based algorithm for CSR Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Provide the host-side implementation in src/base/host/host_matrix_csr.cpp, ensuring input data sanity with assertions. ```cpp template bool HostMatrixCSR::MyNewFunctionality(void) { // Place some asserts to verify sanity of input data // Our algorithm works only for squared matrices assert(this->nrow_ == this->ncol_); assert(this->nnz_ > 0); // place the actual host based algorithm here: // for illustration, we scale the matrix by its inverse diagonal for(int i = 0; i < this->nrow_; ++i) { int row_begin = this->mat_.row_offset[i]; int row_end = this->mat_.row_offset[i + 1]; bool diag_found = false; ValueType inv_diag; // Find the diagonal entry for(int j = row_begin; j < row_end; ++j) { if(this->mat_.col[j] == i) { diag_found = true; inv_diag = static_cast(1) / this->mat_.val[j]; } } // Our algorithm works only with full rank assert(diag_found == true); // Scale the row for(int j = row_begin; j < row_end; ++j) { this->mat_.val[j] *= inv_diag; } } return true; } ``` -------------------------------- ### Read and Write MTX Files with LocalMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Use these methods to import or export matrices in the Matrix Market format. ```cpp LocalMatrix mat; mat.ReadFileMTX("my_matrix.mtx"); ``` ```cpp LocalMatrix mat; // Allocate and fill mat // ... mat.WriteFileMTX("my_matrix.csr"); ``` -------------------------------- ### Implement LocalMatrix functionality Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Implement the routine in the LocalMatrix class, ensuring proper matrix format conversion and backend handling. ```cpp template void LocalMatrix::MyNewFunctionality(void) { // Debug logging log_debug(this, "LocalMatrix::MyNewFunctionality()"); #ifdef DEBUG_MODE // If we are in debug mode, perform an additional matrix sanity check this->Check(); #endif // If no non-zero entries, do nothing if(this->GetNnz() > 0) { // As we want to implement our function only for CSR, we first need to convert // the matrix to CSR format unsigned int format = this->GetFormat(); int blockdim = this->GetBlockDimension(); this->ConvertToCSR(); // Call the corresponding base matrix implementation bool err = this->matrix_->MyNewFunctionality(); // Check its return type if((err == false) && (this->is_host_() == true)) { // If our matrix is on the host, the function call failed. LOG_INFO("Computation of LocalMatrix::MyNewFunctionality() failed"); this->Info(); FATAL_ERROR(__FILE__, __LINE__); } // Run backup algorithm on host, in case the accelerator version failed if(err == false) { // Move matrix to host bool is_accel = this->is_accel_(); this->MoveToHost(); // Try again if(this->matrix_->MyNewFunctionality() == false) { LOG_INFO("Computation of LocalMatrix::MyNewFunctionality() failed"); this->Info(); FATAL_ERROR(__FILE__, __LINE__); } // On a successful host call, move the data back to the accelerator // if initial data was on the accelerator if(is_accel == true) { // Print a warning, that the algorithm was performed on the host // even though the initial data was on the device LOG_VERBOSE_INFO(2, "*** warning: LocalMatrix::MyNewFunctionality() was performed on the host"); this->MoveToAccelerator(); } } // Convert the matrix back to CSR format if(format != CSR) { // Print a warning, that the algorithm was performed in CSR format // even though the initial matrix format was different LOG_VERBOSE_INFO(2, "*** warning: LocalMatrix::MyNewFunctionality() was performed in CSR format"); this->ConvertTo(format, blockdim); } } #ifdef DEBUG_MODE // Perform additional sanity check in debug mode, because this is a non-const function this->Check(); #endif } ``` -------------------------------- ### rocalution::BaseAMG::BuildHierarchy Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Creates the AMG hierarchy for the solver. ```APIDOC ## virtual void rocalution::BaseAMG::BuildHierarchy(void) ### Description Create the AMG hierarchy required for the solver execution. ``` -------------------------------- ### Read and Write CSR Files with LocalMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Use these methods to handle rocALUTION binary CSR format files. ```cpp LocalMatrix mat; mat.ReadFileCSR("my_matrix.csr"); ``` ```cpp // Header out << "#rocALUTION binary csr file" << std::endl; // rocALUTION version out.write((char*)&version, sizeof(int)); // CSR matrix data out.write((char*)&m, sizeof(int)); out.write((char*)&n, sizeof(int)); out.write((char*)&nnz, sizeof(int64_t)); out.write((char*)csr_row_ptr, (m + 1) * sizeof(int)); out.write((char*)csr_col_ind, nnz * sizeof(int)); out.write((char*)csr_val, nnz * sizeof(double)); ``` ```cpp LocalMatrix mat; // Allocate and fill mat // ... mat.WriteFileCSR("my_matrix.csr"); ``` -------------------------------- ### rocalution::BaseRocalution::MoveToAccelerator Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-accelerators.html Moves the object to the accelerator backend. ```APIDOC ## virtual void rocalution::BaseRocalution::MoveToAccelerator(void) ### Description Move the object to the accelerator backend. ``` -------------------------------- ### FSAI::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the Factorized Sparse Approximate Inverse preconditioner using either a power parameter or an external sparsity pattern. ```APIDOC ## void rocalution::FSAI::Set(int power) ### Description Sets the power of the system matrix sparsity pattern for the FSAI preconditioner. ### Parameters - **power** (int) - The power of the system matrix sparsity pattern. ## void rocalution::FSAI::Set(const OperatorType &pattern) ### Description Sets an external sparsity pattern for the FSAI preconditioner. ### Parameters - **pattern** (const OperatorType &) - The external sparsity pattern provided as a LocalMatrix. ``` -------------------------------- ### Copy Local Matrix Data Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html The CopyFrom() method transfers values and structure between local matrices, supporting cross-platform copying between host and accelerator backends. ```cpp LocalMatrix mat1, mat2; // Allocate and initialize mat1 and mat2 // ... // Move mat1 to accelerator // mat1.MoveToAccelerator(); // Now, mat1 is on the accelerator (if available) // and mat2 is on the host // Copy mat1 to mat2 (or vice versa) will move data between host and // accelerator backend mat1.CopyFrom(mat2); ``` -------------------------------- ### Retrieve rocALUTION version information Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-api-basics.html Use these pre-defined macros to check the library version within an application. ```cpp #define __ROCALUTION_VER_MAJOR // version major #define __ROCALUTION_VER_MINOR // version minor #define __ROCALUTION_VER_PATCH // version patch #define __ROCALUTION_VER_TWEAK // commit id (sha-1) #define __ROCALUTION_VER_PRE // version pre-release (alpha or beta) #define __ROCALUTION_VER // version ``` -------------------------------- ### rocalution::BaseAMG::BuildSmoothers Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Creates the smoothers for the AMG solver. ```APIDOC ## virtual void rocalution::BaseAMG::BuildSmoothers(void) ### Description Create the AMG smoothers used during the iteration process. ``` -------------------------------- ### Adding a solver to CMakeLists.txt Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Register the new solver's source file in the CMake compilation list located in src/solvers/CMakeLists.txt. ```cmake ... set(SOLVERS_SOURCES solvers/krylov/cg.cpp solvers/krylov/mysolver.cpp solvers/krylov/fcg.cpp ... ``` -------------------------------- ### Initialize and Finalize rocALUTION Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-api-basics.html Basic structure for a rocALUTION application, requiring initialization and finalization calls. ```cpp #include using namespace rocalution; int main(int argc, char* argv[]) { init_rocalution(); // ... stop_rocalution(); return 0; } ``` -------------------------------- ### Extract test matrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/install/rocALUTION-install.html Decompress the downloaded matrix file. ```bash gzip -d gr_30_30.mtx.gz ``` -------------------------------- ### Matrix Inversion Solver Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Matrix inversion class based on QR decomposition. ```cpp template class Inversion : public rocalution::DirectLinearSolver ``` -------------------------------- ### rocalution::ILU::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Initializes the ILU(p) factorization based on levels or power. ```APIDOC ## void rocalution::ILU::Set(int p, bool level = true) ### Description Initialize ILU(p) factorization. This method computes a sparse lower and sparse upper triangular matrix such that A=LU-R. ### Parameters - **p** (int) - The power parameter for the factorization. - **level** (bool) - If true, build the structure based on levels; if false, build based on power(p+1). ``` -------------------------------- ### Set Local Preconditioner for Block-Jacobi Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the local preconditioner used within the Block-Jacobi wrapper. ```cpp void rocalution::BlockJacobi::Set(Solver, LocalVector, ValueType> &precond) ``` -------------------------------- ### Solver Interface Methods Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Virtual methods defining the standard interface for solver operations. ```cpp void rocalution::Solver::SetOperator(const OperatorType &op) ``` ```cpp virtual void rocalution::Solver::Build(void) ``` ```cpp virtual void rocalution::Solver::Clear(void) ``` ```cpp virtual void rocalution::Solver::Solve(const VectorType &rhs, VectorType *x) = 0 ``` ```cpp virtual void rocalution::Solver::Print(void) const = 0 ``` ```cpp virtual void rocalution::Solver::ReBuildNumeric(void) ``` ```cpp virtual void rocalution::Solver::MoveToHost(void) ``` ```cpp virtual void rocalution::Solver::MoveToAccelerator(void) ``` -------------------------------- ### rocalution::BaseRocalution::Info Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Prints object information, including properties and backend data, for any rocALUTION object. ```APIDOC ## void rocalution::BaseRocalution::Info(void) const ### Description Prints object information, including properties and backend data, for any rocALUTION object. ### Example ```cpp mat.Info(); vec.Info(); ``` ``` -------------------------------- ### rocalution::BlockPreconditioner::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the number, size, and diagonal solvers for the block preconditioner. ```APIDOC ## void rocalution::BlockPreconditioner::Set(int n, const int *size, Solver **D_solver) ### Description Sets the number of blocks, the size of each block, and the diagonal solvers for the block-type preconditioner. ### Parameters - **n** (int) - Number of blocks. - **size** (const int*) - Array containing the size of each block. - **D_solver** (Solver**) - Array of pointers to solvers for the diagonal blocks. ``` -------------------------------- ### rocalution::BaseRocalution::MoveToHostAsync Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-accelerators.html Initiates an asynchronous move of the object to the host backend. ```APIDOC ## virtual void rocalution::BaseRocalution::MoveToHostAsync(void) ### Description Move the object to the host backend with async move. The object switches to the new location after the Sync() function is called. ``` -------------------------------- ### Importing CSR matrix data Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Shows how to import CSR matrix data into a LocalMatrix object using CopyFromCSR. ```cpp // Allocate the CSR matrix int* csr_row_ptr = new int[100 + 1]; int* csr_col_ind = new int[345]; ValueType* csr_val = new ValueType[345]; // Fill the CSR matrix // ... // rocALUTION local matrix object LocalMatrix mat; // Import CSR matrix to rocALUTION mat.AllocateCSR("my_matrix", 345, 100, 100); mat.CopyFromCSR(csr_row_ptr, csr_col, csr_val); ``` -------------------------------- ### Link rocALUTION in CMakeLists.txt Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/include-rocALUTION.html Use find_package and target_link_libraries to include rocALUTION in your project build. ```cmake find_package(rocalution) target_link_libraries( your_exe PRIVATE roc::rocalution ) ``` -------------------------------- ### rocalution::BaseRocalution::MoveToAcceleratorAsync Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-accelerators.html Initiates an asynchronous move of the object to the accelerator backend. ```APIDOC ## virtual void rocalution::BaseRocalution::MoveToAcceleratorAsync(void) ### Description Move the object to the accelerator backend with async move. The object switches to the new location after the Sync() function is called. ``` -------------------------------- ### QR Decomposition Solver Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html QR decomposition class for decomposing matrices into orthogonal and upper triangular matrices. ```cpp template class QR : public rocalution::DirectLinearSolver ``` -------------------------------- ### DirectLinearSolver Class Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Base class for direct linear solvers including LU, QR, and Inversion. These solvers are intended for small, local-type problems. ```APIDOC ## class rocalution::DirectLinearSolver ### Description Base class for all direct linear solvers. The library provides three direct methods - LU, QR and Inversion (based on QR decomposition). The user can pass a sparse matrix, internally it will be converted to dense and then the selected method will be applied. ### Template Parameters - **OperatorType**: Can be LocalMatrix - **VectorType**: Can be LocalVector - **ValueType**: Can be float, double, std::complex or std::complex ``` -------------------------------- ### Extracting raw data from a LocalMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Demonstrates how to steal raw data from a LocalMatrix object in CSR format, leaving the matrix object empty. ```cpp // rocALUTION CSR matrix object LocalMatrix mat; // Allocate the CSR matrix mat.AllocateCSR("my_matrix", 345, 100, 100); // Fill CSR matrix // ... int* csr_row_ptr = NULL; int* csr_col_ind = NULL; ValueType* csr_val = NULL; // Get (steal) the data from the matrix, this will leave the local matrix // object empty mat.LeaveDataPtrCSR(&csr_row_ptr, &csr_col_ind, &csr_val); ``` -------------------------------- ### Perform Connectivity Ordering Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Sorts the matrix based on the number of non-zero entries per row. ```cpp LocalVector conn; mat.ConnectivityOrder(&conn); mat.Permute(conn); ``` -------------------------------- ### Declare new functionality in BaseMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Add the new routine as a public virtual member function in the base_matrix.hpp header file. ```cpp ... virtual bool ILU0Factorize(void); /// Perform MyNewFunctionality algorithm virtual bool MyNewFunctionality(void); /// Perform LU factorization ... ``` -------------------------------- ### rocalution::LocalVector::SetDataPtr Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Initializes a LocalVector on the host using externally allocated data. ```APIDOC ## rocalution::LocalVector::SetDataPtr ### Description Initializes a LocalVector on the host with externally allocated data. The original pointer passed to this method will be set to NULL. ### Parameters - **ptr** (ValueType**) - Required - Pointer to the external data. - **name** (std::string) - Required - Name of the vector. - **size** (int64_t) - Required - Size of the vector. ``` -------------------------------- ### rocalution::LocalMatrix::LeaveDataPtr Methods Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Methods to extract raw data from various LocalMatrix formats (COO, CSR, MCSR, ELL, DIA, DENSE). ```APIDOC ## LocalMatrix LeaveDataPtr Methods ### Description These methods allow extraction of raw data from LocalMatrix objects based on their storage format. The matrix object is left empty after the operation. ### Methods - **LeaveDataPtrCOO(int **row, int **col, ValueType **val)** - **LeaveDataPtrCSR(PtrType **row_offset, int **col, ValueType **val)** - **LeaveDataPtrMCSR(int **row_offset, int **col, ValueType **val)** - **LeaveDataPtrELL(int **col, ValueType **val, int &max_row)** - **LeaveDataPtrDIA(int **offset, ValueType **val, int &num_diag)** - **LeaveDataPtrDENSE(ValueType **val)** ``` -------------------------------- ### IterativeLinearSolver Methods Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Methods available for configuring and interacting with iterative linear solvers. ```APIDOC ## void Init(double abs_tol, double rel_tol, double div_tol, int max_iter) ### Description Initializes the solver with absolute, relative, and divergence tolerances, and the maximum number of iterations. ## void Init(double abs_tol, double rel_tol, double div_tol, int min_iter, int max_iter) ### Description Initializes the solver with absolute, relative, and divergence tolerances, and both minimum and maximum number of iterations. ## void InitMinIter(int min_iter) ### Description Sets the minimum number of iterations required before the solver can stop. ## void InitMaxIter(int max_iter) ### Description Sets the maximum number of iterations allowed. ## void InitTol(double abs, double rel, double div) ### Description Updates the absolute, relative, and divergence tolerances for the solver. ## void RecordResidualHistory(void) ### Description Enables recording of the residual history during solver execution. ## void RecordHistory(const std::string &filename) const ### Description Writes the recorded solver history to the specified file. ## void Verbose(int verb = 1) ### Description Sets the verbosity level of the solver output (0: no output, 2: detailed output). ## void SetPreconditioner(Solver &precond) ### Description Assigns a preconditioner to the linear solver. ## void SetResidualNorm(int resnorm) ### Description Sets the residual norm type: 1 for L1, 2 for L2, or 3 for L∞. ## int64_t GetAmaxResidualIndex(void) ### Description Returns the index of the maximum residual value when using the L∞ norm. ## int GetSolverStatus(void) ### Description Returns the current status of the solver (0: running, 1: abs tol reached, 2: rel tol reached, 3: div tol reached, 4: max iterations reached). ``` -------------------------------- ### rocalution::AS::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the Additive Schwarz preconditioner with the number of blocks, overlap, and an array of preconditioners. ```APIDOC ## void rocalution::AS::Set(int nb, int overlap, Solver **preconds) ### Description Sets the number of blocks, the overlap size, and the array of preconditioners for the Additive Schwarz method. ### Parameters - **nb** (int) - Required - Number of blocks. - **overlap** (int) - Required - Overlap size. - **preconds** (Solver**) - Required - Array of preconditioners. ``` -------------------------------- ### rocalution::Solver::Solve Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Executes the solver to find the solution vector. ```APIDOC ## virtual void rocalution::Solver::Solve(const VectorType &rhs, VectorType *x) ### Description Solves the linear system Ax = rhs, where A is the operator set by SetOperator(). ### Parameters - **rhs** (const VectorType &) - The right-hand-side vector b. - **x** (VectorType *) - Pointer to the vector where the solution will be stored. ``` -------------------------------- ### Matrix Inversion Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Performs full matrix inversion based on QR decomposition. ```APIDOC ## class rocalution::Inversion ### Description Full matrix inversion based on QR decomposition. ``` -------------------------------- ### Clone a LocalMatrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Clones the entire matrix including values, structure, and backend descriptor from a source LocalMatrix. ```cpp LocalMatrix mat; // Allocate and initialize mat (host or accelerator) // ... LocalMatrix tmp; // By cloning mat, tmp will have identical values and structure and will be on // the same backend as mat tmp.CloneFrom(mat); ``` -------------------------------- ### Allocate and Clear LocalVector Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Demonstrates how to allocate a local vector with a specified name and size, and subsequently clear it. ```cpp LocalVector vec; vec.Allocate("my vector", 100); vec.Clear(); ``` -------------------------------- ### Declare host implementation in CSR matrix Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/how-to/extending-rocALUTION.html Add the function declaration to the host CSR matrix header file. ```cpp ... virtual bool ILUTFactorize(double t, int maxrow); virtual bool MyNewFunctionality(void); virtual void LUAnalyse(void); ... ``` -------------------------------- ### Copy CSR matrix from host Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Allocates and copies host CSR matrix data into a LocalMatrix object. ```cpp void rocalution::LocalMatrix::CopyFromHostCSR(const PtrType *row_offset, const int *col, const ValueType *val, const std::string &name, int64_t nnz, int64_t nrow, int64_t ncol) ``` -------------------------------- ### FSAI::SetPrecondMatrixFormat Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Sets the matrix format for the FSAI preconditioner. ```APIDOC ## void rocalution::FSAI::SetPrecondMatrixFormat(unsigned int mat_format, int blockdim = 1) ### Description Configures the internal matrix format used by the FSAI preconditioner. ### Parameters - **mat_format** (unsigned int) - The matrix format identifier. - **blockdim** (int) - The block dimension, defaults to 1. ``` -------------------------------- ### LU Decomposition Solver Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html LU decomposition class for factoring square matrices into lower and upper triangular matrices. ```cpp template class LU : public rocalution::DirectLinearSolver ``` -------------------------------- ### Preconditioner Class Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html The base class for all preconditioners in the rocALUTION library, supporting both local and global operators. ```APIDOC ## class Preconditioner ### Description Base class for all preconditioners in the rocALUTION library. All preconditioners support local operators and can be used as a global preconditioner via the block-jacobi scheme. ### Template Parameters - **OperatorType** - Can be LocalMatrix or GlobalMatrix. - **VectorType** - Can be LocalVector or GlobalVector. - **ValueType** - Can be float, double, std::complex, or std::complex. ``` -------------------------------- ### Solver Class Definition Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html The base class template for all solvers and preconditioners in the rocALUTION library. ```cpp template class Solver : public rocalution::RocalutionObj ``` -------------------------------- ### LocalMatrix::CloneFrom Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Clones the entire matrix, including values, structure, and backend descriptor from another LocalMatrix. ```APIDOC ## void rocalution::LocalMatrix::CloneFrom(const LocalMatrix &src) ### Description Clone the matrix. ### Parameters - **src** (const LocalMatrix &) - Required - LocalMatrix to clone from. ``` -------------------------------- ### Clone a Backend Descriptor Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Clones the backend descriptor from another object without copying data, ensuring multiple objects reside on the same backend. ```cpp LocalVector vec; LocalMatrix mat; // Allocate and initialize vec and mat // ... LocalVector tmp; // By cloning backend, tmp and vec will have the same backend as mat tmp.CloneBackend(mat); vec.CloneBackend(mat); // The following matrix vector multiplication will be performed on the backend // selected in mat mat.Apply(vec, &tmp); ``` -------------------------------- ### DirectLinearSolver Base Class Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Base class for all direct linear solvers in rocALUTION. ```cpp template class DirectLinearSolver : public rocalution::Solver ``` -------------------------------- ### rocalution::LocalMatrix::CopyFrom Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Copies values and structure from one LocalMatrix to another, supporting cross-platform transfers. ```APIDOC ## void rocalution::LocalMatrix::CopyFrom(const LocalMatrix &src) ### Description Copies values and structure from a source LocalMatrix to a destination LocalMatrix. Both matrices must be in the same format. This function supports cross-platform copying between host and accelerator backends. ### Parameters - **src** (LocalMatrix) - Required - The source local matrix to copy from. ``` -------------------------------- ### AIChebyshev::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Sets the order and eigenvalue bounds for the Approximate Inverse Chebyshev preconditioner. ```APIDOC ## void rocalution::AIChebyshev::Set(int p, ValueType lambda_min, ValueType lambda_max) ### Description Sets the order (p), minimum eigenvalue (lambda_min), and maximum eigenvalue (lambda_max) for the Chebyshev polynomial approximation. ### Parameters - **p** (int) - The order of the Chebyshev polynomial. - **lambda_min** (ValueType) - The minimum eigenvalue estimate. - **lambda_max** (ValueType) - The maximum eigenvalue estimate. ``` -------------------------------- ### Perform Cuthill-McKee Ordering Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Generates a permutation vector to minimize the bandwidth of a sparse matrix using the CMK algorithm. ```cpp LocalVector cmk; mat.CMK(&cmk); mat.Permute(cmk); ``` -------------------------------- ### Setting LocalVector raw data pointer Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Initializes a LocalVector with externally allocated host memory using SetDataPtr. ```cpp // Allocate vector ValueType* ptr_vec = new ValueType[200]; // Fill vector // ... // rocALUTION local vector object LocalVector vec; // Set the vector data, ptr_vec will become invalid vec.SetDataPtr(&ptr_vec, "my_vector", 200); ``` -------------------------------- ### rocalution::SPAI::SetPrecondMatrixFormat Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Sets the matrix format for the SPAI preconditioner. ```APIDOC ## void rocalution::SPAI::SetPrecondMatrixFormat(unsigned int mat_format, int blockdim = 1) ### Description Sets the matrix format of the preconditioner. ### Parameters - **mat_format** (unsigned int) - Required - The matrix format to be used. - **blockdim** (int) - Optional - The block dimension, defaults to 1. ``` -------------------------------- ### rocalution::LocalVector::WriteFileBinary Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Writes a LocalVector to a binary file. ```APIDOC ## void rocalution::LocalVector::WriteFileBinary(const std::string &filename) const ### Description Writes the contents of a LocalVector to a binary file. ### Parameters - **filename** (const std::string &) - Required - The path to the file to write. ``` -------------------------------- ### rocalution::TNS::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the computation mode for the TNS preconditioner. ```APIDOC ## void rocalution::TNS::Set(bool imp) ### Description Sets the computation mode to implicit or explicit. ### Parameters - **imp** (bool) - Required - Set to true for implicit computation, false for explicit. ``` -------------------------------- ### BaseRocalution::CloneBackend Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Clones the backend descriptor from another object without copying any data. ```APIDOC ## virtual void rocalution::BaseRocalution::CloneBackend(const BaseRocalution &src) ### Description Clone the Backend descriptor from another object. This is useful if several objects should reside on the same backend while keeping their original data. ### Parameters - **src** (const BaseRocalution &) - Required - Object where the backend should be cloned from. ``` -------------------------------- ### rocalution::Solver::Clear Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-solvers.html Frees all local data associated with the solver. ```APIDOC ## virtual void rocalution::Solver::Clear(void) ### Description Clears the solver by freeing all allocated local data. ``` -------------------------------- ### Symmetric Gauss-Seidel and SSOR Preconditioner Class Definition Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Defines the SGS preconditioner class template for symmetric iterative approximation. ```cpp template class SGS : public rocalution::Preconditioner ``` -------------------------------- ### Setting LocalMatrix raw data pointer for CSR Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-single-node-comp.html Initializes a LocalMatrix with externally allocated CSR data using SetDataPtrCSR. ```cpp // Allocate a CSR matrix int* csr_row_ptr = new int[100 + 1]; int* csr_col_ind = new int[345]; ValueType* csr_val = new ValueType[345]; // Fill the CSR matrix // ... // rocALUTION local matrix object LocalMatrix mat; // Set the CSR matrix data, csr_row_ptr, csr_col and csr_val pointers become // invalid mat.SetDataPtrCSR(&csr_row_ptr, &csr_col, &csr_val, "my_matrix", 345, 100, 100); ``` -------------------------------- ### Define Block-Jacobi Preconditioner Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Class definition for the Block-Jacobi preconditioner designed for MPI environments. ```cpp template class BlockJacobi : public rocalution::Preconditioner ``` -------------------------------- ### rocalution::ILUT::Set Source: https://rocm.docs.amd.com/projects/rocALUTION/en/latest/reference/rocALUTION-preconditioners.html Configures the drop-off threshold and fill-in limits for ILUT factorization. ```APIDOC ## void rocalution::ILUT::Set(double t) ## void rocalution::ILUT::Set(double t, int maxrow) ### Description Sets the parameters for Incomplete LU Factorization based on threshold. Fill-in values are dropped depending on the threshold and optional maximum fill-ins per row. ### Parameters - **t** (double) - The drop-off threshold. - **maxrow** (int) - The maximum number of fill-ins allowed per row. ```