### Run dbt-bouncer with Example Configuration Source: https://godatadriven.github.io/dbt-bouncer/stable/CONTRIBUTING Executes the dbt-bouncer script using a provided example configuration file. This command assumes that a virtual environment is activated and dbt-bouncer is installed. ```bash uv run dbt-bouncer --config-file dbt-bouncer-example.yml ``` -------------------------------- ### Install dbt-bouncer and Dependencies with make and uv Source: https://godatadriven.github.io/dbt-bouncer/stable/CONTRIBUTING Installs dbt-bouncer, its dependencies, and the 'prek' tool. This setup ensures that local code changes are immediately reflected in dbt-bouncer runs. Requires a virtual environment to be set up first. ```bash make install uv run prek install ``` -------------------------------- ### Configure dbt-bouncer in GitHub Actions Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Sets up a GitHub Actions workflow to run dbt-bouncer as part of a CI pipeline. It includes steps for checking out code, generating dbt artifacts, and using the dbt-bouncer action with various configuration options. ```yaml name: CI pipeline on: pull_request: branches: - main jobs: run-dbt-bouncer: permissions: pull-requests: write # Required to write a comment on the PR runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Generate or fetch dbt artifacts run: ... - uses: godatadriven/dbt-bouncer@vX.X with: check: '' # optional, comma-separated check names to run config-file: ./ only: manifest_checks # optional, defaults to running all checks output-file: results.json # optional, default does not save a results file output-format: json # optional, one of: csv, json, junit, sarif, tap. Defaults to json output-only-failures: false # optional, defaults to true send-pr-comment: true # optional, defaults to true show-all-failures: false # optional, defaults to false verbose: false # optional, defaults to false ``` -------------------------------- ### Install dbt-bouncer using pip Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Installs the dbt-bouncer package from PyPI using pip. This is the standard method for installing Python packages and makes the `dbt-bouncer` command available in your environment. ```bash pip install dbt-bouncer ``` -------------------------------- ### Run dbt-bouncer with a configuration file Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Executes the dbt-bouncer tool, specifying the path to its configuration file. This command validates your dbt project against the rules defined in the configuration. ```bash dbt-bouncer --config-file ``` -------------------------------- ### dbt-bouncer Configuration for Custom Checks Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Configuration file for dbt-bouncer, specifying the directory for custom checks and defining which manifest checks to run and on which models. ```yaml custom_checks_dir: my_custom_checks manifest_checks: - name: check_model_deprecation_date include: ^models/staging/legacy_erp ``` -------------------------------- ### Install and Run Code Checks with Prek Source: https://godatadriven.github.io/dbt-bouncer/stable/CONTRIBUTING The `prek` tool is used to enforce code formatting and linting standards. It can be installed locally using `uv run prek install`. After installation, `prek` integrates with git pre-commit hooks to automatically ensure code quality. ```bash uv run prek install ``` -------------------------------- ### Configure Macro Argument Description Check (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/manifest/check_macros YAML configuration for the `check_macro_arguments_description_populated` manifest check. This example shows how to set a minimum description length for macro arguments. ```yaml manifest_checks: - name: check_macro_arguments_description_populated min_description_length: 25 # Setting a stricter requirement for description length ``` -------------------------------- ### Run dbt-bouncer using uvx Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Executes dbt-bouncer as a standalone Python executable using `uvx`. This method ensures a consistent execution environment, especially when managing multiple Python projects. ```bash uvx dbt-bouncer --config-file ``` -------------------------------- ### Example Configuration for CheckColumnNames (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/catalog/check_columns This YAML configuration demonstrates how to use the `check_column_names` within the `catalog_checks` section of a dbt-bouncer configuration file. It specifies the `column_name_pattern` to enforce lowercase column names with underscores allowed. ```yaml catalog_checks: - name: check_column_names column_name_pattern: [a-z_] # Lowercase only, underscores allowed ``` -------------------------------- ### Run dbt-bouncer using Docker Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Executes dbt-bouncer within a Docker container. This approach isolates the tool and its dependencies, ensuring consistent execution across different environments. It mounts the current directory to allow access to dbt artifacts and configuration files. ```bash docker run --rm \ --volume "$PWD":/app \ ghcr.io/godatadriven/dbt-bouncer:vX.X.X \ --config-file /app/ ``` -------------------------------- ### Programmatically invoke dbt-bouncer in Python Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Demonstrates how to run dbt-bouncer directly from a Python script using the `run_bouncer` function. This allows for integration into custom Python workflows and provides control over configuration and output options. The function returns the exit code of the dbt-bouncer process. ```python from pathlib import Path from dbt_bouncer.main import run_bouncer exit_code = run_bouncer( config_file=Path("path/to/dbt-bouncer.yml"), output_file=Path("results.json"), # optional output_format="json", # optional, one of: "csv", "json", "junit", "sarif", "tap". Defaults to "json" ) ``` -------------------------------- ### dbt-bouncer Check Inclusion Example (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/config_file Illustrates how to use the `include` argument at the check level in dbt-bouncer configuration. This example ensures that the `check_model_names` only runs on models located within the `./models/staging` directory. ```yaml # Specify `include` at the check level only manifest_checks: - name: check_model_names include: ^models/staging model_name_pattern: ^stg_ ``` -------------------------------- ### Monorepo dbt-bouncer Configuration Example Source: https://godatadriven.github.io/dbt-bouncer/stable/faq This YAML configuration sets up dbt-bouncer for a monorepo environment. It specifies the directory for dbt artifacts and uses an 'include' directive to target a specific dbt project within the monorepo, along with a manifest check. ```yaml dbt_artifacts_dir: dbt-project/target include: ^dbt-project manifest_checks: - name: check_exposure_based_on_non_public_models ``` -------------------------------- ### Run Performance Tests with Make Source: https://godatadriven.github.io/dbt-bouncer/stable/CONTRIBUTING Performance tests for the `dbt-bouncer` CLI can be executed using the `make test-perf` command. This functionality relies on the `bencher` and `hyperfine` tools being installed in the environment. ```makefile make test-perf ``` -------------------------------- ### dbt-bouncer CI Configuration Example Source: https://godatadriven.github.io/dbt-bouncer/stable/faq This YAML configuration defines checks for dbt-bouncer to be used in a CI pipeline. It includes checks for column descriptions, model directories, and maximum execution time. These checks help ensure data quality and performance within dbt projects. ```yaml catalog_checks: - name: check_column_description_populated include: ^models/marts manifest_checks: - name: check_model_directories include: ^models permitted_sub_directories: - intermediate - marts - staging - utilities run_results_checks: - name: check_run_results_max_execution_time max_execution_time_seconds: 10 ``` -------------------------------- ### dbt-bouncer init Command Source: https://godatadriven.github.io/dbt-bouncer/stable/cli Initializes a basic dbt-bouncer.yml configuration file. ```APIDOC ## dbt-bouncer init Command ### Description This command creates a basic `dbt-bouncer.yml` file in the current directory if it does not already exist. ### Method CLI Command ### Endpoint N/A ### Parameters #### Options - **--help** (flag) - Optional - Show this message and exit. ### Request Example ```bash dbt-bouncer init ``` ### Response #### Success Response Creation of `dbt-bouncer.yml` file. #### Response Example (No direct output, but a file is created) ``` -------------------------------- ### Run dbt-bouncer in verbose mode Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Executes dbt-bouncer with verbose logging enabled, providing detailed output about the configuration loading and validation process. This is useful for debugging. ```bash dbt-bouncer --config-file -v ``` -------------------------------- ### Initialize dbt-bouncer Configuration Source: https://godatadriven.github.io/dbt-bouncer/stable/cli Command to create a basic dbt-bouncer.yml configuration file. This command will raise a RuntimeError if the configuration file already exists. ```bash dbt-bouncer init [OPTIONS] ``` -------------------------------- ### Python Custom Check for dbt-bouncer Source: https://godatadriven.github.io/dbt-bouncer/stable/getting_started Defines a custom check inheriting from BaseCheck for dbt-bouncer. It requires a model with a 'deprecation_date' and fails if it's not set. The check is named 'check_model_deprecation_date'. ```python from typing import TYPE_CHECKING, Literal from pydantic import Field from dbt_bouncer.check_base import BaseCheck from dbt_bouncer.utils import get_clean_model_name if TYPE_CHECKING: import warnings with warnings.catch_warnings(): warnings.filterwarnings("ignore", category=UserWarning) from dbt_bouncer.parsers import DbtBouncerModelBase class CheckModelDepcrecationDate(BaseCheck): model: "DbtBouncerModelBase" = Field(default=None) name: Literal["check_model_deprecation_date"] def execute(self) -> None: """Execute the check.""" assert self.model.deprecation_date is not None, f"`{get_clean_model_name(self.model.unique_id)}` requires a `deprecation_date` to be set." ``` -------------------------------- ### Implement CheckColumnDescriptionPopulated in Python Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/catalog/check_columns This Python code defines the `CheckColumnDescriptionPopulated` class, inheriting from `BaseCheck`. It includes parameters for minimum description length and configuration options like include/exclude patterns and severity. The `execute` method iterates through catalog nodes and their columns, checking if descriptions are populated based on the adapter type and manifest information. It raises a `DbtBouncerFailedCheckError` if any non-complying columns are found. ```python class CheckColumnDescriptionPopulated(BaseCheck): """Columns must have a populated description. Parameters: min_description_length (int | None): Minimum length required for the description to be considered populated. Receives: catalog_node (CatalogNodes): The CatalogNodes object to check. manifest_obj (DbtBouncerManifest): The DbtBouncerManifest object parsed from `manifest.json`. models (list[DbtBouncerModelBase]): List of DbtBouncerModelBase objects parsed from `manifest.json`. Other Parameters: description (str | None): Description of what the check does and why it is implemented. exclude (str | None): Regex pattern to match the model path. Model paths that match the pattern will not be checked. include (str | None): Regex pattern to match the model path. Only model paths that match the pattern will be checked. severity (Literal["error", "warn"] | None): Severity level of the check. Default: `error`. Example(s): ```yaml manifest_checks: - name: check_column_description_populated include: ^models/marts ``` ```yaml manifest_checks: - name: check_column_description_populated min_description_length: 25 # Setting a stricter requirement for description length ``` """ catalog_node: "CatalogNodes | None" = Field(default=None) manifest_obj: "DbtBouncerManifest | None" = Field(default=None) min_description_length: int | None = Field(default=None) models: list["DbtBouncerModelBase"] = Field(default=[]) name: Literal["check_column_description_populated"] def execute(self) -> None: """Execute the check. Raises: DbtBouncerFailedCheckError: If description is not populated. """ if self.catalog_node is None: raise DbtBouncerFailedCheckError("self.catalog_node is None") if self.manifest_obj is None: raise DbtBouncerFailedCheckError("self.manifest_obj is None") if self.is_catalog_node_a_model(self.catalog_node, self.models): model = next( m for m in self.models if m.unique_id == self.catalog_node.unique_id ) non_complying_columns = [] for _, v in self.catalog_node.columns.items(): # Snowflake saves column descriptions in the 'comment' field in catalog.json if self.manifest_obj.manifest.metadata.adapter_type in ["snowflake"]: description = getattr(v, "comment", "") or "" else: columns = model.columns or {} column_from_manifest = columns.get(v.name) description = "" if column_from_manifest: description = column_from_manifest.description or "" if not self._is_description_populated( description, self.min_description_length ): non_complying_columns.append(v.name) if non_complying_columns: raise DbtBouncerFailedCheckError( f"Columns {non_complying_columns} in model {model.name} do not have populated descriptions." ) ``` -------------------------------- ### GitHub Actions Workflow for dbt Cloud Artifacts Source: https://godatadriven.github.io/dbt-bouncer/stable/faq This GitHub Actions workflow demonstrates how to download dbt artifacts from dbt Cloud and then run dbt-bouncer. It requires checkout, downloading artifacts using a specific action, and then executing dbt-bouncer. The workflow is triggered on pull requests to the main branch. ```yaml name: CI pipeline on: pull_request: branches: - main jobs: download-artifacts: runs-on: ubuntu-latest permissions: pull-requests: write steps: - name: Checkout uses: actions/checkout@v4 - name: Download dbt artifacts uses: pgoslatara/dbt-cloud-download-artifacts-action@v1 with: commit-sha: ${{ github.event.pull_request.head.sha }} dbt-cloud-api-token: ${{ secrets.DBT_CLOUD_API_TOKEN }} - name: Run dbt-bouncer uses: godatadriven/dbt-bouncer@vX.X ``` -------------------------------- ### dbt-bouncer Model Name Check with Description (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/config_file Demonstrates how to add a descriptive message to a dbt-bouncer manifest check. This specific example configures the `check_model_names` to ensure models in the staging layer start with 'stg_', providing a clear explanation in the failure message. ```yaml manifest_checks: - name: check_model_names description: Models in the staging layer should always start with "stg_". include: ^models/staging model_name_pattern: ^stg_ ``` -------------------------------- ### dbt-bouncer CLI Usage Source: https://godatadriven.github.io/dbt-bouncer/stable/cli The main entrypoint for the dbt-bouncer CLI. It accepts various options to control configuration, check execution, and output. The command structure is 'dbt-bouncer [OPTIONS] COMMAND [ARGS]...'. ```bash dbt-bouncer [OPTIONS] COMMAND [ARGS]... ``` -------------------------------- ### Configure Max Lines Check in dbt_project.yml (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/manifest/check_models This YAML configuration demonstrates how to enable and set the maximum number of lines for dbt models using the `check_model_max_number_of_lines` check within a dbt project. It shows a basic setup and an example with a custom line limit. ```yaml manifest_checks: - name: check_model_max_number_of_lines ``` ```yaml manifest_checks: - name: check_model_max_number_of_lines max_number_of_lines: 150 ``` -------------------------------- ### CheckMacroDescriptionPopulated Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/manifest/check_macros Ensures that macros have a populated description. ```APIDOC ## CheckMacroDescriptionPopulated ### Description Macros must have a populated description. ### Method N/A (This is a configuration check, not an API endpoint) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body This check is configured within the `manifest_checks` section of a dbt project configuration file. - **`name`** (string) - Required - The name of the check to enable, which is `check_macro_description_populated`. - **`exclude`** (string | null) - Optional - Regex pattern to match the macro path. Macro paths that match the pattern will not be checked. - **`include`** (string | null) - Optional - Regex pattern to match the macro path. Only macro paths that match the pattern will be checked. - **`severity`** (literal['error', 'warn'] | null) - Optional - Severity level of the check. Default: `error`. ### Request Example ```yaml manifest_checks: - name: check_macro_description_populated exclude: "macros/utils/.*\.sql$" severity: warn ``` ### Response #### Success Response (200) N/A (This check runs during dbt's manifest generation or validation process and reports issues directly.) #### Response Example N/A ``` -------------------------------- ### CheckMacroArgumentsDescriptionPopulated Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/manifest/check_macros Ensures that macro arguments have a populated description, with an optional minimum length requirement. ```APIDOC ## CheckMacroArgumentsDescriptionPopulated ### Description Macro arguments must have a populated description. ### Method N/A (This is a configuration check, not an API endpoint) ### Endpoint N/A ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body This check is configured within the `manifest_checks` section of a dbt project configuration file. - **`name`** (string) - Required - The name of the check to enable, which is `check_macro_arguments_description_populated`. - **`min_description_length`** (integer | null) - Optional - Minimum length required for the description to be considered populated. Defaults to null (any non-empty description is sufficient). - **`exclude`** (string | null) - Optional - Regex pattern to match the macro path. Macro paths that match the pattern will not be checked. - **`include`** (string | null) - Optional - Regex pattern to match the macro path. Only macro paths that match the pattern will be checked. - **`severity`** (literal['error', 'warn'] | null) - Optional - Severity level of the check. Default: `error`. ### Request Example ```yaml manifest_checks: - name: check_macro_arguments_description_populated min_description_length: 10 exclude: "models/staging/.*\.sql$" severity: warn ``` ### Response #### Success Response (200) N/A (This check runs during dbt's manifest generation or validation process and reports issues directly.) #### Response Example N/A ``` -------------------------------- ### dbt-bouncer Manifest Check Configuration (YAML) Source: https://godatadriven.github.io/dbt-bouncer/stable/checks/manifest/check_models Example YAML configurations for the check_model_max_chained_views manifest check in dbt-bouncer. These examples demonstrate how to enable the check and specify parameters like materializations_to_include and max_chained_views. ```yaml manifest_checks: - name: check_model_max_chained_views ``` ```yaml manifest_checks: - name: check_model_max_chained_views materializations_to_include: - ephemeral - my_custom_materialization - view max_chained_views: 5 ``` -------------------------------- ### dbt-bouncer Configuration in TOML Source: https://godatadriven.github.io/dbt-bouncer/stable/config_file Example of a dbt-bouncer configuration file written in TOML format. This configuration mirrors the YAML example, specifying the dbt artifacts directory and defining manifest checks with their respective parameters. ```toml [tool.dbt-bouncer] # [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `$DBT_PROJECT_DIR/target` if $DBT_PROJECT_DIR is set, else `./target`. dbt_artifacts_dir = "target" [[tool.dbt-bouncer.manifest_checks]] name = "check_macro_name_matches_file_name" [[tool.dbt-bouncer.manifest_checks]] name = "check_model_names" include = "^models/staging" model_name_pattern = "^stg_" ```