### Install MkDocs Material and Serve Docs Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/contributing.md Install the necessary package for local documentation serving and then start the local development server. Docs are automatically pushed to the website via CI/CD. ```bash pip install mkdocs-material mkdocs serve ``` -------------------------------- ### JSON Output Example Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md This is an example of the JSON output format, suitable for programmatic consumption and automation. ```json [ { "test_name": "dbt_project_evaluator.marts.dag.is_empty_fct_model_fanout_", "results": [ {"resource_name": "my_model", "num_dependents": 5} ] }, { "test_name": "dbt_project_evaluator.marts.tests.is_empty_fct_missing_primary_key_tests_", "results": [ {"resource_name": "stg_orders", "resource_type": "model"} ] } ] ``` -------------------------------- ### Staging Directory Structure Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Example of a well-organized staging directory structure with subdirectories for each data source (e.g., 'braintree', 'stripe'). ```bash ├── dbt_project.yml └── models ├── marts └── staging ├── braintree └── stripe ``` -------------------------------- ### Model Naming Convention Example Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Illustrates a dbt project structure where a model is nested in the 'marts' subdirectory and requires a prefix like 'fct_' or 'dim_'. ```bash ├── dbt_project.yml └── models ├── marts └── model_8.sql ``` -------------------------------- ### Configure Model Access for Exposures (YAML) Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/governance.md Example YAML configuration showing how to set the 'access' property for dbt models. Ensure models that are depended upon by exposures are set to 'public'. ```yaml models: - name: fct_model_6 description: very important OKR reporting model access: public config: materialized: table contract: enforced: true columns: - name: id description: the primary key of my OKR model data_type: integer - name: dim_model_7 description: excellent model access: private ``` ```yaml models: - name: fct_model_6 description: very important OKR reporting model access: public config: materialized: table contract: enforced: true columns: - name: id description: the primary key of my OKR model data_type: integer - name: dim_model_7 description: excellent model access: public ``` -------------------------------- ### Filter JSON Output with `jq` Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Examples of using `jq` to filter and extract specific information from the JSON output. ```bash # Get all test names dbt build --select package:dbt_project_evaluator -q | jq '.[].test_name' ``` ```bash # Get results for a specific test dbt build --select package:dbt_project_evaluator -q | jq '.[] | select(.test_name | contains("fanout"))' ``` ```bash # Count total violations dbt build --select package:dbt_project_evaluator -q | jq '[.[].results | length] | add' ``` -------------------------------- ### Model Directory Rule: Staging Model Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Shows an example of a staging model ('stg_model_3') incorrectly nested in 'source_1' instead of its correct parent source directory 'source_2'. ```bash ├── dbt_project.yml └── models ├── marts └── staging └── source_1 ├── stg_model_3.sql ``` -------------------------------- ### Public Model Without Documentation Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/governance.md Example of a public model missing model-level and column-level descriptions. This check highlights models that may be difficult for consumers to understand and use. ```yaml # public model without documentation models: - name: report_1 access: public columns: - name: id ``` -------------------------------- ### Public Model Without Contract Configuration Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/governance.md Example of a public model definition missing the contract configuration. This check flags models that lack data type and column enforcement. ```yaml # public model without a contract models: - name: report_1 description: very important OKR reporting model access: public ``` -------------------------------- ### Override dbt Project Evaluator Variables Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Customize test and documentation coverage targets, and specify primary key test macros by overriding the default variables in your dbt_project.yml file. This example sets coverage targets to 75% and uses `dbt_constraints.test_primary_key` for primary key validation. ```yaml vars: dbt_project_evaluator: documentation_coverage_target: 75 test_coverage_target: 75 primary_key_test_macros: [["dbt_constraints.test_primary_key"]] ``` -------------------------------- ### Override chained_views_threshold Variable Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Adjust the threshold for unacceptable chain lengths of views. The default is 4, but this example sets it to 8. ```yaml vars: dbt_project_evaluator: # set your chained views threshold to 8 instead of 4 chained_views_threshold: 8 ``` -------------------------------- ### Disable dbt Project Evaluator Models in Production Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/ci-check.md Optionally disable dbt Project Evaluator models in production environments by setting an environment variable. This example shows how to enable models only when the environment variable is set to 'true'. ```yaml models: dbt_project_evaluator: +enabled: "{{ env_var('DBT_PROJECT_EVALUATOR_ENABLED', 'true') | lower == 'true' | as_bool }}" ``` -------------------------------- ### Intermediate and Marts Model Directory Structure Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Shows the recommended directory structure for intermediate and mart models, ensuring files are nested closest to their model type folder. ```bash ├── dbt_project.yml └── models └── marts ├── fct_model_6.sql └── intermediate └── int_model_5.sql ``` -------------------------------- ### Run dbt Build with Custom Seed Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/exceptions.md Execute this command to build the dbt Project Evaluator package along with your custom exception seed. ```bash dbt build --select package:dbt_project_evaluator dbt_project_evaluator_exceptions ``` -------------------------------- ### Configure Output Format and Quoting Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Customize the output format to 'table', 'csv', or 'json', and specify quoting characters for database and schema names. ```yaml on-run-end: - "{{ dbt_project_evaluator.print_dbt_project_evaluator_issues(format='json', quote='"') }}" ``` -------------------------------- ### Model Test Directory Structure Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Model tests and documentation should be located in a '.yml' file within the same subdirectory as the corresponding model, under 'models/'. This ensures tests are co-located with the models they validate. ```bash ├── dbt_project.yml └── models └── marts ├── int_model_4.sql └── staging ├── staging.yml ``` ```bash ├── dbt_project.yml └── models └── marts ├── int_model_4.sql ├── marts.yml └── staging ├── staging.yml ``` -------------------------------- ### Configure `use_native_agate_printing` in `dbt_project.yml` Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Set `use_native_agate_printing` to `false` to use a pure Jinja implementation for printing issues. This is necessary for environments where agate's native printing functions do not work, such as dbt Cloud CLI/IDE, or when using `format='csv'` with dbt Fusion. ```yaml vars: dbt_project_evaluator: # use pure Jinja printing for dbt Cloud CLI, IDE, or Fusion with CSV format use_native_agate_printing: false ``` -------------------------------- ### Document Public Model with Columns Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/governance.md Add model-level and column-level descriptions to public models to improve usability for consumers. This is a stricter check than general model documentation. ```yaml models: - name: report_1 description: very important OKR reporting model access: public columns: - name: id description: the primary key of my OKR model ``` -------------------------------- ### Configure `on-run-end` Hook Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Add this line to your `dbt_project.yml` to automatically print evaluation issues to the logs after each run. ```yaml on-run-end: - "{{ dbt_project_evaluator.print_dbt_project_evaluator_issues() }}" ``` -------------------------------- ### Direct Join to Source Model Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md Identifies models that directly reference both another model and a source. This pattern is flagged to encourage using staging models as an intermediary. ```sql SELECT * FROM {{ ref('int_model_4') }} WHERE {{ source('source_1', 'table_5') }} AND {{ ref('model_1') }} ``` -------------------------------- ### Exclude Package and Paths Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/excluding-packages-and-paths.md Combine `exclude_packages` and `exclude_paths_from_project` to exclude both a specific package and multiple paths within the project. ```yaml vars: exclude_packages: ["upstream_package"] exclude_paths_from_project: ["models/legacy/", "/my_date_spine.sql"] ``` -------------------------------- ### Enable Pure Jinja Printing for dbt Cloud Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Set this variable in `dbt_project.yml` to use a Jinja-based printing method, which is required for dbt Cloud CLI/IDE and for CSV output in dbt Fusion. ```yaml vars: use_native_agate_printing: false ``` -------------------------------- ### Calculate Documentation Coverage Percentage Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/documentation.md This SQL model calculates the percentage of enabled models in a dbt project that have a configured description. It can warn if the coverage is below 100% and allows overriding the target threshold. ```sql --8<-- "models/marts/documentation/fct_documentation_coverage.sql" ``` -------------------------------- ### AWS Athena profiles.yml Configuration Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/integration_tests/README.md This sample profiles.yml file is used for local Athena tests. Ensure environment variables are set. ```yaml athena: # for local tests only type: athena s3_staging_dir: {{ env_var('ATHENA_S3_STAGING_DIR') }} s3_data_dir: {{ env_var('ATHENA_S3_DATA_DIR') }} s3_data_naming: schema_table_unique region_name: {{ env_var('ATHENA_REGION') }} schema: {{ env_var('ATHENA_SCHEMA') }} database: awsdatacatalog work_group: {{ env_var('ATHENA_WORKGROUP') }} num_retries: 2 threads: 4 ``` -------------------------------- ### Run dbt Build Commands in CI Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/ci-check.md Execute dbt build commands as part of your CI job. This includes building modified resources and then running the dbt Project Evaluator package, optionally including exception tests. ```bash dbt build --select state:modified+ --exclude package:dbt_project_evaluator dbt build --select package:dbt_project_evaluator ``` ```bash dbt build --select state:modified+ --exclude package:dbt_project_evaluator dbt build --select package:dbt_project_evaluator dbt_project_evaluator_exceptions ``` -------------------------------- ### Model Directory Rule: Marts Model Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Demonstrates a marts model ('dim_model_7') incorrectly nested within the 'intermediate' subdirectory instead of the 'marts' directory. ```bash ├── dbt_project.yml └── models └── marts └── intermediate ├── dim_model_7.sql ``` -------------------------------- ### Run with `--quiet` and `jq` for Clean JSON Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Combine dbt's `--quiet` flag with `jq` to suppress dbt logs and process the clean JSON output for automation tasks. ```bash dbt build --select package:dbt_project_evaluator -q | jq '.' ``` -------------------------------- ### Source Definition Directory Structure Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Source definitions should reside in a 'source.yml' file within a subdirectory named after the source, located under 'models/staging/'. This promotes organization when managing multiple data sources. ```bash ├── dbt_project.yml └── models ├── marts └── staging └── source_1 ├── source.yml ``` ```bash ├── dbt_project.yml └── models ├── marts └── staging ├── source_1 └── source_2 ├── source.yml ``` ```bash ├── dbt_project.yml └── models ├── marts └── staging ├── braintree └── stripe ``` -------------------------------- ### List Undocumented Sources Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/documentation.md This SQL model identifies and lists all dbt sources that do not have a description configured. It stresses the importance of source documentation for understanding data loading processes. ```sql fct_undocumented_sources ``` ```yaml sources: - name: my_source description: This is the source description tables: - name: my_table ``` -------------------------------- ### Set Environment Variables for AWS Athena Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/integration_tests/README.md Configure these environment variables before running local integration tests with AWS Athena. ```bash ATHENA_S3_STAGING_DIR= ATHENA_S3_DATA_DIR= ATHENA_REGION= ATHENA_SCHEMA= ATHENA_WORKGROUP= ``` -------------------------------- ### Configure dbt_project.yml for Specific Adapters Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/README.md This configuration is necessary for adapters like Databricks/Spark, DuckDB, Redshift, ClickHouse, and Fabric. It ensures the project can run across these adapters by managing macro overrides. ```yaml # dbt_project.yml dispatch: - macro_namespace: dbt search_order: ['dbt_project_evaluator', 'dbt'] ``` -------------------------------- ### Log Custom Rules with `is_empty` Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/issues-in-log.md Apply the `dbt_project_evaluator.is_empty` data test to custom models to log their results. ```yaml models: - name: my_custom_rule_model description: This is my custom project evaluator check data_tests: - dbt_project_evaluator.is_empty ``` -------------------------------- ### Model Directory Rule: Intermediate Model Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/structure.md Illustrates an intermediate model ('int_model_4') incorrectly nested within the 'marts' subdirectory instead of the 'intermediate' subdirectory. ```bash ├── dbt_project.yml └── models └── marts ├── int_model_4.sql ``` -------------------------------- ### Duplicate Source Definitions Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md Highlights database objects that are mapped to by more than one dbt source node. This can lead to inaccurate lineage. ```yaml sources: - name: source_1 schema: real_schema database: real_database tables: - name: table_5 - name: raw_table_5 identifier: table_5 ``` -------------------------------- ### List Undocumented Source Tables Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/documentation.md This SQL model lists all source tables in a dbt project that lack a configured description. It highlights the value of documenting source tables for project contributors. ```sql fct_undocumented_source_tables ``` ```yaml sources: - name: my_source tables: - name: my_table description: This is the source table description ``` -------------------------------- ### Configure Markdownlint for VSCode Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/contributing.md Configure the markdownlint extension in VSCode to enforce specific Markdown formatting rules. This JSON snippet should be placed within your VSCode settings. ```json { "markdownlint.config": { "ul-indent": {"indent": 4}, "MD036": false, "MD046": false } } ``` -------------------------------- ### Customize Model Naming Conventions Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Define custom model types, folder names, and prefixes to align with your project's structure. Update `model_types`, and add new variables like `_folder_name` and `_prefixes` in your dbt_project.yml. ```yaml vars: dbt_project_evaluator: model_types: ['staging', 'intermediate', 'marts', 'other', 'util'] util_folder_name: 'util' util_prefixes: ['util_'] ``` -------------------------------- ### dbt_project.yml Configuration for Specific Adapters Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/index.md This configuration is required for adapters like Databricks/Spark, DuckDB, Redshift, and Fabric. It ensures the package can run by overriding certain dbt core macros. ```yaml dispatch: - macro_namespace: dbt search_order: ['dbt_project_evaluator', 'dbt'] ``` -------------------------------- ### Avoid Hard-Coded References in dbt Models Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md Use `ref` for models and `source` for raw data instead of direct relation references. This ensures dbt can manage dependencies and build order correctly. ```sql with orders as ( select * from my_db.my_schema.orders ), customers as ( select * from my_schema.customers ) select orders.order_id, customers.name from orders left join customers on orders.customer_id = customers.id ``` ```sql with orders as ( select * from {{ ref('orders') }} ), customers as ( select * from {{ ref('customers') }} ) select orders.order_id, customers.name from orders left join customers on orders.customer_id = customers.id ``` -------------------------------- ### List Undocumented Models Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/documentation.md This SQL model identifies and lists all models within a dbt project that are missing a configured description. It emphasizes the importance of model documentation for downstream consumers. ```sql fct_undocumented_models ``` -------------------------------- ### Identify Unused Sources in dbt Project Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md This rule flags sources defined in your dbt project's YAML files that are not referenced by any models. This helps in cleaning up unused configurations. ```yaml sources: - name: some_source database: raw tables: - name: table_1 - name: table_2 - name: table_3 - name: table_4 # <-- remove this line ``` -------------------------------- ### Define Exceptions in CSV Seed Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/exceptions.md Use this CSV format to specify exceptions. The `id_to_exclude` can be a specific value or a `like` pattern. ```csv fct_name,column_name,id_to_exclude,comment fct_multiple_sources_joined,child,stg_%_unioned,Models called _unioned can union multiple sources ``` -------------------------------- ### Downstream Model Dependent on Source Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md Detects marts or intermediate models that directly depend on a source node. This indicates a potential missing staging model. ```sql SELECT * FROM {{ source('source_1', 'table_5') }} WHERE {{ ref('fct_model_9') }} ``` -------------------------------- ### Identify Staging Models Dependent on Marts or Intermediate Models Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md This rule flags staging models that depend on intermediate or marts models, which violates dbt best practices. Staging models should only select from source nodes. ```sql SELECT * FROM {{ ref('fct_staging_dependent_on_marts_or_intermediate') }} ``` -------------------------------- ### Add Contract to Public Model Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/governance.md Configure a public model with a contract to enforce data types and columns. This ensures downstream consumers receive consistent data. ```yaml models: - name: report_1 description: very important OKR reporting model access: public config: contract: enforced: true columns: - name: id data_type: integer ``` -------------------------------- ### Exclude Models/Sources in a Given Path Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/excluding-packages-and-paths.md Use `exclude_paths_from_project` to exclude models or sources located within specific directories or files. ```yaml vars: exclude_paths_from_project: ["models/legacy/"] ``` -------------------------------- ### Exclude a Whole Package Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/excluding-packages-and-paths.md Configure `exclude_packages` to prevent evaluation of models from a specified package. ```yaml vars: exclude_packages: ["upstream_package"] ``` -------------------------------- ### Identify Staging Models Dependent on Other Staging Models Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/rules/modeling.md This rule identifies parent/child relationships where staging models depend on other staging models. This may indicate a naming issue or that the child model should reference a source. ```sql SELECT * FROM {{ ref('fct_staging_dependent_on_staging') }} ``` -------------------------------- ### Disable Default Exception Seed in dbt_project.yml Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/exceptions.md To use a custom exception seed, disable the default one from the package by adding this configuration to your `dbt_project.yml` file. ```yaml seeds: dbt_project_evaluator: dbt_project_evaluator_exceptions: +enabled: false ``` -------------------------------- ### Override DAG Thresholds Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Adjust the `models_fanout_threshold` and `too_many_joins_threshold` to customize DAG complexity checks. Set these in your dbt_project.yml under the `vars` section for `dbt_project_evaluator`. ```yaml vars: dbt_project_evaluator: models_fanout_threshold: 10 too_many_joins_threshold: 6 ``` -------------------------------- ### Override Execution Variables Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/overriding-variables.md Customize 'insert_batch_size' for batch insertion and 'max_depth_dag' to limit the maximum distance between nodes in DAG relationship calculations. The default for 'insert_batch_size' is 10000, and 'max_depth_dag' varies by adapter (9 for BigQuery, Spark, Fabric; -1 for others). ```yaml vars: dbt_project_evaluator: # update the number of records inserted from the graph from 10,000 to 500 to reduce query size insert_batch_size: 500 # set the maximum distance between nodes to 5 max_depth_dag: 5 ``` -------------------------------- ### Override Test Severity in dbt_project.yml Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/ci-check.md Configure the severity of dbt Project Evaluator tests using an environment variable. This allows you to set tests to 'error' in CI environments and 'warn' in others. ```yaml data_tests: dbt_project_evaluator: +severity: "{{ env_var('DBT_PROJECT_EVALUATOR_SEVERITY', 'warn') }}" ``` -------------------------------- ### Disable Specific Model Tests in dbt_project.yml Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/customization.md To disable tests for a specific model, such as `fct_model_fanout`, set `+enabled: false` for that model within the `dag` configuration of the dbt Project Evaluator. This allows fine-grained control over which models are tested. ```yaml models: dbt_project_evaluator: marts: dag: # disable single DAG model fct_model_fanout: +enabled: false ``` -------------------------------- ### Disable Tests in dbt_project.yml Source: https://github.com/dbt-labs/dbt-project-evaluator/blob/main/docs/customization/customization.md To disable all tests from the dbt Project Evaluator package, set `+enabled: false` for the `tests` directory within the package's configuration. This is useful for completely deactivating the test coverage suite. ```yaml models: dbt_project_evaluator: marts: tests: # disable entire test coverage suite +enabled: false ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.