### dbt Project Setup and Execution Example (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt Demonstrates basic dbt commands for cloning a sample project, installing dependencies, running models, and executing tests. This is a fundamental workflow for getting started with dbt. ```bash git clone https://github.com/dbt-labs/jaffle_shop.git cd jaffle_shop dbt deps dbt run dbt test ``` -------------------------------- ### Set up dbt Project with Jaffle Shop (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This sequence of bash commands outlines the process of cloning the Jaffle Shop dbt project, installing its dependencies, loading data via `dbt seed`, running the dbt models, testing data quality, and generating documentation. It serves as a practical guide to starting with a reference dbt project. ```bash # Example: Clone and explore the Jaffle Shop reference project git clone https://github.com/dbt-labs/jaffle_shop.git cd jaffle_shop # Install dependencies dbt deps # Run seeds (load CSV data) dbt seed # Run models dbt run # Test data quality dbt test # Generate documentation dbt docs generate dbt docs serve ``` -------------------------------- ### Build MRR Playbook with dbt (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This example demonstrates setting up and running the dbt MRR (Monthly Recurring Revenue) playbook. It involves cloning the repository, installing dependencies, seeding data, running models, and testing. The playbook focuses on subscription revenue modeling, cohort analysis, and churn calculation. ```bash # Example: Explore MRR (Monthly Recurring Revenue) playbook git clone https://github.com/dbt-labs/mrr-playbook.git cd mrr-playbook # This project demonstrates subscription revenue modeling: # - Customer cohort analysis # - Churn calculation # - MRR tracking and forecasting # - Expansion and contraction revenue dbt deps && dbt seed && dbt run && dbt test ``` -------------------------------- ### Configure SQLFluff for dbt Projects (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This example shows how to install SQLFluff with dbt templating and configure it for dbt projects. It includes setting up a .sqlfluff file to specify the dbt project directory, profiles directory, and dialect. The commands demonstrate linting and fixing SQL files within dbt models. ```bash # Example: Use SQLFluff for SQL linting pip install sqlfluff sqlfluff-templater-dbt # .sqlfluff configuration cat << EOF > .sqlfluff [sqlfluff] templater = dbt dialect = snowflake exclude_rules = L031,L034 max_line_length = 120 [sqlfluff:templater:dbt] project_dir = . profiles_dir = ~/.dbt profile = my_dbt_project target = dev EOF # Lint models sqlfluff lint models/ # Fix auto-fixable issues sqlfluff fix models/staging/ ``` -------------------------------- ### dbt Expectations Package Installation (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt Shows how to declare the `dbt-expectations` package in a dbt project's `packages.yml` file to enable advanced data quality testing capabilities. ```yaml packages: - package: calogica/dbt_expectations version: 0.10.1 ``` -------------------------------- ### Deploy dbt Docs with GitHub Actions (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt A GitHub Actions workflow to automatically generate and deploy dbt documentation as a static website to GitHub Pages. It checks out the code, installs dbt, generates docs, and deploys the output. ```yaml name: Deploy dbt Docs on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Generate dbt docs run: | dbt deps dbt docs generate - name: Deploy to GitHub Pages uses: peaceiris/actions-gh-pages@v3 with: github_token: ${{ secrets.GITHUB_TOKEN }} publish_dir: ./target ``` -------------------------------- ### Contribute to Awesome dbt Repository (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This guide outlines the steps for contributing to the Awesome dbt project. It covers forking the repository, adding new resources to the README.md file (following LIFO order within sections), committing changes, and creating a pull request. It emphasizes adhering to the existing format and ensuring link validity. ```bash # Example: Contributing a new resource # 1. Fork the repository git clone https://github.com/YOUR_USERNAME/awesome-dbt.git cd awesome-dbt # 2. Add your resource at the TOP of the relevant section (LIFO) # Edit README.md # - [Your Resource Title](https://your-link.com) - Brief description. # 3. Commit your changes git add README.md git commit -m "Add [Your Resource Name] to [Section Name]" # 4. Push and create a pull request git push origin main # Remember: # - Add new entries at the TOP of sections (LIFO) # - Follow the existing format # - Ensure links are valid ``` -------------------------------- ### dbt Snowflake Adapter Configuration Example (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt Illustrates the configuration of a dbt adapter for Snowflake, including connection details like account, user, password (via environment variable), role, database, warehouse, schema, and thread count. This is typically defined in a profiles.yml file. ```yaml my_dbt_project: target: dev outputs: dev: type: snowflake account: abc123 user: dbt_user password: "{{ env_var('DBT_PASSWORD') }}" role: transformer database: analytics warehouse: transforming schema: dbt_dev threads: 4 ``` -------------------------------- ### dbt CI with GitHub Actions and SQLFluff (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt A GitHub Actions workflow for dbt Continuous Integration (CI). It sets up Python, installs dbt and Snowflake adapter, runs `dbt debug` and `dbt compile`, performs tests on modified models using Slim CI, and lints SQL code with SQLFluff. ```yaml name: dbt CI on: pull_request: branches: [main] jobs: dbt-ci: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.9' - name: Install dbt run: pip install dbt-snowflake - name: Run dbt debug run: dbt debug - name: Run dbt compile run: dbt compile - name: Run dbt tests on changed models run: | dbt test --select state:modified+ --state ./prod-manifest/ - name: Run SQLFluff linting run: | pip install sqlfluff-templater-dbt sqlfluff lint models/ ``` -------------------------------- ### dbt Slim CI Pattern for Model Selection (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt An example of the Slim CI pattern in dbt, which optimizes CI runs by only testing models that have been modified. It uses the `--select state:modified+` flag and a state reference file. ```bash dbt run --select state:modified+ --state ./prod-manifest/ ``` -------------------------------- ### dbt Expectations Model Testing Example (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt An example of using `dbt-expectations` within a dbt model's `schema.yml` file to define data quality tests. It includes tests for uniqueness, non-nullability, value ranges, and column types. ```yaml version: 2 models: - name: orders columns: - name: order_id tests: - unique - not_null - dbt_expectations.expect_column_values_to_be_between: min_value: 0 max_value: 1000000 - name: order_date tests: - dbt_expectations.expect_column_values_to_be_of_type: column_type: date ``` -------------------------------- ### Declare dbt Packages in packages.yml (YAML) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This YAML configuration lists common dbt packages used to extend dbt functionality. It specifies packages like dbt_utils, codegen, dbt_expectations, audit_helper, and dbt_artifacts with their respective versions. The `dbt deps` command is used to install these packages. ```yaml # Example: Common packages for most dbt projects # packages.yml packages: - package: dbt-labs/dbt_utils version: 1.1.1 - package: dbt-labs/codegen version: 0.12.1 - package: calogica/dbt_expectations version: 0.10.1 - package: dbt-labs/audit_helper version: 0.9.0 - package: brooklyn-data/dbt_artifacts version: 2.6.3 # Install packages # Terminal dbt deps ``` -------------------------------- ### dbt Blue/Green Deployment on Snowflake (SQL) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt Demonstrates a blue/green deployment strategy for dbt models on Snowflake. It involves deploying to a staging schema first, followed by swapping the production and staging schemas after validation. ```sql # Deploy to staging schema first dbt run --target staging # Swap schemas after validation USE DATABASE analytics; ALTER SCHEMA production SWAP WITH staging; ``` -------------------------------- ### Manage dbt YAML Files with dbt-osmosis (Bash) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This snippet illustrates using dbt-osmosis to manage dbt project YAML files. It shows commands for scaffolding YAML files for undocumented models and organizing YAML based on directory structure, simplifying documentation and configuration management. ```bash # Example: Use dbt-osmosis for YAML management pip install dbt-osmosis # Scaffold YAML files for undocumented models dbt-osmosis yaml scaffold # Organize YAML based on directory structure dbt-osmosis yaml organize ``` -------------------------------- ### Orchestrate dbt with Airflow using Cosmos (Python) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt A placeholder comment indicating the use of the Cosmos library for orchestrating dbt projects within Apache Airflow. This suggests a Python-based approach to defining dbt DAGs in Airflow. ```python # Example: Orchestrate dbt with Airflow using Cosmos ``` -------------------------------- ### Generate Surrogate Key with dbt-utils (SQL) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This SQL snippet demonstrates using the `generate_surrogate_key` macro from the dbt-utils package. It creates a unique key for customer records based on `customer_id` and `email`, suitable for dimension tables in a data warehouse. ```sql # Example: Use dbt-utils surrogate_key macro # models/dim_customer.sql {{ config(materialized='table') }} select {{ dbt_utils.generate_surrogate_key(['customer_id', 'email']) }} as customer_key, customer_id, email, first_name, last_name, created_at from {{ ref('stg_customers') }} ``` -------------------------------- ### Create dbt DAG with Cosmos in Airflow (Python) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This snippet demonstrates how to define a dbt DAG within Airflow using the Cosmos library. It configures project, profile, and execution settings, including specifying the dbt executable path and Snowflake connection details. The DAG is scheduled to run daily and does not catch up. ```python from airflow import DAG from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig from cosmos.profiles import SnowflakeUserPasswordProfileMapping from datetime import datetime profile_config = ProfileConfig( profile_name="my_dbt_project", target_name="prod", profile_mapping=SnowflakeUserPasswordProfileMapping( conn_id="snowflake_conn", profile_args={"database": "analytics", "schema": "dbt_prod"}, ) ) dbt_dag = DbtDag( project_config=ProjectConfig("/path/to/dbt/project"), profile_config=profile_config, execution_config=ExecutionConfig(dbt_executable_path="/usr/local/bin/dbt"), schedule_interval="@daily", start_date=datetime(2024, 1, 1), catchup=False, dag_id="dbt_daily_run", ) ``` -------------------------------- ### Temporal Join Macro for dbt Snapshots (SQL) Source: https://context7.com/hiflylabs/awesome-dbt/llms.txt This dbt macro, `trange_join`, facilitates temporal joins between dbt snapshots or other time-aware models. It allows for different join types and dynamically joins two models based on their valid time ranges, using `dbt_valid_from` and `dbt_valid_to` columns. ```sql -- Example: Temporal join macro for dbt snapshots -- macros/trange_join.sql {% macro trange_join(left_model, right_model, left_key, right_key, join_type='inner') %} from {{ ref(left_model) }} as l {{ join_type }} join {{ ref(right_model) }} as r on l.{{ left_key }} = r.{{ right_key }} and l.dbt_valid_from <= r.dbt_valid_to and l.dbt_valid_to >= r.dbt_valid_from {% endmacro %} -- Usage in model select orders.order_id, customers.customer_name, orders.order_date {{ trange_join('orders_snapshot', 'customers_snapshot', 'customer_id', 'customer_id') }} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.