### Reusable default_args Example Source: https://docs.datacoves.com/docs/how-tos/airflow/send-emails This example demonstrates how to place default_args in a separate file for reusability, promoting a DRY (Don't Repeat Yourself) approach in your Airflow DAGs. ```python # Example of importing default_args from a utility file # from orchestrate.utils.default_args import default_args ``` -------------------------------- ### Install dbt Packages Source: https://docs.datacoves.com/docs/how-tos/dbt/compilation-errors Run this command in your terminal to install necessary dbt packages. Ensure you are in your dbt project directory. ```bash dbt deps ``` -------------------------------- ### Datacoves Environment Variables Example Source: https://docs.datacoves.com/docs/reference/airflow/airflow-config-defaults Example of Datacoves environment variables, illustrating how environment slugs are formatted. ```text DATACOVES__DAGS_FOLDER : The folder where Airflow will look for DAGs. This is defined when you set your `python DAGs path` in the environment setup. DATACOVES__ENVIRONMENT_SLUG : The slug for your environment. This is randomly generated upon creation of the environment. The development slug can be seen on the launchpad screen: `https://dev123.datacoves.localhost` in this case `DATACOVES__ENVIRONMENT_SLUG=dev123` ``` -------------------------------- ### Initialize DataHub CLI Source: https://docs.datacoves.com/docs/how-tos/datahub/how_to_datahub_cli Run this command in your VS Code terminal to start the DataHub CLI configuration process. You will be prompted to enter your DataHub host URL. ```bash datahub init ``` -------------------------------- ### Install ipdb and set debug environment variable Source: https://docs.datacoves.com/docs/how-tos/dbt/advenced-dbt-debug Install the ipdb package and set the DBT_MACRO_DEBUGGING environment variable to enable the debug() macro functionality for dbt macro debugging. ```bash pip install ipdb export DBT_MACRO_DEBUGGING=1 ``` -------------------------------- ### Authentication Header Example Source: https://docs.datacoves.com/docs/how-tos/datacoves/how_to_worker_usage Include your API key in the Authorization header for authenticated requests. Ensure the token is a Project-level token. ```http Authorization: Token YOUR_API_KEY ``` -------------------------------- ### Bundle Object Roles into Template Roles Source: https://docs.datacoves.com/docs/how-tos/snowflake/warehouses-schemas-roles Example of creating a template role (`z_base_analyst`) by granting multiple object roles. ```yaml - z_base_analyst: member_of: - z_db_raw - z_db_balboa - z_db_balboa_tst - z_db_balboa_dev - z_db_snowflake - z_db_starschema_covid19 - z_schema_raw - z_schema_snapshots - z_schema_source_account_usage - z_schema_source_country_data - z_schema_source_starschema_covid19 - z_schema_seeds - z_schema_source_dbt_artifacts - z_schema_dbt_metrics - z_schema_dbt_test__audit - z_schema_bay_country - z_schema_bay_covid - z_schema_bay_observability - z_schema_cove_covid - z_wh_transforming ``` -------------------------------- ### GET /api/v1/billing/ Source: https://docs.datacoves.com/docs/how-tos/datacoves/how_to_worker_usage Retrieves usage data for all environments within your account. Requires a Project-level token for authentication. ```APIDOC ## GET /api/v1/billing/ ### Description Returns usage data for all environments in your account. ### Method GET ### Endpoint `/` ### Query Parameters - **start_date** (string YYYY-MM-DD) - Optional - The start date for the usage period. Defaults to 30 days ago. - **end_date** (string YYYY-MM-DD) - Optional - The end date for the usage period. Defaults to today's date. - **service** (string) - Optional - Filter by service type: `airflow`, `airbyte`, or `unknown`. Defaults to all services. ### Request Example (No request body for GET requests) ### Response #### Success Response (200) - **environment_slug** (string) - The environment identifier. - **service** (string) - The service type: `airflow`, `airbyte`, or `unknown`. - **date** (datetime) - The timestamp of the usage record. - **amount_minutes** (integer) - The total minutes of worker usage excluding "warm-up" time. #### Response Example ```json [ { "environment_slug": "abc123", "service": "airflow", "date": "2025-11-15T10:30:00Z", "amount_minutes": 120 }, { "environment_slug": "abc123", "service": "airbyte", "date": "2025-11-15T14:20:00Z", "amount_minutes": 60 } ] ``` ``` -------------------------------- ### GET /api/v1/billing/{env-slug}/ Source: https://docs.datacoves.com/docs/how-tos/datacoves/how_to_worker_usage Retrieves usage data for a specific environment. Requires a Project-level token for authentication. ```APIDOC ## GET /api/v1/billing/{env-slug}/ ### Description Returns usage data for the environment specified by `env-slug`. ### Method GET ### Endpoint `/{env-slug}/` ### Path Parameters - **env-slug** (string) - Required - The slug identifier for your environment (e.g., `abc123`). ### Query Parameters - **start_date** (string YYYY-MM-DD) - Optional - The start date for the usage period. Defaults to 30 days ago. - **end_date** (string YYYY-MM-DD) - Optional - The end date for the usage period. Defaults to today's date. - **service** (string) - Optional - Filter by service type: `airflow`, `airbyte`, or `unknown`. Defaults to all services. ### Request Example (No request body for GET requests) ### Response #### Success Response (200) - **environment_slug** (string) - The environment identifier. - **service** (string) - The service type: `airflow`, `airbyte`, or `unknown`. - **date** (datetime) - The timestamp of the usage record. - **amount_minutes** (integer) - The total minutes of worker usage excluding "warm-up" time. #### Response Example ```json [ { "environment_slug": "abc123", "service": "airflow", "date": "2025-11-15T10:30:00Z", "amount_minutes": 120 }, { "environment_slug": "abc123", "service": "airbyte", "date": "2025-11-15T14:20:00Z", "amount_minutes": 60 } ] ``` ``` -------------------------------- ### Example Airflow DAG for Data Sync Source: https://docs.datacoves.com/docs/how-tos/airflow/sync-database This example DAG demonstrates how to use the `datacoves_airflow_db_sync` task decorator to synchronize the Airflow database to a Snowflake warehouse. Ensure the 'main' service connection is configured in Datacoves. ```python from pendulum import datetime from airflow.decorators import dag, task @dag( default_args={ "start_date": datetime(2026, 1, 1), "owner": "Bruno", "email": "bruno@example.com", "email_on_failure": False, "retries": 3 }, description="Sample DAG for dbt build", schedule="0 0 1 */12 *", tags=["extract_and_load"], catchup=False, ) def airflow_data_sync(): @task.datacoves_airflow_db_sync( db_type="snowflake", destination_schema="airflow_dev", connection_id="main", # tables=["additional_table_1", "additional_table_2"], ) def sync_airflow_db(): pass sync_airflow_db() airflow_data_sync() ``` -------------------------------- ### Example Workflow with Path Filters Source: https://docs.datacoves.com/docs/how-tos/dataops/trigger-cicd-workflow This workflow configuration uses path filters, which means it will NOT be triggered by an empty commit because no files are changed. ```yaml on: push: branches: - main paths: - .github/workflows/** - automate/** - transform/** ``` -------------------------------- ### Create Sample Pandas DataFrame Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/external-python-dag This Python script defines a function to create and print a Pandas DataFrame. It requires the pandas library to be installed. ```python import pandas as pd def print_sample_dataframe(): # Creating a simple DataFrame data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35], 'City': ['New York', 'San Francisco', 'Los Angeles']} df = pd.DataFrame(data) # Displaying the DataFrame print("DataFrame created using Pandas:") print(df) print_sample_dataframe() ``` -------------------------------- ### Prepare Lambda Deployment Package Source: https://docs.datacoves.com/docs/how-tos/airflow/api-triggered-dag These shell commands are used to create a zip file for deploying an AWS Lambda function. They install the `requests` library into a local package directory, zip its contents, and then add the `lambda_function.py` script to the same zip file. ```bash pip install --target ./package requests cd package zip -r ../deployment-package.zip . cd .. zip -g deployment-package.zip lambda_function.py ``` -------------------------------- ### Verify Public Key Configuration in Snowflake Source: https://docs.datacoves.com/docs/how-tos/snowflake/snowflake-key-based-auth Retrieve and display the configured public key fingerprint from the Snowflake user to verify the setup. This is done by querying user properties. ```sql DESC USER SVC_GITHUB_ACTIONS; SELECT SUBSTR((SELECT "value" FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())) WHERE "property" = 'RSA_PUBLIC_KEY_FP'), LEN('SHA256:') + 1); ``` -------------------------------- ### Grant Template Roles to Functional Roles Source: https://docs.datacoves.com/docs/how-tos/snowflake/warehouses-schemas-roles Example of creating functional roles (`analyst`, `analyst_pii`) by inheriting from template roles and adding specific grants. ```yaml - analyst: member_of: - z_base_analyst - z_tables_views_general - z_policy_row_region_all - analyst_pii: member_of: - analyst - z_policy_unmask_pii ``` -------------------------------- ### Complete DAG for Retrying dbt Failures Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/retry-dbt-tasks This complete DAG example demonstrates setting up a dbt task with retry capabilities using the datacoves_dbt decorator, including default arguments, schedule, and description. ```python " ## Retry dbt Example This DAG demonstrates how to retry a DAG that fails during a run " from airflow.decorators import dag, task from orchestrate.utils import datacoves_utils @dag( doc_md = __doc__, catchup = False, default_args=datacoves_utils.set_default_args( owner = "Your Name", owner_email = "your.email@example.com" ), schedule = datacoves_utils.set_schedule("0 0 1 */12 *"), description="Sample DAG demonstrating how to run the dbt models that fail", tags=["dbt_retry"], ) def retry_dbt_failures(): @task.datacoves_dbt( connection_id="your_connection", dbt_api_enabled=True, download_run_results=True, ) def dbt_build(expected_files: list = []): if expected_files: return "dbt build -s result:error+ --state logs" else: return "dbt build -s model_a+ model_b+" dbt_build(expected_files=["run_results.json"]) retry_dbt_failures() ``` -------------------------------- ### Import Variables and Connections Source: https://docs.datacoves.com/docs/reference/airflow/datacoves-commands Use this command to import variables and connections from Team Airflow to My Airflow. This is typically a one-time setup or done when new items are added to Team Airflow. Secret values will be prompted for during execution. ```bash datacoves my import ``` -------------------------------- ### Airflow DAG with dynamic schedule Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/dynamically-set-schedule Import and use the get_schedule function to set the DAG's schedule dynamically. This example sets a schedule of '0 1 * * *' for production environments. ```python from airflow.decorators import dag, task from pendulum import datetime from orchestrate.utils.get_schedule import get_schedule @dag( default_args={"start_date": datetime(2022, 10, 10), "owner": "Noel Gomez", "email": "gomezn@example.com", "email_on_failure": True,}, is_paused_on_creation=True, catchup=False, tags=["version_8"], description="Datacoves Sample DAG", schedule=get_schedule('0 1 * * *'), # Replace with desired schedule ) def datacoves_sample_dag(): @task.datacoves_dbt(connection_id="main") def run_dbt_task(): return "dbt debug" run_dbt_task() datacoves_sample_dag() ``` -------------------------------- ### Configure Kubernetes Pod Override with Resource Requests in Python Source: https://docs.datacoves.com/docs/how-tos/airflow/request-resources-on-workers Use this Python code to define resource requests for memory and CPU when running an Airflow task on a Kubernetes execution environment. Ensure the `kubernetes.client` library is installed. ```python from datetime import datetime from airflow.decorators import dag, task from kubernetes.client import models as k8s # Configuration for Kubernetes Pod Override with Resource Requests TRANSFORM_CONFIG = { "pod_override": k8s.V1Pod( spec=k8s.V1PodSpec( containers=[ k8s.V1Container( name="transform", resources=k8s.V1ResourceRequirements( requests={"memory": "8Gi", "cpu": "1000m"} ), ) ] ) ), } @dag( default_args={ "start_date": datetime(2022, 10, 10), "owner": "Noel Gomez", "email": "gomezn@example.com", "email_on_failure": True, }, description="Sample DAG with custom resources", schedule="0 0 1 */12 *", tags=["version_2"], catchup=False, ) def request_resources_dag(): @task.datacoves_bash(executor_config=TRANSFORM_CONFIG) def transform(): return "echo 'Resource request DAG executed successfully!'" transform() request_resources_dag() ``` -------------------------------- ### Get Airbyte Connection Host in DAG Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/use-variables-and-connections This example demonstrates how to fetch connection details, specifically the host, for an Airbyte connection. It requires the 'pendulum' and 'airflow' libraries. The connection ID used is 'AIRBYTE_CONNECTION'. ```python from pendulum import datetime from airflow.decorators import dag, task from airflow.models import Connection @dag( default_args={"start_date": datetime(2024, 1, 1)}, description="DAG that outputs Airbyte Hostname", schedule="0 0 1 */12 *", tags=["version_1"], catchup=False, ) def connections_dag(): @task.datacoves_bash() def echo_airbyte_host(): airbyte_connection = Connection.get_connection_from_secrets(conn_id="AIRBYTE_CONNECTION") return f"echo 'Airbyte hostname is {airbyte_connection.host}'" echo_airbyte_host() connections_dag() ``` -------------------------------- ### Get Airflow Variable in DAG Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/use-variables-and-connections Use this snippet to retrieve a variable's value from Airflow. If the variable name contains 'SECRET', its value will be hidden. Ensure the 'pendulum' and 'airflow' libraries are installed. ```python from pendulum import datetime from airflow.decorators import dag, task from airflow.models import Variable @dag( default_args={"start_date": datetime(2024, 1, 1)}, description="DAG that outputs a Variable", schedule="0 0 1 */12 *", tags=["version_1"], catchup=False, ) def variables_dag(): @task.datacoves_bash(="main") def transform(): daily_run_tag = Variable.get("DBT_DAILY_RUN_TAG") return f"dbt build -s 'tag:{daily_run_tag}'" transform() variables_dag() ``` -------------------------------- ### Sample Error Response: Start Date After End Date Source: https://docs.datacoves.com/docs/how-tos/datacoves/how_to_worker_usage This JSON object signifies an error where the provided start date occurs after the end date. Ensure the start date precedes the end date. ```json { "error": "start_date must be before end_date." } ``` -------------------------------- ### Initialize Datacoves Repository Source: https://docs.datacoves.com/docs/getting-started/Admin/configure-repository-using-dbt-coves Run this command to create a new, fully configured Datacoves repository. It will prompt you for configuration details like data warehouse, components, and project naming conventions. ```bash # Create a new Datacoves repository dbt-coves setup ``` -------------------------------- ### Create New Workspace Directory Source: https://docs.datacoves.com/docs/how-tos/vs-code/reset-user-env Create a new, empty directory named 'workspace'. This will serve as the location for your new project environment. ```bash mkdir workspace ``` -------------------------------- ### Instantiate DAG with YAML Configuration Source: https://docs.datacoves.com/docs/how-tos/airflow/send-slack-notifications Instantiates a DAG using a YAML configuration file. This DAG includes Slack notifications for success and failure, custom image, and resource requests for tasks. ```python dag = yaml_slack_dag() ``` -------------------------------- ### Set DAG Start Date and Catchup Source: https://docs.datacoves.com/docs/reference/airflow/airflow-best-practices When defining a DAG, set the start date to the day before or earlier and disable `catchup` to prevent unintended runs. This ensures predictable scheduling. ```python from pendulum import datetime from airflow.decorators import dag @dag( default_args=("start_date": datetime(2023, 12, 29), # Set this to the day before or earlier "owner": "Noel Gomez", "email": "gomezn@example.com", "email_on_failure": True, ), dag_id="sample_dag", schedule="@daily", catchup=False, # Set this to false to avoid additional catchup runs tags=["version_1"], description="Datacoves Sample dag", ) ... ``` -------------------------------- ### Example Airflow DAG generated from YAML definitions Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/generate-dags-from-yml This Python script is an example of an Airflow DAG automatically generated by `dbt-coves generate airflow-dags`. It uses the `DatacovesDbtOperator` to run a specific dbt model. ```python from pendulum import datetime from airflow.decorators import dag from operators.datacoves.dbt import DatacovesDbtOperator @dag( default_args={\n "start_date": datetime(2022, 10, 10), "owner": "Noel Gomez", "email": "gomezn@example.com", "email_on_failure": True, }, description="Sample DAG for dbt build", schedule="0 0 1 */12 *", tags=["version_2"], catchup=False, ) def yml_dbt_dag(): run_dbt = DatacovesDbtOperator( task_id="run_dbt", bash_command="dbt run -s personal_loans" ) dag = yml_dbt_dag() ``` -------------------------------- ### Check Jinja Syntax for References Source: https://docs.datacoves.com/docs/how-tos/dbt/compilation-errors Verify the correct syntax for Jinja `ref()` functions. Ensure all opening and closing brackets and braces are properly matched. ```jinja {# Correct #} {{ ref('model_name') }} ``` ```jinja {# Incorrect, missing ) #} {{ ref('model_name' }} ``` ```jinja {# Incorrect, missing } #} {{ ref('model_name') } ``` ```jinja {# Incorrect, missing { and } #} { ref('model_name') } ``` -------------------------------- ### Navigate to Home Directory Source: https://docs.datacoves.com/docs/how-tos/vs-code/reset-user-env Change the current directory to your home directory. This is a prerequisite for managing workspace files. ```bash cd ~ ``` -------------------------------- ### Define German Analyst Role Source: https://docs.datacoves.com/docs/how-tos/snowflake/warehouses-schemas-roles Example of a role (`de_analyst`) with region-specific row access, inheriting from a base analyst role. ```yaml - de_analyst: member_of: - z_base_analyst - z_tables_views_general - z_policy_row_region_de - de_business_analyst_pii: member_of: - de_analyst - z_policy_unmask_pii ``` -------------------------------- ### List My Airflow API Keys Source: https://docs.datacoves.com/docs/reference/airflow/datacoves-commands View all environments where My Airflow is enabled, including their API URLs and existing API keys. This is useful for managing programmatic access. ```bash datacoves my api-key list ``` -------------------------------- ### Define Object Roles in YAML Source: https://docs.datacoves.com/docs/how-tos/snowflake/warehouses-schemas-roles Example of defining object roles for warehouses, databases, and schemas in a YAML configuration file. ```yaml - z_db_raw: - z_db_raw_write: - z_schema_raw: - z_schema_source_dbt_artifacts: - z_schema_bay_country: - z_schema_cove_covid: - z_wh_loading: - z_wh_transforming: ``` -------------------------------- ### Create and Switch to a New Git Branch Source: https://docs.datacoves.com/docs/getting-started/developer/using-git Create a new branch and immediately switch to it using a single command. You can specify a reference branch to branch from. ```git git checkout -b ``` -------------------------------- ### Create Snowflake Warehouses Source: https://docs.datacoves.com/docs/how-tos/snowflake/warehouses-schemas-roles Configure warehouses in `warehouses.yml` and create them using the `create_snowflake_objects.py` script. Use `--dry-run` to preview changes. ```python secure/create_snowflake_objects.py -s warehouses -t prd ``` -------------------------------- ### Example Circular Dependency Error Source: https://docs.datacoves.com/docs/how-tos/dbt/dependency-errors This error indicates a circular dependency where models reference each other in a loop, violating dbt's DAG structure. ```text RuntimeError: Found a cycle: model.balboa.my_model --> model.balboa.some_model.v2 ``` -------------------------------- ### Create JSON Stage Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/s3-to-snowflake Create a Snowflake stage configured to use the custom JSON file format. ```sql CREATE OR REPLACE STAGE JSON_STAGE FILE_FORMAT = MY_JSON_FORMAT; ``` -------------------------------- ### Configure My Airflow API Credentials Source: https://docs.datacoves.com/docs/how-tos/my_airflow/use-my-airflow-api Add your My Airflow API URL and API Key to a .env file in your orchestrate/ directory. Ensure this file is added to your .gitignore. ```dotenv MY_AIRFLOW_API_URL = "https://your-slug-api-airflow-env.domain.com/api/v1/" MY_AIRFLOW_API_KEY = "your-api-key-here" ``` -------------------------------- ### Define Custom Aliases Source: https://docs.datacoves.com/docs/how-tos/vs-code/bash-custom Create custom command aliases in your .bash_custom file for frequently used commands. For example, 'll' for 'ls -la' and 'cls' for 'clear'. ```bash # Aliases alias ll='ls -la' alias cls='clear' ``` -------------------------------- ### View Datacoves Environment Variables Source: https://docs.datacoves.com/docs/reference/vscode/datacoves-env-vars Run this command in your terminal to list all environment variables prefixed with DATACOVES and sort them alphabetically. This is useful for verifying your Datacoves setup. ```bash env | grep DATACOVES | sort ``` -------------------------------- ### DataHub Host URL for Development Environment Source: https://docs.datacoves.com/docs/how-tos/datahub/how_to_datahub_cli Use this URL format when your DataHub instance is in the development environment. Replace `{environment-slug}` with your specific environment identifier. ```bash http://{environment-slug}-datahub-datahub-gms:8080 ``` -------------------------------- ### SQL Query Optimization: Subqueries vs. CTEs Source: https://docs.datacoves.com/docs/how-tos/dbt/database-errors Replace inefficient subqueries with Common Table Expressions (CTEs) for better readability and performance in SQL. This example shows the transformation. ```sql -- Instead of this SELECT * FROM table_a WHERE id IN (SELECT id FROM table_b WHERE status = 'active') -- Use this WITH active_ids AS ( SELECT id FROM table_b WHERE status = 'active' ) SELECT * FROM table_a WHERE id IN (SELECT id FROM active_ids) ``` -------------------------------- ### Python function to get dynamic schedule Source: https://docs.datacoves.com/docs/how-tos/airflow/DAGs/dynamically-set-schedule Use this function to return None for development environments and a specified schedule for others. Ensure the DATACOVES__ENVIRONMENT_SLUG environment variable is set correctly. ```python # get_schedule.py import os from typing import Union DEV_ENVIRONMENT_SLUG = "dev123" # Replace with your environment slug def get_schedule(default_input: Union[str, None]) -> Union[str, None]: """ Sets the application's schedule based on the current environment setting. Allows you to set the the default for dev to none and the the default for prod to the default input. This function checks the Datacoves Slug through 'DATACOVES__ENVIRONMENT_SLUG' variable to determine if the application is running in a specific environment (e.g., 'dev123'). If the application is running in the 'dev123' environment, it indicates that no schedule should be used, and hence returns None. For all other environments, the function returns the given 'default_input' as the schedule. Parameters: - default_input (Union[str, None]): The default schedule to return if the application is not running in the dev environment. Returns: - Union[str, None]: The default schedule if the environment is not 'dev123'; otherwise, None, indicating that no schedule should be used in the dev environment. """ env_slug = os.environ.get("DATACOVES__ENVIRONMENT_SLUG", "").lower() if env_slug == DEV_ENVIRONMENT_SLUG: return None else: return default_input ``` -------------------------------- ### OpenAI Custom Model Configuration Source: https://docs.datacoves.com/docs/how-tos/vs-code/datacoves-copilot/v2 Use `openAiCustomModelInfo` to specify advanced settings for fine-tuned models, such as `maxTokens` and `contextWindow`. ```json "openAiCustomModelInfo": { "maxTokens": -1, "contextWindow": 128000, "supportsImages": true, "supportsPromptCache": false, "inputPrice": 0, "outputPrice": 0, "reasoningEffort": "medium" } ``` -------------------------------- ### Delete Ingested Data by Platform Source: https://docs.datacoves.com/docs/how-tos/datahub/how_to_datahub_cli Use the `datahub delete` command to remove ingested data. Specify the platform using the `--platform` flag, for example, to delete all dbt-related data. ```bash datahub delete --platform dbt ```