### Clone dmp-af Repository with Git Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Clones the dmp-af repository from GitHub to your local machine. This is the first step to get the example project and all required files for setting up dmp-af. ```bash git clone git@github.com:dmp-labs/dmp-af.git dmp-af ``` -------------------------------- ### Install dmp-af Package via pip Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Install dmp-af with optional test and examples dependencies using pip package manager. This is required if not using Docker Compose method for Airflow setup. ```bash pip install dmp-af[tests,examples] ``` -------------------------------- ### Install dmp-af for development Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Set up dmp-af for development by cloning the repository and installing dependencies. It recommends using 'uv' for synchronization but also provides a pip alternative. ```bash # Clone the repository git clone https://github.com/dmp-labs/dmp-af.git cd dmp-af # Install with uv (recommended) uv sync --all-packages --all-groups --all-extras # Or with pip pip install -e ".[dev]" ``` -------------------------------- ### Development Installation using uv Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Clones the dmp-af repository and installs in development mode using uv package manager (recommended). Includes testing tools (pytest, pytest-cov), linting tools (ruff, pre-commit), example dependencies (dbt-postgres), and documentation tools. ```bash git clone https://github.com/dmp-labs/dmp-af.git cd dmp-af uv sync --all-packages --all-groups --all-extras ``` -------------------------------- ### Install and Initialize Local Airflow Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Installs Apache Airflow locally and initializes the database with an admin user. This is an alternative to Docker Compose for development environments. ```bash # Set Airflow home export AIRFLOW_HOME=~/airflow # Install Airflow pip install apache-airflow==2.10.5 # Initialize database airflow db init # Create admin user airflow users create \ --username admin \ --password admin \ --firstname Admin \ --lastname User \ --role Admin \ --email admin@example.com # Start services airflow webserver --port 8080 & airflow scheduler & ``` -------------------------------- ### Development Installation using pip Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Clones the dmp-af repository and installs in editable development mode using pip. Includes development dependencies for testing, linting, and documentation. ```bash git clone https://github.com/dmp-labs/dmp-af.git cd dmp-af pip install -e ".[dev]" ``` -------------------------------- ### Install dmp-af via pip Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Install the dmp-af package using pip. This is the standard method for adding the library to your Python environment. ```bash pip install dmp-af ``` -------------------------------- ### Build dbt Manifest with Shell Script Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Executes the build_manifest.sh script to install dbt dependencies, compile the Jaffle Shop projects, and generate the manifest.json file required by dmp-af. ```bash cd dags ./build_manifest.sh ``` -------------------------------- ### Start Airflow with Docker Compose Script Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Uses the manage-airflow.sh script to start Airflow with Docker Compose. The script handles environment setup, dbt manifest building, and service initialization. Supports both Airflow 2.x (stable) and 3.x (latest). ```bash cd dmp-af/examples # Start Airflow 2.x (stable) ./manage-airflow.sh up 2 # Or start Airflow 3.x (latest) ./manage-airflow.sh up 3 ``` -------------------------------- ### Connect to Postgres Database Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Connect to the PostgreSQL database using the Docker setup. This command allows you to execute SQL queries and inspect tables within the 'jaffle_shop' schema. ```bash docker exec -it postgres psql -U airflow -d airflow ``` -------------------------------- ### Install dmp-af with optional features via pip Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Install dmp-af with specific optional features like Monte Carlo Data (mcd) or Tableau integration using pip. The '[all]' option installs all available features. ```bash # Install with Monte Carlo Data integration pip install dmp-af[mcd] # Install with Tableau integration pip install dmp-af[tableau] # Install all optional features pip install dmp-af[all] ``` -------------------------------- ### Install dmp-af with All Optional Features Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Installs dmp-af with all optional integrations and features enabled, including Monte Carlo Data and Tableau support. Provides complete functionality for all supported platforms and integrations. ```bash pip install dmp-af[all] ``` -------------------------------- ### Install dbt-postgres for dbt-core Compatibility Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Installs dbt-postgres adapter with compatible version constraints (>=1.7.0,<2). Ensures proper dbt-core functionality for PostgreSQL database projects. ```bash pip install dbt-postgres>=1.7.0,<2 ``` -------------------------------- ### Configure dbt_project.yml for dmp-af Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Example configuration within 'dbt_project.yml' to define cluster settings for different types of dbt tasks (SQL and Python). This helps dmp-af route tasks to the correct environments. ```yaml # dbt_project.yml models: my_project: my_domain: sql_cluster: "dev" daily_sql_cluster: "dev" py_cluster: "dev" bf_cluster: "dev" ``` -------------------------------- ### Start Airflow with Docker Compose Script Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/docker-compose.md Shell commands to navigate to the examples directory and start Apache Airflow using the provided management script. Supports both Airflow 2.x and 3.x versions. ```bash cd examples ./manage-airflow.sh up 2 # For Airflow 2.x ./manage-airflow.sh up 3 # For Airflow 3.x ``` -------------------------------- ### Display dmp-af CLI Help Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Shows available command-line interface (CLI) tools for dmp-af. Displays help information for the manifest tests CLI command. ```bash dmp-af-manifest-tests --help ``` -------------------------------- ### Verify dmp-af Installation Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Verifies that dmp-af is properly installed by importing the module and printing its version. Use this command to confirm successful installation and check the installed version. ```bash python -c "import dmp_af; print(dmp_af.__version__)" ``` -------------------------------- ### Check Manifest File and Recompile dbt Project Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Verify the existence and readability of the dbt manifest file and recompile the dbt project if necessary. This addresses 'Manifest Not Found' errors. ```bash ls -l /path/to/manifest.json cd /path/to/dbt/project dbt compile --target dev ``` -------------------------------- ### Start Airflow 2.x with Container Recreation Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Start Airflow 2.x services with the --force-recreate flag to force recreation of all containers, useful when configuration changes need to be applied. ```bash ./manage-airflow.sh up 2 --force-recreate ``` -------------------------------- ### Build dbt Manifest Script Execution Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Navigate to examples/dags directory and execute the build_manifest.sh script to manually rebuild the dbt manifest. This is automatically handled by manage-airflow.sh but can be run independently for manual setup. ```bash cd examples/dags ./build_manifest.sh ``` -------------------------------- ### Python Development Environment Setup Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/contributing.md Setup commands for initializing a development environment using either uv (recommended) or pip. Installs all project dependencies including development and testing tools required for local development. ```bash git clone https://github.com/YOUR-USERNAME/dmp-af.git cd dmp-af uv sync --all-packages --all-groups --all-extras ``` ```bash pip install -e ".[dev]" ``` -------------------------------- ### Start Airflow 2.x Instance with Docker Compose Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Start an Airflow 2.x instance using the manage-airflow.sh script. The script automatically creates the .env file, builds the dbt manifest, and creates required pools. ```bash ./manage-airflow.sh up 2 ``` -------------------------------- ### Setup Development Environment with uv Source: https://github.com/dmp-labs/dmp-af/blob/main/CONTRIBUTING.md Clones the dmp-af repository and installs project dependencies using uv. It also installs pre-commit hooks to validate commit messages. ```bash git clone https://github.com/YOUR-USERNAME/dmp-af.git cd dmp-af uv sync --all-packages --all-groups --all-extras uv run pre-commit install --hook-type commit-msg ``` -------------------------------- ### Install dmp-af with Monte Carlo Data Integration Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Installs dmp-af with optional Monte Carlo Data catalog integration support. Adds airflow-mcd (>=0.3.3,<0.4.0) and pycarlo (>=0.9) dependencies for data quality and catalog features. ```bash pip install dmp-af[mcd] ``` -------------------------------- ### Verify Docker and Docker Compose Installation Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/docker-compose.md Commands to check if Docker and Docker Compose are installed and their versions. This is a prerequisite for using the Docker Compose setup. ```bash docker --version docker compose version ``` -------------------------------- ### Compile dbt project manifest Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Command to compile your dbt project, which generates the 'manifest.json' file required by dmp-af. This step is crucial before creating dbt-related DAGs. ```bash cd /path/to/your/dbt/project dbt compile ``` -------------------------------- ### Set up Airflow pools via CLI Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Commands to create necessary Airflow pools using the Airflow CLI. Pools are essential for dmp-af to manage concurrent task execution and avoid warnings. ```bash # Via Airflow CLI airflow pools set dbt_dev 4 "Development pool" airflow pools set dbt_sensor_pool 10 "Sensor pool" ``` -------------------------------- ### Create Airflow Pools via CLI Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Create dbt_dev and dbt_sensor_pool pools in Airflow using the Airflow CLI with specified slot counts. Start with small numbers to avoid resource overflow on local machines. ```bash airflow pools set dbt_dev 4 "dev" ``` -------------------------------- ### Create Airflow Pools for dmp-af Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Configures required Airflow pools for task management in dmp-af. The dbt_dev pool handles development tasks and dbt_sensor_pool manages sensor operations. ```bash airflow pools set dbt_dev 4 "Development pool" airflow pools set dbt_sensor_pool 10 "Sensor pool" ``` -------------------------------- ### Manage Airflow Pools Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md List available Airflow pools and set a new pool with a specified slot count and description. This is useful for managing task concurrency and resource allocation. ```bash airflow pools list airflow pools set dbt_dev 4 "Dev pool" ``` -------------------------------- ### Update Package Installation (requirements.txt) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/migration.md Replace the 'dbt-af' package with 'dmp-af' in your requirements.txt file for updated installations. This ensures you are using the new fork with active maintenance and features. ```diff - dbt-af + dmp-af ``` -------------------------------- ### Troubleshoot Missing dmp-af Module Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Diagnostic and recovery steps for resolving 'ModuleNotFoundError: No module named dmp_af'. Verifies installation status, checks Python environment, and provides reinstallation command. ```bash pip list | grep dmp-af which python pip install --force-reinstall dmp-af ``` -------------------------------- ### Inspect Jaffle Shop Tables Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md List all tables in the 'jaffle_shop' schema and select the first 5 rows from the 'customers' table. This is useful for verifying data integrity after dbt runs. ```sql \dt jaffle_shop.* SELECT * FROM jaffle_shop.customers LIMIT 5; ``` -------------------------------- ### Restart Specific Airflow Service Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Restart a specific service (e.g., airflow-webserver) in the Airflow 2.x Docker Compose setup to apply configuration changes. ```bash ./manage-airflow.sh restart 2 airflow-webserver ``` -------------------------------- ### Update Package Installation (setup.py) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/migration.md Change the 'dbt-af' package to 'dmp-af' in the install_requires list within your setup.py file. This modification is crucial for packages distributed using setuptools. ```diff install_requires=[ - "dbt-af", + "dmp-af", ] ``` -------------------------------- ### Create dmp-af DAG Configuration in Python Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Defines a dmp-af DAG configuration by creating a Config object with dbt project settings, then compiling DAGs from the manifest.json file. This registers the generated DAGs with Airflow. ```python # dags/my_first_dag.py # LABELS: dag, airflow from dmp_af.dags import compile_dmp_af_dags from dmp_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig # Configure your dbt project config = Config( dbt_project=DbtProjectConfig( dbt_project_name='jaffle_shop', dbt_project_path='/path/to/examples/dags/jaffle_shop', dbt_models_path='/path/to/examples/dags/jaffle_shop/dbt/models', dbt_profiles_path='/path/to/examples/dags', dbt_target_path='/path/to/examples/dags/jaffle_shop/target', dbt_log_path='/path/to/examples/dags/jaffle_shop/logs', dbt_schema='jaffle_shop', ), dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'), ) # Compile DAGs from manifest dags = compile_dmp_af_dags( manifest_path='/path/to/examples/dags/jaffle_shop/target/manifest.json', config=config, ) # Register with Airflow for dag_name, dag in dags.items(): globals()[dag_name] = dag ``` -------------------------------- ### Switching Between Airflow Versions Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/docker-compose.md Instructions on how to safely switch between Airflow 2.x and 3.x by stopping the current version before starting the new one, addressing potential port conflicts. ```bash # Stop Airflow 2.x ./manage-airflow.sh down 2 # Start Airflow 3.x ./manage-airflow.sh up 3 ``` -------------------------------- ### Commit Examples - Patch, Minor, and Major Releases Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/releases.md Demonstrates practical examples of commit messages for different version bump types (patch, minor, major), including commits with scopes, bodies, footers, and breaking change indicators. Shows both standard and shorthand syntax for breaking changes. ```bash # Patch release (0.14.6 → 0.14.7) git commit -m "fix: correct snapshot task generation for new dbt format" # Minor release (0.14.6 → 0.15.0) git commit -m "feat: add support for dbt Cloud metadata export" # With scope git commit -m "feat(operators): add new Python venv operator" # With body and footer git commit -m "fix(sensors): timeout handling in external sensor Previously the sensor would hang indefinitely when the upstream task failed. Now it respects the timeout parameter. Fixes #123" # Major release (0.14.6 → 1.0.0) git commit -m "feat: redesign DAG configuration API BREAKING CHANGE: The DmpAfConfig class now requires explicit domain specification. The old automatic domain detection has been removed." ``` -------------------------------- ### Install Semantic Release Tools - Python Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/releases.md Installs the semantic-release package and its dependencies using the uv package manager. This command fetches all required tools for version prediction and release automation. ```bash uv sync --group release ``` -------------------------------- ### Dependency Version Configuration in pyproject.toml Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Defines package dependency version constraints for dmp-af in pyproject.toml. Specifies compatibility ranges for Apache Airflow (>=2.6,<3), dbt-core (>=1.7,<2), and Pydantic (>=1.10,<3.0.0) to ensure stable integration. ```python dependencies = [ "apache-airflow >=2.6,<3", "dbt-core >=1.7,<2", "pydantic >=1.10,<3.0.0", ] ``` -------------------------------- ### Install Dagger CLI Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/testing.md Instructions for installing the Dagger CLI on macOS and Linux systems. This is a prerequisite for using Dagger to run tests. ```bash # macOS brew install dagger/tap/dagger # Linux curl -L https://dl.dagger.io/dagger/install.sh | sh # Or see https://docs.dagger.io/install ``` -------------------------------- ### Troubleshoot DAG Import Errors Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/quick-start.md Check for Python syntax errors in DAG files and list DAG import errors in Airflow. This helps in diagnosing why DAGs might not be appearing in the Airflow UI. ```bash python dags/my_first_dag.py airflow dags list-import-errors ``` -------------------------------- ### Conventional Commit Message Examples Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/contributing.md Example commit messages demonstrating the Conventional Commits format used for automatic semantic versioning. Includes examples for features, fixes, documentation, and breaking changes. ```bash feat: add support for dbt snapshots in maintenance DAGs ``` ```bash fix: resolve dependency resolution for cross-domain sources ``` ```bash docs: update contributing guide with PR template info ``` ```bash feat!: change configuration structure for domain schedules ``` -------------------------------- ### Valid and Invalid Commit Message Examples Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/releases.md Demonstrates examples of commit messages that pass and fail validation. Valid messages follow the Conventional Commits specification with proper type prefixes, while invalid messages lack proper formatting. ```bash # ✅ Valid git commit -m "feat: add new sensor type" # ❌ Invalid (will be rejected) git commit -m "added new sensor" git commit -m "WIP: testing something" ``` -------------------------------- ### dbt Project Domain Start Date Configuration Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/configuration/model-config.md Specifies the start date for a model's domain, used to minimize Airflow catchup DagRuns. This is typically set in the `dbt_project.yml` file for a given project or domain. ```yaml # dbt_project.yml models: project_name: domain_name: domain_start_date: "2024-01-01T00:00:00" ``` -------------------------------- ### Serve dmp-af Documentation Locally (Bash) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/BUILD.md Starts a local development server for the dmp-af documentation with live reloading. Requires Dagger CLI and Docker. Changes to markdown files will be reflected in the browser automatically. ```bash make docs-serve # or make serve ``` -------------------------------- ### Resolve Airflow Version Conflicts Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/installation.md Resolves Apache Airflow dependency conflicts by creating isolated environment and installing Airflow before dmp-af. Prevents version incompatibility issues during installation. ```bash pip install apache-airflow==2.10.5 pip install dmp-af ``` -------------------------------- ### View Airflow Logs with Tail Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Display the last 100 lines of logs from Airflow 3.x services to diagnose issues or monitor activity. ```bash ./manage-airflow.sh logs 3 --tail=100 ``` -------------------------------- ### Manage Airflow Docker Compose Instances via manage-airflow.sh Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Execute manage-airflow.sh script to start, stop, restart, build, monitor logs, check status, or list containers for Airflow 2.x or 3.x instances. Supports passing additional docker compose arguments for advanced configurations. ```bash cd examples ./manage-airflow.sh [ACTION] [VERSION] [DOCKER_ARGS...] ``` -------------------------------- ### Update Package Installation (CLI) Source: https://github.com/dmp-labs/dmp-af/blob/main/MIGRATION.md After updating dependency files, use pip to reinstall the packages, ensuring that 'dmp-af' is installed instead of 'dbt-af'. This command updates the package to the latest version. ```bash pip install -U dmp-af ``` -------------------------------- ### Update Package Installation (requirements.txt, pyproject.toml) Source: https://github.com/dmp-labs/dmp-af/blob/main/MIGRATION.md Replace 'dbt-af' with 'dmp-af' in your project's dependency files (requirements.txt or pyproject.toml) to update the package installation. After modifying the files, reinstall the dependencies using pip. ```diff # requirements.txt - dbt-af + dmp-af # pyproject.toml [project] dependencies = [ - "dbt-af", + "dmp-af", ] ``` -------------------------------- ### Configure Airflow Environment Variables Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/docker-compose.md Example environment variables that can be added to the `.env` file to customize Airflow's behavior, such as disabling example DAGs, pausing DAGs at creation, and setting dbt project configurations. ```bash # Airflow configuration AIRFLOW__CORE__LOAD_EXAMPLES=False AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True # dmp-af specific DBT_TARGET=dev DBT_PROJECT_DIR=/opt/airflow/dags/jaffle_shop ``` -------------------------------- ### Update Configuration Variables (Python) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/migration.md Adjust variable names in your Python code that reference dbt-af configurations. For example, change 'dbt_af_config' to 'dmp_af_config' and 'dbt_af_graph' to 'dmp_af_graph'. ```diff - dbt_af_config = Config(...) - dbt_af_graph = DbtAfGraph.from_manifest(...) + dmp_af_config = Config(...) + dmp_af_graph = DmpAfGraph.from_manifest(...) ``` -------------------------------- ### Build dmp-af Static Site (Bash) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/BUILD.md Generates the static HTML version of the dmp-af documentation. The output is placed in the 'site/' directory. Requires Dagger CLI and Docker. ```bash make docs-build # or make build ``` -------------------------------- ### Get Test Matrix and Run All Configurations with Dagger (Bash) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/contributing.md This set of commands retrieves the test matrix configurations and then iterates through them for testing. First, it exports the matrix to a JSON file. The second part (commented out, referencing a GitHub workflow) indicates how to run tests for each configuration defined in the matrix, ensuring comprehensive testing across all supported environments. ```bash # Get the test matrix dagger call -m ./.ci tests get-versions-matrix export --path=./matrix.json # Run tests for each configuration in the matrix # (See .github/workflows/integration_tests.yml for the full automation) ``` -------------------------------- ### Stop Airflow Services and Remove Volumes Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Stop Airflow 3.x services and remove associated volumes using the -v flag to clean up persistent data. ```bash ./manage-airflow.sh down 3 -v ``` -------------------------------- ### Serve Documentation Locally (Bash) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/development/contributing.md This command serves the project's user-facing documentation locally using MkDocs. It allows developers to preview changes before deployment. The documentation can then be accessed at http://localhost:8000. ```bash make docs-serve # Visit http://localhost:8000 ``` -------------------------------- ### Check Airflow Services Status Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Display the status of all Airflow 2.x services managed by Docker Compose to verify running state. ```bash ./manage-airflow.sh status 2 ``` -------------------------------- ### Customize Docker Compose Webserver Port Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/docker-compose.md An example snippet from a `docker-compose.yaml` file demonstrating how to change the host port mapping for the Airflow webserver service from the default 8080 to 9090. ```yaml webserver: ports: - "9090:8080" # Changed from 8080:8080 ``` -------------------------------- ### Rebuild Docker Images Without Cache Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Rebuild Airflow 3.x Docker images from scratch without using cached layers, useful for applying system updates or dependency changes. ```bash ./manage-airflow.sh build 3 --no-cache ``` -------------------------------- ### Update Package Installation (pyproject.toml) Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/migration.md Modify the 'dbt-af' dependency to 'dmp-af' within the [project].dependencies section of your pyproject.toml file. This is necessary for projects using this build system to adopt the new package. ```diff [project] dependencies = [ - "dbt-af", + "dmp-af", ] ``` -------------------------------- ### Create dmp-af DAG in Airflow Source: https://github.com/dmp-labs/dmp-af/blob/main/docs/getting-started/index.md Python code to define and compile dbt DAGs using dmp-af. It configures dbt project paths, targets, and schema, then uses 'compile_dmp_af_dags' to generate Airflow DAGs from the manifest. ```python # dags/my_dmp_af_dags.py # LABELS: dag, airflow from dmp_af.dags import compile_dmp_af_dags from dmp_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig config = Config( dbt_project=DbtProjectConfig( dbt_project_name='my_dbt_project', dbt_project_path='/path/to/my_dbt_project', dbt_models_path='/path/to/my_dbt_project/models', dbt_profiles_path='/path/to/my_dbt_project', dbt_target_path='/path/to/my_dbt_project/target', dbt_log_path='/path/to/my_dbt_project/logs', dbt_schema='my_dbt_schema', ), dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'), dry_run=False, ) # Compile DAGs from manifest dags = compile_dmp_af_dags( manifest_path='/path/to/my_dbt_project/target/manifest.json', config=config, ) # Register DAGs with Airflow for dag_name, dag in dags.items(): globals()[dag_name] = dag ``` -------------------------------- ### Stop Airflow 3.x Instance Source: https://github.com/dmp-labs/dmp-af/blob/main/examples/README.md Stop running Airflow 3.x services using the manage-airflow.sh script with the down action. ```bash ./manage-airflow.sh down 3 ```