### Output of Kubernetes Environment Setup Source: https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst Example output displayed after successfully running the `breeze k8s setup-env` command, showing the initialization and installation steps. ```text Initializing K8S virtualenv in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv Reinstalling PIP version in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv Installing necessary packages in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv ``` -------------------------------- ### KubernetesStartKueueJobOperator Example Source: https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/docs/operators.rst Example demonstrating how to start a Kueue job using KubernetesStartKueueJobOperator. ```python from __future__ import annotations import pendulum from airflow.models.dag import DAG from airflow.providers.cncf.kubernetes.operators.kueue import KubernetesStartKueueJobOperator with DAG( dag_id="kubernetes_kueue_start_job_example", schedule=None, start_date=pendulum.datetime(2023, 10, 26, tz="UTC"), catchup=False, tags=["kubernetes", "kueue", "example"], ) as dag: # Example of KubernetesStartKueueJobOperator k8s_kueue_start_job = KubernetesStartKueueJobOperator( task_id="kubernetes_kueue_start_job_task", namespace="default", image="kueue/kueue-job:latest", name="kueue-job-starter", is_delete_operator_pod=False, get_logs=True, job_name="my-kueue-job", queue="default", ) ``` -------------------------------- ### Install Breeze and Start Airflow Source: https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst Installs Breeze, prek, and sets up the development environment for Airflow. This includes installing Python dependencies and starting the Airflow development container. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh uv tool install prek prek install -f prek install -f --hook-type pre-push # for running mypy checks when pushing to repo ./scripts/tools/setup_breeze uv run dev/ide_setup/setup_vscode.py breeze start-airflow ``` -------------------------------- ### KubernetesInstallKueueOperator Example Source: https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/docs/operators.rst Example demonstrating how to install the Kueue component using KubernetesInstallKueueOperator. ```python from __future__ import annotations import pendulum from airflow.models.dag import DAG from airflow.providers.cncf.kubernetes.operators.kueue import KubernetesInstallKueueOperator with DAG( dag_id="kubernetes_kueue_install_example", schedule=None, start_date=pendulum.datetime(2023, 10, 26, tz="UTC"), catchup=False, tags=["kubernetes", "kueue", "example"], ) as dag: # Example of KubernetesInstallKueueOperator k8s_kueue_install = KubernetesInstallKueueOperator( task_id="kubernetes_kueue_install_task", namespace="default", image="kueue/kueue:latest", name="kueue-installer", is_delete_operator_pod=False, get_logs=True, ) ``` -------------------------------- ### Install Airflow chart with example DAGs Source: https://github.com/apache/airflow/blob/main/chart/docs/quick-start.rst Installs the Airflow Helm chart with example DAGs enabled by setting the AIRFLOW__CORE__LOAD_EXAMPLES environment variable to True. ```bash helm install $RELEASE_NAME apache-airflow/airflow \ --namespace $NAMESPACE \ --set-string "env[0].name=AIRFLOW__CORE__LOAD_EXAMPLES" \ --set-string "env[0].value=True" ``` -------------------------------- ### Start Airflow with example DAGs loaded Source: https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst Starts the Airflow development container with example DAGs loaded. This is useful for exploring Airflow's features and for testing. ```bash breeze start-airflow --load-example-dags ``` -------------------------------- ### Combined multi-module setup with Python version Source: https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm_intellij.rst Example of a complex setup combining multi-module mode, a specific Python version, and exclusions. ```bash $ uv run dev/ide_setup/setup_idea.py --multi-module --python 3.12 --exclude providers ``` -------------------------------- ### Setup Lang-SDK Coordinator System Test Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/05_test_commands.rst Provisions a Kubernetes cluster for the lang-SDK coordinator system test. This includes building Go and Java example bundles, deploying LocalStack S3, uploading artifacts, rendering coordinator pod-template image placeholders, and installing the Helm release. ```bash breeze k8s setup-lang-sdk-test ``` -------------------------------- ### Start Documentation Server - Bash Source: https://github.com/apache/airflow/blob/main/devel-common/src/docs/README.rst Bash script to start a lightweight documentation server for Apache Airflow. Requires Python to be installed. Serves documentation at http://localhost:8000 without needing to launch the full webserver. ```bash docs/start_doc_server.sh ``` -------------------------------- ### Start Airflow with Kafka Integration Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/03_developer_tasks.rst Example command to start Airflow with the Kafka integration, using Python 3.10 and a PostgreSQL backend. ```bash breeze --python 3.10 --backend postgres --integration kafka ``` -------------------------------- ### Example Docker buildx builder configuration output Source: https://github.com/apache/airflow/blob/main/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md Expected output showing a properly configured multi-builder setup with local and remote builder instances. ```text airflow_cache docker-container airflow_cache0 unix:///var/run/docker.sock airflow_cache1 tcp://127.0.0.1:2375 ``` -------------------------------- ### Verify GPG Signature Example Source: https://github.com/apache/airflow/blob/main/providers-summary-docs/installing-from-sources.rst Example output demonstrating successful GPG signature verification for an Apache Airflow Providers package. Shows the signature details including key ID, issuer, and verification status. ```console $ gpg --verify apache-airflow-providers-airbyte-1.0.0-source.tar.gz.asc apache-airflow-providers-airbyte-1.0.0-source.tar.gz gpg: Signature made Sat 11 Sep 12:49:54 2021 BST gpg: using RSA key CDE15C6E4D3A8EC4ECF4BA4B6674E08AD7DE406F gpg: issuer "kaxilnaik@apache.org" gpg: Good signature from "Kaxil Naik " [unknown] gpg: aka "Kaxil Naik " [unknown] gpg: WARNING: The key's User ID is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: CDE1 5C6E 4D3A 8EC4 ECF4 BA4B 6674 E08A D7DE 406F ``` -------------------------------- ### Setup Kubernetes Environment and Shell Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md Use the 'breeze' tool to set up the Kubernetes environment and enter a shell with necessary Python dependencies installed, including towncrier. ```shell breeze k8s setup-env breeze k8s shell ``` -------------------------------- ### Install UI Dependencies and Start Development Server Manually (Bash) Source: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/CONTRIBUTING.md These commands are part of the manual setup for Airflow UI development. First, `pnpm install` fetches all project dependencies, and then `pnpm dev` starts the UI development server, enabling live changes and testing. ```bash pnpm install && pnpm dev ``` -------------------------------- ### Package Example Project Source: https://github.com/apache/airflow/blob/main/java-sdk/README.md Navigate to the example directory and package the project using Gradle. ```bash ../gradlew bundle ``` -------------------------------- ### Start Airflow in Development Mode using Breeze (Bash) Source: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ui/CONTRIBUTING.md This command utilizes the Airflow Breeze tool to start Airflow in development mode, streamlining the setup and management of the development environment. It's a quick way to get a development instance running. ```bash breeze start-airflow --dev-mode ``` -------------------------------- ### Set Up Kubernetes Testing Environment Source: https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst Prepare the Kubernetes testing environment by creating a virtual environment and installing necessary tools like `kind`, `kubectl`, and `helm`. This command is typically run once for initial setup. ```bash breeze k8s setup-env ``` -------------------------------- ### Install PyMySQL for Airflow Database Connection (Bash) Source: https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_vscode.rst This command sequence activates the `airflow-env` Python environment and then installs the `PyMySQL` package. This is a required dependency for connecting Airflow to a MySQL database, enabling the use of MySQL as the backend for Airflow's metadata store. ```bash pyenv activate airflow-env pip install PyMySQL ``` -------------------------------- ### Example: Build documentation for Apache Beam provider package using Breeze Source: https://github.com/apache/airflow/blob/main/devel-common/src/docs/README.rst This provides a concrete example of building documentation for a specific Apache Airflow provider, namely `apache-airflow-providers-apache-beam`. It demonstrates using the shorthand `apache.beam` as the `PACKAGE_ID` for targeted documentation generation. ```bash breeze build-docs apache.beam ``` -------------------------------- ### Install Apache Airflow Google Cloud Providers via pip Source: https://github.com/apache/airflow/blob/main/providers/google/docs/operators/_partials/prerequisite_tasks.rst This command installs the necessary Apache Airflow providers for Google Cloud services, including all required dependencies. It uses `pip`, Python's package installer, to fetch and set up the `apache-airflow[google]` package. This step is crucial for enabling interaction with Google Cloud APIs from Airflow. ```bash pip install 'apache-airflow[google]' ``` -------------------------------- ### Install QEMU for multi-platform Docker builds Source: https://github.com/apache/airflow/blob/main/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md Enable multi-platform image building via QEMU emulation. This approach is simple but significantly slower (10x+) than hardware-backed builds. ```bash docker run --privileged --rm tonistiigi/binfmt --install all ``` -------------------------------- ### Run Breeze Setup Helper Script Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/01_installation.rst Execute the helper script to set up the breeze shim at ~/.local/bin/breeze. This script checks for uv, verifies no legacy global breeze install exists, and creates the executable shim. ```bash ./scripts/tools/setup_breeze ``` -------------------------------- ### Install Docker prerequisites and set up repository on Ubuntu Source: https://github.com/apache/airflow/blob/main/contributing-docs/03_contributors_quick_start.rst This snippet updates package lists and installs necessary dependencies for Docker, then sets up the Docker GPG key and repository for Ubuntu. ```bash sudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ``` -------------------------------- ### Install Elasticsearch Extra for Airflow Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst Install the Elasticsearch extra package for Airflow to get Elasticsearch hooks and Log Handler. ```bash pip install 'apache-airflow[elasticsearch]' ``` -------------------------------- ### Test Docker installation Source: https://github.com/apache/airflow/blob/main/contributing-docs/03_contributors_quick_start.rst This snippet runs a simple 'hello-world' container to verify that Docker is correctly installed and configured. ```bash docker run hello-world ``` -------------------------------- ### Install ArangoDB Extra for Airflow Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst Install the ArangoDB extra package for Airflow to get ArangoDB operators, sensors, and hooks. ```bash pip install 'apache-airflow[arangodb]' ``` -------------------------------- ### Start a Celery Worker Source: https://github.com/apache/airflow/blob/main/providers/celery/docs/celery_executor.rst Use this command to start an Airflow Celery worker. Ensure Airflow and necessary dependencies are installed. ```bash airflow celery worker ``` -------------------------------- ### Open IDE after setup completion Source: https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm_intellij.rst Use --open-ide to launch the IDE automatically after setup; can be combined with --confirm for non-interactive runs. ```bash $ uv run dev/ide_setup/setup_idea.py --open-ide ``` ```bash # Combine with --confirm for fully non-interactive setup + open $ uv run dev/ide_setup/setup_idea.py --confirm --open-ide ``` -------------------------------- ### Example Comparison Output Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md This is an example of the expected output when the built packages are byte-by-byte identical to the SVN versions, indicating a successful reproducible build verification. ```text apache_airflow_providers_amazon-8.12.0.tar.gz:No diff found apache_airflow_providers_apache_impala-1.2.1.tar.gz:No diff found apache_airflow_providers_atlassian_jira-2.3.0.tar.gz:No diff found apache_airflow_providers_cncf_kubernetes-7.10.0.tar.gz:No diff found apache_airflow_providers_common_io-1.1.0.tar.gz:No diff found apache_airflow_providers_common_sql-1.8.1.tar.gz:No diff found apache_airflow_providers_databricks-5.0.1.tar.gz:No diff found apache_airflow_providers_dbt_cloud-3.4.1.tar.gz:No diff found apache_airflow_providers_docker-3.8.2.tar.gz:No diff found apache_airflow_providers_elasticsearch-5.2.0.tar.gz:No diff found apache_airflow_providers_google-10.12.0.tar.gz:No diff found apache_airflow_providers_microsoft_azure-8.3.0.tar.gz:No diff found apache_airflow_providers_odbc-4.2.0.tar.gz:No diff found apache_airflow_providers_openai-1.0.1.tar.gz:No diff found apache_airflow_providers_opsgenie-5.3.0.tar.gz:No diff found apache_airflow_providers_papermill-3.5.0.tar.gz:No diff found apache_airflow_providers_redis-3.4.1.tar.gz:No diff found apache_airflow_providers_snowflake-5.1.2.tar.gz:No diff found apache_airflow_providers_trino-5.4.1.tar.gz:No diff found apache_airflow_providers_amazon-8.12.0-py3-none-any.whl:No diff found apache_airflow_providers_apache_impala-1.2.1-py3-none-any.whl:No diff found apache_airflow_providers_atlassian_jira-2.3.0-py3-none-any.whl:No diff found apache_airflow_providers_cncf_kubernetes-7.10.0-py3-none-any.whl:No diff found apache_airflow_providers_common_io-1.1.0-py3-none-any.whl:No diff found apache_airflow_providers_common_sql-1.8.1-py3-none-any.whl:No diff found apache_airflow_providers_databricks-5.0.1-py3-none-any.whl:No diff found apache_airflow_providers_dbt_cloud-3.4.1-py3-none-any.whl:No diff found apache_airflow_providers_docker-3.8.2-py3-none-any.whl:No diff found apache_airflow_providers_elasticsearch-5.2.0-py3-none-any.whl:No diff found apache_airflow_providers_google-10.12.0-py3-none-any.whl:No diff found apache_airflow_providers_microsoft_azure-8.3.0-py3-none-any.whl:No diff found apache_airflow_providers_odbc-4.2.0-py3-none-any.whl:No diff found apache_airflow_providers_openai-1.0.1-py3-none-any.whl:No diff found apache_airflow_providers_opsgenie-5.3.0-py3-none-any.whl:No diff found apache_airflow_providers_papermill-3.5.0-py3-none-any.whl:No diff found apache_airflow_providers_redis-3.4.1-py3-none-any.whl:No diff found apache_airflow_providers_snowflake-5.1.2-py3-none-any.whl:No diff found apache_airflow_providers_trino-5.4.1-py3-none-any.whl:No diff found ``` -------------------------------- ### Install Celery Extra for Airflow Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst Install the Celery extra package for Airflow to get Celery dependencies and sensors, enabling the CeleryExecutor. ```bash pip install 'apache-airflow[celery]' ``` -------------------------------- ### Start Release Process Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/09_release_management_tasks.rst Automates some of the steps needed when preparing the final release. ```bash breeze release-management start-release ``` -------------------------------- ### Start Airflow DAG Processor Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/installation/upgrading_to_airflow3.rst Command to start the DAG processor independently in Airflow 3. This is required even for local development setups. ```bash airflow dag-processor ``` -------------------------------- ### Start RC Process with Version Parameters Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md Initialize the release candidate process with version, previous version, task SDK version, and sync branch parameters. Run this after ensuring the script is up-to-date via git pull. ```shell git checkout main git pull # Ensure that the script is up-to-date breeze release-management start-rc-process \ --version ${VERSION_RC} \ --previous-version ${PREVIOUS_VERSION} \ --task-sdk-version ${TASK_SDK_VERSION_RC} \ --sync-branch ${SYNC_BRANCH} ``` -------------------------------- ### Verify Python installation and module import Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/installation/installing-from-pypi.rst After installing Python, use these commands to start the Python interpreter and confirm that a module can be imported successfully. ```bash python ``` ```python >>> import setproctitle # Success! ``` -------------------------------- ### Install Docker Credential Helper Source: https://github.com/apache/airflow/blob/main/contributing-docs/03_contributors_quick_start.rst Installs 'golang-docker-credential-helpers' to resolve Docker credential initialization errors, allowing Breeze to build images. ```bash sudo apt install golang-docker-credential-helpers ``` -------------------------------- ### Install Airflow with pip and venv on Debian/Ubuntu Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/start.rst Provides a step-by-step guide to set up a virtual environment and install Apache Airflow using pip on Debian/Ubuntu systems. This includes upgrading pip and verifying the installation. ```bash # For Windows after WSL2 install, restart computer, then in WSL Ubuntu terminal sudo apt update sudo apt install python3-pip python3-venv # Go to Linux home directory (not Windows mount) cd ~ # Create airflow directory mkdir -p ~/airflow cd ~/airflow # Create virtual environment python3 -m venv airflow_venv # Activate source airflow_venv/bin/activate # Upgrade pip pip install --upgrade pip # Install Airflow with correct Python version constraints pip install apache-airflow[celery]==3.1.0 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-3.1.0/constraints-3.12.txt # Verify installation airflow version ``` -------------------------------- ### Configuring Composite-Key Mapping with PartitionedAssetTimetable Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/authoring-and-scheduling/assets.rst Demonstrates using ProductMapper with IdentityMapper and StartOfDayMapper to handle composite keys for partitioned assets. This allows for flexible mapping of different segments of a partition key. ```python from airflow.sdk import ( Asset, IdentityMapper, PartitionedAssetTimetable, ProductMapper, StartOfDayMapper, ) regional_sales = Asset(uri="file://incoming/sales/regional.csv", name="regional_sales") with DAG( dag_id="aggregate_regional_sales", schedule=PartitionedAssetTimetable( assets=regional_sales, default_partition_mapper=ProductMapper(IdentityMapper(), StartOfDayMapper()), ), ): ... ``` -------------------------------- ### Install Apache Airflow Python client via pip Source: https://github.com/apache/airflow/blob/main/clients/python/README.md Install the apache-airflow-client package from PyPI, the standard and recommended method for getting the latest version. ```bash pip install apache-airflow-client ``` -------------------------------- ### Run Airflow Docker Compose Tests with Breeze Source: https://github.com/apache/airflow/blob/main/contributing-docs/testing/docker_compose_tests.rst This command sequence builds the Airflow production image for Python 3.10 and then executes the Docker Compose tests using the Breeze tool. It verifies the functionality of the Docker Compose setup as described in the documentation, ensuring Airflow can start and process example DAGs. ```bash breeze prod-image build --python 3.10 breeze testing docker-compose-tests ``` -------------------------------- ### Build and Publish Documentation Locally Source: https://github.com/apache/airflow/blob/main/docs/README.md Use these commands to build documentation for specified packages and then publish them to a local archive directory. Ensure the path to the archive repository is correctly specified. ```bash breeze build-docs "" "" --docs-only breeze release-management publish-docs --override-versioned --airflow-site-directory ``` -------------------------------- ### Install CNCF Kubernetes Extra for Airflow Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/extra-packages-ref.rst Install the CNCF Kubernetes extra package for Airflow to get Kubernetes client libraries and the KubernetesPodOperator, enabling the KubernetesExecutor. ```bash pip install 'apache-airflow[cncf-kubernetes]' ``` -------------------------------- ### Start Release Candidate Process Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/09_release_management_tasks.rst Automates some of the steps needed when preparing a release candidate. ```bash breeze release-management start-rc-process ``` -------------------------------- ### Start Airflow with Breeze for Testing Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md Initiates an Airflow 3.X.Y environment using Breeze, pre-configured for testing. After starting, you can install provider release candidates using pip. ```shell breeze start-airflow --use-airflow-version 3.1.3 --python 3.10 --backend postgres \ --load-example-dags --load-default-connections ``` ```shell pip install apache-airflow-providers-==rc ``` -------------------------------- ### Install Breeze for Airflow 2.11 with Legacy Setup Source: https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md If using the legacy global Breeze installation, run these commands after checking out 'v2-11-test' to reinstall Breeze for Airflow 2.11 compatibility. ```bash git checkout v2-11-test uv tool install --force -e ./dev/breeze ``` -------------------------------- ### Install Breeze with pipx Source: https://github.com/apache/airflow/blob/main/dev/breeze/doc/01_installation.rst Install Breeze using pipx as an alternative to uv tool, creating a fully-installed binary at ~/.local/bin/breeze. ```bash pipx install -e ./dev/breeze ``` -------------------------------- ### HttpOperator: Make a GET request with parameters Source: https://github.com/apache/airflow/blob/main/providers/http/docs/operators.rst This example shows how to use `HttpOperator` to perform a `GET` request, passing parameters to the endpoint. The task succeeds regardless of the response content. ```python task_get_op = HttpOperator( task_id="task_get_op", http_conn_id="http_default", endpoint="get", method="GET", data={"param1": "value1", "param2": "value2"}, log_response=True, ) ``` -------------------------------- ### Example Output of Release Candidate Files Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PYTHON_CLIENT.md This is an example output showing the expected files for a release candidate in the SVN repository, including tar.gz, .asc, and .sha512 checksums, as well as wheel files. ```shell (airflow) bugra@dev:~/PycharmProjects/asf-dist/dev/airflow/clients/python/3.1.8rc1$ ls -la total 57448 drwxrwxr-x 2 bugra bugra 4096 Mar 25 21:29 . drwxrwxr-x 3 bugra bugra 4096 Mar 25 21:28 .. -rw-rw-r-- 1 bugra bugra 386033 Mar 25 21:29 apache_airflow_client-3.1.8-py3-none-any.whl -rw-rw-r-- 1 bugra bugra 228 Mar 25 21:29 apache_airflow_client-3.1.8-py3-none-any.whl.asc -rw-rw-r-- 1 bugra bugra 175 Mar 25 21:29 apache_airflow_client-3.1.8-py3-none-any.whl.sha512 -rw-rw-r-- 1 bugra bugra 223960 Mar 25 21:29 apache_airflow_client-3.1.8.tar.gz -rw-rw-r-- 1 bugra bugra 228 Mar 25 21:29 apache_airflow_client-3.1.8.tar.gz.asc -rw-rw-r-- 1 bugra bugra 165 Mar 25 21:29 apache_airflow_client-3.1.8.tar.gz.sha512 -rw-rw-r-- 1 bugra bugra 58173692 Mar 25 21:29 apache_airflow_python_client-3.1.8-source.tar.gz -rw-rw-r-- 1 bugra bugra 228 Mar 25 21:29 apache_airflow_python_client-3.1.8-source.tar.gz.asc -rw-rw-r-- 1 bugra bugra 179 Mar 25 21:29 apache_airflow_python_client-3.1.8-source.tar.gz.sha512 ``` -------------------------------- ### Get Plugins Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/security/api_permissions_ref.rst Retrieves information about installed Airflow plugins. Requires 'View.PLUGINS' permission. ```APIDOC ## GET /api/v2/plugins ### Description Retrieves a list of all available plugins in Airflow. ### Method GET ### Endpoint /api/v2/plugins ### Permissions View.PLUGINS ``` -------------------------------- ### Define Airflow Teardown with Setup in a Single Line Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/setup-and-teardown.rst This example shows a convenient way to define a teardown task and associate it with its setup task in a single line using the `setups` argument within the `as_teardown()` method. This simplifies the DAG definition for common resource management patterns, ensuring `create_cluster` is the setup for `delete_cluster`. ```python create_cluster >> run_query >> delete_cluster.as_teardown(setups=create_cluster) ``` -------------------------------- ### Verify SVN Artifacts Listing Source: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_MYPY.md Example output showing the expected files in the release directory, including .tar.gz and .whl files with their corresponding .asc and .sha512 checksums. ```shell (apache-airflow) bugra@dev:~/PycharmProjects/asf-dist/dev/airflow/apache-airflow-mypy/0.1.0rc2$ ls -la total 48 drwxrwxr-x 2 bugra bugra 4096 Jun 10 19:00 . drwxrwxr-x 3 bugra bugra 4096 Jun 10 19:00 .. -rw-rw-r-- 1 bugra bugra 11070 Jun 10 19:00 apache_airflow_mypy-0.1.0-py3-none-any.whl -rw-rw-r-- 1 bugra bugra 906 Jun 10 19:00 apache_airflow_mypy-0.1.0-py3-none-any.whl.asc -rw-rw-r-- 1 bugra bugra 173 Jun 10 19:00 apache_airflow_mypy-0.1.0-py3-none-any.whl.sha512 -rw-rw-r-- 1 bugra bugra 9116 Jun 10 19:00 apache_airflow_mypy-0.1.0.tar.gz -rw-rw-r-- 1 bugra bugra 906 Jun 10 19:00 apache_airflow_mypy-0.1.0.tar.gz.asc -rw-rw-r-- 1 bugra bugra 163 Jun 10 19:00 apache_airflow_mypy-0.1.0.tar.gz.sha512 ``` -------------------------------- ### Install Custom Requirements During Docker Build Source: https://github.com/apache/airflow/blob/main/docker-stack-docs/build.rst This example demonstrates how to build an Airflow Docker image while installing custom Python requirements from a `requirements.txt` file located in the `docker-context-files` directory. It also explicitly installs the `apache-airflow` package to ensure version consistency. ```bash #!/usr/bin/env bash # Build Airflow image with custom requirements # The --build-arg DOCKER_CONTEXT_FILES=docker-context-files flag tells Docker to # look for custom files in the 'docker-context-files' directory. # The requirements.txt file in this directory will be installed by pip. # Explicitly installing apache-airflow ensures version consistency. export AIRFLOW_VERSION=2.7.1 export PYTHON_VERSION="3.10" docker build \ --build-arg \ AIRFLOW_VERSION=${AIRFLOW_VERSION} \ --build-arg \ PYTHON_VERSION=${PYTHON_VERSION} \ --build-arg \ DOCKER_CONTEXT_FILES=docker-context-files \ -t apache/airflow:latest \ -f docker/Dockerfile . ``` -------------------------------- ### Custom Partition Mapper Example Source: https://github.com/apache/airflow/blob/main/airflow-core/docs/authoring-and-scheduling/assets.rst This example demonstrates how to create a custom partition mapper that strips a namespace prefix from upstream keys. ```python from airflow.partition_mappers.base import PartitionMapper class CustomPartitionMapper(PartitionMapper): def get_partition_keys(self, upstream_keys): return [key.split('::')[-1] for key in upstream_keys] ```