### Run NiPyAPI Quick Start Examples Source: https://nipyapi.readthedocs.io/en/latest/_sources/nipyapi-docs/examples.rst.txt Commands to initialize the sandbox environment and execute specific workflow scripts. ```console $ make sandbox NIPYAPI_PROFILE=single-user $ python examples/sandbox.py single-user ``` ```console $ python examples/fdlc.py ``` -------------------------------- ### Copy Example Profiles File Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt Copies the example profiles.yml file to the user's configuration directory to serve as a starting point for custom profiles. ```bash cp examples/profiles.yml ~/.nipyapi/profiles.yml ``` -------------------------------- ### Start Sandbox Environment Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/examples.html Use these make targets to set up a ready-to-use sandbox environment for quick experimentation. The recommended option is 'single-user' for a simple setup. ```bash $ make sandbox NIPYAPI_PROFILE=single-user # Recommended - simple setup $ make sandbox NIPYAPI_PROFILE=secure-ldap # LDAP authentication $ make sandbox NIPYAPI_PROFILE=secure-mtls # Certificate authentication (advanced) ``` -------------------------------- ### Start Complete NiFi Environment with Docker Source: https://nipyapi.readthedocs.io/en/latest/readme.html Sets up and starts a single-user NiFi environment using Docker Compose. This is a quick way to get a NiFi instance running for testing. ```bash make certs && make up NIPYAPI_PROFILE=single-user && make wait-ready NIPYAPI_PROFILE=single-user ``` -------------------------------- ### Setup & Installation Make Targets Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt Common Makefile targets for setting up the development environment and managing project artifacts. ```makefile make dev-install # Install with dev dependencies (uses uv if available, pip otherwise) make docs-install # Install documentation dependencies make clean # Remove build, pyc, and temp artifacts make clean-all # Nuclear clean: removes ALL including generated code ``` -------------------------------- ### NiPyAPI CLI CI/CD Operations Examples Source: https://nipyapi.readthedocs.io/en/latest/readme.html Provides examples of common CI/CD operations using the NiPyAPI CLI, such as deploying, starting, and getting the status of NiFi flows. ```bash nipyapi ci deploy_flow --bucket flows --flow my-flow nipyapi ci start_flow --process_group_id nipyapi ci get_status --process_group_id ``` -------------------------------- ### Docker Environment Setup for Development Source: https://nipyapi.readthedocs.io/en/latest/migration.html Commands to set up a Docker environment for development, including certificate generation and starting the NiFi profile. ```bash make certs && make up NIPYAPI_PROFILE=single-user ``` -------------------------------- ### Complete Workflow Example Source: https://nipyapi.readthedocs.io/en/latest/ci.html A comprehensive shell script demonstrating a typical NiFi flow deployment workflow, including ensuring a registry client, deploying a flow, configuring parameters, starting the flow, and verifying its status. ```bash #!/bin/bash set -e # Configuration export NIFI_API_ENDPOINT="https://nifi.example.com/nifi-api" export NIFI_BEARER_TOKEN="$NIFI_TOKEN" export GH_REGISTRY_TOKEN="$GITHUB_TOKEN" # 1. Ensure registry client exists RESULT=$(nipyapi ci ensure_registry --repo myorg/nifi-flows) REGISTRY_ID=$(echo "$RESULT" | jq -r '.registry_client_id') echo "Registry client: $REGISTRY_ID" # 2. Deploy the flow RESULT=$(nipyapi ci deploy_flow \ --registry_client_id "$REGISTRY_ID" \ --bucket connectors \ --flow postgresql \ --version "$FLOW_VERSION") PG_ID=$(echo "$RESULT" | jq -r '.process_group_id') echo "Deployed process group: $PG_ID" # 3. Configure parameters nipyapi ci configure_params \ --process_group_id "$PG_ID" \ --parameters '{"db_host": "'"$DB_HOST"'", "db_password": "'"$DB_PASSWORD"'"}' # 4. Start the flow nipyapi ci start_flow --process_group_id "$PG_ID" # 5. Verify status nipyapi ci get_status --process_group_id "$PG_ID" ``` -------------------------------- ### Run NiPyAPI Sandbox Environment Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/examples.html Execute the sandbox example script with a single-user profile. This is a good starting point for new users. ```bash $ make sandbox NIPYAPI_PROFILE=single-user $ python examples/sandbox.py single-user ``` -------------------------------- ### Verify Installation Source: https://nipyapi.readthedocs.io/en/latest/installation.html Check that the library is correctly installed by printing the version. ```bash $ python -c "import nipyapi; print(f'NiPyAPI {nipyapi.__version__} installed successfully')" ``` -------------------------------- ### Verify installation Source: https://nipyapi.readthedocs.io/en/latest/_sources/devnotes.rst.txt Verify the package installation in a clean environment. ```shell # Test installation in clean environment pip install nipyapi=={version} python -c "import nipyapi; print(nipyapi.__version__)" ``` -------------------------------- ### GET /nars Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html Retrieves summary information for installed NARs. ```APIDOC ## GET /nars ### Description Retrieves summary information for installed NARs. ### Method GET ### Endpoint /nars ### Response #### Success Response (200) - **NarSummariesEntity** (object) - The response data. ``` -------------------------------- ### Set Up Local Development Environment Source: https://nipyapi.readthedocs.io/en/latest/contributing.html Commands to create a virtual environment and install development dependencies. ```bash # using venv $ python -m venv .venv && source .venv/bin/activate $ cd nipyapi/ $ make dev-install # uses uv if available, falls back to pip ``` -------------------------------- ### Set up local development environment Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt Commands to create a virtual environment and install development dependencies using the Makefile. ```shell # using venv $ python -m venv .venv && source .venv/bin/activate $ cd nipyapi/ $ make dev-install # uses uv if available, falls back to pip # or using uv (faster) $ uv venv .venv && source .venv/bin/activate $ make dev-install ``` -------------------------------- ### GET /nars/{id} Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html Retrieves the component types available from the installed NARs. ```APIDOC ## GET /nars/{id} ### Description Retrieves the component types available from the installed NARs. ### Method GET ### Endpoint /nars/{id} ### Parameters #### Path Parameters - **id** (str) - Required - The id of the NAR. ### Response #### Success Response (200) - **NarDetailsEntity** (object) - The response data. ``` -------------------------------- ### Quick Start: Configure NiFi and Deploy Flow Source: https://nipyapi.readthedocs.io/en/latest/_sources/ci.rst.txt This console snippet outlines the initial steps for using nipyapi in a CI environment. It covers setting up NiFi connection details via environment variables, ensuring a registry client, and deploying a flow. ```console # 1. Configure NiFi connection export NIFI_API_ENDPOINT="https://nifi.example.com/nifi-api" export NIFI_BEARER_TOKEN="your-jwt-token" # 2. Set up registry client export GH_REGISTRY_TOKEN="ghp_xxxx" nipyapi ci ensure_registry --repo owner/repo # 3. Deploy a flow nipyapi ci deploy_flow --bucket connectors --flow postgresql # 4. Start the flow nipyapi ci start_flow --process_group_id $PROCESS_GROUP_ID ``` -------------------------------- ### Deploy, Start, and Get Flow Status with NiPyAPI Source: https://nipyapi.readthedocs.io/en/latest/ci.html Use this snippet to deploy a flow from a registry, start it, and then retrieve its current status including state and running processors. Ensure the registry_client_id is correctly configured. ```python result = nipyapi.ci.deploy_flow( registry_client_id=registry_id, bucket='connectors', flow='postgresql' ) pg_id = result['process_group_id'] nipyapi.ci.start_flow(process_group_id=pg_id) status = nipyapi.ci.get_status(process_group_id=pg_id) print(f"State: {status['state']}, Running: {status['running_processors']}") ``` -------------------------------- ### Example Workflow for Code Discovery Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt Demonstrates how to search for existing functions and read their documentation to avoid reimplementation. ```bash # Check what's exported from utils head -50 nipyapi/utils.py | grep -A20 "__all__" # Find a specific function grep -n "def wait_to_complete" nipyapi/utils.py # Read the docstring to understand it # (or use your IDE's go-to-definition) ``` -------------------------------- ### Setup GPG for Commit Signing on macOS Source: https://nipyapi.readthedocs.io/en/latest/devnotes.html Commands to install GPG and generate signing keys for commit authentication. ```bash # Install GPG via Homebrew (recommended) brew install gnupg # Generate signing keys (use a strong passphrase) gpg --full-generate-key ``` -------------------------------- ### Reading Example Test File Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt Illustrates how to read an existing test file to understand established patterns for writing new tests. ```bash # Read an example test file for patterns head -100 tests/test_canvas.py ``` -------------------------------- ### Install Dependencies with uv or pip Source: https://nipyapi.readthedocs.io/en/latest/contributing.html Use uv for faster dependency installation if available, otherwise pip will be used. This command sets up the development environment. ```bash uv venv .venv && source .venv/bin/activate make dev-install ``` -------------------------------- ### Docker Environment Setup with Profiles Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt Set up and manage the Docker environment using make commands, integrating NiPyAPI profiles for consistent configurations. ```shell # One-command environment setup make certs && make up NIPYAPI_PROFILE=secure-ldap && make wait-ready NIPYAPI_PROFILE=secure-ldap # Python code matches the environment nipyapi.profiles.switch('secure-ldap') # Testing with the same profile make test NIPYAPI_PROFILE=secure-ldap ``` -------------------------------- ### GET /api/nar_details/{id} Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html Retrieves the component types available from the installed NARs. This method returns the direct response data. ```APIDOC ## GET /api/nar_details/{id} ### Description Retrieves the component types available from the installed NARs. This method returns the direct response data. ### Method GET ### Endpoint /api/nar_details/{id} ### Parameters #### Path Parameters - **id** (str) - Required - The NAR ID. ### Response #### Success Response (200) - **Response Data** - The component types available from the installed NARs. #### Response Example ```json { "example": "response body" } ``` ``` -------------------------------- ### Initialize Infrastructure via Console Source: https://nipyapi.readthedocs.io/en/latest/_sources/profiles.rst.txt Commands to generate certificates and start the Docker environment for a specific profile. ```console # Generate certificates and start Docker environment make certs make up NIPYAPI_PROFILE=single-user make wait-ready NIPYAPI_PROFILE=single-user ``` -------------------------------- ### Common Usage Examples Source: https://nipyapi.readthedocs.io/en/latest/_sources/cli.rst.txt Practical examples for common tasks like connectivity testing and flow deployment. ```console nipyapi system get_nifi_version_info ``` ```console nipyapi canvas list_all_process_groups ``` ```console nipyapi canvas get_process_group "abc-123-def" id ``` ```console nipyapi versioning get_registry_client "MyRegistryClient" ``` ```console nipyapi ci deploy_flow \ --registry_client_id "$REGISTRY_ID" \ --bucket "connectors" \ --flow "postgresql" ``` ```console # Get registry client ID by name REGISTRY_ID=$(nipyapi versioning get_registry_client "MyClient" | jq -r '.id') # List process group names nipyapi canvas list_all_process_groups | jq -r '.[].component.name' ``` -------------------------------- ### Get NAR Summary Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html Retrieve a summary of a specific NAR installed in NiFi using its unique identifier. Returns None if the NAR is not found. ```python >>> # Example usage for get_nar would go here, assuming 'identifier' is known >>> # nar_summary = nipyapi.extensions.get_nar('some-nar-identifier') ``` -------------------------------- ### Get NAR Details Source: https://nipyapi.readthedocs.io/en/latest/_sources/extensions.rst.txt Retrieve detailed information about an installed NAR, including its processor and controller service types. Requires the nipyapi library. ```python import nipyapi # Get NAR with full details details = nipyapi.extensions.get_nar_details(nar_id) print("Processor types:") for proc_type in details.processor_types: print(f" - {proc_type.type}") print("Controller service types:") for cs_type in details.controller_service_types: print(f" - {cs_type.type}") ``` -------------------------------- ### Get Processor Type Version Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html Retrieves a specific processor type with a given bundle version. This is essential for creating processors when multiple versions of the same type are installed. ```APIDOC ## GET /nifi-api/extensions/processor-type-version ### Description Get a processor type with a specific bundle version for creation. ### Method GET ### Endpoint /nifi-api/extensions/processor-type-version ### Parameters #### Query Parameters - **processor_type** (string) - Required - The processor type name (e.g., ‘PrepareRegulatoryFile’) - **version** (string) - Required - The bundle version (e.g., ‘0.0.2-SNAPSHOT’) ### Response #### Success Response (200) - **DocumentedTypeDTO** - The processor type with the specified bundle version, suitable for passing to `nipyapi.canvas.create_processor()`. ### Raises - **ValueError** - If no matching processor type/version found ### Request Example ```python # Get the v2 processor type proc_type = nipyapi.extensions.get_processor_type_version( 'PrepareRegulatoryFile', '0.0.2-SNAPSHOT' ) # Create processor with that specific version proc = nipyapi.canvas.create_processor(pg, proc_type, (0,0), 'MyProc') ``` ``` -------------------------------- ### Get Specific Processor Type Version Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html Retrieve a processor type that corresponds to a specific bundle version. This is essential for creating processors when multiple versions of the same type are installed, ensuring you use the correct one. ```python # Get the v2 processor type proc_type = nipyapi.extensions.get_processor_type_version( 'PrepareRegulatoryFile', '0.0.2-SNAPSHOT' ) # Create processor with that specific version proc = nipyapi.canvas.create_processor(pg, proc_type, (0,0), 'MyProc') ``` -------------------------------- ### Build and documentation targets Source: https://nipyapi.readthedocs.io/en/latest/contributing.html Commands for building distributions and generating project documentation. ```makefile make dist # Build wheel and source distribution make check-dist # Validate distribution files make test-dist # Test that distribution can be imported make docs # Generate Sphinx documentation ``` -------------------------------- ### Create Custom NiFi Profiles Source: https://nipyapi.readthedocs.io/en/latest/migration.html Copy the example profiles.yml file to your home directory and edit it to configure your specific environments. ```bash # Copy the example as a starting point cp examples/profiles.yml ~/.nipyapi/profiles.yml # Edit for your environments vim ~/.nipyapi/profiles.yml ``` -------------------------------- ### Get Processor Initialization Status Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html Check the initialization status of a processor, particularly useful for custom or Python processors that may require additional setup time. Provides details on readiness, property loading, and validation status. ```python status = nipyapi.extensions.get_processor_init_status(proc) if status['is_ready']: print("Processor ready for configuration") ``` -------------------------------- ### Install in a Virtual Environment Source: https://nipyapi.readthedocs.io/en/latest/installation.html Recommended approach to isolate dependencies and avoid conflicts. ```bash $ python -m venv nipyapi-env $ source nipyapi-env/bin/activate # On Windows: nipyapi-env\Scripts\activate $ pip install nipyapi ``` -------------------------------- ### List Available Processor Bundle Versions Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html Get a list of all installed bundle versions for a given processor type. This is useful when multiple versions of the same processor are available from different NARs. Each version includes bundle information, the full type name, and a description. ```python versions = nipyapi.extensions.get_processor_bundle_versions( 'PrepareRegulatoryFile' ) for v in versions: print(f"{v['bundle'].version}") ``` -------------------------------- ### Install NiPyAPI from Source Archive Source: https://nipyapi.readthedocs.io/en/latest/_sources/installation.rst.txt Install NiPyAPI by downloading and installing from a source archive. This method involves downloading the tarball, extracting it, and then installing using pip. ```console curl -OL https://github.com/Chaffelson/nipyapi/tarball/main tar -xzf main cd Chaffelson-nipyapi-* pip install . ``` ```console python -m build pip install dist/*.whl ``` -------------------------------- ### Compare Configuration Complexity Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt Demonstration of the reduction in configuration steps from 0.x to 1.x. ```python # 0.x: Complex multi-step configuration (10+ lines) import os import nipyapi # Step 1: Set global SSL settings nipyapi.config.global_ssl_verify = True nipyapi.config.global_ssl_host_check = False # But this affects BOTH services! nipyapi.config.disable_insecure_request_warnings = True # Step 2: Direct configuration (NIFI_CA_CERT no longer supported) nipyapi.config.nifi_config.ssl_ca_cert = '/path/to/ca.pem' # Step 4: Manually configure each service endpoint nipyapi.utils.set_endpoint("https://localhost:8443/nifi-api", ssl=True, login=True) # Registry SSL settings? More complex configuration needed... # vs. # 1.x: Simple declarative configuration (1 line) ``` -------------------------------- ### Start Flow Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/ci.html Starts a process group, enabling controller services and starting processors. ```APIDOC ## POST /nipyapi/ci/start_flow ### Description Start a process group (enable controllers, start processors). ### Method POST ### Endpoint /nipyapi/ci/start_flow ### Parameters #### Query Parameters - **process_group_id** (string) - Required - ID of the process group. Env: NIFI_PROCESS_GROUP_ID - **enable_controllers** (boolean) - Optional - Enable controller services first (default: True). ### Response #### Success Response (200) - **started** (boolean) - Indicates if the flow was started. - **process_group_name** (string) - The name of the process group. #### Error Response (400) - **error** (string) - Missing required parameters. #### Error Response (500) - **error** (string) - NiFi API errors. ### Request Example ``` nipyapi ci start_flow --process_group_id PG_ID nipyapi ci start_flow --process_group_id PG_ID --enable_controllers false ``` ``` -------------------------------- ### Output Format Configuration Source: https://nipyapi.readthedocs.io/en/latest/_sources/cli.rst.txt Examples of how the CLI formats output for different environments. ```console nipyapi system get_nifi_version_info # Output: {"niFi_version": "2.0.0", ...} ``` ```console # In GitHub Actions workflow nipyapi ci deploy_flow ... >> $GITHUB_OUTPUT # Output: process-group-id=abc-123 ``` ```console # In GitLab CI job nipyapi ci deploy_flow ... > deploy.env # Output: PROCESS_GROUP_ID=abc-123 ``` ```console export NIFI_OUTPUT_FORMAT=json # JSON (default) export NIFI_OUTPUT_FORMAT=github # GitHub Actions format export NIFI_OUTPUT_FORMAT=dotenv # GitLab dotenv format ``` -------------------------------- ### Test Installation in Clean Environment Source: https://nipyapi.readthedocs.io/en/latest/devnotes.html Verify the installation of the released package in a clean environment by installing it using pip and checking the imported version. ```python pip install nipyapi=={version} python -c "import nipyapi; print(nipyapi.__version__)" ``` -------------------------------- ### Install NiPyAPI with Development and Documentation Dependencies Source: https://nipyapi.readthedocs.io/en/latest/installation.html Installs NiPyAPI along with optional development and documentation dependencies. This command is useful for setting up a complete development environment. ```bash pip install -e ".[dev,docs]" ``` -------------------------------- ### Install for Development Source: https://nipyapi.readthedocs.io/en/latest/installation.html Install the package in editable mode with development and documentation dependencies. ```bash # Clone the repository $ git clone https://github.com/Chaffelson/nipyapi.git $ cd nipyapi # Install in development mode with all dependencies $ pip install -e ".[dev,docs]" ```