NiPyAPI

https://nipyapi.readthedocs.io/en/latest/
Admin
NiPyAPI is a Python client SDK for Apache NiFi that provides mid-level functions, command-line...
Tokens:897,320
Snippets:5,773
Trust Score:9.6
Update:6 months ago
Show doc for...
Tokens:
Raw
### Run NiPyAPI Quick Start Examples

Source: https://nipyapi.readthedocs.io/en/latest/_sources/nipyapi-docs/examples.rst.txt

Commands to initialize the sandbox environment and execute specific workflow scripts.

```console
$ make sandbox NIPYAPI_PROFILE=single-user
$ python examples/sandbox.py single-user
```

```console
$ python examples/fdlc.py
```

--------------------------------

### Copy Example Profiles File

Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt

Copies the example profiles.yml file to the user's configuration directory to serve as a starting point for custom profiles.

```bash
cp examples/profiles.yml ~/.nipyapi/profiles.yml
```

--------------------------------

### Start Sandbox Environment

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/examples.html

Use these make targets to set up a ready-to-use sandbox environment for quick experimentation. The recommended option is 'single-user' for a simple setup.

```bash
$ make sandbox NIPYAPI_PROFILE=single-user     # Recommended - simple setup
$ make sandbox NIPYAPI_PROFILE=secure-ldap     # LDAP authentication
$ make sandbox NIPYAPI_PROFILE=secure-mtls     # Certificate authentication (advanced)

```

--------------------------------

### Start Complete NiFi Environment with Docker

Source: https://nipyapi.readthedocs.io/en/latest/readme.html

Sets up and starts a single-user NiFi environment using Docker Compose. This is a quick way to get a NiFi instance running for testing.

```bash
make certs && make up NIPYAPI_PROFILE=single-user && make wait-ready NIPYAPI_PROFILE=single-user
```

--------------------------------

### Setup & Installation Make Targets

Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt

Common Makefile targets for setting up the development environment and managing project artifacts.

```makefile
make dev-install      # Install with dev dependencies (uses uv if available, pip otherwise)
make docs-install     # Install documentation dependencies
make clean            # Remove build, pyc, and temp artifacts
make clean-all        # Nuclear clean: removes ALL including generated code
```

--------------------------------

### NiPyAPI CLI CI/CD Operations Examples

Source: https://nipyapi.readthedocs.io/en/latest/readme.html

Provides examples of common CI/CD operations using the NiPyAPI CLI, such as deploying, starting, and getting the status of NiFi flows.

```bash
nipyapi ci deploy_flow --bucket flows --flow my-flow
nipyapi ci start_flow --process_group_id <id>
nipyapi ci get_status --process_group_id <id>
```

--------------------------------

### Docker Environment Setup for Development

Source: https://nipyapi.readthedocs.io/en/latest/migration.html

Commands to set up a Docker environment for development, including certificate generation and starting the NiFi profile.

```bash
make certs && make up NIPYAPI_PROFILE=single-user
```

--------------------------------

### Complete Workflow Example

Source: https://nipyapi.readthedocs.io/en/latest/ci.html

A comprehensive shell script demonstrating a typical NiFi flow deployment workflow, including ensuring a registry client, deploying a flow, configuring parameters, starting the flow, and verifying its status.

```bash
#!/bin/bash
set -e

# Configuration
export NIFI_API_ENDPOINT="https://nifi.example.com/nifi-api"
export NIFI_BEARER_TOKEN="$NIFI_TOKEN"
export GH_REGISTRY_TOKEN="$GITHUB_TOKEN"

# 1. Ensure registry client exists
RESULT=$(nipyapi ci ensure_registry --repo myorg/nifi-flows)
REGISTRY_ID=$(echo "$RESULT" | jq -r '.registry_client_id')
echo "Registry client: $REGISTRY_ID"

# 2. Deploy the flow
RESULT=$(nipyapi ci deploy_flow \
    --registry_client_id "$REGISTRY_ID" \
    --bucket connectors \
    --flow postgresql \
    --version "$FLOW_VERSION")
PG_ID=$(echo "$RESULT" | jq -r '.process_group_id')
echo "Deployed process group: $PG_ID"

# 3. Configure parameters
nipyapi ci configure_params \
    --process_group_id "$PG_ID" \
    --parameters '{"db_host": "'"$DB_HOST"'", "db_password": "'"$DB_PASSWORD"'"}'

# 4. Start the flow
nipyapi ci start_flow --process_group_id "$PG_ID"

# 5. Verify status
nipyapi ci get_status --process_group_id "$PG_ID"

```

--------------------------------

### Run NiPyAPI Sandbox Environment

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/examples.html

Execute the sandbox example script with a single-user profile. This is a good starting point for new users.

```bash
$ make sandbox NIPYAPI_PROFILE=single-user
$ python examples/sandbox.py single-user

```

--------------------------------

### Verify Installation

Source: https://nipyapi.readthedocs.io/en/latest/installation.html

Check that the library is correctly installed by printing the version.

```bash
$ python -c "import nipyapi; print(f'NiPyAPI {nipyapi.__version__} installed successfully')"
```

--------------------------------

### Verify installation

Source: https://nipyapi.readthedocs.io/en/latest/_sources/devnotes.rst.txt

Verify the package installation in a clean environment.

```shell
# Test installation in clean environment
pip install nipyapi=={version}
python -c "import nipyapi; print(nipyapi.__version__)"
```

--------------------------------

### GET /nars

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html

Retrieves summary information for installed NARs.

```APIDOC
## GET /nars

### Description
Retrieves summary information for installed NARs.

### Method
GET

### Endpoint
/nars

### Response
#### Success Response (200)
- **NarSummariesEntity** (object) - The response data.
```

--------------------------------

### Set Up Local Development Environment

Source: https://nipyapi.readthedocs.io/en/latest/contributing.html

Commands to create a virtual environment and install development dependencies.

```bash
# using venv
$ python -m venv .venv && source .venv/bin/activate
$ cd nipyapi/
$ make dev-install  # uses uv if available, falls back to pip
```

--------------------------------

### Set up local development environment

Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt

Commands to create a virtual environment and install development dependencies using the Makefile.

```shell
# using venv
$ python -m venv .venv && source .venv/bin/activate
$ cd nipyapi/
$ make dev-install  # uses uv if available, falls back to pip

# or using uv (faster)
$ uv venv .venv && source .venv/bin/activate
$ make dev-install
```

--------------------------------

### GET /nars/{id}

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html

Retrieves the component types available from the installed NARs.

```APIDOC
## GET /nars/{id}

### Description
Retrieves the component types available from the installed NARs.

### Method
GET

### Endpoint
/nars/{id}

### Parameters
#### Path Parameters
- **id** (str) - Required - The id of the NAR.

### Response
#### Success Response (200)
- **NarDetailsEntity** (object) - The response data.
```

--------------------------------

### Quick Start: Configure NiFi and Deploy Flow

Source: https://nipyapi.readthedocs.io/en/latest/_sources/ci.rst.txt

This console snippet outlines the initial steps for using nipyapi in a CI environment. It covers setting up NiFi connection details via environment variables, ensuring a registry client, and deploying a flow.

```console
# 1. Configure NiFi connection
export NIFI_API_ENDPOINT="https://nifi.example.com/nifi-api"
export NIFI_BEARER_TOKEN="your-jwt-token"

# 2. Set up registry client
export GH_REGISTRY_TOKEN="ghp_xxxx"
nipyapi ci ensure_registry --repo owner/repo

# 3. Deploy a flow
nipyapi ci deploy_flow --bucket connectors --flow postgresql

# 4. Start the flow
nipyapi ci start_flow --process_group_id $PROCESS_GROUP_ID
```

--------------------------------

### Deploy, Start, and Get Flow Status with NiPyAPI

Source: https://nipyapi.readthedocs.io/en/latest/ci.html

Use this snippet to deploy a flow from a registry, start it, and then retrieve its current status including state and running processors. Ensure the registry_client_id is correctly configured.

```python
result = nipyapi.ci.deploy_flow(
    registry_client_id=registry_id,
    bucket='connectors',
    flow='postgresql'
)
pg_id = result['process_group_id']

nipyapi.ci.start_flow(process_group_id=pg_id)

status = nipyapi.ci.get_status(process_group_id=pg_id)
print(f"State: {status['state']}, Running: {status['running_processors']}")
```

--------------------------------

### Example Workflow for Code Discovery

Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt

Demonstrates how to search for existing functions and read their documentation to avoid reimplementation.

```bash
# Check what's exported from utils
head -50 nipyapi/utils.py | grep -A20 "__all__"

# Find a specific function
grep -n "def wait_to_complete" nipyapi/utils.py

# Read the docstring to understand it
# (or use your IDE's go-to-definition)
```

--------------------------------

### Setup GPG for Commit Signing on macOS

Source: https://nipyapi.readthedocs.io/en/latest/devnotes.html

Commands to install GPG and generate signing keys for commit authentication.

```bash
# Install GPG via Homebrew (recommended)
brew install gnupg

# Generate signing keys (use a strong passphrase)
gpg --full-generate-key
```

--------------------------------

### Reading Example Test File

Source: https://nipyapi.readthedocs.io/en/latest/_sources/contributing.rst.txt

Illustrates how to read an existing test file to understand established patterns for writing new tests.

```bash
# Read an example test file for patterns
head -100 tests/test_canvas.py
```

--------------------------------

### Install Dependencies with uv or pip

Source: https://nipyapi.readthedocs.io/en/latest/contributing.html

Use uv for faster dependency installation if available, otherwise pip will be used. This command sets up the development environment.

```bash
uv venv .venv && source .venv/bin/activate
make dev-install
```

--------------------------------

### Docker Environment Setup with Profiles

Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt

Set up and manage the Docker environment using make commands, integrating NiPyAPI profiles for consistent configurations.

```shell
# One-command environment setup
make certs && make up NIPYAPI_PROFILE=secure-ldap && make wait-ready NIPYAPI_PROFILE=secure-ldap

# Python code matches the environment
nipyapi.profiles.switch('secure-ldap')

# Testing with the same profile
make test NIPYAPI_PROFILE=secure-ldap
```

--------------------------------

### GET /api/nar_details/{id}

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nifi_apis/controller_api.html

Retrieves the component types available from the installed NARs. This method returns the direct response data.

```APIDOC
## GET /api/nar_details/{id}

### Description
Retrieves the component types available from the installed NARs. This method returns the direct response data.

### Method
GET

### Endpoint
/api/nar_details/{id}

### Parameters
#### Path Parameters
- **id** (str) - Required - The NAR ID.

### Response
#### Success Response (200)
- **Response Data** - The component types available from the installed NARs.

#### Response Example
```json
{
  "example": "response body"
}
```
```

--------------------------------

### Initialize Infrastructure via Console

Source: https://nipyapi.readthedocs.io/en/latest/_sources/profiles.rst.txt

Commands to generate certificates and start the Docker environment for a specific profile.

```console
# Generate certificates and start Docker environment
make certs
make up NIPYAPI_PROFILE=single-user
make wait-ready NIPYAPI_PROFILE=single-user
```

--------------------------------

### Common Usage Examples

Source: https://nipyapi.readthedocs.io/en/latest/_sources/cli.rst.txt

Practical examples for common tasks like connectivity testing and flow deployment.

```console
nipyapi system get_nifi_version_info
```

```console
nipyapi canvas list_all_process_groups
```

```console
nipyapi canvas get_process_group "abc-123-def" id
```

```console
nipyapi versioning get_registry_client "MyRegistryClient"
```

```console
nipyapi ci deploy_flow \
    --registry_client_id "$REGISTRY_ID" \
    --bucket "connectors" \
    --flow "postgresql"
```

```console
# Get registry client ID by name
REGISTRY_ID=$(nipyapi versioning get_registry_client "MyClient" | jq -r '.id')

# List process group names
nipyapi canvas list_all_process_groups | jq -r '.[].component.name'
```

--------------------------------

### Get NAR Summary

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html

Retrieve a summary of a specific NAR installed in NiFi using its unique identifier. Returns None if the NAR is not found.

```python
>>> # Example usage for get_nar would go here, assuming 'identifier' is known
>>> # nar_summary = nipyapi.extensions.get_nar('some-nar-identifier')

```

--------------------------------

### Get NAR Details

Source: https://nipyapi.readthedocs.io/en/latest/_sources/extensions.rst.txt

Retrieve detailed information about an installed NAR, including its processor and controller service types. Requires the nipyapi library.

```python
import nipyapi

# Get NAR with full details
details = nipyapi.extensions.get_nar_details(nar_id)

print("Processor types:")
for proc_type in details.processor_types:
    print(f"  - {proc_type.type}")

print("Controller service types:")
for cs_type in details.controller_service_types:
    print(f"  - {cs_type.type}")
```

--------------------------------

### Get Processor Type Version

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html

Retrieves a specific processor type with a given bundle version. This is essential for creating processors when multiple versions of the same type are installed.

```APIDOC
## GET /nifi-api/extensions/processor-type-version

### Description
Get a processor type with a specific bundle version for creation.

### Method
GET

### Endpoint
/nifi-api/extensions/processor-type-version

### Parameters
#### Query Parameters
- **processor_type** (string) - Required - The processor type name (e.g., ‘PrepareRegulatoryFile’)
- **version** (string) - Required - The bundle version (e.g., ‘0.0.2-SNAPSHOT’)

### Response
#### Success Response (200)
- **DocumentedTypeDTO** - The processor type with the specified bundle version, suitable for passing to `nipyapi.canvas.create_processor()`.

### Raises
- **ValueError** - If no matching processor type/version found

### Request Example
```python
# Get the v2 processor type
proc_type = nipyapi.extensions.get_processor_type_version(
    'PrepareRegulatoryFile', '0.0.2-SNAPSHOT'
)
# Create processor with that specific version
proc = nipyapi.canvas.create_processor(pg, proc_type, (0,0), 'MyProc')
```
```

--------------------------------

### Get Specific Processor Type Version

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html

Retrieve a processor type that corresponds to a specific bundle version. This is essential for creating processors when multiple versions of the same type are installed, ensuring you use the correct one.

```python
# Get the v2 processor type
proc_type = nipyapi.extensions.get_processor_type_version(
    'PrepareRegulatoryFile', '0.0.2-SNAPSHOT'
)
# Create processor with that specific version
proc = nipyapi.canvas.create_processor(pg, proc_type, (0,0), 'MyProc')
```

--------------------------------

### Build and documentation targets

Source: https://nipyapi.readthedocs.io/en/latest/contributing.html

Commands for building distributions and generating project documentation.

```makefile
make dist             # Build wheel and source distribution
make check-dist       # Validate distribution files
make test-dist        # Test that distribution can be imported
make docs             # Generate Sphinx documentation
```

--------------------------------

### Create Custom NiFi Profiles

Source: https://nipyapi.readthedocs.io/en/latest/migration.html

Copy the example profiles.yml file to your home directory and edit it to configure your specific environments.

```bash
# Copy the example as a starting point
cp examples/profiles.yml ~/.nipyapi/profiles.yml

# Edit for your environments
vim ~/.nipyapi/profiles.yml
```

--------------------------------

### Get Processor Initialization Status

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html

Check the initialization status of a processor, particularly useful for custom or Python processors that may require additional setup time. Provides details on readiness, property loading, and validation status.

```python
status = nipyapi.extensions.get_processor_init_status(proc)
if status['is_ready']:
    print("Processor ready for configuration")
```

--------------------------------

### Install in a Virtual Environment

Source: https://nipyapi.readthedocs.io/en/latest/installation.html

Recommended approach to isolate dependencies and avoid conflicts.

```bash
$ python -m venv nipyapi-env
$ source nipyapi-env/bin/activate  # On Windows: nipyapi-env\Scripts\activate
$ pip install nipyapi
```

--------------------------------

### List Available Processor Bundle Versions

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/extensions.html

Get a list of all installed bundle versions for a given processor type. This is useful when multiple versions of the same processor are available from different NARs. Each version includes bundle information, the full type name, and a description.

```python
versions = nipyapi.extensions.get_processor_bundle_versions(
    'PrepareRegulatoryFile'
)
for v in versions:
    print(f"{v['bundle'].version}")
```

--------------------------------

### Install NiPyAPI from Source Archive

Source: https://nipyapi.readthedocs.io/en/latest/_sources/installation.rst.txt

Install NiPyAPI by downloading and installing from a source archive. This method involves downloading the tarball, extracting it, and then installing using pip.

```console
curl -OL https://github.com/Chaffelson/nipyapi/tarball/main
tar -xzf main
cd Chaffelson-nipyapi-*
pip install .
```

```console
python -m build
pip install dist/*.whl
```

--------------------------------

### Compare Configuration Complexity

Source: https://nipyapi.readthedocs.io/en/latest/_sources/migration.rst.txt

Demonstration of the reduction in configuration steps from 0.x to 1.x.

```python
# 0.x: Complex multi-step configuration (10+ lines)
import os
import nipyapi

# Step 1: Set global SSL settings
nipyapi.config.global_ssl_verify = True
nipyapi.config.global_ssl_host_check = False  # But this affects BOTH services!
nipyapi.config.disable_insecure_request_warnings = True

# Step 2: Direct configuration (NIFI_CA_CERT no longer supported)
nipyapi.config.nifi_config.ssl_ca_cert = '/path/to/ca.pem'

# Step 4: Manually configure each service endpoint
nipyapi.utils.set_endpoint("https://localhost:8443/nifi-api", ssl=True, login=True)
# Registry SSL settings? More complex configuration needed...

# vs.

# 1.x: Simple declarative configuration (1 line)
```

--------------------------------

### Start Flow

Source: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/core_modules/ci.html

Starts a process group, enabling controller services and starting processors.

```APIDOC
## POST /nipyapi/ci/start_flow

### Description
Start a process group (enable controllers, start processors).

### Method
POST

### Endpoint
/nipyapi/ci/start_flow

### Parameters
#### Query Parameters
- **process_group_id** (string) - Required - ID of the process group. Env: NIFI_PROCESS_GROUP_ID
- **enable_controllers** (boolean) - Optional - Enable controller services first (default: True).

### Response
#### Success Response (200)
- **started** (boolean) - Indicates if the flow was started.
- **process_group_name** (string) - The name of the process group.

#### Error Response (400)
- **error** (string) - Missing required parameters.
#### Error Response (500)
- **error** (string) - NiFi API errors.

### Request Example
```
nipyapi ci start_flow --process_group_id PG_ID
nipyapi ci start_flow --process_group_id PG_ID --enable_controllers false
```
```

--------------------------------

### Output Format Configuration

Source: https://nipyapi.readthedocs.io/en/latest/_sources/cli.rst.txt

Examples of how the CLI formats output for different environments.

```console
nipyapi system get_nifi_version_info
# Output: {"niFi_version": "2.0.0", ...}
```

```console
# In GitHub Actions workflow
nipyapi ci deploy_flow ... >> $GITHUB_OUTPUT
# Output: process-group-id=abc-123
```

```console
# In GitLab CI job
nipyapi ci deploy_flow ... > deploy.env
# Output: PROCESS_GROUP_ID=abc-123
```

```console
export NIFI_OUTPUT_FORMAT=json    # JSON (default)
export NIFI_OUTPUT_FORMAT=github  # GitHub Actions format
export NIFI_OUTPUT_FORMAT=dotenv  # GitLab dotenv format
```

--------------------------------

### Test Installation in Clean Environment

Source: https://nipyapi.readthedocs.io/en/latest/devnotes.html

Verify the installation of the released package in a clean environment by installing it using pip and checking the imported version.

```python
pip install nipyapi=={version}
python -c "import nipyapi; print(nipyapi.__version__)"
```

--------------------------------

### Install NiPyAPI with Development and Documentation Dependencies

Source: https://nipyapi.readthedocs.io/en/latest/installation.html

Installs NiPyAPI along with optional development and documentation dependencies. This command is useful for setting up a complete development environment.

```bash
pip install -e ".[dev,docs]"

```

--------------------------------

### Install for Development

Source: https://nipyapi.readthedocs.io/en/latest/installation.html

Install the package in editable mode with development and documentation dependencies.

```bash
# Clone the repository
$ git clone https://github.com/Chaffelson/nipyapi.git
$ cd nipyapi

# Install in development mode with all dependencies
$ pip install -e ".[dev,docs]"
```
Tokens:
Raw