CLOE - Project Accelerator Toolbox

https://white-rock-0cabbc003.1.azurestaticapps.net
Admin
CLOE is an accelerator toolbox offering automation solutions for all parts of data warehousing...
Tokens:443,917
Snippets:1,140
Trust Score:5.1
Update:1 year ago
Show doc for...
Tokens:
Raw
### User Guide Directory Structure

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/Documentation

Shows the expected directory layout for the User Guide section, including 'how-to' guides, installation instructions, and tutorials.

```tree
. 
├── how-to
│   └── how_to_use_submodule_1.md
│   └── ...
├── installation.md
└── tutorials
    └── create-a-simple-etl-pipeline.md
    └── ...
```

--------------------------------

### Implement Logging Guide

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/How-To's/how_to_implement_logging

This section provides guidance on implementing logging within the Nessy project. It covers the configuration of loggers and offers practical examples for developers to follow.

```APIDOC
Implement Logging:
  Purpose: To guide developers on setting up and utilizing logging mechanisms within the Nessy project.
  Sections:
    - Configuration of Loggers: Details on how to configure logging levels, handlers, and formatters.
    - Examples: Demonstrates practical usage scenarios for logging within the application.
  Dependencies: Standard Python logging module.
  Usage:
    1. Configure loggers according to project standards.
    2. Use logger instances to record events, errors, and debug information.
  Limitations:
    - Specific logger configurations might vary based on the environment (e.g., development vs. production).
    - Ensure log levels are set appropriately to avoid excessive log output.
```

--------------------------------

### Implement Logging Guide

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/How-To%27s/how_to_implement_logging

This section provides guidance on implementing logging within the Nessy project. It covers the configuration of loggers and offers practical examples for developers to follow.

```APIDOC
Implement Logging:
  Purpose: To guide developers on setting up and utilizing logging mechanisms within the Nessy project.
  Sections:
    - Configuration of Loggers: Details on how to configure logging levels, handlers, and formatters.
    - Examples: Demonstrates practical usage scenarios for logging within the application.
  Dependencies: Standard Python logging module.
  Usage:
    1. Configure loggers according to project standards.
    2. Use logger instances to record events, errors, and debug information.
  Limitations:
    - Specific logger configurations might vary based on the environment (e.g., development vs. production).
    - Ensure log levels are set appropriately to avoid excessive log output.
```

--------------------------------

### Local Development Setup with Git and UV

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/contribute/workflow

Steps to clone a repository, create and activate a Python virtual environment using the `uv` package manager, and install project dependencies.

```Shell
git clone <repository-url>
cd <repository-name>
```

```Python
# Create and activate environment with uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
uv pip install -e .
```

--------------------------------

### Databricks Configuration Profile Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/Development-Environment

Example structure for the `.databrickscfg` file used to configure Databricks connection details, including host, authentication type, and cluster ID.

```ini
[DEFAULT]
host      = https://adb-XXXXXXXXXXXXXXXXX.azuredatabricks.net/
auth_type = databricks-cli
cluster_id = XXXX-XXXXXX-XXXXXXX

```

--------------------------------

### Install MkDocs and Material Theme

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/contribute/how_to_document

Installs the necessary packages for MkDocs and the Material theme, which are essential for building static documentation websites with a professional look.

```shell
pip install mkdocs mkdocs-material
```

--------------------------------

### Implement Logging with LoggingMixin in Python

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/How-To%27s/how_to_implement_logging

This example demonstrates how to use the LoggingMixin from the cloe_nessy.logging package to add logging functionality to a new class. It shows the initialization of both a console logger and a tabular logger, along with basic usage examples for each.

```python
from cloe_nessy.logging import LoggingMixin

class MyNewClass(LoggingMixin):
    def __init__(self):
        # This will add a regular logger that prints to the console.
        self._console_logger = self.get_console_logger()

        # This will add a tabular logging, where the log messages
        # persisted in a tabular format. The target can be defined
        # via environment variables or when calling the method.
        self._tabular_logger = self.get_tabular_logger(
            add_log_analytics_logger = True,
            log_type = "MyNewClass"
        )

        # use the loggers as you would use any other logger
        self.console_logger.info("MyNewClass was initialized.")
        # keep in mind that the tabular logger logs in a tabular format
        # the key and value separators can be defined
        self.tabular_logger.info("COL1 : VALUE | COL2 : VALUE2")

```

--------------------------------

### Implement Logging with LoggingMixin in Python

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/How-To's/how_to_implement_logging

This example demonstrates how to use the LoggingMixin from the cloe_nessy.logging package to add logging functionality to a new class. It shows the initialization of both a console logger and a tabular logger, along with basic usage examples for each.

```python
from cloe_nessy.logging import LoggingMixin

class MyNewClass(LoggingMixin):
    def __init__(self):
        # This will add a regular logger that prints to the console.
        self._console_logger = self.get_console_logger()

        # This will add a tabular logging, where the log messages
        # persisted in a tabular format. The target can be defined
        # via environment variables or when calling the method.
        self._tabular_logger = self.get_tabular_logger(
            add_log_analytics_logger = True,
            log_type = "MyNewClass"
        )

        # use the loggers as you would use any other logger
        self.console_logger.info("MyNewClass was initialized.")
        # keep in mind that the tabular logger logs in a tabular format
        # the key and value separators can be defined
        self.tabular_logger.info("COL1 : VALUE | COL2 : VALUE2")

```

--------------------------------

### Read API: Configuration with Method, Timeout, Headers, and Params

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions/read_api

This example extends the basic configuration by including the HTTP method (e.g., GET), a request timeout, custom headers, and query parameters for the API request.

```CONFIG
Read API:
    action: READ_API
    options:
        base_url: https://some_url.com/api/
        endpoint: my/endpoint/
        method: GET
        timeout: 90
        headers:
            key1: value1
            key2: value2
        params:
            key1: value1
            key2: value2

```

--------------------------------

### Databricks Project Configuration (`.databricks`)

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/Development-Environment

Example of the `.databricks` project configuration file, specifying workspace paths for code synchronization and connection details. Essential for preventing sync errors.

```json
{
  "host": "https://adb-XXXXXXXXXXXXXXXXXXX.azuredatabricks.net/",
  "authType": "databricks-cli",
  "databricksPath": "/home/vscode/.vscode-server/extensions/databricks.databricks-1.3.0-linux-x64/bin/databricks",
  "clusterId": "XXXX-XXXXXX-XXXXXXX",
  "workspacePath": "/Users/firstname.lastname@initions-consulting.com/.ide/cloe-nessy-py-f5a0aa5e"
}

```

--------------------------------

### Example Schema Metadata

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/User-Guide/model_metadata

An example demonstrating the structure and values for schema metadata, including catalog, name, and storage path.

```yaml
catalog: "my_catalog"
name: "raw"
storage_path: "abfss://mycontainer@mystorageaccount.blob.core.windows.net/schemas/"
```

--------------------------------

### Install Cloe Nessi via PyPI

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/User-Guide/installation

Installs the 'cloe_nessy' package from the public PyPI repository. This is the recommended method for most users. Ensure Python 3.11 or 3.12 is installed.

```shell
pip install cloe_nessy # specify the version if needed
```

--------------------------------

### Combined Authentication Headers Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions

An example showing the resulting JSON structure of combined headers from different authentication types used in the ReadAPIAction.

```JSON
{
    "header_key_1": "value_from_environment_variable",
    "header_key_2": "value_from_secret",
    "header_key_3": "value_from_secret",
    "Authorization": "Bearer <access_token> (from azure_oauth)",
    "Authorization": "Basic am9obkBleGFtcGxlLmNvbTphYmMxMjM= (from basic)"
}
```

--------------------------------

### Create New MkDocs Project

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/contribute/how_to_document

Initializes a new MkDocs project directory and navigates into it, setting up the basic structure for your documentation site.

```shell
mkdocs new my-project
cd my-project
```

--------------------------------

### Faker Localization Examples

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/faker-integration

Provides examples of generating locale-specific data, such as US phone numbers and German addresses, using the `locale` option in Faker functions.

```YAML
columns:
  # US-specific phone numbers
  - name: "us_phone"
    data_type: "string"
    faker_function: "phone_number"
    faker_options:
      locale: "en_US"

  # German addresses
  - name: "de_address"
    data_type: "string"
    faker_function: "address"
    faker_options:
      locale: "de_DE"
```

--------------------------------

### Basic Usage Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions

Demonstrates the basic configuration for the ReadAPIAction, specifying the action and the base URL and endpoint for the API request.

```YAML
Read API:
    action: READ_API
    options:
        base_url: https://some_url.com/api/
        endpoint: my/endpoint/
```

--------------------------------

### Install WSL Distribution

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/Development-Environment

Installs a Windows Subsystem for Linux distribution, typically Ubuntu, to provide a Linux environment on Windows. This is a prerequisite for using devcontainers effectively on Windows.

```shell
wsl --install Ubuntu
```

--------------------------------

### Basic Usage Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions/read_api

Demonstrates the basic configuration for the ReadAPIAction, specifying the action and the base URL and endpoint for the API request.

```YAML
Read API:
    action: READ_API
    options:
        base_url: https://some_url.com/api/
        endpoint: my/endpoint/
```

--------------------------------

### Python Project Structure Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/contribute/how_to_python

Illustrates a standard project directory layout for Python projects. It includes directories for source code (`src/`), tests (`tests/`), configuration (`pyproject.toml`), documentation (`docs/`), and a README file.

```python
project_name/
│
├── src/
│   └── your_package/
│       ├── __init__.py
│       ├── module1.py
│       └── module2.py
│
├── tests/
│   ├── __init__.py
│   └── test_module1.py
│
├── pyproject.toml
├── README.md
└── docs/
    └── index.md

```

--------------------------------

### Generated Configuration Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/table-discovery

An example of a generated YAML configuration file for synthetic data generation, specifying column details, Faker functions, options, and target write parameters. This file guides the data generation process.

```yaml
name: Employees Data Generation
num_records: 1000
columns:
- name: employee_id
  data_type: string
  nullable: false
  faker_function: uuid4
  faker_options: {}

- name: first_name
  data_type: string
  nullable: false
  faker_function: first_name
  faker_options: {}

- name: last_name
  data_type: string
  nullable: false
  faker_function: last_name
  faker_options: {}

- name: email
  data_type: string
  nullable: true
  faker_function: email
  faker_options:
    domain: company.com

- name: department
  data_type: string
  nullable: true
  faker_function: random_element
  faker_options:
    elements: ['Engineering', 'Sales', 'Marketing', 'HR', 'Finance', 'Operations']

- name: salary
  data_type: double
  nullable: true
  faker_function: pyfloat
  faker_options:
    left_digits: 5
    right_digits: 2
    positive: true
    min_value: 30000
    max_value: 200000

- name: hire_date
  data_type: date
  nullable: true
  faker_function: date_between
  faker_options:
    start_date: -10y
    end_date: today

target:
  catalog: main
  schema: hr_data
  table: employees
  write_mode: overwrite

```

--------------------------------

### Example Table Metadata Configuration

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/User-Guide/model_metadata

A practical example demonstrating how to structure table metadata, including partitioning, clustering, properties, constraints, foreign keys, and column definitions.

```yaml
name: customers
is_external : false
partition_by:
    - "region"
liquid_clustering: true
data_source_format: "delta"
properties:
    "delta.autoOptimize.optimizeWrite": "true"
    "delta.enableDeletionVectors": "true"
constraints:
  realisticBirthConstraint:
      expression: "(birthDate > '1900-01-01')"
      description: "Ensures birthDate is realistic"
foreign_keys:
    - foreign_key_columns:
        - "service_level"
      parent_table: "service_level_lookup"
      parent_columns:
        - "service_level_name"
      foreign_key_option:
        - "MATCH FULL"
columns:
    - name: "customer_id"
      data_type: "long"
      nullable: false
      generated: "GENERATED ALWAYS AS IDENTITY"
    - name: "customer_name"
      data_type: "string"
      nullable: false
      default_value: "Max Mustermann"
business_properties:
  responsible_manager: "Dr. Michaela Mustermensch"

```

--------------------------------

### Example Copier Commands

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-package-template/features

Illustrative commands demonstrating common workflows for using Copier to manage project templates.

```shell
# Create a new project
copier copy git+https://dev.azure.com/initions-consulting/CLOE/_git/cloe-package-template-py new_project

# Update an existing project
cd existing_project

copier update

# Preview changes before applying
copier diff

```

--------------------------------

### CLOE Faker Integration: Custom Choices

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/configuration

Examples of generating data by selecting from a predefined list of elements using Faker's `random_element` and `random_choices` functions.

```yaml
- name: "status"
  data_type: "string"
  faker_function: "random_element"
  faker_options:
    elements: ["active", "inactive", "pending", "suspended"]

- name: "priority"
  data_type: "string"
  faker_function: "random_choices"
  faker_options:
    elements: ["low", "medium", "high", "critical"]
    length: 1             # Return single choice
```

--------------------------------

### Example CLI Output

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/getting-started

Demonstrates the typical console output when the cloe-synthetic-data-generator command is executed, showing configuration details, connection status, and generation success.

```text
Configuration: Users Data Generation
┌─────────────┬────────────────────────────────┐
│ Property    │ Value                          │
├─────────────┼────────────────────────────────┤
│ Target Table│ main.default.users             │
│ Columns     │ 6                              │
│ Records     │ 100                            │
│ Write Mode  │ overwrite                      │
└─────────────┴────────────────────────────────┘

✅ Connected to Databricks!
✅ Completed successfully!

🎉 Successfully generated 100 records and wrote to main.default.users

```

--------------------------------

### Python Basic Logging Setup

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/contribute/how_to_python

Demonstrates setting up basic logging in Python using the built-in `logging` module. It configures the log level and format, and logs an informational message.

```python
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(message)s")
logging.info("This is an info message")
```

--------------------------------

### Open Repository in VSCode (WSL)

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/Developer-Guide/Development-Environment

Navigates to the cloned repository directory within WSL and opens it in VSCode. This action may trigger the installation of the VSCode server in WSL.

```shell
cd path/to/repository && code .
```

--------------------------------

### Install Cloe Nessi from Source

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/User-Guide/installation

Installs the 'cloe_nessy' package directly from a local source directory. This is useful for development or using unreleased versions. Replace `/path/to/nessy` with the actual path to the cloned repository.

```shell
pip install /path/to/nessy # e.g. /Workspace/Repos/user@domain/cloe-nessy-py
```

--------------------------------

### CLOE Faker Integration: Personal Information

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/configuration

Examples of using Faker functions for generating personal information like names, email addresses, and phone numbers within CLOE configurations.

```yaml
# Names
- name: "first_name"
  data_type: "string"
  faker_function: "first_name"

- name: "last_name"
  data_type: "string"
  faker_function: "last_name"

- name: "full_name"
  data_type: "string"
  faker_function: "name"

# Contact Information
- name: "email"
  data_type: "string"
  faker_function: "email"

- name: "phone"
  data_type: "string"
  faker_function: "phone_number"
```

--------------------------------

### WriteFileAction: YAML Configuration Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions

An example configuration in YAML format demonstrating how to set up the WriteFileAction for writing data. It specifies the action type, output path, file format, partitioning columns, write mode, and additional writer options.

```yaml
Write to File:
    action: WRITE_FILE
    options:
        path: "path/to/location"
        format: "parquet"
        partition_cols: ["date"]
        mode: "append"
        is_stream: False
        options:
            mergeSchema: "true"
```

--------------------------------

### CLOE Faker Integration: Business Data

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/configuration

Configuration examples for generating business-related data using Faker functions, including company names, job titles, and custom department choices.

```yaml
- name: "company_name"
  data_type: "string"
  faker_function: "company"

- name: "job_title"
  data_type: "string"
  faker_function: "job"

- name: "department"
  data_type: "string"
  faker_function: "random_element"
  faker_options:
    elements: ["Engineering", "Sales", "Marketing", "HR"]
```

--------------------------------

### Read Excel Action Configuration Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions/read_excel

An example configuration in YAML format demonstrating how to use the READ_EXCEL action, specifying the file path, columns to use, and referencing further options for the underlying reader.

```yaml
Read Excel Table:
    action: READ_EXCEL
    options:
        file: excel_file_folder/excel_files_june/interesting_excel_file.xlsx
        usecols:
            - key_column
            - interesting_column
        options: <options for the ExcelDataFrameReader read method>
```

--------------------------------

### CLOE Faker Integration: Dates and Times

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/configuration

Examples for generating date and time data using Faker functions, including date ranges for birth dates and time ranges for creation timestamps.

```yaml
- name: "birth_date"
  data_type: "date"
  faker_function: "date_between"
  faker_options:
    start_date: "-65y"    # 65 years ago
    end_date: "-18y"      # 18 years ago

- name: "created_at"
  data_type: "timestamp"
  faker_function: "date_time_between"
  faker_options:
    start_date: "-1y"     # 1 year ago
    end_date: "now"       # Current time
```

--------------------------------

### pre-commit Git Hook Management

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-package-template/tools

pre-commit is a framework for managing pre-commit hooks to improve commit quality. It requires installation and hook setup, after which hooks run automatically on commit or can be run manually.

```Shell
pip install pre-commit
pre-commit install
pre-commit run --all-files
git commit --no-verify
```

--------------------------------

### Read API: Basic Configuration

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions/read_api

This configuration demonstrates the basic setup for the ReadAPIAction, specifying the action type, base URL, and the API endpoint to fetch data from.

```CONFIG
Read API:
    action: READ_API
    options:
        base_url: https://some_url.com/api/
        endpoint: my/endpoint/

```

--------------------------------

### UnityCatalogAdapter Initialization

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/models/adapter

Initializes the UnityCatalogAdapter class, setting up the Spark session, console logger, and retrieving available catalogs.

```APIDOC
__init__(spark: SparkSession | None = None)
  Initializes the UnityCatalogAdapter class.

  Parameters:
    spark: An optional SparkSession object. If not provided, it defaults to the Spark session obtained from SessionManager.get_spark_session().

  Source:
    src/cloe_nessy/models/adapter/unity_catalog_adapter.py
```

--------------------------------

### Start Small: Single Schema Discovery

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-synthetic-data-generator-py/table-discovery

Best practice to begin discovery with a single, small schema to understand the output format and tool behavior before scaling to larger datasets. This example shows discovery for a test schema with a limited number of records.

```bash
cloe-synthetic-data-generator discover \
  --catalog main \
  --schema test_schema \
  --num-records 100

```

--------------------------------

### Terraform Deployment Workflow

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/terraform-setup/terraform-setup

Provides the sequence of Terraform commands required to initialize the project, create a deployment plan, and apply the plan to deploy resources. It also mentions handling access tokens if not set via Azure CLI.

```terraform
cd terraform/setup
terraform init
terraform plan -out tfplan
terraform apply tfplan
```

--------------------------------

### SQL Server: Get Files from FileCatalog

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/cloe-etl-adf/v1/setup/database

This SQL Server stored procedure retrieves file details from the `cloe_dwh.FileCatalog` table. It allows filtering by file path and filename patterns and marks the retrieved files as locked (FileStatus = 1) to prevent concurrent processing. It returns the top matching file record.

```SQLSERVER
CREATE PROCEDURE [cloe_dwh].[spGetFilesFromFileCatalog]
    @pFilepathPattern NVARCHAR(255),
    @pFilenamePattern NVARCHAR(255)
 AS
BEGIN
    DECLARE @lock_id NVARCHAR(255)
    SET @lock_id = NEWID()
    UPDATE [cloe_dwh].[FileCatalog]
    SET [FileStatus] = 1, lock_uuid = @lock_id
    WHERE [FilePath] LIKE @pFilepathPattern AND [FileName] LIKE @pFilenamePattern

    SELECT
        TOP 1
        [ID],
        [FilePath],
        [FileName]
    FROM [cloe_dwh].[FileCatalog]
    WHERE FileStatus = 1 AND lock_uuid = @lock_id
    ORDER BY [ID]
END
```

--------------------------------

### Write Table to Catalog Example

Source: https://white-rock-0cabbc003.1.azurestaticapps.net/tool_docs/nessy/reference/cloe_nessy/pipeline/actions

Example configuration in YAML format for writing a table to a catalog, specifying the action, table identifier, write mode, partitioning, and additional options.

```yaml
Write Table to Catalog:
    action: WRITE_CATALOG_TABLE
    options:
        table_identifier: my_catalog.business_schema.sales_table
        mode: append
        partition_by: day
        options:
            mergeSchema: true
```
Tokens:
Raw