### Development Setup with Makefile

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Clones the Wurzel repository and installs all dependencies, including core libraries, optional features, development tools, and pre-commit hooks, using the provided Makefile.

```bash
# Clone the repository
git clone https://github.com/telekom/wurzel.git
cd wurzel

# Install all dependencies including development tools
make install
```

--------------------------------

### Manual Development Environment Setup

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Manually sets up a development environment by creating a virtual environment, activating it, installing Wurzel in editable mode with all extras, installing direct dependencies from a requirements file, and setting up pre-commit hooks.

```bash
# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in development mode with all extras
pip install -e .[all,lint,test,docs]

# Install direct dependencies (see below)
pip install -r DIRECT_REQUIREMENTS.txt

# Set up pre-commit hooks
pre-commit install
```

--------------------------------

### Verify Basic Wurzel Installation

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Checks if the Wurzel library is installed correctly and if the command-line interface (CLI) is accessible.

```python
import wurzel
print('Wurzel installed successfully')
```

```bash
wurzel --help
```

--------------------------------

### Install Optional Dependencies with Pip

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Installs optional features of Wurzel by specifying extras during pip installation. Examples include vector database support (Qdrant, Milvus), document processing (Docling), backend support (Argo), and development tools.

```bash
# For Qdrant vector database
pip install wurzel[qdrant]

# For Milvus vector database
pip install wurzel[milvus]

# For PDF document processing with Docling
pip install wurzel[docling]

# For Argo Workflows backend
pip install wurzel[argo]

# For linting and code quality tools
pip install wurzel[lint]

# For testing framework and tools
pip install wurzel[test]

# For documentation generation
pip install wurzel[docs]

# Install all optional dependencies
pip install wurzel[all]

# Don't forget the direct dependencies!
pip install -r DIRECT_REQUIREMENTS.txt
```

--------------------------------

### Set Up Virtual Environment and Install Wurzel (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Creates a Python virtual environment named '.venv', activates it, and installs the 'wurzel' package. This isolates project dependencies and prevents conflicts.

```bash
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install wurzel
```

--------------------------------

### Run Local and Kubernetes Wurzel Deployments

Source: https://github.com/telekom/wurzel/blob/main/examples/pipeline/README.md

Commands to build and run a local Docker container for the Wurzel pipeline, and instructions for deploying a multi-tenant setup in Kubernetes using Helm and CronJobs.

```bash
docker build -t my/wurzel examples/pipeline/.
docker run -it my/wurzel
```

--------------------------------

### Run ETL Steps with Different Configurations (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/cli.md

Examples demonstrating how to run ETL steps using the `wurzel run` command. Covers basic usage, using custom executors, and specifying multiple input directories.

```bash
# Basic usage
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep \
    --inputs ./markdown-files \
    --output ./processed-output

# With custom executor
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep \
    --inputs ./markdown-files \
    --output ./processed-output \
    --executor PrometheusStepExecutor

# Multiple input folders
wurzel run wurzel.steps.splitter.SimpleSplitterStep \
    --inputs ./docs \
    --inputs ./markdown \
    --inputs ./pdfs \
    --output ./split-output
```

--------------------------------

### Install and Run Basic Commands (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/cli.md

Basic commands for installing the wurzel package and running/inspecting/generating pipelines. Uses pip for installation and wurzel CLI for pipeline operations.

```bash
# Install wurzel
pip install wurzel

# Run a step
wurzel run wurzel.steps.manual_markdown.ManualMarkdownStep --inputs ./data --output ./out

# Inspect a step
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep

# Generate a pipeline
wurzel generate examples.pipeline.pipelinedemo.pipeline
```

--------------------------------

### Install wurzel Library using pip

Source: https://github.com/telekom/wurzel/blob/main/README.md

This command installs the wurzel Python library. It's a prerequisite for using wurzel's functionalities for ETL and RAG systems.

```bash
pip install wurzel
```

--------------------------------

### Install spaCy German Model

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Manually installs the German spaCy model required for semantic text splitting functionality. This is necessary because the model is hosted externally on GitHub releases.

```bash
pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.7.0/de_core_news_sm-3.7.0-py3-none-any.whl
```

--------------------------------

### Docker Image Management

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Pulls the latest Wurzel Docker image from GitHub Container Registry or builds the image locally from the Dockerfile.

```bash
# Pull the latest image
docker pull ghcr.io/telekom/wurzel:latest

# Or build locally
docker build -t wurzel .
```

--------------------------------

### Step Auto-Discovery via TAB Completion (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/cli.md

Illustrates how the Wurzel CLI uses TAB completion to discover available ETL steps. Shows examples for discovering built-in steps, custom steps, and navigating the step hierarchy.

```bash
wurzel run <TAB>                    # Shows all available steps
wurzel run wurzel.steps.<TAB>       # Shows wurzel built-in steps
wurzel run mysteps.<TAB>            # Shows your custom steps
```

--------------------------------

### Install Wurzel with Optional Dependencies (Bash)

Source: https://context7.com/telekom/wurzel/llms.txt

Installs the Wurzel library using pip, with options to include support for specific vector databases (Qdrant, Milvus), text splitting, embedding generation, Argo Workflows, development tools, or all features.

```bash
pip install wurzel
pip install wurzel[qdrant]
pip install wurzel[milvus]
pip install wurzel[splitter]
pip install wurzel[embedding]
pip install wurzel[argo]
pip install wurzel[all]
pip install wurzel[dev]
```

--------------------------------

### Dockerfile for Wurzel Pipeline

Source: https://github.com/telekom/wurzel/blob/main/examples/pipeline/README.md

A Dockerfile to build a custom Wurzel image. It installs dependencies from requirements.txt, copies the pipeline definition, and sets the environment variable for the pipeline to run.

```dockerfile
FROM ghcr.io/telekom/wurzel:latest
# if your steps are located in other dependencies
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
# add your pipeline definition
COPY pipeline.py .
# addressing the last step
ENV WURZEL_PIPELINE="pipeline:pipeline"
```

--------------------------------

### Chain Pipeline Steps with WZ Utility (Python)

Source: https://context7.com/telekom/wurzel/llms.txt

Illustrates how to create and connect pipeline steps using the WZ utility and the `>>` operator for defining data flow. This example shows a linear RAG pipeline and a branching pipeline with parallel processing paths. Wurzel automatically resolves the execution order based on the defined dependencies.

```python
from wurzel.utils import WZ
from wurzel.steps.manual_markdown import ManualMarkdownStep
from wurzel.steps.splitter import SimpleSplitterStep
from wurzel.steps.embedding import EmbeddingStep
from wurzel.steps.qdrant import QdrantConnectorStep
from wurzel.step import TypedStep

def create_rag_pipeline() -> TypedStep:
    """Define a complete RAG pipeline from documents to vector database."""

    # Step 1: Load markdown documents
    source = WZ(ManualMarkdownStep)

    # Step 2: Split documents into chunks
    splitter = WZ(SimpleSplitterStep)

    # Step 3: Generate embeddings
    embedding = WZ(EmbeddingStep)

    # Step 4: Store in Qdrant vector database
    storage = WZ(QdrantConnectorStep)

    # Chain steps together (defines data flow)
    source >> splitter >> embedding >> storage

    # Return final step (Wurzel resolves all dependencies)
    return storage

# For branching pipelines (one source, multiple destinations)
def create_branching_pipeline() -> TypedStep:
    """Pipeline with parallel processing branches."""
    source = WZ(ManualMarkdownStep)

    # Branch 1: Embed and store
    embedding = WZ(EmbeddingStep)
    vector_db = WZ(QdrantConnectorStep)

    # Branch 2: Different processing path
    from myproject.steps import TextAnalysisStep, AnalyticsStorageStep
    analyzer = WZ(TextAnalysisStep)
    analytics_db = WZ(AnalyticsStorageStep)

    # Source feeds both branches
    source >> embedding >> vector_db
    source >> analyzer >> analytics_db

    return vector_db

```

--------------------------------

### Upgrade Pip (Python)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Upgrades the pip package installer to the latest version. Keeping pip updated can resolve issues related to package installation and dependency management.

```python
python -m pip install --upgrade pip
```

--------------------------------

### Reinstall spaCy Model (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Uninstalls the 'de-core-news-sm' spaCy model and then installs it from a specific wheel file hosted on GitHub. This is useful for resolving corrupted model installations.

```bash
pip uninstall de-core-news-sm
pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.7.0/de_core_news_sm-3.7.0-py3-none-any.whl
```

--------------------------------

### Instantiate a Step using WZ Utility

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/building-pipelines.md

Demonstrates how to create an instance of a processing step (ManualMarkdownStep) using the WZ() utility function in Python. This is a fundamental step for building pipelines.

```python
from wurzel.utils import WZ
from wurzel.steps.manual_markdown import ManualMarkdownStep

# Create a step instance
markdown_step = WZ(ManualMarkdownStep)
```

--------------------------------

### Verify spaCy Model Installation (Python)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Checks if the 'de_core_news_sm' spaCy model is installed and loadable in the current Python environment. This helps diagnose issues with NLP model availability.

```python
python -c "import spacy; nlp = spacy.load('de_core_news_sm'); print('Model loaded successfully')"
```

--------------------------------

### Build and Run Docker Compose for Wurzel Pipeline

Source: https://github.com/telekom/wurzel/blob/main/examples/pipeline/README.md

Commands to build and run the Wurzel pipeline using Docker Compose. This includes options for a standard build and a manual build with a specified platform.

```bash
docker-compose up wurzel-pipeline
```

```bash
docker build --platform linux/amd64 -f ../../Dockerfile -t test_wurzel ../../
docker-compose up wurzel-pipeline
```

--------------------------------

### Python Nested Configuration Settings

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/creating-steps.md

Demonstrates how to define complex, nested configuration structures for steps using `wuzel.meta.Settings`. It includes settings for database connections and processing parameters, which can be overridden via environment variables using double underscores (e.g., `DATABASE__HOST`, `PROCESSING__BATCH_SIZE`).

```python
class DatabaseConfig(Settings):
    """Database connection configuration."""
    host: str = "localhost"
    port: int = 5432
    database: str = "wurzel"
    username: str = "user"
    password: str = "password"

class ProcessingConfig(Settings):
    """Processing configuration."""
    batch_size: int = 100
    parallel_workers: int = 4

class ComplexStepSettings(Settings):
    """Complex step configuration."""
    database: DatabaseConfig = DatabaseConfig()
    processing: ProcessingConfig = ProcessingConfig()
    debug_mode: bool = False

# Environment variables:
# DATABASE__HOST=production-db.example.com
# DATABASE__PORT=5433
# PROCESSING__BATCH_SIZE=200
# DEBUG_MODE=true

```

--------------------------------

### Verify spaCy Model Loading

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Tests whether the German spaCy model ('de_core_news_sm') can be successfully loaded, confirming its installation for semantic text splitting.

```python
import spacy
nlp = spacy.load('de_core_news_sm')
print('spaCy model loaded successfully')
```

--------------------------------

### Run Test Suite (Makefile)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Executes the test suite for the Wurzel project using the Makefile. This is the primary method for running tests.

```makefile
make test
```

--------------------------------

### ManualMarkdownStep: Loading Markdown Files

Source: https://context7.com/telekom/wurzel/llms.txt

Details the ManualMarkdownStep for loading markdown documents from a directory. It explains configuration via environment variables and demonstrates direct execution and integration into a pipeline using WZ.

```python
import os
from wurzel.steps.manual_markdown import ManualMarkdownStep
from wurzel.utils import WZ

# Configure via environment variables
os.environ["MANUALMARKDOWNSETTINGS__FOLDER_PATH"] = "./docs"

# Use in pipeline
source = WZ(ManualMarkdownStep)

# Or execute directly
step = ManualMarkdownStep()
documents = step.run(None)

for doc in documents:
    print(f"Loaded: {doc.url}")
    print(f"Keywords: {doc.keywords}")
    print(f"Content length: {len(doc.md)} chars")
```

--------------------------------

### Run Wurzel Container

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Runs the Wurzel Docker container with specified environment variables for Git configuration, DVC paths, and pipeline execution.

```bash
docker run \
  -e GIT_USER=wurzel-demo \
  -e GIT_MAIL=demo@example.com \
  -e DVC_DATA_PATH=/usr/app/data \
  -e DVC_FILE=/usr/app/dvc.yaml \
  -e WURZEL_PIPELINE=pipelinedemo:pipeline \
  ghcr.io/telekom/wurzel:latest
```

--------------------------------

### Run Test Suite (Pytest)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/installation.md

Manually runs the test suite using Python's Pytest module. This offers more granular control over test execution.

```python
python -m pytest
```

--------------------------------

### Basic Data Source Step: Loading Markdown Files (Python)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/creating-steps.md

Implements a basic data source step that reads Markdown files from a specified directory. It uses `glob` to find files matching a pattern and returns them as `MarkdownDataContract` objects. Configuration is handled via a `MySettings` Pydantic model.

```python
from typing import Any
from wurzel.step import TypedStep
from wurzel.datacontract import MarkdownDataContract
from wurzel.meta.settings import Settings

class MySettings(Settings):
    """Configuration for MyDatasourceStep."""
    data_path: str = "./data"
    file_pattern: str = "*.md"

class MyDatasourceStep(TypedStep[MySettings, None, list[MarkdownDataContract]]):
    """Data source step for loading Markdown files from a configurable path."""

    def __init__(self):
        """Initialize the data source."""
        self.settings = MySettings()
        # Setup logic here (validate paths, check permissions, etc.)

    def run(self, inpt: None) -> list[MarkdownDataContract]:
        """Load and return markdown documents."""
        import glob
        import os

        pattern = os.path.join(self.settings.data_path, self.settings.file_pattern)
        files = glob.glob(pattern)

        documents = []
        for file_path in files:
            with open(file_path, 'r', encoding='utf-8') as f:
                content = f.read()

            doc = MarkdownDataContract(
                content=content,
                source=file_path,
                metadata={"file_path": file_path}
            )
            documents.append(doc)

        return documents
```

--------------------------------

### Execute Pipeline Steps Programmatically

Source: https://context7.com/telekom/wurzel/llms.txt

Demonstrates how to execute a single pipeline step or chain multiple steps programmatically using BaseStepExecutor. It shows input and output handling for steps like ManualMarkdownStep and SimpleSplitterStep.

```python
from pathlib import Path
from wurzel.steps.manual_markdown import ManualMarkdownStep
from wurzel.steps.splitter import SimpleSplitterStep
from wurzel.executor import BaseStepExecutor

# Execute a single step
with BaseStepExecutor() as executor:
    result = executor(
        ManualMarkdownStep,
        inputs={Path("./input-docs")},
        output=Path("./output")
    )
    print(f"Processed {len(result)} documents")

# Chain multiple steps programmatically
input_path = Path("./markdown-files")
intermediate_path = Path("./split-output")
final_path = Path("./embedded-output")

with BaseStepExecutor() as executor:
    # Step 1: Load markdown
    docs = executor(ManualMarkdownStep, {input_path}, intermediate_path)

    # Step 2: Split into chunks
    chunks = executor(SimpleSplitterStep, {intermediate_path}, final_path)

    print(f"Split {len(docs)} documents into {len(chunks)} chunks")
```

--------------------------------

### Install Wurzel with Argo Support

Source: https://github.com/telekom/wurzel/blob/main/docs/backends/argoworkflows.md

Installs the Wurzel package with the necessary dependencies for Argo Workflows backend support. This command requires pip to be installed.

```bash
pip install wurzel[argo]
```

--------------------------------

### Define a Basic Wurzel Pipeline

Source: https://github.com/telekom/wurzel/blob/main/examples/pipeline/README.md

Python code defining a simple Wurzel pipeline with steps for manual markdown loading, embedding, and connecting to Qdrant. It chains these steps together.

```python
from wurzel.steps.embedding import EmbeddingStep
from wurzel.steps.manual_markdown import ManualMarkdownStep
from wurzel.steps.qdrant.step import QdrantConnectorStep
from wurzel.utils import WZ

source = WZ(ManualMarkdownStep)
embedding = WZ(EmbeddingStep)
step = WZ(QdrantConnectorStep)

source >> embedding >> step
pipeline = step
```

--------------------------------

### Monitor and Inspect Argo Workflows

Source: https://github.com/telekom/wurzel/blob/main/docs/backends/argoworkflows.md

Commands to monitor the execution of Argo Workflows within a Kubernetes cluster. These commands require the Argo CLI to be installed and configured.

```bash
# Monitor workflow executions
argo list

# Check workflow logs
argo logs <workflow-name>

# Get workflow status
argo get <workflow-name>
```

--------------------------------

### Database Data Source Step: Loading Documents from SQLite (Python)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/creating-steps.md

Creates a data source step to load documents from a SQLite database. It connects to a specified database file, queries a table based on a configurable query, and returns the results as `MarkdownDataContract` objects. The database connection is closed in the `finalize` method.

```python
import sqlite3
from wurzel.step import TypedStep
from wurzel.datacontract import MarkdownDataContract
from wurzel.meta.settings import Settings

class DatabaseSettings(Settings):
    """Database connection settings."""
    db_path: str = "data.db"
    table_name: str = "documents"
    query: str = "SELECT content, source, metadata FROM documents"

class DatabaseSourceStep(TypedStep[DatabaseSettings, None, list[MarkdownDataContract]]):
    """Load documents from a SQLite database."""

    def __init__(self):
        """Initialize database connection."""
        self.settings = DatabaseSettings()
        self.connection = sqlite3.connect(self.settings.db_path)
        self.connection.row_factory = sqlite3.Row  # Enable column access by name

    def run(self, inpt: None) -> list[MarkdownDataContract]:
        """Query database and return documents."""
        cursor = self.connection.cursor()
        cursor.execute(self.settings.query)

        documents = []
        for row in cursor.fetchall():
            doc = MarkdownDataContract(
                content=row['content'],
                source=row['source'],
                metadata=eval(row['metadata']) if row['metadata'] else {}
            )
            documents.append(doc)

        return documents

    def finalize(self) -> None:
        """Close database connection."""
        if self.connection:
            self.connection.close()
```

--------------------------------

### Python Pipeline Composition Example

Source: https://github.com/telekom/wurzel/blob/main/examples/pipeline/demo-data/architecture.md

Demonstrates how to compose processing steps in a Wurzel pipeline using the '>>' operator. This creates a directed acyclic graph (DAG) for efficient execution by DVC.

```python
source >> processor >> sink
```

--------------------------------

### Inspect ETL Steps and Generate Environment Files (Bash)

Source: https://github.com/telekom/wurzel/blob/main/docs/developer-guide/cli.md

Commands for inspecting ETL steps with `wurzel inspect`. Shows basic inspection and how to generate an environment file for a step.

```bash
# Basic inspection
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep

# Generate environment file
wurzel inspect wurzel.steps.manual_markdown.ManualMarkdownStep --gen-env
```