### Install Dependencies with Make

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Use this command to set up or update the project's virtual environment and install all necessary dependencies. Run this after manually editing pyproject.toml.

```shell
make install
```

--------------------------------

### Install SDK using uv

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Installs the Splunk SDK using the 'uv' package manager. Ensure 'uv' is initialized and the SDK is added before syncing.

```bash
uv init
uv add splunk-sdk
uv sync
```

--------------------------------

### Package Splunk App

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

An example workflow for packaging a Splunk App, including installing dependencies to a target directory and creating a tarball. Ensure the `--platform` matches your Splunk environment.

```bash
python3 -m pip install . \
  --target bin/lib/ \
  # Needs to match the platform Splunk is built and
  # ran on, NOT the one you're writing your App on
  --platform manylinux2014_aarch64 \
  --only-binary=:all:

gtar --transform='s,^,<your_app>/,' \
  --exclude="__pycache__" \
  -czf dist/<your_app>.tgz \
  bin default
```

--------------------------------

### Install SDK using pip

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Installs the Splunk SDK using pip within a Python virtual environment. This is an alternative to using 'uv'.

```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install splunk-sdk
```

--------------------------------

### Start Splunk Docker Container

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

This command starts a Splunk Docker container and waits until it reaches an operational state. Ensure you have docker or podman installed.

```sh
# This command starts a Splunk Docker container
# and waits until it reaches an operational state.
SPLUNK_VERSION=latest make docker-start
```

--------------------------------

### Example AI Modular Input Data

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/examples/ai_modinput_app/README.md

This is an example of the structured data generated by the AI modular input app, showing weather information for a specific date.

```text
{ 
date: 2012-01-04
human_readable: On January 4, 2012, it was rainy with 20.3 mm of precipitation, temperatures ranged from 5.6°C to 12.2°C, and there was a light wind of 4.7 m/s.
It was probably not a great day to go outside for most people, due to the rainy weather.
precipitation: 20.3
temp_max: 12.2
temp_min: 5.6
weather: rain
wind: 4.7
}
```

--------------------------------

### Start Splunk Test Container

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Spin up the Docker container for Splunk, which is necessary for running integration and system tests. Ensure the instance is live before proceeding with tests.

```shell
make docker-start
```

--------------------------------

### Class-based Middleware Implementation

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Implement custom middleware logic by extending AgentMiddleware. This example shows how to intercept agent, model, tool, and subagent calls.

```python
from typing import Any, override
from splunklib.ai.messages import SubagentTextResult, ToolResult
from splunklib.ai.middleware import (
    AgentMiddleware,
    AgentMiddlewareHandler,
    AgentRequest,
    ModelMiddlewareHandler,
    ModelRequest,
    ModelResponse,
    SubagentMiddlewareHandler,
    SubagentRequest,
    SubagentResponse,
    ToolMiddlewareHandler,
    ToolRequest,
    ToolResponse,
)
from splunklib.ai.messages import AgentResponse, ToolCall


class ExampleMiddleware(AgentMiddleware):
    @override
    async def agent_middleware(
        self, request: AgentRequest, handler: AgentMiddlewareHandler
    ) -> AgentResponse[Any | None]:
        # Keep retrying until the agent makes at least one tool call.
        resp = await handler(request)
        while not any(m for m in resp.messages if isinstance(m, ToolCall)):
            resp = await handler(request)
        return resp

    @override
    async def model_middleware(
        self, request: ModelRequest, handler: ModelMiddlewareHandler
    ) -> ModelResponse:
        return await handler(
            ModelRequest(
                system_message=request.system_message.replace(
                    "SECRET", "[REDACTED]"
                ),
                state=request.state,
            )
        )

    @override
    async def tool_middleware(
        self, request: ToolRequest, handler: ToolMiddlewareHandler
    ) -> ToolResponse:
        if request.call.name == "temperature":
            return ToolResponse(result=ToolResult(content="25.0", structured_content=None))
        return await handler(request)

    @override
    async def subagent_middleware(
        self, request: SubagentRequest, handler: SubagentMiddlewareHandler
    ) -> SubagentResponse:
        if request.call.name == "SummaryAgent":
            return SubagentResponse(result=SubagentTextResult(content="Executive summary: no critical incidents detected."))
        return await handler(request)
```

--------------------------------

### Log Model Call Steps with a Hook

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Register a hook to log the number of messages before each model call. This example demonstrates using the `before_model` hook to inspect the agent's state.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.hooks import before_model
from splunklib.ai.middleware import ModelRequest
from splunklib.client import connect

import logging

logger = logging.getLogger(__name__)

model = OpenAIModel(...)
service = connect(...)

@before_model
def log_steps(req: ModelRequest) -> None:
    logger.debug(f"Steps: {len(req.state.messages)}")


async with Agent(
    model=model,
    service=service,
    system_prompt="...",
    middleware=[log_steps],
) as agent: ...
```

--------------------------------

### Retry Pattern for Tool Failures

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Implement bounded retries for transient tool failures using the @tool_middleware decorator. This example retries up to 3 times.

```python
from splunklib.ai.middleware import (
    tool_middleware,
    ToolMiddlewareHandler,
    ToolRequest,
    ToolResponse,
)


class RetryableToolError(Exception): pass


@tool_middleware
async def retry_transient_tool_failures(
    request: ToolRequest, handler: ToolMiddlewareHandler
) -> ToolResponse:
    last_error: Exception | None = None
    for _ in range(3):
        try:
            return await handler(request)

```

--------------------------------

### Define Subagent with Input and Output Schemas

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Define a subagent with Pydantic input and output schemas. This constrains inputs and guides the supervisor agent. Use `invoke_with_data` for subagents with input schemas to separate instructions from data and mitigate prompt injection risks.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.client import connect
from pydantic import BaseModel

model = OpenAIModel(...)
service = connect(...)


class Input(BaseModel):
    ... 

class Output(BaseModel):
    ... 

async with Agent(
        model=model,
        service=service,
        system_prompt="..." ,
        name="...",
        description="...",
        input_schema=Input,
        output_schema=Output,
    ) as subagent:
    async with Agent(
        model=model,
        service=service,
        system_prompt="...",
        agents=[subagent],
    ) as agent:
        await agent.invoke(...)
```

--------------------------------

### Implementing Injection Guard Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

An example of custom middleware using `detect_injection` to scan for common injection patterns and raise an error if potential injection is found. This middleware can be applied to an agent to enforce a consistent policy across all `invoke()` calls.

```python
from typing import Any
from splunklib.ai import Agent, OpenAIModel, detect_injection, truncate_input
from splunklib.ai.middleware import (
    agent_middleware,
    AgentMiddlewareHandler,
    AgentRequest,
)
from splunklib.ai.messages import AgentResponse, HumanMessage

@agent_middleware
async def injection_guard(
    request: AgentRequest, handler: AgentMiddlewareHandler
) -> AgentResponse[Any | None]:
    for msg in request.messages:
        if isinstance(msg, HumanMessage) and detect_injection(msg.content):
            raise ValueError("Potential prompt injection detected in input.")
    return await handler(request)

async with Agent(
    model=model,
    service=service,
    system_prompt="...",
    middleware=[injection_guard],
) as agent:
    await agent.invoke([HumanMessage(content=truncate_input(user_input))])
```

--------------------------------

### Run Integration Tests

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Executes the integration tests for the Splunk SDK for Python. It is recommended to test against a clean Splunk install or disable certain apps to ensure test stability.

```sh
# Run the integration tests:
make test-integration
```

--------------------------------

### Audit Logging Tool Calls with Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Example of custom middleware using `tool_middleware` to log information about tool calls, such as the tool name and call status. This helps in observing agent behavior and debugging.

```python
from splunklib.ai.middleware import tool_middleware, ToolMiddlewareHandler, ToolRequest, ToolResponse

@tool_middleware
async def audit_tool_calls(request: ToolRequest, handler: ToolMiddlewareHandler) -> ToolResponse:
    logger.info("tool_call started", extra={"tool": request.call.name})
    return await handler(request)
```

--------------------------------

### Override Default Agent Limits

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Configure custom limits for the agent by passing an `AgentLimits` object to the constructor. This example overrides only the `max_tokens` limit, while other defaults remain active.

```python
from splunklib.ai.limits import AgentLimits

async with Agent(
    ...,
    limits=AgentLimits(max_tokens=50_000),  # overrides default 200 000; other defaults still apply
) as agent: ...
```

--------------------------------

### Decorator for Agent Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Use the @agent_middleware decorator to create middleware that runs once per invoke call. This example ensures at least one tool call is made.

```python
from typing import Any
from splunklib.ai.middleware import (
    agent_middleware,
    AgentMiddlewareHandler,
    AgentRequest,
)
from splunklib.ai.messages import AgentResponse, ToolCall


@agent_middleware
async def force_tool_call(
    request: AgentRequest, handler: AgentMiddlewareHandler
) -> AgentResponse[Any | None]:
    # Keep retrying until the agent makes at least one tool call.
    resp = await handler(request)
    while not any(m for m in resp.messages if isinstance(m, ToolCall)):
        resp = await handler(request)
    return resp
```

--------------------------------

### Add external dependencies to App

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Modifies the system path to include external dependencies installed in the 'bin/lib/' directory of a Splunk App. This is necessary for the App to find and use these dependencies.

```python
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "lib"))
```

--------------------------------

### Initialize OpenAIModel with Agent and Service

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Set up an Agent using OpenAIModel, a system prompt, a Splunk service connection, and tool settings for local MCP tools.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.tool_settings import ToolSettings
from splunklib.client import connect

model = OpenAIModel(...)
service = connect(...)

async with Agent(
    model=model,
    system_prompt="Your name is Stefan",
    service=service,
    tool_settings=ToolSettings(local=True, remote=None),
) as agent: ...
```

--------------------------------

### Basic Agent Usage

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Demonstrates how to initialize an Agent with an OpenAI model and Splunk service, then invoke it with a human message.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.messages import HumanMessage
from splunklib.client import connect

service = connect(
    scheme="https",
    host="localhost",
    port=8089,
    username="user",
    password="password",
    autologin=True,
)

model = OpenAIModel(
    model="gpt-4o-mini",
    base_url="https://api.openai.com/v1",
    api_key="SECRET",
)

async with Agent(
    model=model,
    system_prompt="Your name is Stefan",
    service=service,
) as agent:
    result = await agent.invoke([HumanMessage(content="What is your name?")])

    print(result.final_message.content) # My name is Stefan
```

--------------------------------

### Enable All Local Tools with Shorthand

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Configure an Agent to load all local tools without any filtering by setting local=True.

```python
tool_settings=ToolSettings(local=True, remote=None)
```

--------------------------------

### Endpoint

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Represents a Splunk API endpoint, allowing direct GET and POST requests.

```APIDOC
## Endpoint

### Description
Represents a Splunk API endpoint, allowing direct interaction via GET and POST requests.

### Methods
- **get**: Send a GET request to the endpoint.
- **post**: Send a POST request to the endpoint.
```

--------------------------------

### Google Vertex AI Model Initialization with Service Account

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes a GoogleModel for use with Vertex AI via service account credentials. Requires model name, project ID, and credentials object.

```python
from google.oauth2 import service_account
from splunklib.ai import Agent, GoogleModel

credentials = service_account.Credentials.from_service_account_file(
    "path/to/service-account.json",
    scopes=["https://www.googleapis.com/auth/cloud-platform"],
)

model = GoogleModel(
    model="gemini-2.0-flash",
    project="your-gcp-project-id",
    credentials=credentials,
    # location="us-central1",  # optional, defaults to us-central1
)

async with Agent(model=model) as agent: ...
```

--------------------------------

### Decorator for Subagent Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Apply the @subagent_middleware decorator to intercept subagent calls. This example provides a mock response for the 'SummaryAgent'.

```python
from splunklib.ai.messages import SubagentTextResult
from splunklib.ai.middleware import (
    subagent_middleware,
    SubagentMiddlewareHandler,
    SubagentRequest,
    SubagentResponse,
)


@subagent_middleware
async def mock_subagent(
    request: SubagentRequest, handler: SubagentMiddlewareHandler
) -> SubagentResponse:
    if request.call.name == "SummaryAgent":
        return SubagentResponse(result=SubagentTextResult(content="Executive summary: no critical incidents detected."))
    return await handler(request)
```

--------------------------------

### Configure Remote Tool Allowlist

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Set up an Agent with remote tool settings, specifying allowed tool names and tags for LLM access.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.tool_settings import LocalToolSettings, RemoteToolSettings, ToolAllowlist, ToolSettings
from splunklib.client import connect

model = OpenAIModel(...)
service = connect(...)

async with Agent(
    model=model,
    system_prompt="Your name is Stefan",
    service=service,
    tool_settings=ToolSettings(
        local=True,
        remote=RemoteToolSettings(
            allowlist=ToolAllowlist(names=["tool_name"], tags=["tag1", "tag2"])
        ),
    ), 
) as agent: ...
```

--------------------------------

### Decorator for Model Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Apply the @model_middleware decorator to intercept model calls. This example redacts sensitive information from the system prompt.

```python
from splunklib.ai.middleware import (
    model_middleware,
    ModelMiddlewareHandler,
    ModelRequest,
    ModelResponse,
)

@model_middleware
async def redact_system_prompt(
    request: ModelRequest, handler: ModelMiddlewareHandler
) -> ModelResponse:
    return await handler(
        ModelRequest(
            system_message=request.system_message.replace(
                "SECRET", "[REDACTED]"
            ),
            state=request.state,
        )
    )
```

--------------------------------

### Decorator for Tool Middleware

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Use the @tool_middleware decorator to intercept tool calls. This example mocks the 'temperature' tool to return a fixed value.

```python
from splunklib.ai.middleware import (
    tool_middleware,
    ToolMiddlewareHandler,
    ToolRequest,
    ToolResponse,
)


@tool_middleware
async def mock_temperature(
    request: ToolRequest, handler: ToolMiddlewareHandler
) -> ToolResponse:
    if request.call.name == "temperature":
        return ToolResponse(result=ToolResult(content="25.0", structured_content=None))
    return await handler(request)
```

--------------------------------

### Google Vertex AI Model Initialization with API Key

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes a GoogleModel for use with Vertex AI via API key. Requires model name, API key, and project ID.

```python
from splunklib.ai import Agent, GoogleModel

model = GoogleModel(
    model="gemini-2.0-flash",
    api_key="YOUR_VERTEX_API_KEY",
    project="your-gcp-project-id",
    # location="us-central1",  # optional, defaults to us-central1
)

async with Agent(model=model) as agent: ...
```

--------------------------------

### Run Splunk SDK Test Suite

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Execute the entire suite of unit and integration tests for the Splunk SDK using the 'make test' command. Ensure your development environment is set up correctly before running tests.

```sh
# Run entire test suite:
make test
```

--------------------------------

### Ollama Self-hosted Model with OpenAIModel

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes an OpenAIModel to connect to a self-hosted model via Ollama. Requires model name and Ollama's OpenAI-compatible base URL.

```python
from splunklib.ai import Agent, OpenAIModel

model = OpenAIModel(
    model="llama3.2:3b",
    base_url="http://localhost:11434/v1",
    api_key="",  # required but ignored
)

async with Agent(model=model) as agent: ....
```

--------------------------------

### OpenAI Model Initialization

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes an OpenAIModel for use with the Agent. Requires model name, base URL, and API key.

```python
from splunklib.ai import Agent, OpenAIModel

model = OpenAIModel(
    model="gpt-4o-mini",
    base_url="https://api.openai.com/v1",
    api_key="SECRET",
)

async with Agent(model=model) as agent: ....
```

--------------------------------

### Pass Middleware to Agent

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Demonstrates how to instantiate an Agent and pass a list of middleware functions to its constructor. Middleware can be used to modify requests or responses.

```python
async with Agent(
    model=model,
    service=service,
    system_prompt="...",
    middleware=[redact_system_prompt, mock_temperature, mock_subagent],
) as agent: ...
```

--------------------------------

### Settings

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Manages Splunk instance settings.

```APIDOC
## Settings

### Description
Manages Splunk instance settings, allowing for updates to various configuration parameters.

### Methods
- **update**: Update Splunk settings.
```

--------------------------------

### Subagents with ConversationStore for Multi-Turn Conversations

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Illustrates how to equip a subagent with its own ConversationStore (e.g., InMemoryStore) to enable multi-turn conversations. This allows the supervisor agent to resume prior conversations with a subagent, maintaining context across multiple interactions.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.conversation_store import InMemoryStore
from splunklib.ai.messages import HumanMessage
from splunklib.client import connect

model = OpenAIModel(...)
service = connect(...)

async with (
    Agent(
        model=model,
        service=service,
        system_prompt=(
            "You are a log analysis expert. When asked to analyze a problem, "
            "ask clarifying questions if needed before querying logs."
        ),
        name="log_analyzer_agent",
        description="Analyzes logs and can ask follow-up questions to narrow down a problem.",
        conversation_store=InMemoryStore(),
    ) as log_analyzer_agent,
):
    async with Agent(
        model=model,
        service=service,
        system_prompt="You are a supervisor. Use the log analyzer agent to investigate issues.",
        agents=[log_analyzer_agent],
    ) as agent:
        # The supervisor calls the subagent and may continue the
        # same conversation with a subagent in the agentic loop and
        # across multiple agent loop invocations.
        await agent.invoke(...)

```

--------------------------------

### Pydantic Model with Custom Validator for Output Schema

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Define a Pydantic model with custom validation logic for structured output. This example shows a validator ensuring 'max_score' is greater than 'min_score'.

```python
class Output(BaseModel):
    min_score: float
    max_score: float = Field(description="max_score must be greater than min_score")

    @model_validator(mode="after")
    def max_must_exceed_min(self) -> "Output":
        if self.max_score <= self.min_score:
            raise ValueError("max_score must be greater than min_score")
        return self
```

--------------------------------

### Connect to Splunk with Username/Password

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Establishes a connection to a Splunk Enterprise instance using a username and password. Ensure `autologin=True` is set for immediate authentication.

```python
import splunklib.client as client

service = client.connect(host=<HOST_URL>, username=<USERNAME>, password=<PASSWORD>, autologin=True)
```

--------------------------------

### Stop Agentic Loop with a Custom Hook

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Implement a hook that raises an exception to stop the agentic loop based on a custom condition, such as a message limit. This example defines a factory function to create the hook with a configurable message limit.

```python
from splunklib.ai.hooks import before_model
from splunklib.ai.middleware import AgentMiddleware, ModelRequest

def message_limit(message_limit: int) -> AgentMiddleware:
    @before_model
    def _hook(req: ModelRequest) -> None:
        if len(req.state.messages) >= message_limit:
            raise Exception("Stopping Agentic Loop")

    return _hook


async with Agent(
    ...,
    middleware=[message_limit(message_limit=5)],
) as agent: ...
```

--------------------------------

### Creating and Using Specialized Subagents with a Supervisor

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Demonstrates how to define and use specialized subagents (debugging_agent, log_analyzer_agent) orchestrated by a supervisor agent. Each subagent is configured with a specific model, system prompt, name, description, and tool settings. This approach helps mitigate context bloat by breaking down complex tasks.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.messages import HumanMessage
from splunklib.ai.tool_settings import LocalToolSettings, ToolAllowlist, ToolSettings
from splunklib.client import connect

model = OpenAIModel(...)
service = connect(...)

async with (
    Agent(
        model=highly_specialized_model,
        service=service,
        system_prompt=(
            "You are a highly specialized debugging agent, your job is to provide as much"
            "details as possible to resolve issues."
            "You have access to debugging-related tools, which you should leverage for your job."
        ),
        name="debugging_agent",
        description="Agent, that provided with logs will analyze and debug complex issues",
        tool_settings=ToolSettings(
            local=LocalToolSettings(allowlist=ToolAllowlist(tags=["debugging"])),
            remote=None,
        ),
    ) as debugging_agent,
    Agent(
        model=low_cost_model,
        service=service,
        system_prompt=(
            "You are a log analyzer agent. Your job is to query logs, based on the details that you receive and"
            "return a summary of interesting logs, that can be used for further analysis."
        ),
        name="log_analyzer_agent",
        description="Agent, that provided with a problem details will return logs, that could be related to the problem",
        tool_settings=ToolSettings(
            local=LocalToolSettings(allowlist=ToolAllowlist(tags=["spl"])),
            remote=None,
        ),
    ) as log_analyzer_agent,
):
    async with Agent(
        model=low_cost_model,
        service=service,
        system_prompt="You are a supervisor agent, use available subagents to perform requested operations.",
        agents=[debugging_agent, log_analyzer_agent],
    ) as agent:
        result = await agent.invoke(
            [
                HumanMessage(
                    content=(
                        "We are facing a production issue, users report that some pages do not exist, but it seems like the 404 are not deterministic."
                        "Query the logs in the index 'main', and try to debug the root cause of this issue."
                    ),
                )
            ]
        )

```

--------------------------------

### Run Integration and System Tests

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Execute integration and system tests, which require a running Splunk instance. Ensure Splunk services are available, potentially using the docker targets, before running.

```shell
make test-integration
```

--------------------------------

### Initialize AnthropicModel with Ollama

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Configure and initialize an AnthropicModel to interact with an Ollama instance. Requires a model name and base URL.

```python
from splunklib.ai import Agent, AnthropicModel

model = AnthropicModel(
    model="llama3.2:3b",
    base_url="http://localhost:11434",
    api_key="",  # required but ignored
)

async with Agent(model=model) as agent: ....
```

--------------------------------

### Manual Prompt Creation with create_structured_prompt

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

If building messages manually, `create_structured_prompt` provides the same separation as `invoke_with_data` and can be used directly within a `HumanMessage`.

```python
from splunklib.ai import create_structured_prompt
from splunklib.ai.messages import HumanMessage

result = await agent.invoke([
    HumanMessage(content=create_structured_prompt(
        instructions="Summarize this security alert and assess its severity.",
        data=alert_payload,
    ))
])
```

--------------------------------

### Copy .env.template to .env for Development

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Duplicate the .env.template file to .env to store Splunk connection arguments for development convenience. Adjust the new .env file to match your local environment.

```sh
cp .env.template .env
```

--------------------------------

### Anthropic Model Initialization

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes an AnthropicModel for use with the Agent. Requires model name, base URL, and API key.

```python
from splunklib.ai import Agent, AnthropicModel

model = AnthropicModel(
    model="claude-haiku-4-5-20251001",
    base_url="https://api.anthropic.com",
    api_key="SECRET",
)

async with Agent(model=model) as agent: ....
```

--------------------------------

### ConfigurationFile

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Represents a Splunk configuration file.

```APIDOC
## ConfigurationFile

### Description
Represents a Splunk configuration file. Inherits common entity methods.
```

--------------------------------

### connect

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/binding.rst

Establishes a connection to a Splunk instance. This is the primary function for initiating communication with Splunk.

```APIDOC
## connect

### Description
Establishes a connection to a Splunk instance. This is the primary function for initiating communication with Splunk.

### Method
connect

### Parameters
(Details for parameters are not provided in the source text)

### Response
(Details for response are not provided in the source text)
```

--------------------------------

### Google Gemini API Model Initialization

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initializes a GoogleModel for use with the Gemini API. Requires model name and API key.

```python
from splunklib.ai import Agent, GoogleModel

model = GoogleModel(
    model="gemini-2.0-flash",
    api_key="YOUR_GOOGLE_API_KEY",
)

async with Agent(model=model) as agent: ...
```

--------------------------------

### create_structured_prompt Function

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/index.rst

Creates a structured prompt string from given components, potentially for use with AI models.

```APIDOC
## Function: create_structured_prompt

### Description
Assembles a formatted prompt string from various input elements, designed for clarity and consistency in AI interactions.

### Parameters
(Details for parameters would be listed here if available in the source)

### Returns
(Details for return value would be listed here if available in the source)
```

--------------------------------

### Configurations

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Manages Splunk configuration files.

```APIDOC
## Configurations

### Description
Manages Splunk configuration files, allowing for the creation and deletion of configuration stanzas.

### Methods
- **create**: Create a new configuration stanza.
- **delete**: Delete a configuration stanza.
```

--------------------------------

### Run Unit Tests Only

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Execute only the unit tests for the fastest feedback loop during development. This is recommended for quick checks after making code changes.

```shell
make test-unit
```

--------------------------------

### Run Only Splunk SDK Unit Tests

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Execute only the unit tests for the Splunk SDK by running the 'make test-unit' command. This is useful for quickly verifying core logic changes.

```sh
# Run only the unit tests:
make test-unit
```

--------------------------------

### Configure Local Tool Allowlist with Custom Predicate

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Define local tool settings using a custom predicate function to filter tools based on their names.

```python
tool_settings=ToolSettings(
    local=LocalToolSettings(
        allowlist=ToolAllowlist(custom_predicate=lambda tool: tool.name.startswith("my_"))
    ),
    remote=None,
)
```

--------------------------------

### Agent with InMemoryStore for Conversation History

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Initialize an Agent with an InMemoryStore to maintain conversation history across multiple invocations.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.ai.conversation_store import InMemoryStore
from splunklib.ai.messages import HumanMessage
from splunklib.client import connect

model = OpenAIModel(...)
service = connect(...)

async with Agent(
    model=model,
    service=service,
    system_prompt="",
    conversation_store=InMemoryStore(),
) as agent:
    await agent.invoke([HumanMessage(content="Hi, my name is Chris.")])
    result = await agent.invoke([HumanMessage(content="What is my name?")])
    print(result.final_message.content)  # Chris
```

--------------------------------

### Application

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Represents a Splunk application.

```APIDOC
## Application

### Description
Represents a Splunk application, providing access to its setup information, package details, and update information.

### Methods
- **setupInfo**: Get the setup information for the application.
- **package**: Get the package details for the application.
- **updateInfo**: Get the update information for the application.
```

--------------------------------

### Tool Settings

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/ai.rst

Configuration options for tools, including local, remote, and allowlist settings.

```APIDOC
## Tool Settings

### Description
Provides classes for configuring and managing tool usage.

### Classes
- **splunklib.ai.tool_settings.ToolSettings**: Base class for tool settings.
- **splunklib.ai.tool_settings.LocalToolSettings**: Settings for local tools.
- **splunklib.ai.tool_settings.RemoteToolSettings**: Settings for remote tools.
- **splunklib.ai.tool_settings.ToolAllowlist**: Manages a list of allowed tools.
```

--------------------------------

### Generate Events with Helper Method in Custom Generating Commands

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Utilize the gen_record() method from SearchCommand to properly generate new events with key=value pairs in a Custom Generating Command. This ensures correct event formatting.

```diff
@Configuration()
class GeneratorTest(GeneratingCommand):
    def generate(self):
        yield {'_time': time.time(), 'one': 1}
        yield {'_time': time.time(), 'two': 2}
```

```diff
@Configuration()
class GeneratorTest(GeneratingCommand):
    def generate(self):
        yield self.gen_record(_time=time.time(), one=1)
        yield self.gen_record(_time=time.time(), two=2)
```

--------------------------------

### Connect to Splunk with Session Key

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/README.md

Establishes a connection to Splunk Enterprise using a session key. This is a common method for authenticated API requests.

```python
import splunklib.client as client

service = client.connect(host=<HOST_URL>, token=<SESSION_KEY>, autologin=True)
```

--------------------------------

### Input

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Represents a Splunk input.

```APIDOC
## Input

### Description
Represents a Splunk input, providing methods to update its configuration.

### Methods
- **update**: Update the input configuration.
```

--------------------------------

### dispatch

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/searchcommands.rst

Dispatches a search command class, handling argument parsing and input/output streams. This is the main entry point for executing a custom search command.

```APIDOC
## dispatch

### Description
Dispatches a search command class, handling argument parsing and input/output streams. This is the main entry point for executing a custom search command.

### Function Signature
def dispatch(command_class, argv=sys.argv, input_file=sys.stdin, output_file=sys.stdout, module_name=None, allow_empty_input=True)

### Parameters
- **command_class**: The class of the search command to dispatch.
- **argv** (list, optional): The list of command-line arguments. Defaults to `sys.argv`.
- **input_file** (file-like object, optional): The input stream. Defaults to `sys.stdin`.
- **output_file** (file-like object, optional): The output stream. Defaults to `sys.stdout`.
- **module_name** (str, optional): The name of the module. Defaults to None.
- **allow_empty_input** (bool, optional): Whether to allow empty input. Defaults to True.
```

--------------------------------

### Run Full Pytest Suite

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Execute all unit, integration, and system tests to ensure code quality and stability. This command requires Splunk services to be available for integration tests.

```shell
make test
```

--------------------------------

### Run Splunk Query with ToolContext

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Execute a Splunk search query using the service object available in ToolContext and process the JSON results.

```python
from splunklib.ai.registry import ToolContext, ToolRegistry
from splunklib.results import JSONResultsReader

registry = ToolRegistry()


@registry.tool()
def run_splunk_query(ctx: ToolContext) -> list[str]:
    stream = ctx.service.jobs.oneshot(
        "| makeresults count=10 | streamstats count as row_num",
        output_mode="json",
    )

    result = JSONResultsReader(stream)
    output: list[str] = []
    for r in result:
        if isinstance(r, dict):
            output.append(r["row_num"])

    return output

if __name__ == "__main__":
    registry.run()
```

--------------------------------

### Search for App Information in Internal Logs

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/examples/ai_modinput_app/README.md

Use this Splunk search query to look for any mentions of the AI modular input app within the internal Splunk logs.

```spl
index="_internal" ai_modinput_app
```

--------------------------------

### HttpLib

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/binding.rst

Provides low-level HTTP client functionalities for the Splunk SDK for Python.

```APIDOC
## HttpLib

### Description
Provides low-level HTTP client functionalities for the Splunk SDK for Python.

### Class Members
- **delete**: Performs an HTTP DELETE request.
- **get**: Performs an HTTP GET request.
- **post**: Performs an HTTP POST request.
- **request**: Performs a generic HTTP request.
```

--------------------------------

### Configure Agent for Specific Local Tools

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Enable an Agent to access specific local tools by name or tag using LocalToolSettings. Alternatively, set local=True to enable all.

```python
from splunklib.ai.tool_settings import LocalToolSettings, ToolAllowlist, ToolSettings

async with Agent(
    model=model,
    service=service,
    system_prompt="...",
    tool_settings=ToolSettings(
        # local=True,  # enable all local tools
        local=LocalToolSettings(
            allowlist=ToolAllowlist(names=["tool1"], tags=["tag1"])
        ),
        remote=None,
    ),
) as agent: ...
```

--------------------------------

### Format Code with Ruff

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/AGENTS.md

Apply code formatting and linting rules using Ruff. This command should be run after editing any Python file to ensure code style consistency.

```shell
make lint
```

--------------------------------

### execute

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/searchcommands.rst

Executes a search command. This function is typically called internally by `dispatch`.

```APIDOC
## execute

### Description
Executes a search command. This function is typically called internally by `dispatch`.

### Function Signature
def execute(command_class, argv=sys.argv, input_file=sys.stdin, output_file=sys.stdout, module_name=None, allow_empty_input=True)

### Parameters
- **command_class**: The class of the search command to execute.
- **argv** (list, optional): The list of command-line arguments. Defaults to `sys.argv`.
- **input_file** (file-like object, optional): The input stream. Defaults to `sys.stdin`.
- **output_file** (file-like object, optional): The output stream. Defaults to `sys.stdout`.
- **module_name** (str, optional): The name of the module. Defaults to None.
- **allow_empty_input** (bool, optional): Whether to allow empty input. Defaults to True.
```

--------------------------------

### NoSuchCapability

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Exception raised when a capability is not found.

```APIDOC
## NoSuchCapability

### Description
Exception raised when a requested Splunk capability does not exist.
```

--------------------------------

### Configure Agent for Remote MCP Tools

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Enable an Agent to access remote tools provided by the Splunk MCP Server App. Specify allowed tools by name or tag.

```python
from splunklib.ai.tool_settings import RemoteToolSettings, ToolAllowlist, ToolSettings

async with Agent(
    model=model,
    service=service,
    system_prompt="...",
    tool_settings=ToolSettings(
        remote=RemoteToolSettings(
            allowlist=ToolAllowlist(names=["splunk_get_indexes"], tags=["tag1"])
        )
    ),
) as agent: ...
```

--------------------------------

### GeneratingCommand

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/searchcommands.rst

Base class for search commands that generate events and write them to standard output. These commands do not read from standard input.

```APIDOC
## GeneratingCommand

### Description
Base class for search commands that generate events and write them to standard output. These commands do not read from standard input.

### Methods
- **generate(self)**: Abstract method to be implemented by subclasses to generate events.
- **process(self, args=sys.argv, input_file=sys.stdin, output_file=sys.stdout, allow_empty_input=True)**: Processes the search command, generating and writing events.
```

--------------------------------

### Enable Detailed Agent Logging

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Configure detailed tracing and debugging for the agent's lifecycle by passing a configured logger object to the `Agent` constructor. This enables logging for events like model interactions and tool calls.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.client import connect
import logging

model = OpenAIModel(...)
service = connect(...)

logger = logging.getLogger("test")
logger.setLevel(logging.DEBUG)

async with Agent(
        model=model,
        service=service,
        system_prompt="..." ,
        logger=logger,
    ) as agent: ...
```

--------------------------------

### splunklib.modularinput

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/index.rst

Facilitates the creation of modular inputs for Splunk, providing classes for arguments, events, input definitions, schemes, and script execution.

```APIDOC
## Module: splunklib.modularinput

### Description
Facilitates the creation of modular inputs for Splunk.

### Classes
- `Argument`: Represents an argument for a modular input.
- `Event`: Represents an event generated by a modular input.
- `EventWriter`: Writes events from a modular input.
- `InputDefinition`: Defines the input parameters for a modular input.
- `Scheme`: Defines the scheme for a modular input.
- `Script`: Represents the script for a modular input.
- `ValidationDefinition`: Defines validation rules for modular input arguments.
```

--------------------------------

### Register Local Tool with ToolRegistry

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Define and register a custom local tool using the ToolRegistry decorator. The tool's signature and docstring define its interface.

```python
from splunklib.ai.registry import ToolRegistry

registry = ToolRegistry()

@registry.tool()
def hello(name: str) -> str:
    """Returns a hello message"""
    return f"Hello {name}!"


if __name__ == "__main__":
    registry.run()
```

--------------------------------

### splunklib.searchcommands

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/index.rst

Enables the development of custom search commands for Splunk, supporting different command types and option definitions.

```APIDOC
## Module: splunklib.searchcommands

### Description
Enables the development of custom search commands for Splunk.

### Classes
- `EventingCommand`: Base class for eventing search commands.
- `GeneratingCommand`: Base class for generating search commands.
- `ReportingCommand`: Base class for reporting search commands.
- `StreamingCommand`: Base class for streaming search commands.
- `ExternalSearchCommand`: Base class for external search commands.
- `Option`: Defines an option for a search command.

### Functions
- `dispatch()`: Dispatches a search command.
- `execute()`: Executes a search command.

### Data
- `SearchMetric`: Represents search metrics.
```

--------------------------------

### NotSupportedError

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Exception raised for unsupported operations.

```APIDOC
## NotSupportedError

### Description
Exception raised when an operation is not supported by the Splunk API or the client library.
```

--------------------------------

### Using invoke_with_data for Security

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Use `invoke_with_data` when passing external or user-supplied data to the agent to reduce the risk of prompt injection by separating instructions from untrusted data.

```python
from splunklib.ai.messages import HumanMessage

# Use invoke for plain conversational messages.
result = await agent.invoke([HumanMessage(content="What are the top threats this week?")])

# Use invoke_with_data when passing external data to the agent.
result = await agent.invoke_with_data(
    instructions="Summarize this security alert and assess its severity.",
    data=alert_payload,  # str or dict
)
```

--------------------------------

### StreamingCommand

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/searchcommands.rst

Base class for search commands that stream events from standard input to standard output. These commands typically perform transformations on each event as it passes through.

```APIDOC
## StreamingCommand

### Description
Base class for search commands that stream events from standard input to standard output. These commands typically perform transformations on each event as it passes through.

### Methods
- **stream(self, record)**: Method to be implemented by subclasses to process individual events.
- **process(self, args=sys.argv, input_file=sys.stdin, output_file=sys.stdout, allow_empty_input=True)**: Processes the search command, streaming events.
```

--------------------------------

### Define and Use Pydantic Model for Structured Output

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/splunklib/ai/README.md

Configure an Agent with a Pydantic model to receive structured output. Use invoke_with_data for secure input processing. Access parsed results via the structured_output attribute.

```python
from splunklib.ai import Agent, OpenAIModel
from splunklib.client import connect
from typing import Literal
from pydantic import BaseModel, Field

model = OpenAIModel(...)
service = connect(...)

class Output(BaseModel):
    service_name: str = Field(
        description="Name of the service or component where the failure occurred",
        min_length=1,
    )

    severity: Literal["critical", "high", "medium", "low", "info"] = Field(
        description="Assessed severity of the failure based on impact and urgency"
    )

    summary: str = Field(
        description="Concise human-readable summary of what went wrong",
        min_length=1,
    )

async with Agent(
    model=model,
    service=service,
    system_prompt="You are an agent, whose job is to determine the details of provided failure logs",
    output_schema=Output,
) as agent:
    # Use invoke_with_data when passing external data to the agent to reduce
    # the risk of prompt injection.
    result = await agent.invoke_with_data(
        instructions="Analyze this log and determine the failure details.",
        data=log,
    )

    # Make use of the output.
    result.structured_output.service_name
    result.structured_output.severity
    result.structured_output.summary
```

--------------------------------

### EventingCommand

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/searchcommands.rst

Base class for search commands that process events from standard input and write results to standard output. Commands inheriting from this class typically transform or filter events.

```APIDOC
## EventingCommand

### Description
Base class for search commands that process events from standard input and write results to standard output. Commands inheriting from this class typically transform or filter events.

### Methods
- **transform(self, records)**: Abstract method to be implemented by subclasses to process a list of records.
- **process(self, args=sys.argv, input_file=sys.stdin, output_file=sys.stdout)**: Processes the search command, reading from input and writing to output.
```

--------------------------------

### splunklib.client

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/index.rst

Offers client-side functionalities for interacting with Splunk resources, including service connection, endpoint management, and various entity and collection classes.

```APIDOC
## Module: splunklib.client

### Description
Provides client-side functionalities for interacting with Splunk resources, including service connection, endpoint management, and access to various entities and collections.

### Functions
- `connect()`: Establishes a client connection to Splunk.

### Classes
- `Service`: Represents the Splunk service connection.
- `Endpoint`: Base class for Splunk endpoints.
- `Entity`: Base class for Splunk entities.
- `Collection`: Represents a collection of Splunk entities.
- `ReadOnlyCollection`: Represents a read-only collection of Splunk entities.
- `Application`: Represents a Splunk application.
- `AlertGroup`: Represents an alert group.
- `ConfigurationFile`: Represents a Splunk configuration file.
- `Stanza`: Represents a stanza within a configuration file.
- `Configurations`: Manages Splunk configurations.
- `Index`: Represents a Splunk index.
- `Indexes`: Manages Splunk indexes.
- `Input`: Represents a Splunk input.
- `Inputs`: Manages Splunk inputs.
- `Job`: Represents a Splunk job.
- `Jobs`: Manages Splunk jobs.
- `KVStoreCollection`: Represents a KV Store collection.
- `KVStoreCollectionData`: Represents data within a KV Store collection.
- `KVStoreCollections`: Manages KV Store collections.
- `Loggers`: Manages Splunk loggers.
- `Message`: Represents a Splunk message.
- `ModularInputKind`: Represents a modular input kind.
- `Role`: Represents a Splunk role.
- `Roles`: Manages Splunk roles.
- `SavedSearch`: Represents a saved search.
- `SavedSearches`: Manages saved searches.
- `Settings`: Manages Splunk settings.
- `StoragePassword`: Represents a storage password.
- `StoragePasswords`: Manages storage passwords.
- `User`: Represents a Splunk user.
- `Users`: Manages Splunk users.

### Exceptions
- `AmbiguousReferenceException`: Raised when a reference is ambiguous.
- `IllegalOperationException`: Raised for illegal operations.
- `IncomparableException`: Raised when entities are incomparable.
- `InvalidNameException`: Raised for invalid names.
- `NoSuchCapability`: Raised when a capability is not found.
- `NotSupportedError`: Raised when an operation is not supported.
- `OperationError`: Raised for general operation errors.
```

--------------------------------

### Service

Source: https://github.com/splunk/splunk-sdk-python/blob/develop/docs/client.rst

Represents a connection to a Splunk instance and provides access to various Splunk resources.

```APIDOC
## Service

### Description
Represents a connection to a Splunk instance. It serves as a gateway to access and manage various Splunk resources like apps, configurations, indexes, jobs, and more.

### Methods
- **apps**: Access the collection of installed Splunk applications.
- **confs**: Access the collection of configuration files.
- **capabilities**: Access the collection of Splunk capabilities.
- **event_types**: Access the collection of event types.
- **fired_alerts**: Access the collection of fired alerts.
- **indexes**: Access the collection of indexes.
- **info**: Retrieve information about the Splunk instance.
- **inputs**: Access the collection of inputs.
- **job**: Access a specific job by its ID.
- **jobs**: Access the collection of jobs.
- **kvstore**: Access the collection of KV Store collections.
- **loggers**: Access the collection of loggers.
- **messages**: Access the collection of messages.
- **modular_input_kinds**: Access the collection of modular input kinds.
- **parse**: Parse Splunk search results.
- **restart**: Restart the Splunk instance.
- **restart_required**: Check if a restart is required.
- **roles**: Access the collection of roles.
- **search**: Perform a Splunk search.
- **saved_searches**: Access the collection of saved searches.
- **settings**: Access and update Splunk settings.
- **splunk_version**: Get the Splunk version.
- **storage_passwords**: Access the collection of storage passwords.
- **users**: Access the collection of users.
```