### Build Tools Installation: Development Dependencies

Source: https://docs.ag-ui.com/quickstart/server

Installs essential build utilities including Protocol Buffers for message serialization, Turbo for fast builds, and PNPM for efficient package management. Required for development environment setup.

```bash
brew install protobuf
npm i turbo
curl -fsSL https://get.pnpm.io/install.sh | sh -
```

--------------------------------

### Run AG-UI Development Server with Turbo

Source: https://docs.ag-ui.com/quickstart/middleware

This command compiles the project and starts the development server. It uses Turbo for monorepo management. No inputs required; outputs a running server at localhost:3000. Limitations: Assumes Turbo is installed and project is set up.

```bash
turbo run dev
```

--------------------------------

### Project Scaffolding: Server Template Setup

Source: https://docs.ag-ui.com/quickstart/server

Initializes the project by cloning the AG-UI repository and creating a new OpenAI server integration from the starter template. Sets up the basic directory structure and configuration files.

```bash
git clone git@github.com:ag-ui-protocol/ag-ui.git
cd ag-ui
cp -r integrations/server-starter integrations/openai-server
```

--------------------------------

### Install OpenAI SDK using Poetry (Bash)

Source: https://docs.ag-ui.com/quickstart/server

Installs the OpenAI Python SDK in the project via Poetry, preparing the environment for API calls. Run the commands in the server's Python directory.

```Bash
cd integrations/openai-server/server/python
poetry add openai
```

--------------------------------

### Install Dependencies (pnpm)

Source: https://docs.ag-ui.com/quickstart/middleware

Installs the project dependencies using pnpm.

```bash
pnpm install
```

--------------------------------

### Install Protobuf (Brew)

Source: https://docs.ag-ui.com/quickstart/middleware

Installs the protobuf compiler using Homebrew. This is a build dependency for the project.

```bash
brew install protobuf
```

--------------------------------

### Application Execution: Server and Dojo Startup

Source: https://docs.ag-ui.com/quickstart/server

Commands to start both the Python server and the dojo development environment. The Python server runs the OpenAI integration while the dojo provides the web interface for testing and development.

```bash
cd integrations/openai-server/server/python
poetry install && poetry run dev
```

```bash
# Install dependencies
pnpm install

# Compile the project and run the dojo
turbo run dev
```

--------------------------------

### Install OpenAI SDK with pnpm

Source: https://docs.ag-ui.com/quickstart/middleware

Installs the OpenAI SDK in the integrations/openai directory. Requires pnpm package manager and being in the project root. No inputs; outputs installed dependencies. Limitations: cd command assumes correct directory structure.

```bash
cd integrations/openai
pnpm install openai
```

--------------------------------

### Install Turbo (npm)

Source: https://docs.ag-ui.com/quickstart/middleware

Installs the Turbo package manager using npm. This is a build utility used in the project.

```bash
npm i turbo
```

--------------------------------

### Install pnpm (curl)

Source: https://docs.ag-ui.com/quickstart/middleware

Installs the pnpm package manager using curl. This is another build utility.

```bash
curl -fsSL https://get.pnpm.io/install.sh | sh -
```

--------------------------------

### Environment Setup: OpenAI API Key Configuration

Source: https://docs.ag-ui.com/quickstart/server

Sets up the OpenAI API key as an environment variable required for authentication with OpenAI services. This is the first step in configuring access to OpenAI's GPT models.

```bash
# Set your OpenAI API key
export OPENAI_API_KEY=your-api-key-here
```

--------------------------------

### Start Development Server (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

This command starts the development server using pnpm.

```bash
pnpm dev
```

--------------------------------

### Copy Middleware Starter Template (cp)

Source: https://docs.ag-ui.com/quickstart/middleware

Copies the middleware-starter template to create the OpenAI integration folder.

```bash
cp -r integrations/middleware-starter integrations/openai
```

--------------------------------

### Install AG-UI, Mastra, and AI SDK Dependencies in Bash

Source: https://docs.ag-ui.com/quickstart/clients

These commands install core AG-UI packages, Mastra ecosystem tools for agents and memory, and AI SDK with Zod for validation. Requires pnpm and internet access. Outputs installed dependencies; compatible with Node.js v18+ but may need API keys for runtime.

```bash
# Core AG-UI packages
pnpm add @ag-ui/client @ag-ui/core @ag-ui/mastra

# Mastra ecosystem packages
pnpm add @mastra/core @mastra/memory @mastra/libsql

# AI SDK and utilities
pnpm add @ai-sdk/openai zod@^3.25
```

--------------------------------

### OpenAI Streaming Chat Endpoint with FastAPI

Source: https://docs.ag-ui.com/quickstart/server

Complete FastAPI server implementation that creates a streaming chat endpoint for OpenAI's GPT-4o model. Supports tool calling, event streaming, and AG-UI format compatibility. Handles setup, request processing, streaming responses, and error handling.

```python
# Initialize OpenAI client - uses OPENAI_API_KEY from environment
client = OpenAI()

@app.post("/")
async def agentic_chat_endpoint(input_data: RunAgentInput, request: Request):
    """OpenAI agentic chat endpoint"""
    accept_header = request.headers.get("accept")
    encoder = EventEncoder(accept=accept_header)

    async def event_generator():
        try:
            yield encoder.encode(
                RunStartedEvent(
                    type=EventType.RUN_STARTED,
                    thread_id=input_data.thread_id,
                    run_id=input_data.run_id
                )
            )

            # Call OpenAI's API with streaming enabled
            stream = client.chat.completions.create(
                model="gpt-4o",
                stream=True,
                # Convert AG-UI tools format to OpenAI's expected format
                tools=[
                    {
                        "type": "function",
                        "function": {
                            "name": tool.name,
                            "description": tool.description,
                            "parameters": tool.parameters,
                        }
                    }
                    for tool in input_data.tools
                ] if input_data.tools else None,
                # Transform AG-UI messages to OpenAI's message format
                messages=[
                    {
                        "role": message.role,
                        "content": message.content or "",
                        # Include tool calls if this is an assistant message with tools
                        **({"tool_calls": message.tool_calls} if message.role == "assistant" and hasattr(message, 'tool_calls') and message.tool_calls else {}),
                        # Include tool call ID if this is a tool result message
                        **({"tool_call_id": message.tool_call_id} if message.role == "tool" and hasattr(message, 'tool_call_id') else {}),
                    }
                    for message in input_data.messages
                ],
            )

            message_id = str(uuid.uuid4())

            # Stream each chunk from OpenAI's response
            for chunk in stream:
                # Handle text content chunks
                if chunk.choices[0].delta.content:
                    yield encoder.encode({
                        "type": EventType.TEXT_MESSAGE_CHUNK,
                        "message_id": message_id,
                        "delta": chunk.choices[0].delta.content,
                    })
                # Handle tool call chunks
                elif chunk.choices[0].delta.tool_calls:
                    tool_call = chunk.choices[0].delta.tool_calls[0]

                    yield encoder.encode({
                        "type": EventType.TOOL_CALL_CHUNK,
                        "tool_call_id": tool_call.id,
                        "tool_call_name": tool_call.function.name if tool_call.function else None,
                        "parent_message_id": message_id,
                        "delta": tool_call.function.arguments if tool_call.function else None,
                    })

            yield encoder.encode(
                RunFinishedEvent(
                    type=EventType.RUN_FINISHED,
                    thread_id=input_data.thread_id,
                    run_id=input_data.run_id
                )
            )

        except Exception as error:
            yield encoder.encode(
                RunErrorEvent(
                    type=EventType.RUN_ERROR,
                    message=str(error)
                )
            )

    return StreamingResponse(
        event_generator(),
        media_type=encoder.get_content_type()
    )

def main():
    """Run the uvicorn server."""
    port = int(os.getenv("PORT", "8000"))
    uvicorn.run(
        "example_server:app",
        host="0.0.0.0",
        port=port,
        reload=True
    )

if __name__ == "__main__":
    main()
```

--------------------------------

### Install pnpm Package Manager in Bash

Source: https://docs.ag-ui.com/quickstart/clients

This command installs pnpm globally using npm for efficient package management in the Node.js project. It depends on having npm installed via Node.js. The output is a successful installation message; use this if pnpm is not already available.

```bash
# Install pnpm
npm install -g pnpm
```

--------------------------------

### Implement AG-UI streaming endpoint (Python)

Source: https://docs.ag-ui.com/quickstart/server

Defines a FastAPI POST endpoint that streams AG-UI events such as run start, message content, and run finish. It uses EventEncoder to format SSE events and returns a StreamingResponse. Requires fastapi, ag_ui core, and uuid.

```Python
@app.post("/\")
async def agentic_chat_endpoint(input_data: RunAgentInput, request: Request):
    """Agentic chat endpoint"""
    # Get the accept header from the request
    accept_header = request.headers.get("accept")

    # Create an event encoder to properly format SSE events
    encoder = EventEncoder(accept=accept_header)

    async def event_generator():

        # Send run started event
        yield encoder.encode(
          RunStartedEvent(
            type=EventType.RUN_STARTED,
            thread_id=input_data.thread_id,
            run_id=input_data.run_id
          ),
        )

        message_id = str(uuid.uuid4())

        yield encoder.encode(
            TextMessageStartEvent(
                type=EventType.TEXT_MESSAGE_START,
                message_id=message_id,
                role="assistant"
            )
        )

        yield encoder.encode(
            TextMessageContentEvent(
                type=EventType.TEXT_MESSAGE_CONTENT,
                message_id=message_id,
                delta="Hello world!"
            )
        )

        yield encoder.encode(
            TextMessageEndEvent(
                type=EventType.TEXT_MESSAGE_END,
                message_id=message_id
            )
        )

        # Send run finished event
        yield encoder.encode(
          RunFinishedEvent(
            type=EventType.RUN_FINISHED,
            thread_id=input_data.thread_id,
            run_id=input_data.run_id
          ),
        )

 return StreamingResponse(
        event_generator(),
        media_type=encoder.get_content_type()
    )
```

```Python
import os
import uuid
import uvicorn
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from ag_ui.core import (
    RunAgentInput,
    EventType,
    RunStartedEvent,
    RunFinishedEvent,
    RunErrorEvent,
)
from ag_ui.encoder import EventEncoder
from openai import OpenAI

app = FastAPI(title="AG-UI OpenAI Server")
```

--------------------------------

### Install TypeScript Dev Dependencies in Bash

Source: https://docs.ag-ui.com/quickstart/clients

This installs TypeScript, Node.js types, and tsx for running TypeScript files directly. It requires pnpm and targets development use. Outputs installed packages in node_modules; run in the project root.

```bash
pnpm add -D typescript @types/node tsx
```

--------------------------------

### Add 'open' Package (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

This command installs the 'open' package using pnpm.

```bash
pnpm add open
```

--------------------------------

### ReasoningStart Event

Source: https://docs.ag-ui.com/concepts/events

Marks the start of reasoning. This draft event supports LLM reasoning visibility and continuity.

```APIDOC
## ReasoningStart Event

### Description
Marks the start of reasoning. This draft event supports LLM reasoning visibility and continuity.

### Method
EVENT

### Endpoint
/reasoning/start

### Parameters
#### Request Body
- **messageId** (string) - Required - Unique identifier of this reasoning
- **encryptedContent** (string) - Optional - Optional encrypted content

### Request Example
{
  "messageId": "reason-456",
  "encryptedContent": "encrypted-payload-here"
}

### Response
#### Success Response (200)
- **status** (string) - Status of the event processing

#### Response Example
{
  "status": "reasoning started"
}
```

--------------------------------

### Clone AG-UI Repository (git)

Source: https://docs.ag-ui.com/quickstart/middleware

Clones the AG-UI repository from GitHub. This provides the base project structure.

```bash
git clone git@github.com:ag-ui-protocol/ag-ui.git
```

--------------------------------

### Package Configuration: Metadata and Dependencies

Source: https://docs.ag-ui.com/quickstart/server

Configures package metadata for the new OpenAI server integration including name, author, and version information. Also manages dependencies in the dojo application configuration.

```json
{
  "name": "@ag-ui/openai-server",
  "author": "Your Name <your-email@example.com>",
  "version": "0.0.1",

  ... rest of package.json
}
```

```json
{
  "name": "demo-viewer",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@ag-ui/agno": "workspace:*",
    "@ag-ui/langgraph": "workspace:*",
    "@ag-ui/mastra": "workspace:*",
    "@ag-ui/middleware-starter": "workspace:*",
    "@ag-ui/server-starter": "workspace:*",
    "@ag-ui/server-starter-all-features": "workspace:*",
    "@ag-ui/vercel-ai-sdk": "workspace:*",
    "@ag-ui/openai-server": "workspace:*",

  ... rest of package.json
}
```

--------------------------------

### Install ag-ui-protocol package

Source: https://docs.ag-ui.com/sdk/python/encoder/overview

Command to install the ag-ui-protocol package using pip. This package contains the necessary modules for encoding agent events.

```bash
pip install ag-ui-protocol
```

--------------------------------

### ActivitySnapshotEvent JSON example

Source: https://docs.ag-ui.com/drafts/activity-events

Sample JSON payload of an ActivitySnapshotEvent showing a PLAN activity with a list of tasks.

```JSON
{
  "id": "evt_001",
  "ts": 1714064100000,
  "type": "ACTIVITY_SNAPSHOT",
  "messageId": "msg_789",
  "activityType": "PLAN",
  "content": {
    "tasks": ["check reddit", "search X.com"]
  }
}
```

--------------------------------

### Add OpenAI Package to Dojo Dependencies (JSON)

Source: https://docs.ag-ui.com/quickstart/middleware

Adds the `@ag-ui/openai` package to the `dependencies` section of `apps/dojo/package.json`.

```json
  "@ag-ui/openai": "workspace:*", <- Add this line
```

--------------------------------

### Handle Tool Call Start Event in TypeScript

Source: https://docs.ag-ui.com/concepts/tools

This snippet shows the ToolCallStart event structure for initiating a tool call with a unique ID. It integrates with AG-UI's event system; inputs include tool details, outputs signal the start of execution. No dependencies beyond AG-UI; used for tracking and sequencing tool interactions without exposing results.

```typescript
{
  type: EventType.TOOL_CALL_START,
  toolCallId: "tool-123",
  toolCallName: "confirmAction",
  parentMessageId: "msg-456" // Optional reference to a message
}
```

--------------------------------

### Run AG-UI Assistant (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

This bash command initiates the AG-UI assistant development server. It assumes the project dependencies have been installed using a package manager like pnpm. This command is used to test the CLI interface created in Step 4.

```bash
pnpm dev

```

--------------------------------

### Integration Implementation: Server Agent and Menu Configuration

Source: https://docs.ag-ui.com/quickstart/server

Implements the core server agent class by extending HttpAgent and configures integration in the dojo application through menu and agent configuration. Enables agentic chat functionality for the OpenAI server.

```typescript
// Change the name to OpenAIServerAgent to add a minimal middleware for your integration.
// You can use this later on to add configuration etc.
export class OpenAIServerAgent extends HttpAgent {}
```

```typescript
// ...
export const menuIntegrations: MenuIntegrationConfig[] = [
  // ...

  {
    id: "openai-server",
    name: "OpenAI Server",
    features: ["agentic_chat"],
  },
]
```

```typescript
// ...
import { OpenAIServerAgent } from "@ag-ui/openai-server"

export const agentsIntegrations: AgentIntegrationConfig[] = [
  // ...

  {
    id: "openai-server",
    agents: async () => {
      return {
        agentic_chat: new OpenAIServerAgent(),
      }
    },
  },
]
```

--------------------------------

### MetaEvents TypeScript Definition and Examples

Source: https://docs.ag-ui.com/drafts/meta-events

TypeScript type definition for MetaEvent and JSON examples showing user feedback, annotations, and external system events. Includes thumbs up/down reactions, user notes, tag assignments, analytics tracking, and content moderation flags.

```typescript
type MetaEvent = BaseEvent & {
  type: EventType.META
  /**
   * Application-defined type of the meta event.
   * Examples: "thumbs_up", "thumbs_down", "tag", "note"
   */
  metaType: string

  /**
   * Application-defined payload.
   * May reference other entities (e.g., messageId) or contain freeform data.
   */
  payload: Record<string, unknown>
}
```

```json
{
  "id": "evt_123",
  "ts": 1714063982000,
  "type": "META",
  "metaType": "thumbs_up",
  "payload": {
    "messageId": "msg_456",
    "userId": "user_789"
  }
}
```

```json
{
  "id": "evt_124",
  "ts": 1714063985000,
  "type": "META",
  "metaType": "thumbs_down",
  "payload": {
    "messageId": "msg_456",
    "userId": "user_789",
    "reason": "inaccurate",
    "comment": "The calculation seems incorrect"
  }
}
```

```json
{
  "id": "evt_789",
  "ts": 1714064001000,
  "type": "META",
  "metaType": "note",
  "payload": {
    "text": "Important question to revisit",
    "relatedRunId": "run_001",
    "author": "user_123"
  }
}
```

```json
{
  "id": "evt_890",
  "ts": 1714064100000,
  "type": "META",
  "metaType": "tag",
  "payload": {
    "tags": ["important", "follow-up"],
    "threadId": "thread_001"
  }
}
```

```json
{
  "id": "evt_901",
  "ts": 1714064200000,
  "type": "META",
  "metaType": "analytics",
  "payload": {
    "event": "conversation_shared",
    "properties": {
      "shareMethod": "link",
      "recipientCount": 3
    }
  }
}
```

```json
{
  "id": "evt_902",
  "ts": 1714064300000,
  "type": "META",
  "metaType": "moderation",
  "payload": {
    "action": "flag",
    "messageId": "msg_999",
    "category": "inappropriate_content",
    "confidence": 0.95
  }
}
```

--------------------------------

### Update package.json (JSON)

Source: https://docs.ag-ui.com/quickstart/middleware

Modifies the `package.json` file's name, author, and version. Ensures the package is properly configured within the project.

```json
{
  "name": "@ag-ui/openai",
  "author": "Your Name <your-email@example.com>",
  "version": "0.0.1",

  ... rest of package.json
}
```

--------------------------------

### JSON Examples for Run Interrupt and Resume Flows

Source: https://docs.ag-ui.com/drafts/interrupts

These JSON examples demonstrate minimal and complex interrupt/resume interactions between an agent and client. They show how agents send RUN_FINISHED events with interrupt payloads for approval, and how clients resume with confirmation payloads. The structures include threadId, runId, interruptId, and arbitrary payloads for proposals, modifications, or approvals, ensuring graceful handling of user responses.

```json
{
  "type": "RUN_FINISHED",
  "threadId": "t1",
  "runId": "r1",
  "outcome": "interrupt",
  "interrupt": {
    "id": "int-abc123",
    "reason": "human_approval",
    "payload": {
      "proposal": {
        "tool": "sendEmail",
        "args": { "to": "a@b.com", "subject": "Hi", "body": "…" }
      }
    }
  }
}

{
  "threadId": "t1",
  "runId": "r2",
  "resume": {
    "interruptId": "int-abc123",
    "payload": { "approved": true }
  }
}

{
  "type": "RUN_FINISHED",
  "threadId": "thread-456",
  "runId": "run-789",
  "outcome": "interrupt",
  "interrupt": {
    "id": "approval-001",
    "reason": "database_modification",
    "payload": {
      "action": "DELETE",
      "table": "users",
      "affectedRows": 42,
      "query": "DELETE FROM users WHERE last_login < '2023-01-01'",
      "rollbackPlan": "Restore from backup snapshot-2025-01-23",
      "riskLevel": "high"
    }
  }
}

{
  "threadId": "thread-456",
  "runId": "run-790",
  "resume": {
    "interruptId": "approval-001",
    "payload": {
      "approved": true,
      "modifications": {
        "batchSize": 10,
        "dryRun": true
      }
    }
  }
}
```

--------------------------------

### Add Integration to Dojo Menu (TypeScript)

Source: https://docs.ag-ui.com/quickstart/middleware

Adds the OpenAI integration to the Dojo menu. This makes the integration visible and accessible to users.

```typescript
  {
    id: "openai",
    name: "OpenAI",
    features: ["agentic_chat"]
  },
```

--------------------------------

### Configure HttpAgent with AgentConfig Interface (TypeScript)

Source: https://docs.ag-ui.com/concepts/agents

Shows how to configure an HttpAgent using the AgentConfig interface, providing options like agentId, description, threadId, initialMessages, and initialState. This allows for customized agent setup during instantiation.

```typescript
interface AgentConfig {
  agentId?: string
  description?: string
  threadId?: string
  initialMessages?: Message[]
  initialState?: State
}

const agent = new HttpAgent({
  agentId: "my-agent-123",
  description: "A helpful assistant",
  threadId: "thread-456",
  initialMessages: [
    { id: "1", role: "system", content: "You are a helpful assistant." },
  ],
  initialState: { preferredLanguage: "English" },
})
```

--------------------------------

### Set OpenAI API Key (Bash)

Source: https://docs.ag-ui.com/quickstart/middleware

Sets the OpenAI API key as an environment variable. This is required for the OpenAI integration.

```bash
export OPENAI_API_KEY=your-api-key-here
```

--------------------------------

### Complete Conversation with Tool Usage - TypeScript

Source: https://docs.ag-ui.com/concepts/messages

Full practical example showing end-to-end conversation flow with tool integration. Demonstrates user query, assistant tool request, tool execution result, and final response. Illustrates the complete tool call lifecycle in AG-UI.

```typescript
// Conversation history
;[
  // User query
  {
    id: "msg_1",
    role: "user",
    content: "What's the weather in New York?",
  },

  // Assistant response with tool call
  {
    id: "msg_2",
    role: "assistant",
    content: "Let me check the weather for you.",
    toolCalls: [
      {
        id: "call_1",
        type: "function",
        function: {
          name: "get_weather",
          arguments: '{"location": "New York", "unit": "celsius"}',
        },
      },
    ],
  },

  // Tool result
  {
    id: "result_1",
    role: "tool",
    content:
      '{"temperature": 22, "condition": "Partly Cloudy", "humidity": 65}',
    toolCallId: "call_1",
  },

  // Assistant's final response using tool results
  {
    id: "msg_3",
    role: "assistant",
    content:
      "The weather in New York is partly cloudy with a temperature of 22°C and 65% humidity.",
  },
]
```

--------------------------------

### Multiple images with question (JSON)

Source: https://docs.ag-ui.com/drafts/multimodal-messages

JSON example demonstrating multiple URL-referenced images with a comparative question. Shows how to reference external binary content.

```json
{
  "id": "msg-003",
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "What are the differences between these images?"
    },
    {
      "type": "binary",
      "mimeType": "image/png",
      "url": "https://example.com/image1.png"
    },
    {
      "type": "binary",
      "mimeType": "image/png",
      "url": "https://example.com/image2.png"
    }
  ]
}
```

--------------------------------

### ReasoningMessageStart Event

Source: https://docs.ag-ui.com/concepts/events

Signals the start of a reasoning message. This draft event is part of the reasoning visibility mechanism.

```APIDOC
## ReasoningMessageStart Event

### Description
Signals the start of a reasoning message. This draft event is part of the reasoning visibility mechanism.

### Method
EVENT

### Endpoint
/reasoning/message/start

### Parameters
#### Request Body
- **messageId** (string) - Required - Unique identifier of the message
- **role** (string) - Required - Role of the reasoning message

### Request Example
{
  "messageId": "msg-789",
  "role": "assistant"
}

### Response
#### Success Response (200)
- **status** (string) - Status of the event processing

#### Response Example
{
  "status": "message started"
}
```

--------------------------------

### ActivityDeltaEvent JSON example

Source: https://docs.ag-ui.com/drafts/activity-events

Sample JSON payload of an ActivityDeltaEvent applying a JSON Patch to replace a task entry.

```JSON
{
  "id": "evt_002",
  "ts": 1714064120000,
  "type": "ACTIVITY_DELTA",
  "messageId": "msg_789",
  "activityType": "PLAN",
  "patch": [
    {
      "op": "replace",
      "path": "/tasks/0",
      "value": "✓ check reddit"
    }
  ]
}
```

--------------------------------

### Define and Pass Frontend Tools in TypeScript

Source: https://docs.ag-ui.com/concepts/tools

This example demonstrates defining a user confirmation tool in the frontend with parameters using JSON Schema and passing it to an agent for execution. It relies on AG-UI's agent framework; inputs are tool definitions, outputs enable dynamic tool calls. Limitations include frontend-controlled permissions and context-dependent tool availability.

```typescript
// Define tools in the frontend
const userConfirmationTool = {
  name: "confirmAction",
  description: "Ask the user to confirm a specific action before proceeding",
  parameters: {
    type: "object",
    properties: {
      action: {
        type: "string",
        description: "The action that needs user confirmation",
      },
      importance: {
        type: "string",
        enum: ["low", "medium", "high", "critical"],
        description: "The importance level of the action",
      },
    },
    required: ["action"],
  },
}

// Pass tools to the agent during execution
agent.runAgent({
  tools: [userConfirmationTool],
  // Other parameters...
})
```

--------------------------------

### Tool Definition and Agent Execution

Source: https://docs.ag-ui.com/concepts/agents

Defines and executes agents with custom tools from the front-end. The example shows how to create a confirmAction tool with parameters for action details and importance level, then pass it to an agent during execution. Enables AI agents to request specific human actions while maintaining context.

```typescript
// Tool definition
const confirmAction = {
  name: "confirmAction",
  description: "Ask the user to confirm a specific action before proceeding",
  parameters: {
    type: "object",
    properties: {
      action: {
        type: "string",
        description: "The action that needs user confirmation",
      },
      importance: {
        type: "string",
        enum: ["low", "medium", "high", "critical"],
        description: "The importance level of the action",
      },
      details: {
        type: "string",
        description: "Additional details about the action",
      },
    },
    required: ["action"],
  },
}

// Running an agent with tools from the frontend
agent.runAgent({
  tools: [confirmAction], // Frontend-defined tools passed to the agent
  // other parameters
})
```

--------------------------------

### Update package.json Scripts for Development

Source: https://docs.ag-ui.com/quickstart/clients

This adds npm scripts for starting, developing, building, and cleaning the TypeScript project using tsx and tsc. It assumes package.json exists and requires tsx for hot-reloading. Outputs updated scripts section; run with npm or pnpm.

```json
{
  // ...

  "scripts": {
    "start": "tsx src/index.ts",
    "dev": "tsx --watch src/index.ts",
    "build": "tsc",
    "clean": "rm -rf dist"
  }

  // ...
}
```

--------------------------------

### Add Integration to Dojo Agents (TypeScript)

Source: https://docs.ag-ui.com/quickstart/middleware

Registers the OpenAI integration with the Dojo agents. This allows the agent to be used within the AG-UI environment.

```typescript
    {
      id: "openai",
      agents: async () => {
        return {
          agentic_chat: new OpenAIAgent(),
        }
      },
    },
```

--------------------------------

### Simple text message (JSON)

Source: https://docs.ag-ui.com/drafts/multimodal-messages

Backward-compatible JSON example showing a traditional text-only user message. Demonstrates the simplest case where content remains a string.

```json
{
  "id": "msg-001",
  "role": "user",
  "content": "What's in this image?"
}
```

--------------------------------

### Audio transcription request (JSON)

Source: https://docs.ag-ui.com/drafts/multimodal-messages

JSON example showing an audio transcription request with ID-referenced content. Demonstrates using pre-uploaded binary content with an identifier.

```json
{
  "id": "msg-004",
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "Please transcribe this audio recording"
    },
    {
      "type": "binary",
      "mimeType": "audio/wav",
      "filename": "meeting-recording.wav",
      "id": "audio-upload-123"
    }
  ]
}
```

--------------------------------

### TypeScript Agent Configuration Interfaces

Source: https://docs.ag-ui.com/sdk/js/client/abstract-agent

Defines interfaces for configuring agents, including the base AgentConfig and an example extension in a subclass. Inputs are configuration objects; outputs are agent instances. Dependencies: Extends base interface for custom options; limitations: Optional fields may lead to default behaviors.

```typescript
interface AgentConfig {
  agentId?: string // The identifier of the agent
  description?: string // A description of the agent, used by the LLM
  threadId?: string // The conversation thread identifier
  initialMessages?: Message[] // An array of initial messages
  initialState?: State // The initial state of the agent
}
```

```typescript
interface MyAgentConfig extends AgentConfig {
  myConfigOption: string
}

class MyAgent extends AbstractAgent {
  private myConfigOption: string

  constructor(config: MyAgentConfig) {
    super(config)
    this.myConfigOption = config.myConfigOption
  }
}
```

--------------------------------

### Create MastraAgent in TypeScript

Source: https://docs.ag-ui.com/quickstart/clients

This TypeScript code defines a MastraAgent wrapped around a Mastra Agent using OpenAI GPT-4o model and LibSQL for persistent memory. It requires all installed dependencies and an OpenAI API key. The agent handles conversational interactions with instructions for friendly responses; limitations include local database file creation.

```typescript
import { openai } from "@ai-sdk/openai"
import { Agent } from "@mastra/core/agent"
import { MastraAgent } from "@ag-ui/mastra"
import { Memory } from "@mastra/memory"
import { LibSQLStore } from "@mastra/libsql"

export const agent = new MastraAgent({
  agent: new Agent({
    name: "AG-UI Assistant",
    instructions: `
      You are a helpful AI assistant. Be friendly, conversational, and helpful.
      Answer questions to the best of your ability and engage in natural conversation.
    `,
    model: openai("gpt-4o"),
    memory: new Memory({
      storage: new LibSQLStore({
        url: "file:./assistant.db",
      }),
    }),
  }),
  threadId: "main-conversation",
})
```

--------------------------------

### Implement OpenAI Streaming Agent in TypeScript

Source: https://docs.ag-ui.com/quickstart/middleware

Complete implementation of a streaming OpenAI agent that extends AbstractAgent. The agent initializes OpenAI client, streams chat completions from GPT-4o, handles tool calls by converting between AG-UI and OpenAI formats, and emits appropriate AG-UI events for each response chunk.

```typescript
// integrations/openai/src/index.ts
import {
  AbstractAgent,
  RunAgentInput,
  EventType,
  BaseEvent,
} from "@ag-ui/client"
import { Observable } from "rxjs"

import { OpenAI } from "openai"

export class OpenAIAgent extends AbstractAgent {
  private openai: OpenAI

  constructor(openai?: OpenAI) {
    super()
    // Initialize OpenAI client - uses OPENAI_API_KEY from environment if not provided
    this.openai = openai ?? new OpenAI()
  }

  run(input: RunAgentInput): Observable<BaseEvent> {
    return new Observable<BaseEvent>((observer) => {
      // Same as before - emit RUN_STARTED to begin
      observer.next({
        type: EventType.RUN_STARTED,
        threadId: input.threadId,
        runId: input.runId,
      } as any)

      // NEW: Instead of hardcoded response, call OpenAI's API
      this.openai.chat.completions
        .create({
          model: "gpt-4o",
          stream: true, // Enable streaming for real-time responses
          // Convert AG-UI tools format to OpenAI's expected format
          tools: input.tools.map((tool) => ({
            type: "function",
            function: {
              name: tool.name,
              description: tool.description,
              parameters: tool.parameters,
            },
          })),
          // Transform AG-UI messages to OpenAI's message format
          messages: input.messages.map((message) => ({
            role: message.role as any,
            content: message.content ?? "",
            // Include tool calls if this is an assistant message with tools
            ...(message.role === "assistant" && message.toolCalls
              ? {
                  tool_calls: message.toolCalls,
                }
              : {}),
            // Include tool call ID if this is a tool result message
            ...(message.role === "tool"
              ? { tool_call_id: message.toolCallId }
              : {}),
          })),
        })
        .then(async (response) => {
          const messageId = Date.now().toString()

          // NEW: Stream each chunk from OpenAI's response
          for await (const chunk of response) {
            // Handle text content chunks
            if (chunk.choices[0].delta.content) {
              observer.next({
                type: EventType.TEXT_MESSAGE_CHUNK, // Chunk events open and close messages automatically
                messageId,
                delta: chunk.choices[0].delta.content,
              } as any)
            }
            // Handle tool call chunks (when the model wants to use a function)
            else if (chunk.choices[0].delta.tool_calls) {
              let toolCall = chunk.choices[0].delta.tool_calls[0]

              observer.next({
                type: EventType.TOOL_CALL_CHUNK,
                toolCallId: toolCall.id,
                toolCallName: toolCall.function?.name,
                parentMessageId: messageId,
                delta: toolCall.function?.arguments,
              } as any)
            }
          }

          // Same as before - emit RUN_FINISHED when complete
          observer.next({
            type: EventType.RUN_FINISHED,
            threadId: input.threadId,
            runId: input.runId,
          } as any)

          observer.complete()
        })
        // NEW: Handle errors from the API
        .catch((error) => {
          observer.next({
            type: EventType.RUN_ERROR,
            message: error.message,
          } as any)

          observer.error(error)
        })
    })
  }
}
}
```

--------------------------------

### Implement Stub OpenAIAgent in TypeScript

Source: https://docs.ag-ui.com/quickstart/middleware

Extends AbstractAgent to simulate an OpenAI agent by emitting a sequence of events for run lifecycle and text message streaming. Depends on @ag-ui/client and RxJS Observable. Inputs: RunAgentInput; Outputs: Observable of BaseEvent. Limitations: Currently stubs 'Hello world!' response without real OpenAI calls.

```ts
// integrations/openai/src/index.ts
import {
  AbstractAgent,
  BaseEvent,
  EventType,
  RunAgentInput,
} from "@ag-ui/client"
import { Observable } from "rxjs"

export class OpenAIAgent extends AbstractAgent {
  run(input: RunAgentInput): Observable<BaseEvent> {
    const messageId = Date.now().toString()
    return new Observable<BaseEvent>((observer) => {
      observer.next({
        type: EventType.RUN_STARTED,
        threadId: input.threadId,
        runId: input.runId,
      } as any)

      observer.next({
        type: EventType.TEXT_MESSAGE_START,
        messageId,
      } as any)

      observer.next({
        type: EventType.TEXT_MESSAGE_CONTENT,
        messageId,
        delta: "Hello world!",
      } as any)

      observer.next({
        type: EventType.TEXT_MESSAGE_END,
        messageId,
      } as any)

      observer.next({
        type: EventType.RUN_FINISHED,
        threadId: input.threadId,
        runId: input.runId,
      } as any)

      observer.complete()
    })
  }
}
```

--------------------------------

### Add Startup Script to package.json (JSON)

Source: https://docs.ag-ui.com/quickstart/clients

Adds the entry point for the CLI application in the package.json file.

```json
{
  "bin": {
    "weather-assistant": "./dist/index.js"
  }
}
```

--------------------------------

### Shared State Proposal Structure in JSON

Source: https://docs.ag-ui.com/concepts/state

This JSON snippet illustrates a sample state proposal for an agent action, such as sending an email. It serves as an example of how state objects can encode proposals for user review and modification. No dependencies are required as it's plain JSON data; input is application state, output is the proposal object.

```json
{
  "proposal": {
    "action": "send_email",
    "recipient": "client@example.com",
    "content": "Draft email content..."
  }
}
```

--------------------------------

### Update CLI for Tool Event Handling (TypeScript)

Source: https://docs.ag-ui.com/quickstart/clients

This code adds event handlers to the agent.runAgent call in src/index.ts to log tool calls, arguments, and results. It depends on the existing agent setup. Inputs are the agent configuration; outputs are console logs for tool interactions. Limitations: Assumes stdout and console.log are available in the environment.

```typescript
// Add these new event handlers to your agent.runAgent call:
await agent.runAgent(
  {}, // No additional configuration needed
  {
    // ... existing event handlers ...

    onToolCallStartEvent({ event }) {
      console.log("🔧 Tool call:", event.toolCallName)
    },
    onToolCallArgsEvent({ event }) {
      process.stdout.write(event.delta)
    },
    onToolCallEndEvent() {
      console.log("")
    },
    onToolCallResultEvent({ event }) {
      if (event.content) {
        console.log("🔍 Tool call result:", event.content)
      }
    },
  }
)
```

--------------------------------

### Initialize AG-UI Client Project in Bash

Source: https://docs.ag-ui.com/quickstart/clients

These commands create a new directory and initialize a Node.js project using pnpm. They set up the basic project structure without additional dependencies. Inputs are directory name and package details; outputs include created files like package.json.

```bash
mkdir my-ag-ui-client
cd my-ag-ui-client

pnpm init
```

--------------------------------

### JSON Patch Remove Operation Example

Source: https://docs.ag-ui.com/concepts/state

Example of a JSON Patch remove operation that removes a value from the state. This operation would remove temporary data from the state.

```json
{ "op": "remove", "path": "/temporary_data" }
```

--------------------------------

### JSON Patch Replace Operation Example

Source: https://docs.ag-ui.com/concepts/state

Example of a JSON Patch replace operation that replaces a value in the state. This operation would update the conversation state to 'paused'.

```json
{ "op": "replace", "path": "/conversation_state", "value": "paused" }
```

--------------------------------

### Create Tools Directory (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

This command sets up the initial directory structure for storing tools in the project. It requires no additional dependencies and takes no inputs. The output is the creation of the src/tools directory if it does not exist. There are no known limitations.

```bash
mkdir -p src/tools
```

--------------------------------

### JSON Patch Move Operation Example

Source: https://docs.ag-ui.com/concepts/state

Example of a JSON Patch move operation that moves a value from one location to another. This operation would move the first item from pending_items to completed_items.

```json
{ "op": "move", "path": "/completed_items", "from": "/pending_items/0" }
```

--------------------------------

### JSON Patch Add Operation Example

Source: https://docs.ag-ui.com/concepts/state

Example of a JSON Patch add operation that adds a value to an object or array. This operation would add a theme preference to the user's preferences object.

```json
{ "op": "add", "path": "/user/preferences", "value": { "theme": "dark" } }
```

--------------------------------

### Create Production Build (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

This command creates a production build of the application using pnpm.

```bash
pnpm build
```

--------------------------------

### Make Executable (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

Makes the compiled JavaScript file executable using chmod.

```bash
chmod +x dist/index.js
```

--------------------------------

### Define TextMessageStartEvent in TypeScript

Source: https://docs.ag-ui.com/concepts/messages

Defines the type for an event indicating the start of a new text message.

```typescript
interface TextMessageStartEvent {
  type: EventType.TEXT_MESSAGE_START
  messageId: string
  role: string
}
```

--------------------------------

### Link Globally (Bash)

Source: https://docs.ag-ui.com/quickstart/clients

Links the CLI application globally using pnpm.

```bash
pnpm link --global
```

--------------------------------

### onTextMessageStartEvent in TypeScript

Source: https://docs.ag-ui.com/sdk/js/client/subscriber

Triggered when a text message generation starts. Provides event and agent subscriber parameters for state manipulation.

```typescript
onTextMessageStartEvent?(params: { event: TextMessageStartEvent } & AgentSubscriberParams): MaybePromise<AgentStateMutation | void>
```

--------------------------------

### Run an HttpAgent with Messages and Event Handling (TypeScript)

Source: https://docs.ag-ui.com/concepts/agents

Demonstrates how to instantiate an HttpAgent, set initial messages, and run the agent with specified parameters. It includes handling different event types emitted during the agent's execution, such as text message content. Dependencies include HttpAgent and EventType.

```typescript
const agent = new HttpAgent({
  url: "https://your-agent-endpoint.com/agent",
})

agent.messages = [
  {
    id: "1",
    role: "user",
    content: "Hello, how can you help me today?",
  },
]

agent
  .runAgent({
    runId: "run_123",
    tools: [], // Optional tools
    context: [], // Optional context
  })
  .subscribe({
    next: (event) => {
      switch (event.type) {
        case EventType.TEXT_MESSAGE_CONTENT:
          console.log("Content:", event.delta)
          break
      }
    },
    error: (error) => console.error("Error:", error),
    complete: () => console.log("Run complete"),
  })
```

--------------------------------

### Update Agent with Tools (TypeScript)

Source: https://docs.ag-ui.com/quickstart/clients

Updates the agent to include both the weather and browser tools. Defines the agent's instructions and sets up the OpenAI model.

```typescript
import { openai } from "@ai-sdk/openai"
import { Agent } from "@mastra/core/agent"
import { MastraAgent } from "@ag-ui/mastra"
import { Memory } from "@mastra/memory"
import { LibSQLStore } from "@mastra/libsql"
import { weatherTool } from "./tools/weather.tool"
import { browserTool } from "./tools/browser.tool"

export const agent = new MastraAgent({
  agent: new Agent({
    name: "AG-UI Assistant",
    instructions: `
      You are a helpful assistant with weather and web browsing capabilities.

      For weather queries:
      - Always ask for a location if none is provided
      - Use the weatherTool to fetch current weather data

      For web browsing:
      - Always use full URLs (e.g., "https://www.google.com")
      - Use the browserTool to open web pages

      Be friendly and helpful in all interactions!
    `,
    model: openai("gpt-4o"),
    tools: { weatherTool, browserTool }, // Add both tools
    memory: new Memory({
      storage: new LibSQLStore({
        url: "file:./assistant.db",
      }),
    }),
  }),
  threadId: "main-conversation",
})
```

--------------------------------

### Browser Tool Implementation (TypeScript)

Source: https://docs.ag-ui.com/quickstart/clients

Defines a browser tool that opens a URL in the default web browser using the 'open' package. Includes Zod schema validation for input and output.

```typescript
import { createTool } from "@mastra/core/tools"
import { z } from "zod"
import { open } from "open"

export const browserTool = createTool({
  id: "open-browser",
  description: "Open a URL in the default web browser",
  inputSchema: z.object({
    url: z.string().url().describe("The URL to open"),
  }),
  outputSchema: z.object({
    success: z.boolean(),
    message: z.string(),
  }),
  execute: async ({ context }) => {
    try {
      await open(context.url)
      return {
        success: true,
        message: `Opened ${context.url} in your default browser`,
      }
    } catch (error) {
      return {
        success: false,
        message: `Failed to open browser: ${error}`,
      }
    }
  },
})
```

--------------------------------

### Update Agent Class Name (TypeScript)

Source: https://docs.ag-ui.com/quickstart/middleware

Changes the class name within `index.ts` to `OpenAIAgent`. This customizes the agent's identity.

```typescript
export class OpenAIAgent extends AbstractAgent {}
```

--------------------------------

### Event Compaction Before and After

Source: https://docs.ag-ui.com/drafts/serialization

Shows the transformation of verbose event logs into compacted snapshots for messages and state. Helps reduce storage size and improve load times.

```typescript
;[
  { type: "TEXT_MESSAGE_START", messageId: "msg1", role: "user" },
  { type: "TEXT_MESSAGE_CONTENT", messageId: "msg1", delta: "Hello " },
  { type: "TEXT_MESSAGE_CONTENT", messageId: "msg1", delta: "world" },
  { type: "TEXT_MESSAGE_END", messageId: "msg1" },
  { type: "STATE_DELTA", patch: { op: "add", path: "/foo", value: 1 } },
  { type: "STATE_DELTA", patch: { op: "replace", path: "/foo", value: 2 } },
]
```

```typescript
;[
  {
    type: "MESSAGES_SNAPSHOT",
    messages: [{ id: "msg1", role: "user", content: "Hello world" }],
  },
  {
    type: "STATE_SNAPSHOT",
    state: { foo: 2 },
  },
]
```

--------------------------------

### Document analysis request (JSON)

Source: https://docs.ag-ui.com/drafts/multimodal-messages

JSON example for document analysis with URL-referenced PDF. Shows how to request processing of external document files with metadata.

```json
{
  "id": "msg-005",
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "Summarize the key points from this PDF"
    },
    {
      "type": "binary",
      "mimeType": "application/pdf",
      "filename": "quarterly-report.pdf",
      "url": "https://example.com/reports/q4-2024.pdf"
    }
  ]
}
```