Generative UI for Agentic Apps (copilotkit/generative-ui)

Generative UI for Agentic Apps

https://github.com/copilotkit/generative-ui
Admin
Generative UI enables building adaptive applications where AI agents generate, select, or control...

Tokens:7,990
Snippets:50
Trust Score:8.4
Update:1 week ago
Show doc for...
Context Summary (auto-generated)
Raw
# Generative UI for Agentic Apps (CopilotKit)

This repository is a comprehensive guide and reference for implementing Generative UI patterns in agentic applications using CopilotKit and its associated protocols (AG-UI, A2UI, MCP Apps). Generative UI is a paradigm in which AI agents dynamically select, generate, or control parts of the user interface at runtime rather than relying on fully static, developer-predefined screens. The project documents three distinct patterns ranging from highly controlled (developer owns all UI, agent picks what to show) to fully open-ended (agent generates complete HTML surfaces), giving teams a clear framework for choosing the right approach based on their control-vs-freedom trade-offs.

The core functionality revolves around four concrete implementation patterns built on top of the CopilotKit ecosystem: **Controlled Generative UI** via `useFrontendTool` (AG-UI), **Declarative Generative UI** via A2UI and Open-JSON-UI specifications, **Open-ended Generative UI** via MCP Apps middleware, and **Open Generative UI** via `useComponent`. All four patterns share AG-UI as the bidirectional runtime interaction layer that connects agents to frontend applications, providing the agent ↔ application communication channel uniformly across all specifications. The repository is now consolidated into the [CopilotKit monorepo](https://github.com/CopilotKit/CopilotKit) at `examples/showcases/generative-ui`.

---

## APIs and Key Functions

### `useFrontendTool` — Register a Controlled Generative UI tool with lifecycle-aware rendering

`useFrontendTool` is a CopilotKit React hook that registers a named tool the AI agent can call. The developer pre-builds all UI components; the agent decides when to invoke the tool and supplies the data. The `render` callback receives the current execution `status` (`inProgress`, `executing`, `complete`) so the component can show loading states, executing states, and final results as the tool runs — giving full layout and styling ownership to the developer while the agent drives what appears and when.

```typescript
import { useFrontendTool } from "@copilotkit/react-core";
import { z } from "zod";

interface WeatherData {
  location: string;
  temperature: number;
  conditions: string;
  humidity: number;
  windSpeed: number;
}

function getMockWeather(location: string): WeatherData {
  return {
    location,
    temperature: 22,
    conditions: "Partly Cloudy",
    humidity: 58,
    windSpeed: 14,
  };
}

function WeatherCard({ location, temperature, conditions, humidity, windSpeed }: WeatherData) {
  return (
    <div className="weather-card">
      <h3>{location}</h3>
      <p>{temperature}°C — {conditions}</p>
      <p>Humidity: {humidity}% | Wind: {windSpeed} km/h</p>
    </div>
  );
}

function WeatherLoadingState({ location }: { location?: string }) {
  return <div className="loading">Fetching weather for {location ?? "…"}…</div>;
}

// Inside a React component:
useFrontendTool({
  name: "get_weather",
  description: "Get current weather information for a location",
  parameters: z.object({
    location: z.string().describe("The city or location to get weather for"),
  }),
  handler: async ({ location }) => {
    // Simulate async fetch; replace with a real weather API call
    await new Promise((r) => setTimeout(r, 500));
    return getMockWeather(location);
  },
  render: ({ status, args, result }) => {
    if (status === "inProgress" || status === "executing") {
      return <WeatherLoadingState location={args?.location} />;
    }
    if (status === "complete" && result) {
      const data = JSON.parse(result) as WeatherData;
      return (
        <WeatherCard
          location={data.location}
          temperature={data.temperature}
          conditions={data.conditions}
          humidity={data.humidity}
          windSpeed={data.windSpeed}
        />
      );
    }
    return <></>;
  },
});
// When the agent sends: { tool: "get_weather", args: { location: "Tokyo" } }
// → Loading card appears immediately, then WeatherCard renders on completion
```

---

### A2UI Agent — Configure a Python LLM agent to emit declarative A2UI JSONL UI specs

A2UI (from Google) is a JSONL-based, streaming declarative UI specification designed for platform-agnostic rendering. To wire it up on the agent side, inject a few-shot A2UI example into the agent's system prompt so the model learns the three required message envelopes: `surfaceUpdate` (component tree), `dataModelUpdate` (state/data bindings), and `beginRendering` (render trigger). The `get_ui_prompt` helper composes the final prompt string that tells the agent which base URL to use and what A2UI JSONL format to output.

```python
# prompt_builder.py
from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm

LITELLM_MODEL = "openai/gpt-4o"

AGENT_INSTRUCTION = (
    "You are a helpful assistant. When the user asks for a form or structured UI, "
    "respond with valid A2UI JSONL messages."
)

# One-shot example teaching the agent the three required A2UI envelopes
UI_EXAMPLES = """
---BEGIN FORM_EXAMPLE---
{"surfaceUpdate":{"surfaceId":"contact-form","components":[{"id":"form-column","type":"column","children":["name-field","email-field","submit-btn"]},{"id":"name-field","type":"textInput","label":"Full Name","binding":"/name"},{"id":"email-field","type":"textInput","label":"Email","binding":"/email"},{"id":"submit-btn","type":"button","label":"Submit","action":"submit"}]}}
{"dataModelUpdate":{"surfaceId":"contact-form","path":"/","contents":[{"key":"name","value":""},{"key":"email","value":""}]}}
{"beginRendering":{"surfaceId":"contact-form","root":"form-column","styles":{"theme":"light","spacing":"comfortable"}}}
---END FORM_EXAMPLE---
"""

def get_ui_prompt(base_url: str, examples: str) -> str:
    return (
        f"\n\nBase URL for surface actions: {base_url}\n"
        f"When generating UI, output valid A2UI JSONL lines exactly as shown:\n{examples}"
    )

def build_agent(base_url: str) -> LlmAgent:
    instruction = AGENT_INSTRUCTION + get_ui_prompt(base_url, UI_EXAMPLES)
    return LlmAgent(
        model=LiteLlm(model=LITELLM_MODEL),
        name="ui_generator_agent",
        description="Generates dynamic UI via A2UI declarative JSON.",
        instruction=instruction,
        tools=[],
    )

# Usage:
agent = build_agent(base_url="https://myapp.example.com")
# The agent will now emit A2UI JSONL when asked for forms/cards/lists
```

---

### `createA2UIMessageRenderer` — Render streamed A2UI output as interactive UI on the frontend

`createA2UIMessageRenderer` creates a CopilotKit activity-message renderer that intercepts streamed A2UI JSONL lines from the agent and renders them as live React components. Passing it into the `renderActivityMessages` prop of `CopilotKitProvider` connects the declarative agent output to the visual layer, and any user interactions (button clicks, form submissions) are automatically forwarded back to the agent as tool responses.

```typescript
// app/a2ui-page.tsx
import { CopilotKitProvider, CopilotSidebar } from "@copilotkit/react";
import { createA2UIMessageRenderer } from "@copilotkit/a2ui-renderer";

// Define a theme to apply consistent styling to all A2UI surfaces
const a2uiTheme = {
  colors: { primary: "#6963ff", background: "#ffffff", text: "#1a1a2e" },
  spacing: { unit: 8 },
  borderRadius: 8,
};

// Create the renderer once — it is stateless and reusable
const A2UIRenderer = createA2UIMessageRenderer({ theme: a2uiTheme });

export function A2UIPage({ children }: { children: React.ReactNode }) {
  return (
    <CopilotKitProvider
      runtimeUrl="/api/copilotkit-a2ui"          // Points to your Next.js API route
      renderActivityMessages={[A2UIRenderer]}     // Intercepts A2UI JSONL activity messages
    >
      {children}
      <CopilotSidebar
        defaultOpen
        labels={{ modalHeaderTitle: "A2UI Assistant" }}
      />
    </CopilotKitProvider>
  );
}

// When the agent streams:
// {"surfaceUpdate":{"surfaceId":"s1","components":[...]}}
// {"dataModelUpdate":{"surfaceId":"s1","path":"/","contents":[...]}}
// {"beginRendering":{"surfaceId":"s1","root":"root-col","styles":{}}}
// → CopilotKit renders the described form/card live in the sidebar
```

---

### Open-JSON-UI Agent Response — Emit a declarative card payload from the agent

Open-JSON-UI is the open standardisation of OpenAI's internal declarative Generative UI schema. The agent returns a structured JSON object describing UI components (cards, lists, forms), and the frontend renders it. No special renderer setup is required beyond handling the typed payload — the schema is self-describing.

```javascript
// Example agent response payload (Open-JSON-UI specification)
// The agent returns this JSON as a tool result or message content:
const openJsonUiPayload = {
  type: "open-json-ui",
  spec: {
    components: [
      {
        type: "card",
        properties: {
          title: "Q3 Sales Summary",
          subtitle: "July – September 2025",
          content: {
            type: "stat-grid",
            stats: [
              { label: "Total Revenue", value: "$1.24M", trend: "+12%" },
              { label: "New Customers", value: "3,847", trend: "+8%" },
              { label: "Churn Rate", value: "2.1%", trend: "-0.3%" },
            ],
          },
          actions: [
            { label: "Download Report", action: "download_report" },
            { label: "View Details", action: "open_details" },
          ],
        },
      },
    ],
  },
};

// Frontend rendering handler:
function handleAgentMessage(message) {
  if (message.type === "open-json-ui") {
    message.spec.components.forEach((component) => {
      if (component.type === "card") {
        renderCard(component.properties);
        // → Displays a styled card with title, stats, and action buttons
      }
    });
  }
}
```

---

### `MCPAppsMiddleware` — Attach MCP Apps servers to a CopilotKit agent for open-ended UI

`MCPAppsMiddleware` connects a `BuiltInAgent` (or any AG-UI compatible agent) to one or more external MCP (Model Context Protocol) Apps servers. When the agent invokes a tool on the MCP server (e.g., `create_view`), the server returns an iframe URL or full HTML surface, and CopilotKit renders it directly in the chat interface. This pattern requires the least frontend code — the server does the heavy lifting of generating the UI surface.

```typescript
// src/app/api/copilotkit/route.ts  (Next.js App Router)
import {
  CopilotRuntime,
  ExperimentalEmptyAdapter,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { BuiltInAgent } from "@copilotkit/runtime/v2";
import { MCPAppsMiddleware } from "@ag-ui/mcp-apps-middleware";
import { NextRequest } from "next/server";

// Build the agent and attach MCP Apps middleware
const agent = new BuiltInAgent({
  model: "openai/gpt-4o",
  prompt:
    "You are an AI diagramming assistant. When the user describes a diagram, " +
    "call create_view on the Excalidraw MCP server with the Excalidraw element array.",
}).use(
  new MCPAppsMiddleware({
    mcpServers: [
      {
        type: "http",
        // Use environment variable so local dev and production can differ
        url: process.env.MCP_SERVER_URL ?? "http://localhost:3001/mcp",
        serverId: "excalidraw",
      },
    ],
  }),
);

const runtime = new CopilotRuntime({
  agents: { default: agent },
});

// Export the POST handler for the Next.js App Router
export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter: new ExperimentalEmptyAdapter(),
    endpoint: "/api/copilotkit",
  });
  return handleRequest(req);
};

// Result: user types "draw a flowchart for user login"
// → agent calls excalidraw MCP create_view tool
// → CopilotKit renders the returned iframe in the chat sidebar
// → user can edit the diagram live without leaving the chat
```

---

### `useComponent` — Register a named frontend component the agent can invoke directly

`useComponent` is the `useFrontendTool` counterpart from `@copilotkit/react-core/v2` designed for rich, self-contained HTML/SVG/Canvas output. Instead of querying an external MCP server, the agent generates full HTML content as a string and passes it as a tool call argument. The frontend `render` function receives the structured output and displays it in a sandboxed iframe — no server round-trip, no external URL.

```typescript
// components/widget-host.tsx
import { useComponent } from "@copilotkit/react-core/v2";
import { z } from "zod";

// Schema for the agent's tool call payload
const WidgetRendererProps = z.object({
  title: z.string().describe("Short title describing the visualization"),
  description: z.string().describe("One-sentence description of what is shown"),
  html: z.string().describe(
    "Complete self-contained HTML document (may include inline CSS/JS, " +
    "Three.js CDN, D3, etc.) to render inside a sandboxed iframe"
  ),
});

type WidgetProps = z.infer<typeof WidgetRendererProps>;

// The React component that renders the agent's HTML in a sandboxed iframe
function WidgetRenderer({ title, description, html }: WidgetProps) {
  const blob = new Blob([html], { type: "text/html" });
  const src = URL.createObjectURL(blob);
  return (
    <div className="widget-container">
      <h4>{title}</h4>
      <p>{description}</p>
      <iframe
        src={src}
        sandbox="allow-scripts"   // No allow-same-origin → full isolation
        style={{ width: "100%", height: 400, border: "none" }}
        title={title}
      />
    </div>
  );
}

// Register the component inside your React app — the agent calls "widgetRenderer"
function App() {
  useComponent({
    name: "widgetRenderer",
    description:
      "Renders interactive HTML/SVG visualizations in a sandboxed iframe. " +
      "Use for algorithm visualizations, D3 charts, Three.js 3D scenes, simulations.",
    parameters: WidgetRendererProps,
    render: WidgetRenderer,
  });

  return <YourAppShell />;
}

// When user says "show me a bubble sort visualization"
// Agent emits: { tool: "widgetRenderer", args: { title: "Bubble Sort", description: "...", html: "<html>...</html>" } }
// → WidgetRenderer receives the payload and mounts a sandboxed iframe instantly
```

---

## Summary

Generative UI with CopilotKit addresses a spectrum of real-world agentic use cases. **Controlled Generative UI** (`useFrontendTool`) is ideal for production dashboards and customer-facing apps where brand consistency and accessibility matter — developers define every pixel, the agent drives data and timing. **Declarative Generative UI** (A2UI / Open-JSON-UI) fits internal tools, data apps, and form-heavy workflows where the agent needs to compose novel layouts from a shared component vocabulary without the developer needing to anticipate every possible UI in advance. **Open-ended Generative UI** (MCP Apps and `useComponent`) serves power-user environments like AI coding assistants, diagramming tools, data science notebooks, and interactive simulations where the richest possible output matters more than strict visual consistency.

Integration across all four patterns follows the same architectural spine: an AG-UI compatible runtime (CopilotKit Runtime) handles the bidirectional agent ↔ frontend protocol, a service adapter connects to the underlying LLM, and the chosen Generative UI mechanism (hook, renderer, or middleware) slots in as a thin layer on top. This means teams can start with the most controlled pattern (`useFrontendTool`) and progressively adopt more open-ended patterns (`MCPAppsMiddleware`, `useComponent`) for specific features without replacing their existing infrastructure. The [Generative UI Playground](https://go.copilotkit.ai/gen-ui-demo) and the [CopilotKit monorepo](https://github.com/CopilotKit/CopilotKit) at `examples/showcases/generative-ui` provide runnable end-to-end references for all patterns.