### Setup WebRTC Connection and Communication

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Example function demonstrating how to establish a WebRTC connection, including adding local tracks, handling remote tracks, creating a data channel for text communication, and exchanging SDP offers/answers with the server. It also shows how to send ICE candidates as they are gathered.

```typescript
// Example: Setting up WebRTC connection
import { webrtcOffer } from '@/apis'

async function setupWebRTC(
  stream: MediaStream,
  peerConnection: RTCPeerConnection,
  remoteVideoElement: HTMLVideoElement
) {
  // Add local tracks to peer connection
  stream.getTracks().forEach((track) => {
    peerConnection.addTrack(track, stream)
  })

  // Handle incoming remote tracks
  peerConnection.addEventListener('track', (evt) => {
    if (remoteVideoElement.srcObject !== evt.streams[0]) {
      remoteVideoElement.srcObject = evt.streams[0]
    }
  })

  // Create data channel for text communication
  const dataChannel = peerConnection.createDataChannel('text')

  // Create and send offer
  const offer = await peerConnection.createOffer()
  await peerConnection.setLocalDescription(offer)

  const webrtc_id = Math.random().toString(36).substring(7)

  // Send ICE candidates as they're gathered
  peerConnection.onicecandidate = ({ candidate }) => {
    if (candidate) {
      webrtcOffer({
        candidate: candidate.toJSON(),
        webrtc_id,
        type: 'ice-candidate',
      })
    }
  }

  // Send offer and receive answer
  const response = await webrtcOffer({
    sdp: offer.sdp!,
    type: offer.type,
    webrtc_id,
  })

  const serverResponse = await response.json()
  await peerConnection.setRemoteDescription(serverResponse)

  return [dataChannel, webrtc_id]
}
```

--------------------------------

### Environment Configuration (.env) Examples

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Examples of .env file configurations for connecting to local or remote OpenAvatarChat backend services. Ensure VITE_USE_SSL is set correctly based on your deployment.

```env
VITE_SERVER_IP=127.0.0.1
VITE_SERVER_PORT=8282
VITE_USE_SSL=false
```

```env
VITE_SERVER_IP=your-server-ip
VITE_SERVER_PORT=8282
VITE_USE_SSL=true
```

--------------------------------

### Update Frontend Instructions

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Commands to update the frontend by installing dependencies, building the project, and copying the output to the backend's static assets directory.

```bash
# 在 WebUI 项目中
pnpm install
pnpm run build
# 将 dist/ 目录内容复制到 OpenAvatarChat/src/service/frontend_service/frontend/dist/
```

--------------------------------

### Development Mode: Independent Frontend

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Commands to install dependencies, configure environment variables, and start the development server for an independently deployed frontend. Vite's proxy handles cross-origin requests during development.

```bash
# 安装依赖
pnpm install

# 配置 .env 指向后端服务地址
# VITE_SERVER_IP=your-backend-ip
# VITE_SERVER_PORT=8282

# 启动开发服务器（支持HMR热更新）
pnpm run dev
```

--------------------------------

### Fetch and Apply Initialization Configuration

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Example of how to fetch initialization configuration from the backend and apply it to the application's state. This involves setting up RTC configuration, chat mode, avatar settings, and media track constraints.

```typescript
// Example: Fetching and applying init configuration
import { initConfig, makeURL } from '@/apis'

const appStore = useAppStore()
const mediaStore = useMediaStore()

initConfig()
  .then((res) => res.json())
  .then((config) => {
    if (config.detail) {
      console.error('Init failed:', config.detail)
      return
    }

    // Apply RTC configuration
    if (config.rtc_configuration) {
      appStore.rtcConfig = config.rtc_configuration
    }

    // Set chat mode (webrtc or ws)
    appStore.chatMode = config.chat_mode === 'ws' ? 'ws' : 'webrtc'

    // Configure avatar settings
    if (config.avatar_config) {
      appStore.avatarType = config.avatar_config.avatar_type || ''
      appStore.avatarWSRoute = config.avatar_config.avatar_ws_route || ''
      appStore.avatarAssetsPath = config.avatar_config.avatar_assets_path
        ? makeURL(config.avatar_config.avatar_assets_path)
        : ''
    }

    // Apply media track constraints
    if (config.track_constraints) {
      mediaStore.setTrackConstraints(config.track_constraints)
    }
  })
```

--------------------------------

### Use WebSocket Chat Store for Sessions

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Demonstrates initializing the app, starting a WebSocket session, sending text messages, interrupting avatar responses, and monitoring session state. Ensure the chat mode is set to 'ws' before starting a session.

```typescript
// Example: Using the WebSocket chat store
import { useWSVideoChatStore } from '@/store/ws'
import { useAppStore } from '@/store/app'

const wsStore = useWSVideoChatStore()
const appStore = useAppStore()

// Initialize app configuration
await appStore.init()

// Ensure chat mode is set to WebSocket
if (appStore.chatMode === 'ws') {
  // Start WebSocket session
  await wsStore.startSession()

  // Send text message
  wsStore.sendText('What is the weather today?')

  // Interrupt avatar response
  wsStore.interrupt()

  // Monitor session state
  watch(() => wsStore.streamState, (state) => {
    if (state === 'open') {
      console.log('WebSocket session active:', wsStore.sessionId)
    }
  })

  // Stop session
  await wsStore.startSession() // Toggle off when already connected
}
```

--------------------------------

### Extending with New Digital Human Renderers

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Step-by-step guide on how to integrate custom digital human rendering engines.

```APIDOC
## How to Extend with New Digital Human Renderers

To integrate a custom digital human rendering engine (e.g., Live2D, 3D models), follow these steps:

### Step 1: Create a Renderer Class

Create a new renderer file in `src/renderer/src/handlers/avatarRenderers/`.

```typescript
// Example: src/renderer/src/handlers/avatarRenderers/custom.ts
export class CustomRenderer {
  container: HTMLDivElement
  assetsPath: string

  constructor(options: {
    container: HTMLDivElement
    assetsPath: string
    downloadProgress?: (percent: number) => void
    loadProgress?: (percent: number) => void
  }) {
    this.container = options.container
    this.assetsPath = options.assetsPath
    // Initialize your rendering engine...
  }

  // Update expression/motion data
  updateExpression(data: any): void {
    // Apply motion data to your rendering model
  }

  // Destroy the renderer
  destroy(): void {
    // Clean up resources
  }
}
```

### Step 2: Integrate into AvatarHandler

Modify `src/renderer/src/handlers/avatarHandler.ts`.

First, extend the `rendererType` type definition to support the new renderer type:

```typescript
// Extend rendererType in AvatarHandlerOptions interface
interface AvatarHandlerOptions {
  // ...
  rendererType: 'lam' | '' | 'custom'  // Add your renderer type identifier
}
```

Then, add your renderer type branch in the `render()` method:

```typescript
async render(): Promise<void> {
  if (this._rendererType === 'lam') {
    // ... existing LAM renderer logic
  } else if (this._rendererType === 'custom') {
    this._renderer = new CustomRenderer({
      container: this._avatarDivEle,
      assetsPath: this._assetsPath,
      downloadProgress: this._downloadProgress,
      loadProgress: this._loadProgress,
    })
  } else {
    this._renderer = null
  }
}
```

### Step 3: Handle Motion Data

Adapt your data format in `_handleBinaryMessage()` or `Processor` to convert motion data sent by the backend into a format understandable by your renderer.

### Step 4: Register in Store

Register your Handler using `chatStore.setActiveRenderer()` in `useVideoChatStore` or `useWSVideoChatStore`:

```typescript
// In the Store's initAvatarHandler() method
const handler = new AvatarHandler({
  container: visionStore.remoteVideoContainerRef!,
  assetsPath: appStore.avatarAssetsPath,
  ws: ws,
  rendererType: 'custom',  // Your renderer type
})
chatStore.setActiveRenderer(handler)
chatStore.bindAvatarHandler(handler)
```
```

--------------------------------

### WebRTC Video Chat Store Usage Example

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Demonstrates how to use the `useVideoChatStore` Pinia store for managing WebRTC video chat sessions. This includes initializing the app, accessing media devices, starting/stopping WebRTC, sending messages, and monitoring loading progress.

```typescript
// Example: Using the video chat store for WebRTC mode
import { useVideoChatStore } from '@/store/webrtc'
import { useAppStore } from '@/store/app'
import { useChatStore } from '@/store/chat'
import { useMediaStore } from '@/store/media'

const videoChatStore = useVideoChatStore()
const appStore = useAppStore()
const chatStore = useChatStore()
const mediaStore = useMediaStore()

// Initialize application
await appStore.init()

// Access camera/microphone
await mediaStore.accessDevice()

// Start WebRTC session
await videoChatStore.startWebRTC()

// Check connection state
if (videoChatStore.streamState === 'open') {
  console.log('Connected! WebRTC ID:', videoChatStore.webRTCId)
}

// Send text message
videoChatStore.sendText('Hello, avatar!')

// Interrupt avatar response
videoChatStore.interrupt()

// Monitor loading progress for LAM renderer
watch(() => videoChatStore.gsLoadPercent, (percent) => {
  console.log(`Avatar model loading: ${percent}%`)
})

// Toggle volume mute
chatStore.handleVolumeMute()

// Toggle subtitle/chat records display
chatStore.handleSubtitleToggle()

// Stop session
await videoChatStore.startWebRTC() // Toggle off when already connected
```

--------------------------------

### Custom Renderer Class Example

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Example of how to create a custom renderer class for integrating new digital human engines. This class should handle initialization, updating expressions, and cleanup.

```typescript
// 例如：src/renderer/src/handlers/avatarRenderers/custom.ts
export class CustomRenderer {
  container: HTMLDivElement
  assetsPath: string

  constructor(options: {
    container: HTMLDivElement
    assetsPath: string
    downloadProgress?: (percent: number) => void
    loadProgress?: (percent: number) => void
  }) {
    this.container = options.container
    this.assetsPath = options.assetsPath
    // 初始化你的渲染引擎...
  }

  // 更新表情/动作数据
  updateExpression(data: any): void {
    // 将运动数据应用到你的渲染模型
  }

  // 销毁渲染器
  destroy(): void {
    // 清理资源
  }
}
```

--------------------------------

### Example: Initialize and Use AvatarHandler

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Illustrates how to instantiate the AvatarHandler with specified options, listen for various events like state changes and messages, send speech or audio data, control avatar muting, interrupt responses, and clean up the handler.

```typescript
// Example: Creating and using AvatarHandler
import { AvatarHandler } from '@renderer/handlers/avatarHandler'
import { EventTypes } from '@/interface/eventType'
import { TYVoiceChatState } from '@/interface/voiceChat'

const avatarHandler = new AvatarHandler({
  container: document.getElementById('avatar-container') as HTMLDivElement,
  assetsPath: 'https://server.com/assets/avatar-model',
  ws: websocketConnection,
  rendererType: 'lam',
  downloadProgress: (percent) => {
    console.log(`Downloading assets: ${percent}%`)
  },
  loadProgress: (percent) => {
    console.log(`Loading model: ${percent}%`)
  },
})

// Listen to state changes
avatarHandler.on(EventTypes.StateChanged, (state: TYVoiceChatState) => {
  console.log('Avatar state:', state)
  // Handle UI updates based on state
})

// Listen to messages
avatarHandler.on(EventTypes.MessageReceived, (data) => {
  const { role, payload } = data
  console.log(`${role}: ${payload.text}`)
})

// Listen to errors
avatarHandler.on(EventTypes.ErrorReceived, (error) => {
  console.error('Avatar error:', error)
})

// Send text message
avatarHandler.sendSpeech('Hello, how are you?')

// Send audio (16kHz PCM, Int16Array)
const audioData = new Int16Array(16000) // 1 second of audio
avatarHandler.sendAudio(audioData, 'base64')

// Mute/unmute avatar audio
avatarHandler.setAvatarMute(true)

// Interrupt current response
avatarHandler.interrupt()

// Get current conversation state
const state = avatarHandler.getChatState()

// Clean up
avatarHandler.exit()
```

--------------------------------

### WebSocket Message Sending Examples

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Provides example functions for sending various messages via WebSocket, including initializing a session, sending text, audio, interrupt signals, and heartbeats. Ensure the WS object is properly initialized and connected before use.

```typescript
// Example: Sending protocol messages via WebSocket
import { WsProtocol } from '@/interface/eventType'
import { nanoid } from 'nanoid'

// Initialize avatar session
function initializeSession(ws: WS) {
  ws.send(JSON.stringify({
    header: {
      name: WsProtocol.InitializeAvatarSession,
      request_id: nanoid(),
    },
    payload: {
      audio: {
        format: 'PCM',
        sample_rate: 16000,
        channels: 1,
      },
      subscriptions: ['human_text', 'avatar_text', 'motion_data'],
    },
  }))
}

// Send user text input
function sendText(ws: WS, text: string) {
  ws.send(JSON.stringify({
    header: {
      name: WsProtocol.SendHumanText,
      request_id: nanoid(),
    },
    payload: {
      request_id: nanoid(),
      stream_key: nanoid(),
      mode: 'full_text',
      text: text,
      end_of_speech: true,
    },
  }))
}

// Send audio data (base64 encoded)
function sendAudio(ws: WS, pcmData: Int16Array) {
  const base64 = Buffer.from(pcmData.buffer).toString('base64')
  ws.send(JSON.stringify({
    header: {
      name: WsProtocol.SendHumanAudio,
      request_id: nanoid(),
    },
    payload: {
      transport: 'base64',
      data_base64: base64,
    },
  }))
}

// Send interrupt signal
function interrupt(ws: WS, maxBatchId?: number) {
  ws.send(JSON.stringify({
    header: {
      name: WsProtocol.Interrupt,
      request_id: nanoid(),
    },
    payload: { maxBatchId },
  }))
}

// Send heartbeat (every 10 seconds)
function triggerHeartbeat(ws: WS) {
  ws.send(JSON.stringify({
    header: {
      name: WsProtocol.TriggerHeartbeat,
      request_id: nanoid(),
    },
  }))
}
```

--------------------------------

### Initialization Configuration API (GET /openavatarchat/initconfig)

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Endpoint called on frontend startup to fetch backend initialization configuration, determining chat mode, avatar rendering, and media parameters.

```APIDOC
## Initialization Configuration API (GET /openavatarchat/initconfig)

Frontend calls `GET /openavatarchat/initconfig` on startup to get backend initialization configuration, which dictates the frontend's chat mode, avatar rendering method, and media capture parameters.

### Response Data Structure

```typescript
interface InitConfigResponse {
  // Error message (if present, indicates initialization failure)
  detail?: string

  // WebRTC configuration (standard RTCConfiguration object, including ICE servers, etc.)
  rtc_configuration?: RTCConfiguration

  // Chat mode: 'webrtc' (real-time audio/video) or 'ws' (WebSocket audio/text)
  chat_mode?: 'webrtc' | 'ws'

  // Avatar configuration
  avatar_config?: {
    avatar_type: string          // Rendering type identifier, e.g., 'lam' (on-device rendering) or '' (audio only)
    avatar_ws_route: string      // Avatar WebSocket route, e.g., '/ws/webrtc/avatar'
    avatar_assets_path: string   // Path to avatar model assets
    ws_session_route?: string    // Session WebSocket route (fallback)
  }

  // Fallback session WebSocket route
  ws_session_route?: string

  // Media track constraints (controls camera/microphone capture parameters)
  track_constraints?: {
    audio?: boolean | MediaTrackConstraints
    video?: boolean | MediaTrackConstraints
  }
}
```

### Field Description

| Field                     | Frontend Storage Location | Purpose                                                                 |
|---------------------------|---------------------------|-------------------------------------------------------------------------|
| `rtc_configuration`       | `appStore.rtcConfig`      | Initialize WebRTC PeerConnection                                        |
| `chat_mode`               | `appStore.chatMode`       | Determines WebRTC or WebSocket chat mode                                |
| `avatar_config.avatar_type` | `appStore.avatarType`     | Selects avatar rendering engine                                         |
| `avatar_config.avatar_ws_route` | `appStore.avatarWSRoute`  | Establishes avatar WebSocket channel                                    |
| `avatar_config.avatar_assets_path` | `appStore.avatarAssetsPath` | Loads avatar model assets (auto-converted to full URL)                  |
| `avatar_config.ws_session_route` | `appStore.wsSessionRoute` | Fallback session route, used if `avatar_ws_route` is not set            |
| `ws_session_route`        | `appStore.wsSessionRoute` | Fallback session route, used if `avatar_ws_route` and `avatar_config.ws_session_route` are not set |
| `track_constraints`       | `mediaStore.trackConstraints` | Controls frontend audio/video capture parameters                          |

### Configuration Priority

1. `avatar_config.avatar_ws_route` is prioritized for WebSocket connections.
2. If not set, `avatar_config.ws_session_route` is tried, followed by `ws_session_route`.
3. `avatar_assets_path` is automatically converted to a full resource URL using `makeURL()`.
```

--------------------------------

### Example: Avatar WebSocket Communication

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Demonstrates how to use the createWS function to establish a WebSocket connection for avatar communication, handle different WebSocket events, send messages, and close the connection.

```typescript
// Example: Using WebSocket for avatar communication
import { createWS } from '@/apis'
import { WsEventTypes } from '@/interface/eventType'

const sessionId = crypto.randomUUID()
const ws = createWS('/ws/webrtc/avatar', sessionId)

ws.on(WsEventTypes.WS_OPEN, () => {
  console.log('WebSocket connected')
  // Initialize session after connection
})

ws.on(WsEventTypes.WS_MESSAGE, (data: Blob | string) => {
  if (typeof data === 'string') {
    const message = JSON.parse(data)
    console.log('Received:', message.header?.name, message.payload)
  } else {
    // Handle binary motion data
    console.log('Received binary data:', data.size, 'bytes')
  }
})

ws.on(WsEventTypes.WS_CLOSE, () => {
  console.log('WebSocket closed')
})

ws.on(WsEventTypes.WS_ERROR, (event) => {
  console.error('WebSocket error:', event)
})

// Send data
ws.send(JSON.stringify({ header: { name: 'Ping' } }))

// Close connection
ws.stop()
```

--------------------------------

### Registering Custom Handler in Store

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Example of how to register your custom AvatarHandler with the store, specifying your custom renderer type during initialization.

```typescript
// 在 Store 的 initAvatarHandler() 方法中
const handler = new AvatarHandler({
  container: visionStore.remoteVideoContainerRef!,
  assetsPath: appStore.avatarAssetsPath,
  ws: ws,
  rendererType: 'custom',  // 你的渲染器类型
})
chatStore.setActiveRenderer(handler)
chatStore.bindAvatarHandler(handler)
```

--------------------------------

### Implement Custom Avatar Renderer

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Example implementation of a custom avatar renderer using a 3D model. Includes asset downloading, model loading with progress reporting, expression updates, and resource cleanup.

```typescript
// Example: Creating a custom avatar renderer
// File: src/renderer/src/handlers/avatarRenderers/custom.ts

export class CustomRenderer {
  container: HTMLDivElement
  assetsPath: string
  private model: any // Your 3D model instance

  constructor(options: {
    container: HTMLDivElement
    assetsPath: string
    downloadProgress?: (percent: number) => void
    loadProgress?: (percent: number) => void
  }) {
    this.container = options.container
    this.assetsPath = options.assetsPath

    // Initialize your rendering engine
    this.initRenderer(options)
  }

  private async initRenderer(options: RendererOptions) {
    // Download assets with progress reporting
    options.downloadProgress?.(0)
    const assets = await this.downloadAssets(this.assetsPath)
    options.downloadProgress?.(100)

    // Load model with progress reporting
    options.loadProgress?.(0)
    this.model = await this.loadModel(assets)
    options.loadProgress?.(100)
  }

  // Update facial expressions/motion from ARKit data
  updateExpression(arkitData: Record<string, number>): void {
    if (!this.model) return

    // Map ARKit blend shapes to your model
    Object.entries(arkitData).forEach(([blendShape, value]) => {
      this.model.setBlendShape(blendShape, value)
    })

    this.model.render()
  }

  // Clean up resources
  dispose(): void {
    this.model?.dispose()
    this.container.innerHTML = ''
  }

  private async downloadAssets(path: string): Promise<ArrayBuffer> {
    const response = await fetch(path)
    return response.arrayBuffer()
  }

  private async loadModel(assets: ArrayBuffer): Promise<any> {
    // Load your 3D model from assets
    return { /* model instance */ }
  }
}

// Integration in AvatarHandler (avatarHandler.ts)
// Add 'custom' to rendererType union: 'lam' | '' | 'custom'
// In render() method:
async render(): Promise<void> {
  if (this._rendererType === 'lam') {
    // ... existing LAM renderer
  } else if (this._rendererType === 'custom') {
    this._renderer = new CustomRenderer({
      container: this._avatarDivEle,
      assetsPath: this._assetsPath,
      downloadProgress: this._downloadProgress,
      loadProgress: this._loadProgress,
    })
  } else {
    this._renderer = null // Audio-only mode
  }
}
```

--------------------------------

### Use Manager Store for Dashboard Monitoring

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Demonstrates starting the manager store connection, monitoring its status, accessing sorted sessions, selecting a specific session, and retrieving its data. Includes actions for sending interrupts, reconnecting, and stopping the connection.

```typescript
// Example: Using the manager store for session monitoring
import { useManagerStore } from '@/store/manager'

const managerStore = useManagerStore()

// Start manager WebSocket connection
managerStore.start()

// Monitor connection status
watch(() => managerStore.status, (status) => {
  console.log('Manager connection:', status)
})

// Get sorted sessions (active first, then by last updated)
const sessions = computed(() => managerStore.sortedSessions)

// Select a specific session
managerStore.selectSession('session-123')

// Get active session data
const activeSession = computed(() => managerStore.activeSession)
const chatMessages = computed(() => managerStore.activeChatMessages)

// Get flow visualization data for signal pipeline
const flowNodes = computed(() => managerStore.flowNodes)
const flowEdges = computed(() => managerStore.flowEdges)

// Send interrupt to active session
managerStore.sendInterrupt()

// Reconnect WebSocket
managerStore.reconnect()

// Disconnect and cleanup
managerStore.stop()

// Access current backend runtime configuration
const config = computed(() => managerStore.currentConfig)

```

--------------------------------

### Build Command: Electron Desktop App (Windows)

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Command to build the Electron desktop application specifically for Windows.

```bash
pnpm run build:win
```

--------------------------------

### Build Command: Electron Desktop App (Linux)

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Command to build the Electron desktop application specifically for Linux.

```bash
pnpm run build:linux
```

--------------------------------

### Build Platform-Specific Electron Applications

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Use these commands to build the macOS, Windows, or Linux applications using pnpm.

```bash
pnpm run build:mac
```

```bash
pnpm run build:win
```

```bash
pnpm run build:linux
```

--------------------------------

### Production Build: Independent Frontend

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Command to build the frontend for production. The output in the 'dist/' directory can then be deployed to any web server.

```bash
pnpm run build
# 输出目录：dist/
# 将 dist/ 部署到任意 Web 服务器（Nginx、Apache 等）
```

--------------------------------

### Build Command: Electron Desktop App (macOS)

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Command to build the Electron desktop application specifically for macOS.

```bash
pnpm run build:mac
```

--------------------------------

### NPM/PNPM Build and Development Scripts

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Common scripts for managing the development server, building for production, and packaging the Electron application.

```bash
# Web Development
pnpm run dev              # Start Vite dev server with HMR
pnpm run build            # Production build (output to dist/)

# Electron Development
pnpm run electron:dev     # Start Electron in development mode
pnpm run electron:build   # Build Electron application code
```

--------------------------------

### Deployment Methods

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Overview of different ways to deploy the OpenAvatarChat WebUI, including integrated deployment, independent deployment, and Electron desktop application.

```APIDOC
## Deployment Methods

### 1. Deploy with OpenAvatarChat (Recommended)

Simplest method where the frontend is served as static resources by the backend FastAPI service.

**Process**:

1. Deploy the backend service following the [OpenAvatarChat](https://github.com/HumanAIGC-Engineering/OpenAvatarChat) documentation.
2. Frontend build artifacts are included in `OpenAvatarChat/src/service/frontend_service/frontend/dist/`.
3. Access `https://your-server:8282` after starting the backend service to be automatically redirected to the frontend.

**Features**:

- Same-origin deployment for backend and frontend, avoiding cross-origin issues.
- No separate `.env` configuration required.
- Frontend updates require recompilation and syncing the `dist` folder to the backend project.

**To update the frontend**:

```bash
# In the WebUI project
pnpm install
pnpm run build
# Copy contents of dist/ to OpenAvatarChat/src/service/frontend_service/frontend/dist/
```

### 2. Independent Frontend Deployment

Suitable for scenarios requiring separate frontend and backend development or custom frontend deployment.

**Development Mode**:

```bash
# Install dependencies
pnpm install

# Configure .env to point to the backend service address
# VITE_SERVER_IP=your-backend-ip
# VITE_SERVER_PORT=8282

# Start development server (supports HMR hot updates)
pnpm run dev
```

**Production Build**:

```bash
pnpm run build
# Output directory: dist/
# Deploy the contents of dist/ to any web server (Nginx, Apache, etc.)
```

**Features**:

- Independent deployment of frontend and backend offers high flexibility.
- Development mode uses Vite proxy to resolve cross-origin issues (proxies routes like `/openavatarchat`, `/webrtc/offer`, `/ws`).
- Production deployment requires configuring the backend address via `.env` or forwarding API requests through a reverse proxy.
- Ideal for frontend development debugging and custom deployment.

### 3. Electron Desktop Application Deployment

Packages the frontend into a native desktop application for enhanced desktop integration.

**Development Mode**:

```bash
pnpm run electron:dev
```

**Build Commands**:

```bash
# macOS
pnpm run build:mac

# Windows
pnpm run build:win

# Linux
pnpm run build:linux
```

**Features**:

- Native desktop application experience, runs in an independent window.
- Supports borderless transparent windows.
- Uses Electron IPC for communication between main and renderer processes.
- Requires configuring the backend server address within the application.

### Comparison of Deployment Methods

| Feature           | Deploy with OpenAvatarChat | Independent Frontend | Electron |
|-------------------|----------------------------|----------------------|----------|
| Deployment Complexity | Low (auto-integrated)      | Medium (proxy/address config) | High (packaging/distribution) |
| Cross-Origin Handling | None (same origin)         | Requires proxy config | None (built-in requests) |
| .env Configuration | Not required               | Required             | Required |
| Hot Update Dev    | Not supported              | Supported            | Supported |
| Use Case          | Production deployment      | Frontend dev/custom deploy | Desktop app distribution |
```

--------------------------------

### Environment Variable Configuration (.env)

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Configuration for connecting the frontend to the backend service using environment variables.

```APIDOC
## Environment Variable Configuration (.env)

`.env` file located at the project root for configuring frontend connection to backend services.

### Variable Description

| Environment Variable | Type   | Purpose                                      | Default Value                               |
|----------------------|--------|----------------------------------------------|---------------------------------------------|
| `VITE_SERVER_IP`     | String | IP address of the OpenAvatarChat backend server | Automatically read from `location.hostname` |
| `VITE_SERVER_PORT`   | String | Backend server port                          | Automatically read from `location.port`     |
| `VITE_USE_SSL`       | String | Whether to use SSL/HTTPS connection (`true`/`false`) | Inferred from `location.protocol`           |

### Configuration Rules

- **When deployed with OpenAvatarChat**: No `.env` configuration needed; frontend automatically gets server address from browser `location` (same origin deployment).
- **For independent frontend development/deployment**: Configure backend server address in `.env`.
- When `VITE_USE_SSL=false`, hostname must be `127.0.0.1` or `localhost` due to browser security restrictions.
- Non-local network access requires SSL certificates (`VITE_USE_SSL=true`).

### Example

```env
# Connect to local backend service (no SSL)
VITE_SERVER_IP=127.0.0.1
VITE_SERVER_PORT=8282
VITE_USE_SSL=false

# Connect to remote backend service (requires SSL)
VITE_SERVER_IP=your-server-ip
VITE_SERVER_PORT=8282
VITE_USE_SSL=true
```
```

--------------------------------

### Environment Configuration for Standalone Deployment

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Configuration variables for connecting to the OpenAvatarChat backend server in standalone mode. These are typically set in a .env file.

```bash
# .env file in project root

# OpenAvatarChat backend server IP address
# Default: auto-detected from location.hostname
VITE_SERVER_IP=127.0.0.1

# Backend server port
# Default: auto-detected from location.port
VITE_SERVER_PORT=8282

# Whether to use SSL/HTTPS connection
# Default: auto-detected from location.protocol
# Note: When false, hostname must be 127.0.0.1 or localhost
VITE_USE_SSL=false
```

--------------------------------

### POST /webrtc/offer

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Handles WebRTC offer and ICE candidate exchange for establishing real-time audio/video communication with the avatar backend.

```APIDOC
## POST /webrtc/offer

### Description
Establishes WebRTC peer connections for real-time audio/video communication with the avatar backend by sending SDP offers and receiving answers, as well as handling ICE candidates.

### Method
POST

### Endpoint
/webrtc/offer

### Parameters
#### Request Body
- **sdp** (string) - The Session Description Protocol (SDP) string.
- **type** (RTCSdpType) - The type of the SDP message (e.g., 'offer', 'answer').
- **webrtc_id** (string) - A unique identifier for the WebRTC connection.
- **candidate** (RTCIceCandidateInit) - The ICE candidate information (for ICE candidate messages).
- **type** ('ice-candidate') - Indicates the message type is an ICE candidate.

### Request Example
```json
{
  "sdp": "v=0...",
  "type": "offer",
  "webrtc_id": "a1b2c3d4"
}
```

```json
{
  "candidate": {
    "candidate": "candidate:1 1 UDP 2122260223 192.168.1.100 50000",
    "sdpMid": "0",
    "sdpMLineIndex": 0
  },
  "webrtc_id": "a1b2c3d4",
  "type": "ice-candidate"
}
```

### Response
#### Success Response (200)
- **sdp** (string) - The SDP answer from the server.
- **type** (RTCSdpType) - The type of the SDP message ('answer').

#### Response Example
```json
{
  "sdp": "v=0...",
  "type": "answer"
}
```
```

--------------------------------

### TypeScript Interface for InitConfigResponse

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Defines the structure of the initialization configuration response from the backend. This interface dictates chat mode, avatar rendering, and media constraints.

```typescript
interface InitConfigResponse {
  // 错误信息（若存在则表示初始化失败）
  detail?: string

  // WebRTC 配置（标准 RTCConfiguration 对象，包含 ICE 服务器等）
  rtc_configuration?: RTCConfiguration

  // 对话模式：'webrtc'（实时音视频）或 'ws'（WebSocket 音频/文本）
  chat_mode?: 'webrtc' | 'ws'

  // 数字人配置
  avatar_config?: {
    avatar_type: string          // 渲染类型标识，如 'lam'（端侧渲染）或 ''（纯音频）
    avatar_ws_route: string      // 数字人 WebSocket 路由，如 '/ws/webrtc/avatar'
    avatar_assets_path: string   // 数字人模型资源路径
    ws_session_route?: string    // 会话 WebSocket 路由（备选）
  }

  // 备选会话 WebSocket 路由
  ws_session_route?: string

  // 媒体轨道约束（控制摄像头/麦克风采集参数）
  track_constraints?: {
    audio?: boolean | MediaTrackConstraints
    video?: boolean | MediaTrackConstraints
  }
}
```

--------------------------------

### Internal Usage of Environment Variables

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Demonstrates how environment variables are accessed and utilized within the application for server configuration and network requests. Includes custom fetch logic with authentication.

```typescript
// Example: How environment variables are used internally
// File: src/renderer/src/apis/base.ts

const isDEV = import.meta.env.DEV

// SSL configuration
export const useSSL = isDEV
  ? location.protocol === 'https:'
  : typeof import.meta.env.USE_SSL === 'undefined'
    ? location.protocol === 'https:'
    : import.meta.env.USE_SSL

// Server configuration
export const serverIP = import.meta.env.SERVER_IP || location.hostname
export const serverPort = import.meta.env.SERVER_PORT || location.port
export const serverHost = isDEV ? location.host : `${serverIP}:${serverPort}`
export const serverProtocol = useSSL ? 'https' : 'http'
export const serverOrigin = `${serverProtocol}://${serverHost}`

// Custom fetch with auth token
export function fetch(input: RequestInfo | URL, init?: RequestInit): Promise<Response> {
  const token = localStorage.getItem('auth_openavatarchat')
  const headers = new Headers(init?.headers || {})
  if (token) {
    headers.set('Authorization', `Bearer ${token}`)
  }
  return window.fetch(`${serverOrigin}${input}`, { ...init, headers })
}
```

--------------------------------

### Interface for Initialization Configuration Response

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines the structure of the response received when fetching initialization configuration from the backend. This includes settings for WebRTC/WebSocket modes, avatar rendering, and media constraints.

```typescript
// GET /openavatarchat/initconfig
// Returns configuration for WebRTC/WebSocket mode, avatar settings, and media constraints

interface InitConfigResponse {
  detail?: string                    // Error message if initialization failed
  rtc_configuration?: RTCConfiguration  // WebRTC ICE server configuration
  chat_mode?: 'webrtc' | 'ws'        // Conversation mode
  avatar_config?: {
    avatar_type: string              // Renderer type: 'lam' or '' (audio-only)
    avatar_ws_route: string          // WebSocket route for avatar
    avatar_assets_path: string       // Avatar model assets path
    ws_session_route?: string        // Fallback session WebSocket route
  }
  ws_session_route?: string          // Alternative session WebSocket route
  track_constraints?: {
    audio?: boolean | MediaTrackConstraints
    video?: boolean | MediaTrackConstraints
  }
}
```

--------------------------------

### Development Mode: Electron Desktop App

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/README.md

Command to run the Electron desktop application in development mode, allowing for live reloading and debugging.

```bash
pnpm run electron:dev
```

--------------------------------

### Run Code Quality Checks

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Execute code quality tools such as ESLint for linting, Prettier for formatting, and TypeScript for type checking using pnpm.

```bash
pnpm run lint
```

```bash
pnpm run format
```

```bash
pnpm run typecheck
```

--------------------------------

### Avatar Handler Initialization Options

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Interface defining the configuration options required for initializing an AvatarHandler instance.

```typescript
// AvatarHandler 初始化选项
interface AvatarHandlerOptions {
  container: HTMLDivElement           // 渲染容器 DOM 元素
  assetsPath: string                  // 数字人模型资源路径
  ws: WS                              // WebSocket 连接实例
  downloadProgress?: (percent: number) => void  // 资源下载进度回调
  loadProgress?: (percent: number) => void      // 资源加载进度回调
  rendererType: 'lam' | ''            // 渲染模式：'lam'=LAM本地渲染, ''=纯音频模式
}
```

--------------------------------

### Create WebSocket Connections

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Functions to create WebSocket connections for avatar session communication and manager data tools. Includes logic for adding authentication tokens.

```typescript
// WebSocket endpoints:
// - /ws/{ws_route}/{session_id} - Avatar session communication
// - /ws/manager/data_tool - Manager dashboard data stream

import { WS } from '@/helpers/ws'
import { serverHost, useSSL } from './base'

// Create WebSocket for avatar session
function createWS(ws_route: string, webRTCId: string): WS {
  const token = localStorage.getItem('auth_openavatarchat')
  let url = `${useSSL ? 'wss' : 'ws'}://${serverHost}${ws_route}/${webRTCId}`
  if (token) {
    url += `?token=${encodeURIComponent(token)}`
  }
  return new WS(url)
}

// Create WebSocket for manager data tool
function createDataToolWS(): WS {
  const token = localStorage.getItem('auth_openavatarchat')
  let url = `${useSSL ? 'wss' : 'ws'}://${serverHost}/ws/manager/data_tool`
  if (token) {
    url += `?token=${encodeURIComponent(token)}`
  }
  return new WS(url)
}
```

--------------------------------

### Event System

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Describes the events emitted by the AvatarHandler.

```APIDOC
## Event System

AvatarHandler emits the following events:

- `EventTypes.StateChanged`: Dialogue state change.
- `EventTypes.MessageReceived`: Text message received.
- `EventTypes.SignalReceived`: Stream signal received.
- `EventTypes.ErrorReceived`: Error received.
```

--------------------------------

### AvatarHandler Core Concepts

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Overview of the AvatarHandler's architecture, state machine, and core interfaces.

```APIDOC
## AvatarHandler Architecture

AvatarHandler is an event-driven component based on EventEmitter3, responsible for:

- WebSocket communication management
- Audio and video data processing
- Digital human rendering engine management
- Heartbeat mechanism
- Dialogue state machine management

### State Machine

```
Idle ──[Start Speaking]──→ Listening
  ↑                    │
  │                    ↓
  └────[Dialogue Complete]── Responding ──[Speaking End]──→ Idle
```

**State Definitions (`TYVoiceChatState`)**:

```typescript
export enum TYVoiceChatState {
  Idle = 'Idle',             // Idle/Standby
  Listening = 'Listening',   // Currently listening to user input
  Responding = 'Responding', // Digital human is responding
  Thinking = 'Thinking',     // Reserved state for thinking
}
```

### Core Interfaces

**`AvatarHandlerOptions`**: Options for initializing AvatarHandler.

```typescript
interface AvatarHandlerOptions {
  container: HTMLDivElement           // DOM element for rendering container
  assetsPath: string                  // Path to digital human model assets
  ws: WS                              // WebSocket connection instance
  downloadProgress?: (percent: number) => void  // Callback for asset download progress
  loadProgress?: (percent: number) => void      // Callback for asset load progress
  rendererType: 'lam' | ''            // Rendering mode: 'lam' for LAM local rendering, '' for audio-only mode
}
```

**`AvatarLike`**: Interface for digital human renderers.

```typescript
interface AvatarLike {
  setAvatarMute?(isMute: boolean): void  // Mute control
  interrupt?(): void                      // Interrupt current response
}
```
```

--------------------------------

### WebRTC Offer API Request Interfaces

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines the expected request payload structures for sending WebRTC SDP offers and ICE candidates to the backend. These are used during the WebRTC negotiation phase.

```typescript
// POST /webrtc/offer
// Sends SDP offer and receives answer for WebRTC negotiation

interface WebRTCOfferRequest {
  sdp: string
  type: RTCSdpType
  webrtc_id: string
}

// Also handles ICE candidates:
interface ICECandidateRequest {
  candidate: RTCIceCandidateInit
  webrtc_id: string
  type: 'ice-candidate'
}
```

--------------------------------

### Built-in Renderers

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Information about the currently integrated digital human renderers.

```APIDOC
## Current Built-in Renderers

| Renderer | Type Identifier | Description |
|---|---|---|
| LAMRenderer | `'lam'` | Client-side digital human rendering based on Gaussian Splatting, using the `gaussian-splat-renderer-for-lam` library. |
| Audio-Only Mode | `''` | No digital human avatar is rendered; only voice dialogue is processed. |
```

--------------------------------

### AvatarHandler Options and States

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines the configuration options for the AvatarHandler class, including rendering container, assets path, WebSocket instance, and progress callbacks. Also defines the possible conversation states for the avatar.

```typescript
interface AvatarHandlerOptions {
  container: HTMLDivElement           // Rendering container DOM element
  assetsPath: string                  // Avatar model assets path
  ws: WS                              // WebSocket connection instance
  downloadProgress?: (percent: number) => void  // Asset download progress callback
  loadProgress?: (percent: number) => void      // Asset loading progress callback
  rendererType: 'lam' | ''            // Render mode: 'lam' = LAM local render, '' = audio-only
}

// Conversation states
enum TYVoiceChatState {
  Idle = 'Idle',             // Idle/standby
  Listening = 'Listening',   // Listening to user input
  Responding = 'Responding', // Avatar is responding
  Thinking = 'Thinking',     // Thinking (reserved state)
}
```

--------------------------------

### Video Chat State Interface

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines the TypeScript interface for the state management of WebRTC-based video chat sessions within the Pinia store. This interface outlines the properties related to the stream, peer connection, and avatar rendering.

```typescript
interface VideoChatState {
  streamState: StreamState           // closed | waiting | open
  peerConnection: RTCPeerConnection | null
  webRTCId: string
  gsLoadPercent: number              // Gaussian splatting load progress
  localAvatarRenderer: AvatarHandler | null
  chatDataChannel: RTCDataChannel | null
}
```

--------------------------------

### Define Custom Avatar Renderer Interface

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines the required interface for custom avatar renderers and the options structure for their constructors. Implement `AvatarLike` for basic controls.

```typescript
interface AvatarLike {
  setAvatarMute?(isMute: boolean): void  // Volume control
  interrupt?(): void                      // Interrupt current response
}

// Renderer constructor options
interface RendererOptions {
  container: HTMLDivElement
  assetsPath: string
  downloadProgress?: (percent: number) => void
  loadProgress?: (percent: number) => void
}
```

--------------------------------

### Avatar Handler State Machine

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Visual representation of the voice chat state transitions within the AvatarHandler.

```mermaid
stateDiagram-v2
Idle --> Listening: Start Speaking
Listening --> Responding: Response Complete
Responding --> Idle: Speaking End
Idle <-- Responding: Dialogue End
```

--------------------------------

### Communication Protocol

Source: https://github.com/humanaigc-engineering/openavatarchat-webui/blob/main/docs/extending-avatar-renderer.md

Details the WebSocket communication protocol between the frontend and backend.

```APIDOC
## Communication Protocol

AvatarHandler communicates with the backend via WebSocket. The primary protocol messages are:

**Sending (Frontend → Backend)**:

- `InitializeAvatarSession`: Initialize digital human session.
- `SendHumanText`: Send user text input.
- `SendHumanAudio`: Send user voice (16kHz PCM, base64 encoded).
- `Interrupt`: Interrupt digital human response.
- `TriggerHeartbeat`: Heartbeat for keep-alive (every 10 seconds).

**Receiving (Backend → Frontend)**:

- `AvatarSessionInitialized`: Session initialization complete.
- `EchoHumanText`: User text echo.
- `EchoAvatarText`: Digital human text reply.
- `MotionData` / `MotionDataWelcome`: Motion/expression data (for client-side rendering).
- `ChatSignal`: Stream signal (stream_begin/stream_end/stream_cancel).
- `Error`: Error information.
```

--------------------------------

### Define Manager Store Types

Source: https://context7.com/humanaigc-engineering/openavatarchat-webui/llms.txt

Defines TypeScript types for connection status, session state including messages and timestamps, and flow node data for visualization in the manager dashboard.

```typescript
// Connection status types
type ConnectionStatus = 'idle' | 'connecting' | 'open' | 'closed' | 'error'

// Session state
interface SessionState {
  id: string
  owner?: string
  messages: SessionMessage[]
  chatMessages: ChatMessage[]
  lastUpdated: number
}

// Flow visualization data
interface FlowNode {
  id: string
  type: 'handler'
  position: { x: number; y: number }
  data: {
    label: string
    sourceType: string
    sourceName: string
    status: 'active' | 'inactive' | 'timeout'
    startTime?: number
    endTime?: number
  }
}
```