### Application Initialization and Setup Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the application, including checking for HTTPS, setting up the audio player, WebSocket client, voice activity detection, and UI handlers. Starts all components upon successful initialization. ```javascript try { // Check HTTPS requirement if (!navigator.mediaDevices) { throw 'Please use the app via HTTPS in order to enable microphone support!' } // Initialize components const player = new RealTimeAudioPlayer() const websocket = new PbWebsocket(`wss://${window.location.host}/channels/default/audio?buffer-ms=50`) // Initialize voice activity detection const myvad = await MicVAD.new({ onSpeechEnd: async function(audioSamples) { // Convert float samples to WAV and send websocket.send({'audioMessage': riffWaveFromFloat32Array(audioSamples, 16000)}) }, }); // Setup UI handlers setupUI(websocket, player); // Start all components player.start() websocket.start() myvad.start() } catch(e) { // Display error to user const errorEl = document.querySelector('#error') errorEl.innerHTML = e.toString() } ``` -------------------------------- ### Typical Main Server Integration Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Example of a typical main.go file for integrating the AI assistant server. It loads configuration, initializes MCP servers, adds routes, and starts an HTTPS server. ```go func main() { cfg, _ := config.FromFile("config.yaml") mcpServers, _ := mcp.NewServers(context.Background(), cfg.MCPServers) defer mcpServers.Close() mux := http.NewServeMux() server.AddRoutes(context.Background(), cfg, mcpServers, "web/dist", mux) http.ListenAndServeTLS(":8443", "cert.pem", "key.pem", mux) } ``` -------------------------------- ### Configuration-Based Tool Setup Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Illustrates how to configure MCP servers and tools via a configuration file. This setup translates into loading servers and creating a tool provider that respects whitelisted tools. ```yaml tools: - mcpServer: tool-containers allow: - wikipedia - calculator - mcpServer: memory ``` ```go // Load MCP servers servers, _ := mcp.NewServers(ctx, cfg.MCPServers) // Create tool provider that selectively enables tools provider, _ := mcp.ToolProvider(servers, cfg.Tools) // provider.Tools() returns only whitelisted tools ``` -------------------------------- ### NewServers Initialization Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Initializes MCP servers from a given configuration map. It spawns processes, connects via stdin/stdout, and verifies tool capabilities. If any server fails to start, all opened servers are closed, and an error is returned. ```go mcpServers := map[string]config.MCPServer{ "tool-containers": { Command: "/tool-containers-mcp", Args: []string{"--config=/etc/tools.yaml"}, }, } servers, err := mcp.NewServers(context.Background(), mcpServers) if err != nil { log.Fatal(err) } defer servers.Close() // Use servers... ``` -------------------------------- ### Chat Completion Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Demonstrates how to use the LLM.ChatCompletion method. This example initializes an LLM client, defines tools, sets up a conversation, and streams the responses. Ensure the HTTP client has a reasonable timeout. ```go llm := &chat.LLM{ ServerURL: "http://localhost:8080", APIKey: "fake-key", Model: "qwen3-4b", Temperature: 0.7, MaxTurns: 5, HTTPClient: &http.Client{Timeout: 90*time.Second}, } tools := []tools.Tool{myTool1, myTool2} conversation := model.NewConversation("You are helpful", 1) conversation.AddUserRequest(llms.TextPart("What's the weather?")) responses := make(chan chat.ResponseChunk, 50) go func() { err := llm.ChatCompletion(context.Background(), 1, tools, conversation, responses) if err != nil { log.Fatal(err) } }() for chunk := range responses { if chunk.Type == model.MessageTypeChunk { fmt.Print(chunk.Text) } } ``` -------------------------------- ### RealTimeAudioPlayer.start() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Starts the audio playback system, initializing the Web Audio API context and preparing for playback. ```APIDOC ## RealTimeAudioPlayer.start() ### Description Starts the audio playback system, initializing the Web Audio API context and preparing for playback. ### Method `start()` ### Behavior Initializes Web Audio API context and playback. ``` -------------------------------- ### MCPServer Configuration Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Defines the path to an MCP server executable and its command-line arguments. ```yaml mcpServers: : command: args: - - ``` -------------------------------- ### NewServers() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Initializes and starts MCP servers based on provided configuration, connecting via stdin/stdout transport. ```APIDOC ### NewServers() Initialize MCP servers from configuration. **Signature**: ```go func NewServers(ctx context.Context, mcpServers map[string]config.MCPServer) (Servers, error) ``` **Parameters**: | Parameter | Type | Description | |-----------|------|-------------| | ctx | context.Context | Context for server processes | | mcpServers | map[string]config.MCPServer | Server configurations | **Returns**: - **Servers**: Initialized servers - **error**: Startup error (any server failed) **Behavior**: 1. For each server in configuration: - Create MCP client - Connect via stdin/stdout transport (spawn process) - Verify tool capability - Add to servers map 2. If any server fails: - Close all opened servers - Return error 3. All server processes run concurrently **Error Cases**: | Error | Cause | |-------|-------| | "starting MCP server: " | Process startup failed, connection failed | | "MCP server does not support tools" | Server lacks tool capability | **Example**: ```go mcpServers := map[string]config.MCPServer{ "tool-containers": { Command: "/tool-containers-mcp", Args: []string{"--config=/etc/tools.yaml"}, }, } servers, err := mcp.NewServers(context.Background(), mcpServers) if err != nil { log.Fatal(err) } defer servers.Close() // Use servers... ``` ``` -------------------------------- ### PubSub.Publish Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Shows how to publish an event to all active subscribers of a PubSub system. Slow subscribers are logged and removed asynchronously. ```go event := MyEvent{Text: "Hello"} pubsub.Publish(event) // All subscriptions receive the event // Slow subscribers logged and removed ``` -------------------------------- ### PubSub.Subscribe Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Demonstrates how to create a new subscription to a PubSub system and process incoming events. Ensure to stop the subscription when done. ```go pubsub := pubsub.New[MyEvent]() ctx := context.Background() subscription := pubsub.Subscribe(ctx) defer subscription.Stop() for event := range subscription.ResultChan() { handleEvent(event) } ``` -------------------------------- ### Configuration-Based Tool Setup Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Explains how to set up tools based on configuration, allowing selective enabling of tools from MCP servers. ```APIDOC ## Configuration-Based Tool Setup Tools can be configured via a configuration file, specifying which MCP servers to use and which tools are allowed from each. **Configuration Example**: ```yaml tools: - mcpServer: tool-containers allow: - wikipedia - calculator - mcpServer: memory ``` **Code Translation**: ```go // Load MCP servers servers, _ := mcp.NewServers(ctx, cfg.MCPServers) // Create tool provider that selectively enables tools provider, _ := mcp.ToolProvider(servers, cfg.Tools) // provider.Tools() returns only whitelisted tools ``` ``` -------------------------------- ### Create Pub/Sub Client and Handle Events Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Demonstrates how to create a Pub/Sub client, get or create a channel, subscribe to events, publish audio messages, and process incoming responses. ```go // Setup channels := channel.NewChannels(ctx, cfg, mcpServers) // Get channel ch, err := channels.GetOrCreate("default") if err != nil { log.Fatal(err) } // Subscribe to events subscription := ch.Subscribe(context.Background()) def subscription.Stop() // Send audio audioBytes := readWAVFile("input.wav") ch.Publish(model.AudioMessage{ WaveData: audioBytes, }) // Receive responses go func() { for event := range subscription.ResultChan() { if event.Role == model.RoleAssistant { if len(event.WaveData) > 0 { speakerPlayAudio(event.WaveData) } if len(event.Text) > 0 { printText(event.Text) } } } }() ``` -------------------------------- ### Server Integration: Channel Setup Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Shows how to initialize and integrate the Channels manager into the server's HTTP routes, handling channel creation and message routing. ```go // In server startup func AddRoutes(ctx context.Context, cfg config.Configuration, mcpServers mcp.Servers, webDir string, mux *http.ServeMux) { channels := channel.NewChannels(ctx, cfg, mcpServers) mux.HandleFunc("/channels/{channelId}/audio", func(w http.ResponseWriter, req *http.Request) { channelId := req.PathValue("channelId") // Get or create channel c, err := channels.GetOrCreate(channelId) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } // POST: publish audio // GET: subscribe to events (WebSocket or HTTP streaming) }) } ``` -------------------------------- ### Minimal Configuration File Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md A basic YAML configuration file demonstrating essential parameters for server URL, API key, and AI model selection. ```yaml serverURL: http://localhost:8080 apiKey: fake sttModel: whisper-1 chatModel: qwen3-4b ttsModel: voice-en-us-amy-low temperature: 0.7 wakeWord: Computer ``` -------------------------------- ### Wake Word Placeholder Substitution Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Demonstrates how the {wakeWord} placeholder is substituted in the introPrompt. ```yaml wakeWord: Computer introPrompt: "I am {wakeWord}, your AI assistant. Say '{wakeWord}' to address me." ``` -------------------------------- ### PbWebsocket start() Method Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the WebSocket connection for the PbWebsocket client. It automatically handles reconnection attempts in case of errors. ```javascript start() ``` -------------------------------- ### Example: Find Input Audio Device Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/audio-io.md Demonstrates how to use the inputDevice function with a partial name match, a numeric ID, or by requesting the system default. ```go // Substring match dev, err := inputDevice("KLIM") // Matches "KLIM Talk (KLIM Technologies)" // Numeric ID dev, err := inputDevice("1") // Device at index 1 // System default dev, err := inputDevice("") // Uses default input ``` -------------------------------- ### WebSocket Audio Streaming Request Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Example of initiating a WebSocket audio stream. This involves a GET request to the audio endpoint with an 'Upgrade' header. ```http GET wss://localhost:8443/channels/default/audio?buffer-ms=50 Connection: Upgrade ``` -------------------------------- ### Setup UI for WebSocket and Player Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the user interface by setting up event handlers for user interactions and configuring how WebSocket messages are processed. ```javascript export function setupUI(websocket, player) { setupUIEventHandlers(websocket); setupWebsocketMessageHandling(websocket, player); } ``` -------------------------------- ### Template Variable Resolution Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Shows how the {wakeWord} placeholder is resolved within a prompt template. ```yaml wakeWord: "Assistant" prompt: - "I am {wakeWord}. You must address me as {wakeWord} to get my attention." ``` -------------------------------- ### Protobuf Decoding Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Demonstrates how to unmarshal a protobuf chat.Message and publish its content as an AudioMessage. This is used when reading inbound WebSocket messages. ```Go pbMsg := chat.Message{} err = proto.Unmarshal(b, &pbMsg) if err != nil { return err // Invalid protobuf } out.Publish(model.AudioMessage{ WaveData: pbMsg.GetAudioMessage(), Message: model.Message{ Text: pbMsg.GetTextMessage(), }, }) ``` -------------------------------- ### Complete Configuration File Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md An extensive YAML configuration file showcasing all available parameters, including audio devices, VAD settings, prompts, MCP servers, tools, and agents. ```yaml serverURL: http://localhost:8080 apiKey: fake inputDevice: "KLIM Talk" outputDevice: "ALC1220 Analog" minVolume: 450 vadEnabled: true vadModelPath: /models/silero_vad.onnx sttModel: whisper-1 chatModel: qwen3-4b ttsModel: voice-en-us-amy-low temperature: 0.7 wakeWord: Computer introPrompt: | Initially, start the conversation by asking the user how you can help them and explain that they must say '{wakeWord}' in order to address you. prompt: - | You are a helpful assistant. Your name is {wakeWord}. You are interacting with the user via STT and TTS technology. Therefore, don't respond with Markdown or emoticons. mcpServers: tool-containers: command: /tool-containers-mcp args: - --config=/etc/tool-containers-mcp/tools.yaml memory: command: docker args: - run - -i - --rm - -v - /data/memory:/local-directory - mcp/memory tools: - mcpServer: tool-containers - mcpServer: memory agents: - name: research-agent description: Specialized agent for researching information prompt: - You are a research expert. Find detailed information to answer user queries. tools: - mcpServer: tool-containers allow: - wikipedia ``` -------------------------------- ### STT Client.Transcribe Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/stt-tts.md Demonstrates how to transcribe audio data using the STT client. Ensure WAV audio is 16-bit, 16kHz, and mono. ```go client := &stt.Client{ URL: "http://localhost:8080", Model: "whisper-1", Client: &http.Client{Timeout: 30*time.Second}, } wavData := readWAVFile("audio.wav") result, err := client.Transcribe(context.Background(), wavData) if err != nil { log.Fatal(err) } fmt.Println(result.Text) // "What is the weather?" ``` -------------------------------- ### Research Agent Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md An example agent configured for research tasks, with access to Wikipedia and web search tools. ```yaml agents: - name: researcher description: "Find and summarize information from the internet" prompt: - "You are a research expert." - "Find detailed, accurate information to answer user queries." - "Always cite sources when available." tools: - mcpServer: tool-containers allow: - wikipedia - web-search ``` -------------------------------- ### Build the Server Container Image Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/README.md Build the Linux container image for the server component. Requires Docker to be installed. ```sh make build-server ``` -------------------------------- ### Tool Containers Server Configuration Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Example configuration for a tool containers MCP server, specifying the command and arguments. ```yaml tool-containers: command: /tool-containers-mcp args: - --config=/etc/tool-containers-mcp/tools.yaml ``` -------------------------------- ### Selective Tool Access Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Allows specific tools from one server and all tools from another. ```yaml tools: - mcpServer: tool-containers allow: - wikipedia - calculator - mcpServer: memory ``` -------------------------------- ### Custom Tool Implementation Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Provides an example of how to implement a custom tool by satisfying the `tools.Tool` interface, including `Definition()` and `Call()` methods. ```APIDOC ## Custom Tool Implementation ### Implementing `tools.Tool` Interface To create a custom tool, implement the `tools.Tool` interface, which requires `Definition()` and `Call()` methods. ```go // Implement tools.Tool interface type MyTool struct{} func (t *MyTool) Definition() llms.FunctionDefinition { return llms.FunctionDefinition{ Name: "my_tool", Description: "Does something useful", Parameters: jsonschema.Definition{ Type: "object", Properties: map[string]jsonschema.Definition{ "query": { Type: "string", Description: "Search query", }, }, Required: []string{"query"}, }, } } func (t *MyTool) Call(ctx context.Context, params string) (string, error) { var args struct { Query string `json:"query"` } if err := json.Unmarshal([]byte(params), &args); err != nil { return "", err } // Execute logic result := fmt.Sprintf("Result for: %s", args.Query) return result, nil } // Use in chat completion tools := []tools.Tool{&MyTool{}} err := llm.ChatCompletion(ctx, reqNum, tools, conv, ch) ``` ``` -------------------------------- ### Configure STT and TTS Clients Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/stt-tts.md Initialize STT and TTS clients using configuration values. This example shows how to set up the `stt.Transcriber` and `tts.SpeechGenerator` with server URL, model names, an HTTP client, and an API key. ```go // STT transcriber := &stt.Transcriber{ Service: &stt.Client{ URL: cfg.ServerURL, Model: cfg.STTModel, // e.g., "whisper-1" Client: httpClient, }, } // TTS generator := &tts.SpeechGenerator{ Service: &tts.Client{ URL: cfg.ServerURL, Model: cfg.TTSModel, // e.g., "voice-en-us-amy-low" Client: httpClient, APIKey: cfg.APIKey, }, } ``` -------------------------------- ### Agent.AsTool Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Shows how to convert an Agent struct into a callable tool using the AsTool method. This is useful for integrating agents into the main LLM's toolset. ```go agent := &chat.Agent{ Name: "researcher", Description: "Research information", SystemPrompt: "You are a research expert", Tools: toolProvider, LLM: llm, } tool := agent.AsTool(42, responseChan) llmTools = append(llmTools, tool) ``` -------------------------------- ### PbWebsocket send() Method Examples Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Demonstrates sending either a text message or an audio message using the PbWebsocket client's send method. ```javascript // Send text message websocket.send({'textMessage': 'Hello, assistant'}) ``` ```javascript // Send audio message websocket.send({'audioMessage': wavBytes}) ``` -------------------------------- ### Configure Server with TLS Certificates Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Starts the server with TLS/HTTPS support by providing paths to the certificate and key files. This is essential for secure communication, especially for WebSockets. ```bash ./ai-assistant-vui-server \ -tlsCert=/path/to/cert.pem \ -tlsKey=/path/to/key.pem ``` -------------------------------- ### Specify Configuration File Path Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Use the -config flag to specify the path to your YAML configuration file when starting the server or client. ```bash ./ai-assistant-vui-server -config /path/to/config.yaml ./ai-assistant-vui -config /path/to/config.yaml ``` -------------------------------- ### PbWebsocket.connect() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Establishes the WebSocket connection. This method handles connection setup, including registering message and error handlers, and decoding protobuf messages. ```APIDOC ## PbWebsocket.connect() ### Description Establishes the WebSocket connection. This method handles connection setup, including registering message and error handlers, and decoding protobuf messages. ### Method `connect()` ### Behavior 1. Check if already connected or connecting (prevent duplicates) 2. Create WebSocket instance 3. Set binary message mode 4. Register handlers: - `onmessage`: Decode protobuf and call handler - `onopen`: Log successful connection - `onerror`: Log error and trigger reconnect - `onclose`: Log closure and trigger reconnect 5. Handle connection errors gracefully ### Message Decoding ```javascript ws.onmessage = (event) => { const msg = chat.Message.decode(new Uint8Array(event.data)); // msg.role, msg.textMessage, msg.audioMessage this._onmessage(msg); }; ``` ``` -------------------------------- ### Custom Tool Implementation Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Provides an example of implementing the tools.Tool interface for a custom tool, including its definition and call logic. This custom tool can then be used in chat completions. ```go // Implement tools.Tool interface type MyTool struct{} func (t *MyTool) Definition() llms.FunctionDefinition { return llms.FunctionDefinition{ Name: "my_tool", Description: "Does something useful", Parameters: jsonschema.Definition{ Type: "object", Properties: map[string]jsonschema.Definition{ "query": { Type: "string", Description: "Search query", }, }, Required: []string{"query"}, }, } } func (t *MyTool) Call(ctx context.Context, params string) (string, error) { var args struct { Query string `json:"query"` } if err := json.Unmarshal([]byte(params), &args); err != nil { return "", err } // Execute logic result := fmt.Sprintf("Result for: %s", args.Query) return result, nil } // Use in chat completion tools := []tools.Tool{&MyTool{}} er := llm.ChatCompletion(ctx, reqNum, tools, conv, ch) ``` -------------------------------- ### Memory Server via Docker Configuration Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/configuration.md Example configuration for a memory MCP server using Docker, including volume mounts. ```yaml memory: command: docker args: - run - -i - --rm - -v - /data/memory:/local-directory - mcp/memory ``` -------------------------------- ### STT Transcriber.Transcribe Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/stt-tts.md Shows how to use the Transcriber to process a stream of audio messages into a stream of text. Errors are logged but processing continues. ```go transcriber := &stt.Transcriber{ Service: client, // STT client } audioInput := getAudioStream() // Some <-chan AudioMessage textOutput := transcriber.Transcribe(context.Background(), audioInput) for text := range textOutput { fmt.Println(text.Text) // Transcribed user input } ``` -------------------------------- ### AddUserRequestsToConversation Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Demonstrates how to use the AddUserRequestsToConversation method. It initializes a Requester, sets up channels for input messages, completion requests, and feedback events, and then sends a user message. ```go requester := &chat.Requester{ NotificationSampleRate: 16000, } inputMessages := make(chan model.AudioMessage) completionRequests, feedbackEvents := requester.AddUserRequestsToConversation( context.Background(), inputMessages, conversation, ) // Process requests go func() { for req := range completionRequests { // Trigger chat completion for this request } }() // Process feedback (display on UI) go func() { for event := range feedbackEvents { // Play audio or display text } }() // Send user messages inputMessages <- model.AudioMessage{ Message: model.Message{Text: "Hello"}, } ``` -------------------------------- ### Build the VUI Container Image Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/README.md Build the Linux container image for the terminal application (VUI). Requires Docker to be installed. ```sh make build-vui ``` -------------------------------- ### Protobuf Encoding Example Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Shows how to encode a model.Message into a protobuf chat.Message for streaming server events over WebSockets. It handles setting the role, text, and audio data. ```Go pbMsg := chat.Message{} switch msg.Role { case model.RoleUser: pbMsg.SetRole(chat.Role_USER) case model.RoleAssistant: pbMsg.SetRole(chat.Role_ASSISTANT) } if msg.Text != "" { pbMsg.SetTextMessage(msg.Text) } if len(msg.WaveData) > 0 { pbMsg.SetAudioMessage(msg.WaveData) } // Send binary protobuf ws.Write(ctx, websocket.MessageBinary, data) ``` -------------------------------- ### Start AI Assistant VUI Server with TLS Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/endpoints.md Launch the AI Assistant VUI server with custom TLS certificates. This is necessary for enabling HTTPS and secure WebSocket connections (wss://). ```bash ./ai-assistant-vui-server -tlsCert=/path/to/cert.pem -tlsKey=/path/to/key.pem ``` -------------------------------- ### Run the LocalAI API Server Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/README.md Start the LocalAI API server, which serves as the LLM server for the project. This command should be run from the project's root directory. ```sh make run-localai ``` -------------------------------- ### Channel Internal Pipeline Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Illustrates the internal processing pipeline for a Channel, starting from the input channel through audio processing to the output event stream. ```go input channel ↓ audio processing (VAD, STT, wake word, LLM, TTS) ↓ event stream ↓ output pub/sub ``` -------------------------------- ### Create New Conversation Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Initializes a new conversation with a system prompt and a starting request number. Allocates message slice with capacity 100. ```go func NewConversation(systemPrompt string, reqNum int64) *Conversation ``` ```go conversation := model.NewConversation( "You are a helpful assistant named Computer", 1, ) ``` -------------------------------- ### Get Available Tools Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Retrieves a list of tools available from an MCP server. This function queries the server, converts MCP tools to the adapter format, and returns the list. ```go func (p *mcpToolProvider) Tools(ctx context.Context) ([]tools.Tool, error) ``` -------------------------------- ### Configuration Flow Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/INDEX.md Details the flow of configuration data from the config.yaml file through various components, including LLM initialization, tool setup, and audio device selection. This outlines how settings are applied across the system. ```text config.yaml (configuration.md) ↓ Configuration struct (types.md) ↓ LLM initialization (chat-completion.md) ↓ Tool/MCP setup (tools-mcp.md) ↓ Audio device selection (audio-io.md) ↓ Channel creation (pubsub-channels.md) ``` -------------------------------- ### Run the AI Assistant Server Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/README.md Start the main server for the AI assistant. This should be run in a separate terminal after LocalAI is running. Access the web app at https://localhost:8443. ```sh make run-server ``` -------------------------------- ### Import Core Modules Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Imports necessary modules for the web UI application, including VAD, audio processing, WebSocket communication, and UI setup. ```javascript import { MicVAD } from "@ricky0123/vad-web" import { riffWaveFromFloat32Array } from './riffwave.js' import { RealTimeAudioPlayer } from './audioplayer.js' import { PbWebsocket } from './pbwebsocket.js' import { setupUI } from './ui.js' ``` -------------------------------- ### HTTP Audio Streaming (Polling) Request Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Example of requesting an audio stream via HTTP polling. This uses a GET request with an 'Accept' header for audio/wav. ```http GET https://localhost:8443/channels/default/audio?buffer-ms=1250 Accept: audio/wav ``` -------------------------------- ### Initialize MicVAD for Speech Detection Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Sets up and starts the microphone-based Voice Activity Detection (VAD) to capture speech and convert it to WAV format upon speech end. ```javascript const myvad = await MicVAD.new({ onSpeechEnd: async function(audioSamples) { // Triggered when speech ends const wav = riffWaveFromFloat32Array(audioSamples, 16000); websocket.send({'audioMessage': wav}); }, }); myvad.start() // Begin listening ``` -------------------------------- ### Send Audio via curl Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/endpoints.md Example of sending raw audio data to the audio channel endpoint using curl. Ensure the Content-Type header is set to application/octet-stream. ```bash curl -X POST https://localhost:8443/channels/default/audio \ -H "Content-Type: application/octet-stream" \ --data-binary @audio.wav ``` -------------------------------- ### Tool Interface Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Defines the core abstraction for callable functions, including how to get its definition and how to execute it. ```APIDOC ## Tool Interface Core abstraction for callable functions. ```go type Tool interface { Definition() llms.FunctionDefinition // Function schema Call(ctx context.Context, params string) (string, error) // Execute } ``` ``` -------------------------------- ### Initialize and Run Chat Completer Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Initializes a chat completer with an LLM, intro prompt, tools, and agents. It then runs the completer, feeding requests through a channel and consuming responses. ```go completer := &chat.Completer{ LLM: llm, IntroPrompt: "Ask the user how you can help", Tools: toolProvider, Agents: []chat.Agent{researchAgent}, } // Source of requests (e.g., from Requester) requestChan := make(chan chat.ChatCompletionRequest) responses, err := completer.Run(context.Background(), requestChan, conversation) if err != nil { log.Fatal(err) } // Feed requests go func() { requestChan <- chat.ChatCompletionRequest{RequestNum: 1} close(requestChan) }() // Consume responses for chunk := range responses { handleChunk(chunk) } ``` -------------------------------- ### Channels Interface Definition Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/pubsub-channels.md Defines the interface for managing multiple channels, including getting or creating channels by ID. ```go type Channels interface { GetOrCreate(channelId string) (*Channel, error) } ``` -------------------------------- ### Integrate Tools with Chat Completion Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Demonstrates how to fetch tools, add agents as tools, and then call the LLM's ChatCompletion method with the combined list of tools. ```go tools, err := toolProvider.Tools(ctx) if err != nil { return err } // Add agents as tools for _, agent := range agents { tools = append(tools, agent.AsTool(reqNum, ch)) } // Call LLM with tools err = llm.ChatCompletion(ctx, reqNum, tools, conv, ch) ``` -------------------------------- ### List Audio Devices with PortAudio Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/audio-io.md Demonstrates how to list available audio devices using the PortAudio library. This is useful for troubleshooting and device selection. ```go import "github.com/gordonklaus/portaudio" // List devices devices, err := portaudio.Devices() // Get device info dev := devices[0] fmt.Println(dev.Name) fmt.Println(dev.DefaultSampleRate) fmt.Println(dev.MaxInputChannels) ``` -------------------------------- ### Get Full Message History Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Retrieves the complete message history of the conversation. Note that this method is not thread-safe for concurrent access. ```go history := conversation.Messages() // [system_message, user_message_1, assistant_message_1, tool_call, tool_result, ...] ``` -------------------------------- ### Configure Chat LLM Parameters Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Initialize a chat LLM client with parameters derived from a configuration object. Includes settings for server URL, API key, model, temperature, and loop prevention. ```go llm := &chat.LLM{ ServerURL: cfg.ServerURL, APIKey: cfg.APIKey, Model: cfg.ChatModel, Temperature: cfg.Temperature, FrequencyPenalty: 1.5, // Fixed MaxTokens: 0, // No limit MaxTurns: 5, // Infinite loop prevention StripResponsePrefix: fmt.Sprintf("%s:", cfg.WakeWord), HTTPClient: httpClient, } ``` -------------------------------- ### Configure HTTP Client for LLM Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Demonstrates how to set a custom HTTP client for an LLM instance, configuring timeout and transport settings. ```go llm.HTTPClient = &http.Client{ Timeout: 90 * time.Second, Transport: &http.Transport{ MaxIdleConns: 10, }, } ``` -------------------------------- ### Error Handling Reference Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/MANIFEST.txt Comprehensive guide to error types, trigger conditions, and recovery strategies for the AI Assistant VUI. ```APIDOC ## Error Handling Reference ### Description This document enumerates all error types within the AI Assistant VUI, detailing their trigger conditions and recommended recovery strategies. ### File - errors.md ### Content - All error types enumerated - Trigger conditions documented - Recovery strategies provided ``` -------------------------------- ### Get System Prompt Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Retrieves the current system prompt text in a thread-safe manner. Returns the exact text from the first message. ```go func (c *Conversation) SystemPrompt() string ``` ```go prompt := conversation.SystemPrompt() fmt.Println(prompt) ``` -------------------------------- ### RealTimeAudioPlayer Constructor Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the RealTimeAudioPlayer, setting up the necessary components for audio playback. ```APIDOC ## RealTimeAudioPlayer Constructor ### Description Initializes the RealTimeAudioPlayer, setting up the necessary components for audio playback. ### Method `new RealTimeAudioPlayer()` ### Behavior Initializes audio context and playback buffers. ``` -------------------------------- ### sendHTTPAudioStream() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Streams audio via chunked HTTP for non-WebSocket GET requests. It allows configuring a buffer duration for silence padding. ```APIDOC ## sendHTTPAudioStream() ### Description Stream audio via chunked HTTP (non-WebSocket GET). ### Parameters - **c**: Channel to subscribe to - **w**: Response writer - **req**: HTTP request with query parameters ### Query Parameters #### Query Parameters - **buffer-ms** (uint64) - Optional - Default: 1250 - Buffer duration for padding silence ### Response Headers ``` Content-Type: audio/wav Transfer-Encoding: chunked Cache-Control: no-cache, no-store, must-revalidate X-Accel-Buffering: no Pragma: no-cache Expires: 0 ``` ### Audio Format - WAV container (16-bit, 16kHz, mono) - Chunked streaming - Padded with silence to `buffer-ms` after each utterance ``` -------------------------------- ### Initialize OpenAI Compatible Client Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/chat-completion.md Create a new client for an OpenAI-compatible server, specifying the HTTP client, base URL, API key, and model. This is used for integrating with services like LocalAI or ollama. ```go llm, err := openai.New( openai.WithHTTPClient(c.HTTPClient), openai.WithBaseURL(c.ServerURL+"/v1"), openai.WithToken(c.APIKey), openai.WithModel(c.Model), ) ``` -------------------------------- ### RealTimeAudioPlayer Constructor Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the RealTimeAudioPlayer, setting up the necessary Web Audio API context and internal buffers for audio playback. ```javascript new RealTimeAudioPlayer() ``` -------------------------------- ### Clone the AI Assistant VUI Repository Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/README.md Clone the project repository to your local machine. This is the first step to setting up the project. ```sh git clone https://github.com/mgoltzsche/ai-assistant-vui.git cd ai-assistant-vui ``` -------------------------------- ### Tool Timeout Configuration Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Demonstrates setting a custom HTTP client with a timeout to control how long a tool can run before being cancelled due to context deadline exceeded. ```go httpClient := &http.Client{ Timeout: 90 * time.Second, } llm.HTTPClient = httpClient ``` -------------------------------- ### Tool Interface Definition Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Defines the core abstraction for callable functions, including a method to get the function schema and another to execute the function. ```go type Tool interface { Definition() llms.FunctionDefinition // Function schema Call(ctx context.Context, params string) (string, error) // Execute } ``` -------------------------------- ### Conversation.RequestCounter Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Gets the current request sequence number. This number is monotonically increasing and used to detect outdated responses, ensuring thread-safe reads. ```APIDOC ## Conversation.RequestCounter() ### Description Get current request sequence number. ### Signature ```go func (c *Conversation) RequestCounter() int64 ``` ### Returns - **int64**: Current request number (incremented on user request) ### Details - Thread-safe read - Used to detect outdated responses - Monotonically increasing ### Example ```go currentReq := conversation.RequestCounter() if receivedReq < currentReq { // Response from outdated request, ignore } ``` ``` -------------------------------- ### Audio Channel - GET /channels/{channelId}/audio Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/endpoints.md Streams audio response from the server to the client via WebSocket upgrade or chunked HTTP. ```APIDOC ## Audio Channel - GET /channels/{channelId}/audio ### Description Streams audio response from the server to the client via WebSocket upgrade or chunked HTTP. ### Method GET ### Endpoint /channels/{channelId}/audio ### Parameters #### Path Parameters - **channelId** (string) - Required - Unique channel identifier (e.g., "default", UUID) #### Query Parameters - **buffer-ms** (uint64) - Optional - Default: 1250 - Client-side buffer duration in milliseconds. Used to pad output with silence after speech playback. ### Request Headers - **Connection**: Upgrade # For WebSocket - **Accept**: audio/x-raw # Optional, for raw audio format ### Response #### Success Response (200) - Response Headers (WebSocket upgrade): - Upgrade: websocket - Connection: Upgrade - Response Headers (Chunked HTTP): - Content-Type: audio/wav - Transfer-Encoding: chunked - Cache-Control: no-cache, no-store, must-revalidate - X-Accel-Buffering: no - Response Body: - WebSocket: Binary frames containing protobuf-encoded `chat.Message` - HTTP: Chunked WAV audio stream #### Error Responses - Status 400: Invalid buffer-ms value - Status 500: Stream error ``` -------------------------------- ### PbWebsocket.start() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Initializes the WebSocket connection for the PbWebsocket client. It automatically attempts to reconnect if the connection is lost. ```APIDOC ## PbWebsocket.start() ### Description Initializes the WebSocket connection for the PbWebsocket client. It automatically attempts to reconnect if the connection is lost. ### Method `start()` ### Behavior Initiates connection attempt. Automatically reconnects on error. ``` -------------------------------- ### PbWebsocket connect() Method Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/web-ui.md Establishes the WebSocket connection, including setting up message handlers for receiving, errors, and closure. It prevents duplicate connection attempts and handles errors gracefully. ```javascript connect() ``` -------------------------------- ### Get Request Counter Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Retrieves the current request sequence number in a thread-safe manner. This number is monotonically increasing and used to detect outdated responses. ```go func (c *Conversation) RequestCounter() int64 ``` ```go currentReq := conversation.RequestCounter() if receivedReq < currentReq { // Response from outdated request, ignore } ``` -------------------------------- ### Define Configuration Struct Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/types.md Complete system configuration loaded from YAML. Includes settings for API endpoints, audio devices, models, and agent definitions. ```go type Configuration struct { ServerURL string // OpenAI-compatible API server URL APIKey string // API authentication key InputDevice string // Audio input device name or ID OutputDevice string // Audio output device name or ID MinVolume int // Minimum microphone volume threshold VADEnabled bool // Enable voice activity detection VADModelPath string // Path to silero-vad ONNX model STTModel string // Speech-to-text model name (e.g., "whisper-1") TTSModel string // Text-to-speech model name ChatModel string // Chat completion model name Temperature float64 // LLM temperature (0.0 to 2.0) WakeWord string // Wake word for activation (e.g., "Computer") IntroPrompt string // Initial greeting prompt template MCPServers map[string]MCPServer // Configured MCP servers Agents []AgentDefinition // Agent configurations AgentDefinition // Inline agent definition } ``` -------------------------------- ### Tool Lifecycle Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Illustrates the lifecycle of a tool, from configuration loading to execution within the AI assistant's conversation flow. ```APIDOC ## Tool Lifecycle ### Tool Availability Flow 1. Configuration loaded 2. MCP servers started 3. ToolProvider initialized 4. User makes request 5. LLM.ChatCompletion() calls ToolProvider.Tools() 6. Tools listed in function definitions 7. LLM decides which tools to call 8. Tool.Call() executes 9. Result added to conversation 10. LLM continues (may call more tools) ``` -------------------------------- ### Configuration Reference Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/MANIFEST.txt Detailed documentation for all configuration options available for the AI Assistant VUI. ```APIDOC ## Configuration Reference ### Description This section details all configuration options for the AI Assistant VUI, including their types, default values, and usage examples, typically found in `config.yaml`. ### File - configuration.md ### Content - All config options documented - Type and default values specified - Examples provided ``` -------------------------------- ### Noop() Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Creates a ToolProvider that returns no tools, useful for configurations without available tools. ```APIDOC ## Noop() Create a no-op tool provider that returns no tools. ```go func Noop() ToolProvider ``` Used in configurations where no tools are available. ``` -------------------------------- ### Integration with Chat Completion Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/tools-mcp.md Demonstrates how to integrate the `ToolProvider` with the LLM's chat completion process, including fetching tools and passing them to the completion function. ```APIDOC ## Integration with Chat Completion ### Usage Example ```go tools, err := toolProvider.Tools(ctx) if err != nil { return err } // Add agents as tools for _, agent := range agents { tools = append(tools, agent.AsTool(reqNum, ch)) } // Call LLM with tools err = llm.ChatCompletion(ctx, reqNum, tools, conv, ch) ``` ``` -------------------------------- ### Configure Input and Output Audio Devices Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/audio-io.md Sets the input and output audio devices using a YAML configuration file or environment variables. Ensure device names are exact and case-sensitive. ```yaml inputDevice: "KLIM Talk" # or "1" or "" outputDevice: "ALC1220" # or "2" or "" ``` -------------------------------- ### NewConversation Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/conversation-model.md Creates a new, thread-safe conversation instance with an initial system prompt and starting request number. It initializes the message history and sets up the conversation for multi-turn interactions. ```APIDOC ## NewConversation() ### Description Create a new conversation with initial system prompt. ### Signature ```go func NewConversation(systemPrompt string, reqNum int64) *Conversation ``` ### Parameters #### Parameters - **systemPrompt** (string) - Initial system prompt for the AI - **reqNum** (int64) - Starting request counter (typically 1) ### Returns - ** *Conversation**: Initialized conversation ### Details - Allocates message slice with capacity 100 - Sets system prompt as first message - Initializes request counter to `reqNum` ### Example ```go conversation := model.NewConversation( "You are a helpful assistant named Computer", 1, ) ``` ``` -------------------------------- ### Go WebSocket Client for AI Assistant Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/endpoints.md Establish a WebSocket connection using Go, send audio messages as protobuf-encoded binary data, and unmarshal received messages. ```go ctx := context.Background() ws, _, err := websocket.Dial(ctx, "wss://localhost:8443/channels/default/audio", nil) def ws.Close(websocket.StatusNormalClosure, "done") // Send message msg := &chat.Message{AudioMessage: wavBytes} data, _ := proto.Marshal(msg) ws.Write(ctx, websocket.MessageBinary, data) // Receive message msgType, data, _ := ws.Read(ctx) var resp chat.Message proto.Unmarshal(data, &resp) ``` -------------------------------- ### Send Audio via HTTP POST Source: https://github.com/mgoltzsche/ai-assistant-vui/blob/main/_autodocs/api-reference/server-http.md Example of sending audio data for transcription via an HTTP POST request. The audio data should be in the request body with 'Content-Type: application/octet-stream'. ```http POST https://localhost:8443/channels/default/audio Content-Type: application/octet-stream [WAV data bytes] ```