### Running and Testing Open LLM VTuber This section outlines the steps required to run and test the Open LLM VTuber application. It covers dependency installation, configuration setup, starting the main server, connecting the frontend UI, launching the platform-specific script, and observing the system's response to messages sent on the live platform. ```bash # 1. Install dependencies. # pip install -r requirements.txt # 2. Configure conf.yaml. # (Edit conf.yaml with your platform's details) # 3. Start the main program (uvicorn server). # uvicorn main:app --reload # 4. Ensure the frontend UI connects to '/proxy-ws' # (Modify the WebSocket URL in frontend settings). # 5. In another terminal, start the platform script. # uv run python scripts/run_my_platform_live.py # 6. Send messages on the live platform and observe logs and frontend UI. ``` -------------------------------- ### Run the Project Server Command to start the main server script for the project. This assumes all dependencies are installed and the environment is set up correctly. ```bash python run_server.py ``` -------------------------------- ### Install Git on macOS using Homebrew Installs Git on macOS using the Homebrew package manager. It first ensures Homebrew is installed by running the official installation script, then proceeds to install Git. ```bash # If Homebrew is not installed, run this command first, or refer to https://brew.sh/zh-cn/ for installation. /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install Git brew install git ``` -------------------------------- ### Install Git on CentOS/RHEL Installs Git on CentOS or RHEL-based Linux distributions using the dnf package manager. ```bash # CentOS/RHEL sudo dnf install git ``` -------------------------------- ### Install Git on Windows using winget Installs Git on Windows using the winget package manager. This command assumes winget is available on your system. If not, it advises downloading winget from the Microsoft Store or installing Git directly from its official website for older Windows versions. ```bash # In the command line, run winget install Git.Git ``` -------------------------------- ### Project Setup Steps This outlines the core steps for setting up the Open LLM VTuber project, from obtaining the code to configuring and launching the application. ```bash 1. Get project code (e.g., git clone ...) 2. Install project dependencies (e.g., pip install -r requirements.txt) 3. Configure LLM (e.g., edit conf.yaml) 4. Configure other modules 5. Start the project ``` -------------------------------- ### Install Project Dependencies with Pip Installs project dependencies listed in the requirements.txt file using pip. This is a standard method for managing Python package installations. ```bash pip install -r requirements.txt ``` -------------------------------- ### Check Ollama Installation Verifies that Ollama is installed and accessible. ```bash ollama --version ``` -------------------------------- ### Install Git on Ubuntu/Debian Installs Git on Ubuntu or Debian-based Linux distributions using the apt package manager. ```bash # Ubuntu/Debian sudo apt install git ``` -------------------------------- ### TTS Installation Guide Provides guidance on installing and setting up Text-to-Speech (TTS) for the Open LLM Vtuber backend. It humorously notes that TTS installation is the most difficult step, followed by getting others to install it. ```APIDOC TTS Installation: Description: The most challenging part of adding TTS support, followed by ensuring others can also install it. Dependencies: None explicitly mentioned, but implies system-level dependencies for TTS engines. Input: None Output: Functional TTS integration for the backend. Limitations: Assumes user has basic command-line and system configuration knowledge. ``` -------------------------------- ### Check uv Installation Verifies that the uv package manager is installed and accessible. ```bash uv --version ``` -------------------------------- ### Install uv on Windows Installs the 'uv' dependency manager on Windows using PowerShell or winget. It's recommended for managing project dependencies. ```bash # 方法 1: PowerShell powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" # 方法 2: winget winget install --id=astral-sh.uv -e # 重要:对于 winget,安装完成后请重启命令行 / IDE ``` -------------------------------- ### Synchronize Project Dependencies Creates a virtual environment named .venv and installs all project dependencies using uv. ```bash # Ensure you are in the project root directory uv sync ``` -------------------------------- ### Install FFmpeg on Ubuntu/Debian Installs FFmpeg on Ubuntu or Debian-based Linux distributions using the apt package manager. ```bash # Ubuntu/Debian sudo apt install ffmpeg ``` -------------------------------- ### Install FFmpeg on macOS using Homebrew Installs FFmpeg on macOS using the Homebrew package manager. It first ensures Homebrew is installed by running the official installation script, then proceeds to install FFmpeg. ```bash # If Homebrew is not installed, run this command first, or refer to https://brew.sh/zh-cn/ for installation. /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install ffmpeg brew install ffmpeg ``` -------------------------------- ### List Installed Ollama Models Displays a list of all models currently downloaded and available in Ollama. ```bash ollama list ``` -------------------------------- ### Verify FFmpeg Installation Checks if FFmpeg has been successfully installed by running the 'ffmpeg -version' command in the terminal. A successful installation will display the FFmpeg version information. ```bash ffmpeg -version ``` -------------------------------- ### Install FFmpeg on CentOS/RHEL Installs FFmpeg on CentOS or RHEL-based Linux distributions using the dnf package manager. ```bash # CentOS/RHEL sudo dnf install ffmpeg ``` -------------------------------- ### Install FFmpeg on Windows using winget Installs FFmpeg on Windows using the winget package manager. This command assumes winget is available on your system. If not, it advises downloading winget from the Microsoft Store or installing FFmpeg directly from its official website for older Windows versions. ```bash # In the command line, run winget install ffmpeg ``` -------------------------------- ### Verify CUDA Toolkit Installation Command to check the installed CUDA Toolkit version. This confirms that the CUDA compiler (nvcc) is accessible and correctly installed. ```bash nvcc --version ``` -------------------------------- ### Run Backend Server Starts the Open-LLM-VTuber backend service. The first run may take time due to model downloads. ```bash uv run run_server.py ``` -------------------------------- ### BiliBiliLivePlatform Implementation Example A concrete implementation of the LivePlatformInterface for Bilibili, demonstrating how to connect to the proxy and format danmaku messages. ```APIDOC BiliBiliLivePlatform (Implementation of LivePlatformInterface) Location: src/open_llm_vtuber/live/bilibili_live.py Description: Provides a specific implementation for integrating with Bilibili live streams. It utilizes the `blivedm` library to capture danmaku events and forwards them to the Open LLM Vtuber proxy. Key Features: - Connects to the `/proxy-ws` endpoint using the `connect` method. - Uses `blivedm` to listen for danmaku events from a Bilibili live room. - Formats captured danmaku messages into the required `{"type": "text-input", ...}` structure within the `_send_to_proxy` method before sending them to the proxy. Example of `_send_to_proxy` logic: ```python # Inside BiliBiliLivePlatform class def _send_to_proxy(self, message: dict): # message is typically a danmaku event from blivedm # Format it for the proxy: formatted_message = { "type": "text-input", "text": message.get('content', '') # Assuming 'content' holds the danmaku text } # Send using the established WebSocket connection self.websocket_connection.send(json.dumps(formatted_message)) ``` Workflow: 1. A viewer sends danmaku to a Bilibili live room. 2. `blivedm` in `run_bilibili_live.py` receives the danmaku event. 3. The `BiliBiliLivePlatform` instance captures this event. 4. The `_send_to_proxy` method formats the danmaku into `{"type": "text-input", "text": "..."}`. 5. The formatted message is sent via WebSocket to the `/proxy-ws` endpoint in the main Open LLM Vtuber process. ``` -------------------------------- ### Install Project in Editable Mode with Pip Installs the project in editable mode using pip, which links the project files directly into the Python environment. This is useful during development. ```bash pip install -e . ``` -------------------------------- ### Ollama LLM Configuration in conf.yaml Example configuration snippet for Ollama within the conf.yaml file, specifying the model, base URL, and temperature. ```yaml ollama_llm: base_url: http://localhost:11434 # Keep default for local runs model: qwen2.5:latest # Model name obtained from 'ollama list' temperature:0.7 # Controls response randomness (0-1), higher is more random ``` -------------------------------- ### Install uv on macOS/Linux Installs the 'uv' dependency manager on macOS or Linux using curl or Homebrew. It's recommended for managing project dependencies. ```bash # 方法 1: curl curl -LsSf https://astral.sh/uv/install.sh | sh # 或者运行 wget -qO- https://astral.sh/uv/install.sh | sh 如果你的电脑上不存在 curl # 方法 2: homebrew (如果已安装) brew install uv # 重要:安装完成后请运行以下命令重新加载配置文件,或者重启命令行 / IDE source ~/.bashrc # 如果使用 bash # 或 source ~/.zshrc # 如果使用 zsh ``` -------------------------------- ### MyPlatformLive Implementation Example Example implementation of a custom live platform class inheriting from LivePlatformInterface. It demonstrates handling platform events, sending formatted messages to a proxy WebSocket, and the basic structure for the run method. ```python # src/open_llm_vtuber/live/my_platform_live.py (关键部分示例) # ... (imports) ... classMyPlatformLive(LivePlatformInterface): # ... (实现 __init__, connect, disconnect, run 等方法) ... asyncdef_handle_platform_event(self, event_data): message_text = event_data.get('message') if message_text: logger.info(f"Received from My Platform: {message_text}") # 调用发送给代理的方法 await self._send_to_proxy(message_text) asyncdef_send_to_proxy(self, text:str)->bool: ifnot self.is_connected: logger.error("Cannot send message: Not connected to proxy") returnFalse try: # !!! 核心:格式化为 text-input 类型 !!! payload ={"type":"text-input","text": text} await self._websocket.send(json.dumps(payload)) logger.info(f"Sent formatted message to proxy: {text}") returnTrue except Exception as e: # ... (错误处理) returnFalse asyncdefrun(self)->None: # 确保连接到代理 proxy_url = "ws://localhost:12393/proxy-ws" # ... (连接逻辑) ... # --- 你的平台事件监听逻辑 --- # # while self._running: # event = await my_platform_sdk.get_next_event() # await self._handle_platform_event(event) # ----------------------------- # ... (清理逻辑) ... ``` -------------------------------- ### Configuration File Example Provides an example snippet of the `conf.yaml` file, demonstrating how to specify the TTS model and its associated configuration options. ```yaml tts_config: tts_model: "edge_tts" # text to speech model options: # "azure_tts", "pyttsx3_tts", "edge_tts", "bark_tts", # "cosyvoice_tts", "melo_tts", "coqui_tts", ``` -------------------------------- ### Verify NVIDIA Driver Installation Command to check the status and details of the installed NVIDIA graphics driver. It displays information about the driver version and GPU status. ```bash nvidia-smi ``` -------------------------------- ### Navigate to Project Directory Change the current directory to the Open-LLM-VTuber project folder. ```bash cd Open-LLM-VTuber ``` -------------------------------- ### Ollama Installation and Model Management Instructions for installing Ollama, verifying the installation, downloading and running a model (e.g., `qwen2.5:latest`), and listing installed models. ```bash ollama --version ``` ```bash ollama run qwen2.5:latest # After successful execution, you can chat with qwen2.5:latest # You can exit the chat interface (Ctrl/Command + D), but do not close the command line. ``` ```bash ollama list # NAME ID SIZE MODIFIED # qwen2.5:latest 845dbda0ea48 4.7 GB 2 minutes ago ``` -------------------------------- ### Project Navigation Links to different sections of the documentation, including Quick Start, Detailed Guides, API Documentation, and community resources like GitHub and Discord. ```en Quick Start Detailed Guide API Documentation GitHub Discord QQ Group Chat ``` -------------------------------- ### Activate Conda Environment Activates the specified Conda environment, making its Python interpreter and installed packages available in the current shell session. ```bash conda activate ./.conda ``` -------------------------------- ### Start Bilibili Live Client Command to launch the Bilibili live client script. This script captures comments from the Bilibili live room and forwards them to the main Open-LLM-VTuber program via WebSocket. ```bash uv run python scripts/run_bilibili_live.py ``` -------------------------------- ### TTS Engine Configurations Configuration examples for different TTS engines. Each engine has specific parameters for voice, pitch, rate, and API keys. ```yaml azure_tts: api_key:"azure-api-key" region:"eastus" voice:"en-US-AshleyNeural" pitch:"26"# percentage of the pitch adjustment rate:"1"# rate of speak bark_tts: voice:"v2/en_speaker_1" edge_tts: # Check out doc at https://github.com/rany2/edge-tts # Use `edge-tts --list-voices` to list all available voices voice:"en-US-AvaMultilingualNeural"# "en-US-AvaMultilingualNeural" #"zh-CN-XiaoxiaoNeural" # "ja-JP-NanamiNeural" ``` -------------------------------- ### Configuration File Structure Details the basic structure of the configuration file for version 1.0.0. This guide helps users understand how to set up and customize the project's behavior by modifying configuration parameters. ```APIDOC Configuration File (conf.yaml): - Structure for v1.0.0 - Key configuration sections: - ASR_MODEL: Specifies the Automatic Speech Recognition model. - LLM_MODEL: Specifies the Large Language Model backend and model. - AGENT_TYPE: Defines the type of agent, e.g., 'basicmemoryagent'. - TTS_MODEL: Selects the Text-to-Speech engine. - TRANSLATE_PROVIDER: Configures the translation service. - Other parameters for customization and integration. ``` -------------------------------- ### Live2D Model Integration Guide Instructions for adding new Live2D models to the project. This process typically involves three to five steps to ensure proper integration and functionality. ```en To add a new Live2D model to your project, you need to complete the following three to five steps: 1. Prepare your Live2D model assets according to the project's specifications. 2. Place the model files in the designated directory within your project. 3. Configure the model settings in the project's configuration files. 4. (Optional) Test the model integration through the project's interface. 5. (Optional) Adjust parameters for optimal performance. ``` -------------------------------- ### Install FunASR Dependencies Installs the necessary Python packages for the FunASR backend using uv. Includes alternative commands for resolving potential dependency conflicts with llvmlite. ```bash uv add funasr modelscope huggingface_hub onnxconverter_common torch torchaudio onnx ``` ```bash uv pip install funasr modelscope huggingface_hub torch torchaudio onnx onnxconverter_common ``` -------------------------------- ### Live2D Model Configuration Example Example JSON configuration for defining tap motions and associated weights for Live2D models. This configuration links user interactions (like tapping specific body parts) to predefined animations within the model's files. ```json { "name": "shizuku-local", "tapMotions": { "body": { "tap_body": 30, "shake": 30 }, "head": { "flick_head": 40 } } } ``` -------------------------------- ### Remote Deployment & Cross-Device Access This guide is intended for scenarios involving remote deployment and accessing the Open LLM Vtuber application across different devices. It covers the necessary steps and considerations for setting up remote access. ```APIDOC Remote Deployment & Cross-Device Access: - Target Scenarios: Remote server deployment, accessing from various devices. - Configuration: Network settings, port forwarding, security considerations. - Access Methods: How to connect to the application remotely. - Troubleshooting: Common issues and solutions for remote access. ``` -------------------------------- ### Ollama Commands This snippet provides essential Ollama commands for managing models and checking their status. ```bash ollama list # Lists all installed models. ollama run # Runs a specified model. ``` -------------------------------- ### Environment Preparation This section details the necessary software and configurations for preparing the development environment, including Git, FFmpeg, and Python. ```bash # Install Git # Install FFmpeg # NVIDIA GPU Support (if applicable) # Python environment management (e.g., using venv or conda) ``` -------------------------------- ### Add Aliyun Mirror Source to pyproject.toml Appends configuration to the pyproject.toml file to use the Aliyun PyPI mirror for faster dependency downloads in mainland China. ```toml [[tool.uv.index]] url = "https://mirrors.aliyun.com/pypi/simple" default = true ``` -------------------------------- ### Run My Platform Live Client This Python script demonstrates how to initialize and run a live client for a custom platform. It includes steps for setting up the project path, importing necessary modules, reading and validating configuration from a YAML file, instantiating the platform client with specific configurations, and running the client asynchronously. Error handling for import errors and general exceptions is also included. ```python import os import sys import asyncio import logging # Ensure src directory can be imported project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) sys.path.insert(0, project_root) # !!! Import your platform implementation class !!! from src.open_llm_vtuber.live.my_platform_live import MyPlatformLive from src.open_llm_vtuber.config_manager.utils import read_yaml, validate_config logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) async def main(): logger.info("Starting My Platform Live client") try: # Read main configuration config_path = os.path.join(project_root, "conf.yaml") config_data = read_yaml(config_path) config = validate_config(config_data) # !!! Get your platform configuration !!! my_platform_config = config.live_config.my_platform # !!! (Optional) Check for critical configuration existence !!! # if not my_platform_config.channel_id or not my_platform_config.api_token: # logger.error("Missing required configuration for My Platform (channel_id or api_token)") # return logger.info(f"Attempting to connect to My Platform channel: {my_platform_config.channel_id}") # !!! Instantiate your platform client, passing configuration !!! # Convert Pydantic model to dictionary to pass platform = MyPlatformLive(config=my_platform_config.model_dump()) # !!! Start the client !!! await platform.run() except ImportError as e: logger.error(f"Failed to import required modules: {e}") # May need to prompt the user to install specific dependencies: logger.error("Did you install my_platform_sdk?") except Exception as e: logger.error(f"Error starting My Platform Live client: {e}") import traceback logger.debug(traceback.format_exc()) if __name__ == "__main__": try: # Run the main function using asyncio asyncio.run(main()) except KeyboardInterrupt: # Graceful shutdown logger.info("Shutting down My Platform Live client") ``` -------------------------------- ### Run Server to Generate Config Executes the main server script to generate default configuration files. Press Ctrl+C to exit after generation. ```bash uv run run_server.py ``` -------------------------------- ### MCP Configuration Example This snippet shows how to enable MCP servers by listing their names in the `mcp_enabled_servers` list within the `conf.yaml` file. It highlights the dynamic loading of server tools. ```yaml mcp_enabled_servers: - time - ddg-search ``` -------------------------------- ### Clone Project Repository with Submodules Clones the Open-LLM-VTuber project repository and initializes all Git submodules. The '--recursive' flag is essential for including the frontend code. ```bash # 克隆仓库 / 下载最新的 Github Release git clone https://github.com/Open-LLM-VTuber/Open-LLM-VTuber --recursive ``` -------------------------------- ### Edge TTS Example Usage Demonstrates how to use the Edge TTS engine and the expected format for keyword arguments when the TTS model is set to 'edge_tts'. ```shell 当 `tts_model` 设成 `edge_tts` 的时候: * `engine_type` 的传入数值将会是 `edge_tts`。 * `kwargs` 将会是这样的字典 ``` {"voice":"en-US-AvaMultilingualNeural"} ``` ``` -------------------------------- ### Speech Synthesis (TTS) Details how to enable and configure the Text-to-Speech (TTS) engine. Users need to install dependencies and set the TTS_MODEL option in `conf.yaml` to activate their chosen TTS engine. ```APIDOC Speech Synthesis (TTS): - Setup: Install required dependencies. - Configuration: Set `TTS_MODEL` in `conf.yaml`. - Functionality: Converts text responses into spoken audio. - Engine Options: Supports various TTS engines. ``` -------------------------------- ### 运行 Open LLM VTuber 容器 运行已构建的 Docker 容器。此命令将挂载本地的 `conf.yaml` 文件到容器内,并映射端口。请将 `$(pwd)/conf.yaml` 替换为您的配置文件实际路径。 ```docker docker run -it --net=host --rm -v $(pwd)/conf.yaml:/app/conf.yaml -p 12393:12393 open-llm-vtuber ``` -------------------------------- ### Troubleshoot LLM API Connection Errors Comprehensive guide to resolving LLM API connection issues, including checking the base URL, API availability, Ollama/LM Studio setup, and proxy configurations. Covers common error messages and solutions. ```APIDOC Error calling the chat endpoint: Connection error. Failed to connect to the LLM API. Check the configurations and the reachability of the LLM backend. See the logs for details. Troubleshooting with documentation Common causes and solutions: * Verify `base_url` is correctly configured. * Ensure the API is accessible and can be reached. * For Ollama, confirm it's running and accessible via `http://localhost:11434/` (or your Ollama address without `/v1`). * For LM Studio, ensure the server is running and the model is loaded. * If using a proxy, ensure it bypasses local addresses to allow connection to Ollama or other local LLM inference engines. ``` -------------------------------- ### Install gradio_client Command to install the gradio_client package, which is required for the CosyVoice2 TTS integration. ```Shell uv pip install gradio_client ``` -------------------------------- ### Run Ollama Model Downloads and runs the specified Ollama model (e.g., qwen2.5:latest) for interactive chat. Exit the chat with Ctrl/Command + D, but keep the terminal open. ```bash ollama run qwen2.5:latest ``` -------------------------------- ### Windows Installation Error: 'Windows protected your PC' This section addresses the common issue where the Windows desktop client installation is blocked by the 'Windows protected your PC' security feature or reports the file as corrupted. It provides a workaround for users to bypass the security prompt and proceed with the installation. ```Windows Click "More info" and then select "Run anyway". ``` -------------------------------- ### Live2D Expression Configuration Examples Demonstrates how to configure emotion mappings for Live2D models. This involves defining how AI-driven emotions (like 'neutral', 'anger') correspond to specific expressions within the model. Mappings can be done using expression indices (0-based) or expression names as defined in the model's 'Expressions' array in model3.json. The guide also shows how AI can trigger expressions in dialogue using a tag format like `[emotion]`. ```json // Example model3.json Expressions definition: "Expressions":[ {"Name":"f01","File":"f01.exp3.json"}, // Index 0 {"Name":"f02","File":"f02.exp3.json"}, // Index 1 {"Name":"f03","File":"f03.exp3.json"}, // Index 2 {"Name":"f04","File":"f04.exp3.json"} // Index 3 ] ``` ```json // model_dict.json configuration - Mapping by Index: "emotionMap":{ "neutral":0, // Corresponds to f01 "anger":2, // Corresponds to f03 "sadness":1 // Corresponds to f02 } ``` ```json // model_dict.json configuration - Mapping by Name: "emotionMap":{ "neutral":"f01", "anger":"f03", "sadness":"f02" } ``` ```text Example dialogue triggering expression: 噢,该死的![anger] 你说的这番话简直比约翰森叔叔家的鸡蛋豆腐面还要难以下咽。 ``` -------------------------------- ### Project Update Instructions Guidance for updating the Open LLM Vtuber project. Versions below v1.0.0 require a full redeployment due to significant changes in the deployment method. ```en If your project version is below v1.0.0, you must redeploy your project according to the Quick Start documentation. The deployment method for v1.0.0 underwent significant changes, making direct updates difficult. ``` -------------------------------- ### Fish Audio TTS Dependency Installation Command to install the fish-audio-sdk, which is necessary for using the Fish Audio TTS service. ```Shell uv pip install fish-audio-sdk ``` -------------------------------- ### Open LLM Vtuber 可定制性 Open-LLM-VTuber提供了高度的可定制性,允许用户通过简单的配置文件修改来切换功能模块,无需深入代码。用户可以导入自定义Live2D模型以获得独特外观,通过修改Prompt塑造AI伴侣的人设,并进行音色克隆以获得期望的声线。此外,项目支持继承和实现Agent接口,接入任何架构的Agent,并具有良好的可扩展性,允许用户轻松添加自定义LLM、ASR、TTS、MCP工具等模块。 ```APIDOC High Customization: Simple Module Configuration: - Switch functional modules via simple configuration file edits, no deep code changes required. Character Customization: - Import custom Live2D models for unique appearances. - Shape AI companion's personality by modifying Prompts. - Perform voice cloning for desired vocal characteristics. Agent Implementation: - Inherit and implement Agent interfaces to integrate any Agent architecture (e.g., HumeAI EVI, OpenAI Her, Mem0). Extensibility: - Modular design allows easy addition of custom LLM, ASR, TTS, MCP tool modules to extend new features. ``` -------------------------------- ### Install pywhispercpp with Vulkan Installs the pywhispercpp library with Vulkan support. This is an alternative for GPU acceleration on systems that support Vulkan. ```bash GGML_VULKAN=1 pip install git+https://github.com/absadiki/pywhispercpp ``` -------------------------------- ### MiniMax TTS Configuration Example Example configuration for MiniMax TTS in `conf.yaml`. This requires the `group_id` and `api_key` obtained from the Minimax platform. ```YAML minimax_tts: group_id:'' api_key:'' ``` -------------------------------- ### 构建 Docker 镜像 使用此命令构建 Open LLM VTuber 的 Docker 镜像。请确保在执行前已准备好 `conf.yaml` 配置文件。 ```docker docker build -t open-llm-vtuber . ```