### Run Local HTTP Server

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/cosyvoice-js/README.md

Start a local HTTP server in the example directory to serve the web page. Access the test page via your browser.

```bash
python -m http.server 9000
```

--------------------------------

### Run Audio Interaction Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/omni/java/README_EN.md

Execute this script to start the audio-only interaction with the Omni model. It records voice input and plays audio responses.

```shell
cd run_server_vad
./run.sh
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-text-out-audio-html-ai-assistant/python/README_EN.md

Install the Aliyun Bailian SDK and websocket service dependencies using pip.

```command
pip3 install dashscope // Install Aliyun Bailian SDK
```

```command
pip3 install websockets // Install websocket service dependencies
```

--------------------------------

### Push2Talk Mode Example in Java

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/java/multimodal_dialog/README.md

Demonstrates the Push2Talk mode for speech interaction. This mode requires manual control of speech recognition start and stop. It includes building request parameters, starting the conversation, and sending audio data.

```java
public void testPush2Talk() {
    // 构建请求参数
    MultiModalRequestParam params = MultiModalRequestParam.builder()
        .customInput(
            MultiModalRequestParam.CustomInput.builder()
                .workspaceId(WORKSPACE_ID)
                .appId(APP_ID)
                .build())
        .upStream(
            MultiModalRequestParam.UpStream.builder()
                .mode("push2talk")
                .audioFormat("pcm")
                .build())
        .downStream(
            MultiModalRequestParam.DownStream.builder()
                .voice("longxiaochun_v2")
                .sampleRate(48000)
                .build())
        .clientInfo(
            MultiModalRequestParam.ClientInfo.builder()
                .userId("demo_user")
                .device(MultiModalRequestParam.ClientInfo.Device.builder())
                    .uuid("demo_device")
                    .build())
                .build())
        .apiKey(API_KEY)
        .model(MODEL)
        .build();

    // 创建对话实例
    MultiModalDialog conversation = new MultiModalDialog(params, new DialogCallbackImpl());
    
    try {
        // 开始对话
        conversation.start();
        
        // 等待监听状态
        waitForListeningState(conversation);
        
        // 开始语音识别
        conversation.startSpeech();
        
        // 发送音频数据
        sendAudioFromFile(conversation, "path/to/audio.wav");
        
        // 停止语音识别
        conversation.stopSpeech();
        
        // 等待对话完成
        Thread.sleep(5000);
        
        // 停止对话
        conversation.stop();
        
    } catch (Exception e) {
        e.printStackTrace();
    }
}

private void sendAudioFromFile(MultiModalDialog conversation, String filePath) {
    try (AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(filePath))) {
        byte[] buffer = new byte[3200];
        int bytesRead;
        
        while ((bytesRead = audioInputStream.read(buffer)) != -1) {
            conversation.sendAudioData(ByteBuffer.wrap(buffer, 0, bytesRead));
            Thread.sleep(100); // 模拟实时音频流
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}
```

--------------------------------

### Install DashScope SDK for Python

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/PREREQUISITES.md

Install the DashScope SDK for Python using pip. Ensure Python 3.8 or later is installed.

```bash
pip3 install dashscope
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-text-out-audio-html-ai-assistant/python/README.md

Install the Alibaba Cloud Baichuan SDK and WebSocket service dependencies using pip.

```command
pip3 install dashscope //Install Alibaba Cloud Baichuan SDK
pip3 install websockets //Install websocket service dependency
```

--------------------------------

### Summarization Output Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/speech-plus/transcribe-video-and-do-translation-summarization-and-qa/python/README_EN.md

Example console output displaying a summary of the video content, broken down into key points and a final concluding statement.

```text
============= summary === START ===
1. 电气化时代：电灯的发明引领人类进入电气化时代，促进工业、交通发展，加速社会发展进程。
2. 数字化时代变革：当前正处于百年未遇的科技变局，数字化转型改变人类生活、生产方式及生存状态。
3. 云计算新时代：阿里云推动云计算成为新计算时代基础，让计算资源普及，如同电一般无处不在，重塑世界，激发无限想象力。

总结：从电气化到数字化，云计算正如同电一样，深刻改变世界格局，开启创新纪元。
============= summary ===  END  ===
```

--------------------------------

### Run Video Interaction Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/omni/java/README_EN.md

Initiate the video interaction service. After running this, open the web frontend to enable camera input and video streaming.

```shell
cd run_with_camera
./run.sh
```

--------------------------------

### Real-time Recognition Results Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/speech-recognition/recognize_speech_from_files_by_realtime_mode/README.md

This example shows the expected output format for real-time recognition results. Each file is associated with a process ID, and incremental results are returned.

```text
//process: 51389 对应sample_audio.mp3的识别结果1
    [process 51389]RecognitionCallback text:  那河畔的金柳是夕阳中的

    //process: 51392 对应sample_video_story.mp4的识别结果
    [process 51392]RecognitionCallback text:  在一个阳光明媚的早晨，鸭妈妈决定带着小鸭子们

    //process: 51389 对应sample_audio.mp3的识别结果2
    [process 51389]RecognitionCallback text:  那河畔的金柳是夕阳中的新娘。
```

--------------------------------

### Question Answering Output Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/speech-plus/transcribe-video-and-do-translation-summarization-and-qa/python/README_EN.md

Example console output demonstrating a question asked about the video content and the generated answer.

```text
============= QA === START ===
question is: 人类什么时候发明的电灯
result is: 人类发明电灯大约在一百多年前，这标志着电灯首次进入了人类的生活，开启了新的时代。
============= QA ===  END  ===
```

--------------------------------

### Install FFmpeg on macOS using Homebrew

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/ffmpeg.md

Use Homebrew to easily install FFmpeg on macOS. Ensure Homebrew is installed first.

```bash
brew install ffmpeg
```

--------------------------------

### Check FFmpeg Installation

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/ffmpeg.md

Verify that FFmpeg has been successfully installed and added to your system's PATH by running the version command in the terminal.

```bash
ffmpeg -version
```

--------------------------------

### Compile and Install FFmpeg from Source on Linux

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/ffmpeg.md

Compile FFmpeg from source on Linux or other unsupported systems. This involves downloading the source code, configuring build options, and then compiling and installing.

```bash
cd ffmpeg
./configure --prefix=/usr/local/ffmpeg --enable-openssl --disable-x86asm
make && make install
```

--------------------------------

### Run the Speech Recognition Sample

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/recognize_speech_from_video_and_decode_to_opus/python/README.md

Execute the Python script to start the real-time speech recognition process. The script will convert local video files to 16k Opus audio format using ffmpeg and then transcribe the audio to text.

```command
python3 run.py
```

--------------------------------

### Transcription and Translation Output Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/speech-plus/transcribe-video-and-do-translation-summarization-and-qa/python/README_EN.md

Example console output showing the transcribed text and its translation for segments of a video.

```text
============ transcribe and translate === START ===
transcribe==> 一百多年前，电灯第一次进入了人类的生活世界，迎来了新的时代。
translate ==> More than a hundred years ago, electric lights entered human life for the first time, ushering in a new era.
transcribe==> 有了电，人类才发明了电视机，才有了夜生活。
translate ==> With electricity, humans were able to invent television and thus, nightlife came into existence.
transcribe==> 现代工业和交通的出现，创造了现代的城市。
translate ==> The emergence of modern industry and transportation gave rise to modern cities.
transcribe==> 人类社会发展的速度超过了历史上任何时候，进入了伟大的电气化时代。
translate ==> Human society has developed at a faster pace than ever before in history, ushering in the grand era of electrification.
transcribe==> 今天，我们又迎来了一轮百年未遇的科技变局。
translate ==> Today, we are witnessing another round of technological transformations unprecedented in a century.
============= transcribe and translate ===  END  ===
```

--------------------------------

### Python SDK: Speech recognition has started

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/paraformer.md

This indicates a repeated call to the `start` function. The `Recognition` object in the Python SDK is not reusable; ensure `start` is called only once.

```python
Speech recognition has started.
```

--------------------------------

### Install Python Dependencies for Bailian SDK

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-audio-out-text-html/python/README_EN.md

Install the necessary Python packages for the Alibaba Cloud Bailian SDK and WebSocket functionality. Ensure you have Python 3.8 or higher.

```command
pip3 install dashscope // Install Alibaba Cloud Bailian SDK
```

```command
pip3 install websockets // Install WebSocket service dependencies
```

--------------------------------

### Sample Console Output

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/recognize_speech_from_video_and_decode_to_opus/python/README.md

This is an example of the brief text output that will be printed to the console during the speech recognition process.

```text
The brief result is:
横看成岭侧成峰，远近高低各不同。不识庐山真面目，只缘身在此山中。
```

--------------------------------

### Start Real-time Video Stream

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/omni/java/interface/index.html

Initiates a WebSocket connection to the server and begins sending captured video frames. Ensure the server is running at 'ws://localhost:5000/video'. The stream quality can be adjusted by changing the 'image/jpeg' quality parameter.

```javascript
const video = document.getElementById('video');
const startBtn = document.querySelector('.start');
const closeBtn = document.getElementById('closeBtn');
let ws;

// Get camera
navigator.mediaDevices.getUserMedia({ video: true })
  .then(stream => {
    video.srcObject = stream;
  });

function startStream() {
  if (ws && ws.readyState === WebSocket.OPEN) {
    alert("Connection already exists, please close it first.");
    return;
  }

  const canvas = document.createElement('canvas');
  const ctx = canvas.getContext('2d');
  canvas.width = 640;
  canvas.height = 480;

  // Create WebSocket connection
  ws = new WebSocket('ws://localhost:5000/video');

  // Listen for connection open event
  ws.onopen = () => {
    console.log("WebSocket connected");
    setInterval(() => {
      if (video.readyState === video.HAVE_ENOUGH_DATA) {
        ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
        canvas.toBlob(blob => {
          if (blob && ws && ws.readyState === WebSocket.OPEN) {
            ws.send(blob);
          }
        }, 'image/jpeg', 0.7); // Adjust quality as needed
      }
    }, 500); // Send frame every 0.5s

    // Show/hide buttons
    startBtn.style.display = 'none';
    closeBtn.style.display = 'inline-block';
  };

  // Listen for connection error event
  ws.onerror = (err) => {
    console.error("WebSocket connection error:", err);
    alert("Failed to connect to the server. Please check if the service is running!");
    // Do not change button state if there's an error
    if (ws) {
      ws.close();
      ws = null;
    }
  };
}
```

--------------------------------

### Real-time Translation Output Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/speech-recognition/translate_speech_from_files_by_realtime_mode/README.md

This output shows the streaming recognition results for multiple files. Each file's processing is identified by a process ID, and incremental results are returned continuously.

```text
translation with file :hello_world_male_16k_16bit_mono.wav
[process 94459] TranslationRecognizerCallback open.
translation with file :hello_world_male_16k_16bit_mono.wav
[process 94461] TranslationRecognizerCallback open.
[process 94459] Transcript ==>  hello ,word,这里是阿里巴巴语音实验室。
[process 94459] Translate ==>  Hello, world. This is Alibaba's voice lab.
[process 94459] Translation completed
[process 94459] TranslationRecognizerCallback close.
[Metric] requestId: xxxxxxxxx, first package delay ms: 448.789794921875, last package delay ms: 1169.598876953125
[process 94461] Transcript ==>  hello ,word,这里是阿里巴巴语音实验室。
[process 94461] Translate ==>  Hello, world. This is Alibaba's voice lab.
[process 94461] Translation completed
[process 94461] TranslationRecognizerCallback close.
[Metric] requestId: xxxxxxxxx, first package delay ms: 409.506103515625, last package delay ms: 1175.384033203125
```

--------------------------------

### Run Non-server_vad Mode Example

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/omni/java/README_EN.md

Execute this script to simulate conversations without server_vad mode, allowing active control over model response timing.

```shell
cd run_without_server_vad
./run.sh
```

--------------------------------

### Run WebSocket Server and HTTP Server

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-audio-out-text-html/python/README.md

Start the Python WebSocket service on port 9090 for real-time communication and then run an HTTP server on port 9000 to serve the web application files. Configure your Baichuan API key in the environment variables.

```command
export DASHSCOPE_API_KEY=xxxxxxx
python server.py
```

```command
python -m http.server 9000
```

--------------------------------

### Run the Sample

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README_EN.md

Enable SDK internal logging and run the main Python script. Ensure you have configured your app_id, workspace_id, and api_key.

```bash
export DASHSCOPE_LOGGING_LEVEL='info' # Enable dashscope SDK internal logging
python3 run.py
```

--------------------------------

### Run LiveAI (Video Conversation) Demo

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README.md

Execute this command to run the LiveAI demo for video conversations. This involves sending video frames and receiving descriptive responses.

```bash
python3 run_live_ai.py
```

--------------------------------

### Python SDK: Speech synthesizer has not been started

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/cosyvoice.md

This error indicates the speech synthesizer was not started before attempting to use it. Ensure you call `streamingCall` to initiate the task before other related operations.

```python
speech synthesizer has not been started.
```

--------------------------------

### Basic Configuration for Multimodal Dialog

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README_EN.md

Import necessary classes and define configuration parameters including workspace ID, app ID, API key, model name, and WebSocket URL.

```python
from dashscope.multimodal.dialog_state import DialogState
from dashscope.multimodal.multimodal_dialog import MultiModalDialog, MultiModalCallback
from dashscope.multimodal.multimodal_request_params import (
    Upstream, Downstream, ClientInfo, RequestParameters, Device
)

# Configuration parameters
WORKSPACE_ID = "your_workspace_id"
APP_ID = "your_app_id"
API_KEY = "your_api_key"
MODEL = "multimodal-dialog"
WEBSOCKET_URL = "wss://dashscope.aliyuncs.com/api-ws/v1/inference"
```

--------------------------------

### Run Visual Question Answering (VQA) Demo

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README.md

Execute this command to run the VQA demo. This involves sending an image and receiving a text-based answer about the image content.

```bash
python3 run_vqa.py
```

--------------------------------

### Java SDK: State invalid: expect stream input tts state is started but idle

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/cosyvoice.md

This error in the Java SDK suggests an invalid state, expecting the stream input TTS to be started but finding it idle. Verify that the task was initiated correctly and check for any previous task failures.

```java
State invalid: expect stream input tts state is started but idle
```

--------------------------------

### Basic Configuration for Multimodal Dialog Service

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/java/multimodal_dialog/README_EN.md

Set up the necessary configuration parameters and WebSocket API URL for the Multimodal Dialog service. Ensure you replace placeholder values with your actual credentials.

```java
import com.alibaba.dashscope.multimodal.MultiModalDialog;
import com.alibaba.dashscope.multimodal.MultiModalDialogCallback;
import com.alibaba.dashscope.multimodal.MultiModalRequestParam;
import com.alibaba.dashscope.multimodal.State;
import com.alibaba.dashscope.utils.Constants;

public class MultiModalDialogExample {
    
    // Configuration parameters
    private static final String WORKSPACE_ID = "your_workspace_id";
    private static final String APP_ID = "your_app_id";
    private static final String API_KEY = "your_api_key";
    private static final String MODEL = "multimodal-dialog";
    
    public static void main(String[] args) {
        // Set WebSocket API URL
        Constants.baseWebsocketApiUrl = "wss://dashscope.aliyuncs.com/api-ws/v1/inference";
        
        // Create test instance
        MultiModalDialogExample example = new MultiModalDialogExample();
        example.testPush2Talk();
    }
}
```

--------------------------------

### Run Backend WebSocket Service

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-text-out-audio-html-ai-assistant/python/README.md

Start the Python WebSocket server for the AI Assistant backend. This service defaults to running on port 11111.

```python
python server.py
```

--------------------------------

### Configure API Key and Run WebSocket Server

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/gallery/input-audio-out-text-html/python/README_EN.md

Set your Alibaba Cloud API key as an environment variable and run the Python WebSocket service. This service handles real-time audio processing.

```shell
export DASHSCOPE_API_KEY=xxxxxxx
python demo_server.py
```

--------------------------------

### 发送音频文件

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README.md

使用对话管理器的 send_audio_file 方法发送本地音频文件。该方法会处理音频流的识别和发送，并根据对话模式决定是否停止语音识别。

```python
    def send_audio_file(self, file_path: str):
        """发送音频文件"""
        # 等待监听状态
        while self.conversation.get_dialog_state() != DialogState.LISTENING:
            time.sleep(0.1)
        
        # 开始语音识别
        self.conversation.start_speech()
        
        # 流式发送音频数据
        with open(file_path, "rb") as f:
            while True:
                data = f.read(3200)
                if not data:
                    break
                self.conversation.send_audio_data(data)
                time.sleep(0.1)
        
        # 停止语音识别（Push2Talk 模式）
        if self.conversation.get_conversation_mode() == "push2talk":
            self.conversation.stop_speech()
```

--------------------------------

### Close Real-time Video Stream

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/omni/java/interface/index.html

Closes the active WebSocket connection if one exists. This function is called when the 'Close Connection' button is clicked. It also updates the UI to show the 'Start Stream' button again.

```javascript
function closeStream() {
  if (!ws || ws.readyState !== WebSocket.OPEN) {
    alert("No active connection!");
  } else {
    ws.close();
    console.log("WebSocket closed");
  }

  // Show/hide buttons
  startBtn.style.display = 'inline-block';
  closeBtn.style.display = 'none';
}

// Automatically close connection when the page unloads (optional)
window.addEventListener('beforeunload', () => {
  if (ws && ws.readyState === WebSocket.OPEN) {
    ws.close();
  }
});
```

--------------------------------

### Configure API Key via Environment Variable (Windows)

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/docs/QA/cosyvoice.md

Set the DASHSCOPE_API_KEY environment variable for Windows. Verify the setting by echoing the variable.

```bash
$env:DASHSCOPE_API_KEY="YOUR_API_KEY"
# 验证设置生效
echo $env:DASHSCOPE_API_KEY
```

--------------------------------

### 开始对话

Source: https://github.com/aliyun/alibabacloud-bailian-speech-demo/blob/master/samples/conversation/multimodal_dialog/python/README.md

调用对话管理器的 start_conversation 方法来启动多模态对话。此方法会初始化 WebSocket 连接。

```python
    def start_conversation(self):
        """开始对话"""
        self.conversation.start("")
        print("🎯 对话已开始")
```