Neuphonic

https://docs.neuphonic.com
Admin
Neuphonic is an AI voice platform providing text-to-speech synthesis, voice cloning, and...
Tokens:14,129
Snippets:126
Trust Score:7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies and Run Agent

Source: https://docs.neuphonic.com/integrations/livekit

After setting up your environment and configuration, download necessary LiveKit model files and start your agent using pnpm and npm commands.

```bash
pnpm download-files
npm run build
npm start
```

--------------------------------

### Run FastAPI Application

Source: https://docs.neuphonic.com/integrations/twilio

Starts the FastAPI server. Ensure all dependencies are installed and environment variables are set.

```bash
uvicorn.run(app, host="0.0.0.0", port=8000)
```

--------------------------------

### Install Python SDK and dependencies

Source: https://docs.neuphonic.com/

Install the core Python SDK and the optional pyaudio dependency for local audio playback.

```bash
pip install pyneuphonic
```

```bash
pip install pyaudio
```

--------------------------------

### Start FastAPI Server

Source: https://docs.neuphonic.com/integrations/twilio

Executes the main FastAPI application file to start the server. This command assumes 'app.py' is the entry point.

```bash
python app.py
```

--------------------------------

### Install JavaScript SDK

Source: https://docs.neuphonic.com/

Install the Neuphonic JavaScript SDK via npm.

```bash
npm install @neuphonic/neuphonic-js
```

--------------------------------

### Connect via WebSockets

Source: https://docs.neuphonic.com/build-group/text-to-speech

Example for establishing a WebSocket connection for real-time, two-way communication.

```bash
# First do `npm install -g wscat`  
wscat -c "wss://api.neuphonic.com/speak/en?api_key=<API_KEY>&speed=1.0&lang_code=en&sampling_rate=24000"
```

--------------------------------

### Install LiveKit Client SDK

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

Install the necessary LiveKit client SDK packages for JavaScript/TypeScript applications using npm.

```shell
npm install livekit-client @livekit/components-react @livekit/components-styles
```

--------------------------------

### Start ngrok Tunnel

Source: https://docs.neuphonic.com/integrations/twilio

Starts an ngrok tunnel to expose your local server to the internet. The displayed URL should be added to your .env file as SERVER_BASE_URL.

```bash
ngrok http http://localhost:8000
```

--------------------------------

### Install Neuphonic LiveKit Plugin

Source: https://docs.neuphonic.com/integrations/livekit

Install the Neuphonic plugin for LiveKit agents using npm. This command should be run in your project directory.

```bash
npm install @livekit/agents-plugin-neuphonic
```

--------------------------------

### Install Python Dependencies

Source: https://docs.neuphonic.com/integrations/twilio

Installs necessary Python packages including pyneuphonic, twilio, fastapi, python-dotenv, and uvicorn. Ensure your virtual environment is activated.

```bash
pip install "pyneuphonic~=1.8.0" "twilio~=9.3" "fastapi[standard]~=0.114" "python-dotenv~=1.0" "uvicorn[standard]~=0.30"  
```

--------------------------------

### Create and run a voice agent

Source: https://docs.neuphonic.com/how-to-use-agents

Examples for initializing and running agents in English and Spanish. Requires a valid NEUPHONIC_API_KEY environment variable.

```python
import os  
import asyncio  
  
# See AgentConfig model for full list of parameters to configure the agent  
from pyneuphonic import Neuphonic, Agent, AgentConfig  
  
async def main():  
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
    agent_id = client.agents.create(  
        name='Agent 1',  
        prompt='You are a helpful agent. Answer in 10 words or less.',  
        greeting='Hi, how can I help you today?'  
    ).data['agent_id']  
  
    # All additional keyword arguments (such as `agent_id`) are passed as  
    # parameters to the model. See AgentConfig model for full list of parameters.  
    agent = Agent(client, agent_id=agent_id)  
  
    try:  
        await agent.start()  
  
        while True:  
            await asyncio.sleep(1)  
    except KeyboardInterrupt:  
        await agent.stop()  
  
asyncio.run(main())
```

```python
import os  
import asyncio  
  
# See AgentConfig model for full list of parameters to configure the agent  
from pyneuphonic import Neuphonic, Agent, AgentConfig  
  
async def main():  
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
    agent_id = client.agents.create(  
        name='Asistente',  # Assistant  
        prompt='Eres un agente español útil.',  # You are a helpful spanish agent.  
        greeting='¿Cómo puedo ayudarte hoy?',  # How can I help you today?  
    ).data['agent_id']  
  
    # All additional keyword arguments (such as `agent_id`) are passed as  
    # parameters to the model. See AgentConfig model for full list of parameters.  
    agent = Agent(  
        client,  
        agent_id=agent_id,  
        lang_code='es',  
        voice_id='<VOICE_ID>',  
    )  
  
    try:  
        await agent.start()  
  
        while True:  
            await asyncio.sleep(1)  
    except KeyboardInterrupt:  
        await agent.stop()  
  
  
asyncio.run(main())s
```

--------------------------------

### Create Project Directory

Source: https://docs.neuphonic.com/integrations/twilio

Creates a new directory for the Twilio agent project and navigates into it. Ensure you have Python 3.9+ installed.

```bash
mkdir twilio-agent  
cd twilio-agent  
```

--------------------------------

### Generate Speech via SSE

Source: https://docs.neuphonic.com/build-group/text-to-speech

Examples for streaming text-to-speech using the SSE protocol across different interfaces.

```bash
# Replace <API_KEY> with your actual API key.  
# To switch languages, replace the lang_code in the path parameter (e.g., /en) with the desired language code.  
curl -N --request POST \
  --url https://api.neuphonic.com/sse/speak/en \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <API_KEY>' \
  --header 'Accept: text/event-stream' \
  --data '{
  "text": "Hello, world!"
}'
```

```python
import os  
from pyneuphonic import Neuphonic, TTSConfig  
from pyneuphonic.player import AudioPlayer  
  
# Ensure the API key is set in your environment  
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
sse = client.tts.SSEClient()  
  
# TTSConfig is a pydantic model so check out the source code for all valid options  
tts_config = TTSConfig(  
    lang_code='en', # replace the lang_code with the desired language code.  
    sampling_rate=24000,  
)  
  
# Create an audio player with `pyaudio`  
# Make sure you use the same sampling rate as in the TTSConfig  
with AudioPlayer(sampling_rate=24000) as player:  
    response = sse.send('Hello, world!', tts_config=tts_config)  
    player.play(response)
```

```javascript
import fs from 'fs';  
import { createClient, toWav } from '@neuphonic/neuphonic-js';  
const client = createClient({ apiKey: '<API_KEY>'});  
  
const msg = `Hello how are you?<STOP>`;  
  
const sse = await client.tts.sse({  
  speed: 1.15,  
  lang_code: 'en'  
});  
  
const res = await sse.send(msg);  
  
// Saving data to file  
const wav = toWav(res.audio);  
fs.writeFileSync(__dirname + '/sse.wav', wav);
```

--------------------------------

### Clone a Voice using JavaScript SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Clone a voice using the JavaScript SDK. This example demonstrates how to initialize the client with your API key and call the clone function with voice details and file path.

```javascript
import { createClient } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const id = await client.voices.clone({
  voiceName: 'Cloned Name',
  voiceFilePath: __dirname + '/data/voice1.wav',
  voiceTags: ['Tag 1']
});

// New voice ID returned if the voice is cloned successfully
console.log(id);
```

--------------------------------

### SSML Phoneme Tag

Source: https://docs.neuphonic.com/build-group/text-to-speech

Example of using the phoneme tag to control word pronunciation via IPA.

```xml
<phoneme alphabet="ipa" ph="ˈdeɪtə">data</phoneme>
```

--------------------------------

### Implement Custom Agent Callback

Source: https://docs.neuphonic.com/how-to-use-agents

Example of attaching a custom event handler to the Agent class to process different message types.

```python
import os
import asyncio

from pyneuphonic import Neuphonic, Agent, AgentConfig
from pyneuphonic.models import APIResponse, AgentResponse

def on_message(message: APIResponse[AgentResponse]):
    if message.data.type == 'user_transcript':
        print(f'Received user_transcript')
    elif message.data.type == 'llm_response':
        print(f'Received llm_response')
    elif message.data.type == 'audio_response':
        print(f'Received audio_response. Playing audio.')
    elif message.data.type == 'stop_audio_response':
        print(f'Received stop_audio_response.')

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Agent 1',
        prompt='You are a helpful agent. Answer in 10 words or less.',
        greeting='Hi, how can I help you today?'
    ).data['agent_id']

    agent = Agent(
        client,
        agent_id=agent_id,
        tts_model='neu_hq',
        on_message=on_message,  # attach the custom callback
    )

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()

asyncio.run(main())
```

--------------------------------

### List All Voices using Python SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Get a list of all available voices using the Python SDK. Ensure the API key is set in your environment variables.

```python
import os
from pyneuphonic import Neuphonic

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

response = client.voices.list()  # get's all available voices
print(response.data['voices'])  # display list of voices
```

--------------------------------

### Custom Agent Event Handler

Source: https://docs.neuphonic.com/how-to-use-agents

Example of attaching a custom event handler to the Agent class for custom behavior.

```APIDOC
## Custom Agent Event Handler

### Description
The `Agent` class, by default, plays `audio_response` messages and prints `llm_response` and `user_transcript` messages to the console. To add custom behavior, you can attach a custom event handler which perform extra actions on each event.

### Code Example
```python
import os
import asyncio

from pyneuphonic import Neuphonic, Agent, AgentConfig
from pyneuphonic.models import APIResponse, AgentResponse

def on_message(message: APIResponse[AgentResponse]):
    if message.data.type == 'user_transcript':
        print(f'Received user_transcript')
    elif message.data.type == 'llm_response':
        print(f'Received llm_response')
    elif message.data.type == 'audio_response':
        print(f'Received audio_response. Playing audio.')
    elif message.data.type == 'stop_audio_response':
        print(f'Received stop_audio_response.')

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    agent_id = client.agents.create(
        name='Agent 1',
        prompt='You are a helpful agent. Answer in 10 words or less.',
        greeting='Hi, how can I help you today?'
    ).data['agent_id']

    agent = Agent(
        client,
        agent_id=agent_id,
        tts_model='neu_hq',
        on_message=on_message,  # attach the custom callback
    )

    try:
        await agent.start()

        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        await agent.stop()

asyncio.run(main())
```
```

--------------------------------

### List All Voices using cURL

Source: https://docs.neuphonic.com/build-group/voice-cloning

Retrieve a list of all available voices in your library by sending a GET request to the /voices endpoint.

```bash
curl -X GET "http://api.neuphonic.com/voices" \
     -H "X-API-KEY: <API_KEY>"
```

--------------------------------

### Get Specific Agent Details

Source: https://docs.neuphonic.com/how-to-use-agents

Retrieve full details for a specific agent by ID using cURL or the Python SDK.

```bash
curl -X GET "http://api.neuphonic.com/agents/<AGENT_ID>" \
     -H "X-API-KEY: <API_KEY>"
```

```python
import os
from pyneuphonic import Neuphonic

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

# Get specific agent
response = client.agents.get(agent_id='<AGENT_ID>')

print(response.data)  # display response
```

--------------------------------

### Delete a Cloned Voice using JavaScript SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Delete a cloned voice using the JavaScript SDK. This example shows how to initialize the client and call the delete function with the voice name.

```javascript
import { createClient } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const voiceDeleted = await client.voices.delete({
  name: 'Cloned Name'
});

// Will return true if the voice was deleted successfully
console.log(voiceDeleted);
```

--------------------------------

### WebSocket Stop Audio Response Message Structure

Source: https://docs.neuphonic.com/how-to-use-agents

JSON structure for stop audio response messages, indicating the user has started speaking.

```json
{
    "type": "stop_audio_response",
}
```

--------------------------------

### Initialize Weather Server Environment

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Commands to set up the project directory and virtual environment for a Python MCP server.

```bash
# Create a new directory for our server  
mkdir weather  
cd weather  
  
# Create virtual environment and activate it  
python -m venv .venv  
source .venv/bin/activate  
  
# Install dependencies  
pip install mcp requests  
  
# Create our server file  
touch weather.py  
```

```bash
# Create a new directory for our server  
mkdir weather  
cd weather  
  
# Create virtual environment and activate it  
python -m venv .venv  
.venv\Scripts\activate  
  
# Install dependencies  
pip install mcp requests  
  
# Create our server file  
new-item weather.py  
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://docs.neuphonic.com/integrations/twilio

Sets up a Python virtual environment for project isolation. Activate it using the provided command.

```bash
python -m venv .venv  
source .venv/bin/activate  
```

--------------------------------

### Initialize and run a Neuphonic agent in Python

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Configures an agent with a prompt and MCP server URL, then manages the agent lifecycle using an asynchronous loop.

```python
from pyneuphonic import Neuphonic, Agent, AgentConfig  
  
async def main():  
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
    agent_id = client.agents.create(  
        name='Agent 1',  
        prompt='You are a helpful agent. Answer in 10 words or less.',  
        greeting='Hi, how can I help you today?'  
    ).data['agent_id']  
  
    # Pass your ngrok URL into the Agent  
    agent = Agent(  
        client,  
        agent_id=agent_id,  
        mcp_servers=['https://1234-56-789-123-4.ngrok-free.app/sse']  
    )
  
    try:  
        await agent.start()  
  
        while True:  
            await asyncio.sleep(1)  
    except KeyboardInterrupt:  
        await agent.stop()  
  
asyncio.run(main())
```

--------------------------------

### Configure Environment Variables

Source: https://docs.neuphonic.com/integrations/twilio

Define the necessary API keys and phone numbers in your environment configuration file.

```text
# The base url of this server, e.g., if using ngrok it would similar to 123d-45-678-912-3.ngrok-free.app  
SERVER_BASE_URL=  
  
FROM_NUMBER=  
TO_NUMBER=  
  
NEUPHONIC_API_KEY=  
```

--------------------------------

### Configure Neuphonic API Key

Source: https://docs.neuphonic.com/integrations/livekit

Create a .env.local file to store your NEUPHONIC_API_KEY. This key is required for authentication with the Neuphonic service.

```bash
NEUPHONIC_API_KEY=<your key>
```

--------------------------------

### Expose Local Server with ngrok

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Command to expose the local server port to the public internet.

```bash
ngrok http 8000  
```

--------------------------------

### Configure Environment Variables

Source: https://docs.neuphonic.com/integrations/twilio

Sets up the .env file with essential secrets for Twilio integration. Ensure you replace the placeholders with your actual Twilio Account SID and Auth Token.

```dotenv
TWILIO_AUTH_TOKEN=  
TWILIO_ACCOUNT_SID=  
```

--------------------------------

### Initialize and run a Neuphonic agent in the browser

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Uses the browser SDK to authenticate with a JWT, connect to an agent, and handle text and audio events.

```javascript
import { createBrowserClient } from '@neuphonic/neuphonic-js/browser';  
const client = createBrowserClient();  
  
// Pass the token from the server  
client.jwt('<JWT TOKEN>');  
  
const agent = client.createAgent({ agent_id: '<AGENT ID>', mcp_servers=['https://1234-56-789-123-4.ngrok-free.app/sse'] });  
  
// The Agent will try to access the mic and listen  
const chat = await agent.current.start();  
  
// Transcribed messages and agent replies are available through the onText callback.  
chat.onText((role, text) => {  
   addMessage(text, role);  
});  
  
// Event triggered upon audio playback start or stop  
chat.onAudio(async (audio) => {  
   // Indicates whether audio is currently playing (true) or not (false)  
   console.log(audio);  
});
```

--------------------------------

### List All Voices using JavaScript SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Fetch all available voices using the JavaScript SDK. Initialize the client with your API key and call the list function.

```javascript
import { createClient } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const voices = await client.voices.list();
console.log(voices);
```

--------------------------------

### Create an Agent Chatbot

Source: https://docs.neuphonic.com/

Initialize and run an interactive agent with custom prompts and greeting.

```python
import os  
import asyncio  
  
# See AgentConfig model for full list of parameters to configure the agent  
from pyneuphonic import Neuphonic, Agent, AgentConfig  
  
async def main():  
    # Load the API key from the environment  
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
    agent_id = client.agents.create(  
        name='Agent 1',  
        prompt='You are a helpful agent. Answer in 10 words or less.',  
        greeting='Hi, how can I help you today?'  
    ).data['agent_id']  
  
    # All additional keyword arguments (such as `agent_id` and `tts_model`) are passed as  
    # parameters to the model. See AgentConfig model for full list of parameters.  
    agent = Agent(client, agent_id=agent_id, tts_model='neu_hq')  
  
    try:  
        await agent.start()  
  
        while True:  
            await asyncio.sleep(1)  
    except KeyboardInterrupt:  
        await agent.stop()  
  
asyncio.run(main())
```

--------------------------------

### Perform Text to Speech

Source: https://docs.neuphonic.com/

Use the SSEClient to generate audio from text and play it using the AudioPlayer.

```python
import os  
from pyneuphonic import Neuphonic, TTSConfig  
from pyneuphonic.player import AudioPlayer  
  
# Load the API key from the environment  
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))  
  
sse = client.tts.SSEClient()  
  
# TTSConfig is a pydantic model so check out the source code for all valid options  
tts_config = TTSConfig(  
    speed=1.05,  
    lang_code='en', # replace the lang_code with the desired language code.  
    voice_id='<VOICE_ID>'  # use client.voices.list() to view all available voices  
)  
  
# Create an audio player with `pyaudio`  
with AudioPlayer() as player:  
    response = sse.send('Hello, world!', tts_config=tts_config)  
    player.play(response)  
  
    player.save_audio('output.wav')  # save the audio to a .wav file
```

--------------------------------

### Implement Weather MCP Server

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Python code using FastMCP to define a weather retrieval tool.

```python
import requests  
from mcp.server.fastmcp import FastMCP  
  
# Initialize FastMCP server  
mcp = FastMCP("weather")  
  
# Define a single tool that allows you to get the weather  
@mcp.tool()  
async def get_current_weather(city: str) -> str:  
    print(f"[debug-server] get_current_weather({city})")  
    url = f"https://wttr.in/{city}?format=4"  
    response = requests.get(url)  
  
    if not response.ok:  
        return f"Something went wrong. Unable to fetch forecast data for this city: {city}"  
  
    print(f"[debug-server] {response.text}")  
    return response.text  
  
# Run the server on port 8000  
if __name__ == "__main__":  
    mcp.run(transport='sse')  
```

--------------------------------

### Connect MCP Server to Agent

Source: https://docs.neuphonic.com/how-to-use-agents/connecting-mcp-servers

Initial imports for connecting an MCP server to a Neuphonic Agent.

```python
import os  
import asyncio  
  
```

--------------------------------

### Configure environment variables

Source: https://docs.neuphonic.com/

Set the NEUPHONIC_API_KEY environment variable for different operating systems.

```bash
# Set the environment variable globally, macOS / Linux  
echo "export NEUPHONIC_API_KEY=<YOUR API KEY HERE>" >> ~/.zshrc  
source ~/.zshrc
```

```bash
# Set the environment variable globally, example for Windows (Command Prompt)  
setx NEUPHONIC_API_KEY "<YOUR API KEY HERE>"
```

```bash
# Set environment variable locally with a .env file  
export NEUPHONIC_API_KEY=<YOUR API KEY HERE>
```

--------------------------------

### List All Agents

Source: https://docs.neuphonic.com/how-to-use-agents

Retrieve a list of all existing agents using cURL or the Python SDK.

```bash
curl -X GET "https://api.neuphonic.com/agents" \
     -H "X-API-KEY: <API_KEY>"
```

```python
import os
from pyneuphonic import Neuphonic

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

# List all available agents
response = client.agents.list()

print(response.data)  # display response
```

--------------------------------

### Clone a Voice using Python SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Clone a voice using the Python SDK. Ensure your API key is set as an environment variable. Provide the voice name, language code, optional tags, and the path to your audio file.

```python
import os
from pyneuphonic import Neuphonic

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

response = client.voices.clone(
    voice_name='<VOICE_NAME>',
    lang_code='<LANG_CODE>'
    voice_tags=['tag1', 'tag2'],  # optional, add descriptive tags of what your voice sounds like
    voice_file_path='<FILE_PATH>.wav'  # replace with file path to a sample of the voice to clone
)

print(response.data)  # this will contain a success message with the voice_id of the cloned voice
voice_id = response.data['voice_id']  # store the voice_id for later use
```

--------------------------------

### Connect to LiveKit Room with React Component

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

A React component that connects to a LiveKit room using the provided token and displays real-time transcriptions. Ensure your token is never exposed client-side.

```javascript
'use client';

import {
  LiveKitRoom,
  useTranscriptions,
  AudioConference
} from '@livekit/components-react';
import '@livekit/components-styles';
import { useEffect, useRef } from 'react';

const LIVEKIT_URL = 'https://agents.neuphonic.com';

const Transcript = () => {
  const transcriptions = useTranscriptions();
  const scrollRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (scrollRef.current) {
      scrollRef.current.scrollTop = scrollRef.current.scrollHeight;
    }
  }, [transcriptions]);

  return (
    <div
      ref={scrollRef}
      className="flex min-h-0 flex-1 w-full flex-col gap-4 overflow-y-auto rounded-xl border border-slate-200 bg-white p-4 dark:border-neutral-700 dark:bg-neutral-800"
    >
      {transcriptions.length === 0 ? (
        <div className="flex h-full items-center justify-center text-sm text-gray-400 dark:text-neutral-500">
          Conversation will appear here...
        </div>
      ) : (
        transcriptions.map((t, idx) => {
          const isAgent = t.participantInfo.identity.startsWith('agent-');
          return (
            <div
              key={idx}
              className={`flex ${isAgent ? 'justify-start' : 'justify-end'}`}
            >
              <div
                className={`max-w-[80%] rounded-lg px-3 py-2 text-base ${ 
                  isAgent
                    ? 'text-neutral-700 dark:text-neutral-200'
                    : 'bg-themeContrast2 text-themeWhite shadow-md shadow-slate-300 dark:border dark:border-darkThemeSecondary/30 dark:bg-darkThemeSecondary/20 dark:shadow-neutral-800'
                }`}
              >
                {t.text}
              </div>
            </div>
          );
        })
      )}
    </div>
  );
};

export const Agent = ({ token }: { token: string }) => {
  return (
    <div>
      <LiveKitRoom
        serverUrl={LIVEKIT_URL}
        token={token}
        data-lk-theme="default"
        connect={true}
        audio={true}
      >
        <AudioConference />
        <Transcript />
      </LiveKitRoom>
    </div>
  );
};

```

--------------------------------

### Implement FastAPI Twilio Agent Handler

Source: https://docs.neuphonic.com/integrations/twilio

Create the application logic to handle incoming Twilio calls and manage the real-time WebSocket stream with the Neuphonic agent.

```python
from dotenv import load_dotenv  
from starlette.websockets import WebSocketState  
import logging  
import json  
import base64  
from pyneuphonic import Neuphonic, WebsocketEvents, AgentConfig  
from pyneuphonic.models import APIResponse, TTSResponse  
import os  
from twilio.twiml.voice_response import VoiceResponse, Connect  
from fastapi.responses import HTMLResponse  
from fastapi import (  
    FastAPI,  
    APIRouter,  
    WebSocket,  
    WebSocketDisconnect,  
    Request  
)  
  
load_dotenv(override=True)  
  
app = FastAPI()  
  
router = APIRouter()  
  
  
@app.get('/ping')  
def ping():  
    return {'message': 'pong'}  
  
  
@app.api_route("/twilio/inbound_call", methods=["GET", "POST"])  
async def handle_incoming_call(request: Request):  
    """Handle incoming call and return TwiML response."""  
    response = VoiceResponse()  
    connect = Connect()  
    connect.stream(url=f'wss://{os.getenv("SERVER_BASE_URL")}/twilio/agent')  
    response.append(connect)  
    return HTMLResponse(content=str(response), media_type="application/xml")  
  
  
@app.websocket('/twilio/agent')  
async def agent_websocket(  
    websocket: WebSocket,  
):  
    """Handles the WebSocket connection for a Twilio agent, allowing real-time voice conversation."""  
    await websocket.accept()  
    stream_sid = None  
  
    client = Neuphonic(api_key=os.getenv('NEUPHONIC_API_KEY'))  
    neuphonic_agent_websocket = client.agents.AsyncWebsocketClient()  
  
    async def on_message(message: APIResponse[TTSResponse]):  
        """Handles messages that are returned from the Neuphonic server to the websocket client."""  
        if stream_sid is not None and message.data.type == 'audio_response':  
            # Forward audio to the users's phone  
            await websocket.send_json(  
                {  
                    'event': 'media',  
                    'streamSid': stream_sid,  
                    'media': {  
                        'payload': base64.b64encode(message.data.audio).decode('utf-8')  
                    },  
                }  
            )  
        elif message.data.type == 'user_transcript':  
            logging.info(f'user_transcript: {message.data.text}')  
        elif message.data.type == 'llm_response':  
            logging.info(f'llm_response: {message.data.text}')  
        elif message.data.type == 'stop_audio_response':  
            # If the user interrupts then stop playing any currently queued audio  
            if stream_sid is not None:  
                await websocket.send_json(  
                    {  
                        'event': 'clear',  
                        'streamSid': stream_sid,  
                    }  
                )  
  
    neuphonic_agent_websocket.on(WebsocketEvents.MESSAGE, on_message)  
  
    await neuphonic_agent_websocket.open(  
        agent_config=AgentConfig(  
            incoming_sampling_rate=8000,  
            return_sampling_rate=8000,  
            incoming_encoding='pcm_mulaw',  
            return_encoding='pcm_mulaw',  
            voice_id='<VOICE_ID>',  
        )  
    )  
  
    try:  
        while websocket.client_state == WebSocketState.CONNECTED:  
            message = await websocket.receive()  
  
            if message is None or message['type'] == 'websocket.disconnect':  
                continue  
  
            data = json.loads(message['text'])  
  
            if data['event'] == 'connected':  
                logging.info(f'Connected Message received: {message}')  
  
            if data['event'] == 'start':  
                stream_sid = data['start']['streamSid']  
  
            if data['event'] == 'media':  
                # Send incoming audio to the Neuphonic agent  
                await neuphonic_agent_websocket.send(  
                    {'audio': data['media']['payload']}  
                )  
  
            if data['event'] == 'closed':  
                logging.info(f'Closed Message received: {message}')  
                break  
    except WebSocketDisconnect as e:  
        logging.error(f'WebSocketDisconnect: {e}')  
    except Exception as e:  
        logging.error(f'Error occured: {e}')  
    finally:  
        await neuphonic_agent_websocket.close()  
  
  
if __name__ == "__main__":  
    import uvicorn
```

--------------------------------

### Stream TTS with Python SDK

Source: https://docs.neuphonic.com/build-group/text-to-speech

Uses the AsyncWebsocketClient to stream audio and play it back via an asynchronous audio player.

```python
import os
import asyncio
from pyneuphonic import Neuphonic, TTSConfig, WebsocketEvents
from pyneuphonic.models import APIResponse, TTSResponse
from pyneuphonic.player import AsyncAudioPlayer

async def main():
    client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

    # Create a websocket object from the client
    ws = client.tts.AsyncWebsocketClient()

    # Set the desired configuration, see the TTSConfig model for more configuration options
    tts_config = TTSConfig(lang_code='es',voice_id='<VOICE_ID>', sampling_rate=24000, speed=1.0, encoding="pcm_linear")

    # Create and start the audio player, which will output audio to your device's speaker
    player = AsyncAudioPlayer()
    await player.open()

    # Attach event handlers. Check WebsocketEvents enum for all valid events.
    async def on_message(message: APIResponse[TTSResponse]):
        """Play audio through the speaker as it is received."""
        await player.play(message.data.audio)

    async def on_close():
        """Close the audio player when the websocket closes."""
        await player.close()

    ws.on(WebsocketEvents.MESSAGE, on_message)
    ws.on(WebsocketEvents.CLOSE, on_close)

    await ws.open(tts_config=tts_config)  # create the connection with the server

    # A special symbol ' <STOP>' must be sent to the server, otherwise the server will wait for
    # more text to be sent before generating the last few snippets of audio
    await ws.send('¡Hola, cómo estás!', autocomplete=True)
    await ws.send('¡Hola, cómo estás! <STOP>')  # Both the above line, and this line, are equivalent

    await asyncio.sleep(15)  # let the audio play
    player.save_audio('output.wav')  # save the audio to a .wav file
    await ws.close()  # close the websocket and terminate the audio resources

asyncio.run(main())
```

--------------------------------

### Stream TTS with JavaScript SDK

Source: https://docs.neuphonic.com/build-group/text-to-speech

Demonstrates collecting audio chunks from a WebSocket stream and saving them as a WAV file.

```javascript
import fs from 'fs';
import { createClient, toWav } from '@neuphonic/neuphonic-js';
const client = createClient({ apiKey: '<API_KEY>'});

const msg = `Hello how are you?<STOP>`;

const ws = await client.tts.websocket({
  speed: 1.15,
  lang_code: 'en'
});

let byteLen = 0;
const chunks = [];

// Websocket allow us to get chunk of the data as soon as they ready
// which can make API more responsve
for await (const chunk of ws.send(msg)) {
  // here you can send the data to the client
  // or collect it in array and save as a file
  chunks.push(chunk.audio);
  byteLen += chunk.audio.byteLength;
}

// Merging all chunks into single Uint8Array array
let offset = 0;
const allAudio = new Uint8Array(byteLen);
chunks.forEach(chunk => {
  allAudio.set(chunk, offset);
  offset += chunk.byteLength;
})

// Saving data to file
const wav = toWav(allAudio);
fs.writeFileSync(__dirname + '/ws.wav', wav);

await ws.close(); // closing the socket if we don't want to send anything
```

--------------------------------

### View Agents - List

Source: https://docs.neuphonic.com/how-to-use-agents

Retrieve a list of all your existing agents.

```APIDOC
## View Agents - List

### Description
To retrieve a list of all your existing agents.

### cURL
```bash
curl -X GET "https://api.neuphonic.com/agents" \
     -H "X-API-KEY: <API_KEY>"
```

### Python SDK
```python
import os
from pyneuphonic import Neuphonic

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

# List all available agents
response = client.agents.list()

print(response.data)  # display response
```
```

--------------------------------

### POST /agents/{agent_id}/token

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

Obtain a token for agent interaction. Authentication is handled via the X-API-KEY header. Exactly one of `api_key` or `jwt_token` must be provided in the request body.

```APIDOC
## POST /agents/{agent_id}/token

### Description
Obtain a token for agent interaction. This token is a signed LiveKit JWT valid for one session.

### Method
POST

### Endpoint
`/agents/{agent_id}/token`

### Parameters
#### Path Parameters
- **agent_id** (string) - Required - The ID of the agent.

#### Query Parameters
None

#### Request Body
- **api_key** (string) - Yes* - Your Neuphonic API key. Required if `jwt_token` is not provided.
- **jwt_token** (string) - Yes* - Your Neuphonic JWT token. Required if `api_key` is not provided.
- **room_name** (string) - No - A name for the LiveKit room. A unique name is auto-generated if omitted.
- **greeting** (string) - No - Override the agent's opening greeting for this session.
- **instructions** (string) - No - Override the agent's system prompt for this session.
- **voice_id** (string) - No - Voice ID to use for TTS.
- **lang_code** (string) - No - Language code for the agent (e.g. `en`, `es`).
- **mcp_servers** (string[]) - No - List of MCP server URLs to make available to the agent.
- **return_sampling_rate** (integer) - No - Sample rate (Hz) of the audio returned by the agent. Defaults to `24000`.
- **return_encoding** (string) - No - Encoding of the audio returned by the agent (e.g. `pcm_linear`).
- **incoming_sampling_rate** (integer) - No - Sample rate (Hz) of the audio you send to the agent. Defaults to `16000`.
- **incoming_encoding** (string) - No - Encoding of the audio you send to the agent (e.g. `pcm_linear`, `opus`).

*Exactly one of `api_key` or `jwt_token` must be provided.

### Request Example
```json
{
  "api_key": "YOUR_NEUPHONIC_API_KEY",
  "room_name": "my-custom-room",
  "greeting": "Hello there!",
  "instructions": "You are a helpful assistant.",
  "voice_id": "nova",
  "lang_code": "en",
  "mcp_servers": ["https://mcp.example.com"],
  "return_sampling_rate": 48000,
  "return_encoding": "pcm_f32le",
  "incoming_sampling_rate": 44100,
  "incoming_encoding": "opus"
}
```

### Response
#### Success Response (200)
- **data.token** (string) - A signed LiveKit JWT. Valid for one session.
- **data.room_name** (string) - The room name passed in the request.

#### Response Example
```json
{
  "data": {
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
    "room_name": "my-custom-room"
  }
}
```
```

--------------------------------

### Neuphonic LiveKit Server URL

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

The base URL for the Neuphonic LiveKit server. Use this when connecting your LiveKit client.

```text
https://agents.neuphonic.com
```

--------------------------------

### Initialize Neuphonic TTS Plugin

Source: https://docs.neuphonic.com/integrations/livekit

Configure the Neuphonic TTS plugin within the defineAgent function's entry method. Ensure your NEUPHONIC_API_KEY is correctly set in the environment variables. You can customize langCode, voiceId, and speed.

```typescript
tts: new neuphonic.TTS({
  apiKey: process.env.NEUPHONIC_API_KEY!,
  langCode: 'en',
  voiceId: '9626c31c-bec5-4cca-baa8-f8ba9e84c8bc', // change to your preferred voice ID
  speed: 1.0, // optional, default is 1.0
}),
```

--------------------------------

### Initiate Outbound Call Script

Source: https://docs.neuphonic.com/integrations/twilio

Runs the Python script responsible for initiating an outbound call through Twilio.

```bash
python make_outbound_call.py
```

--------------------------------

### Optional Agent Parameters for Token Request

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

Include additional fields in the token request body to override agent settings like voice, language, prompt, and audio configurations.

```json
{
  "api_key": "<API_KEY>",
  "jwt_token": "<JWT_TOKEN>",
  "room_name": "my-room",
  "lang_code": "es",
  "voice_id": "<VOICE_ID>",
  "greeting": "¡Hola! ¿En qué puedo ayudarte?",
  "instructions": "You are a helpful Spanish-speaking assistant.",
  "mcp_servers": ["https://my-mcp-server.example.com"],
  "return_sampling_rate": 24000,
  "return_encoding": "pcm_linear",
  "incoming_sampling_rate": 16000,
  "incoming_encoding": "pcm_linear"
}
```

--------------------------------

### Clone a Voice using cURL

Source: https://docs.neuphonic.com/build-group/voice-cloning

Use this cURL command to send a POST request to the Neuphonic API for voice cloning. Ensure you replace placeholders with your actual API key, voice name, language code, tags, and file path. The audio file must be in WAV format.

```bash
curl -X POST "http://api.neuphonic.com/voices?voice_name=<VOICE_NAME>&lang_code=<LANG_CODE>" \
     -H "X-API-KEY: <API_KEY>" \
     -F "voice_tags=<VOICE_TAGS>" \
     -F "voice_file=@<FILE_PATH>.wav;type=audio/wav"
```

```bash
# Example:
# curl -X POST "http://api.neuphonic.com/voices?voice_name=Mike" \
#      -H "X-API-KEY: <API_KEY>" \
#      -F "voice_tags=Male, Thirties" \
#      -F "voice_file=@./audio_sample.wav;type=audio/wav"
```

--------------------------------

### Import Neuphonic Module

Source: https://docs.neuphonic.com/integrations/livekit

Add this import statement at the top of your src/agent.ts file to use the Neuphonic module.

```typescript
import * as neuphonic from '@livekit/agents-plugin-neuphonic';
```

--------------------------------

### Request Agent Token with cURL

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

Request a short-lived token from the Neuphonic API to authenticate your LiveKit connection. This request must be made from your backend.

```shell
curl -X POST "https://api.neuphonic.com/agents/<AGENT_ID>/token" \
     -H "X-API-KEY: <API_KEY>" \
     -H "Content-Type: application/json" \
     -d '{"room_name": "my-room", "api_key": "<API_KEY>"}'
```

--------------------------------

### Token Response Structure

Source: https://docs.neuphonic.com/how-to-use-agents/using-webrtc

The JSON response from the token request, containing the LiveKit JWT and room name.

```json
{
  "data": {
    "token": "<LIVEKIT_JWT>",
    "room_name": "my-room"
  }
}
```

--------------------------------

### Provide a context ID with Neuphonic SDKs

Source: https://docs.neuphonic.com/build-group/text-to-speech

Use a context ID to associate audio responses with specific input requests. This identifier is returned alongside the audio data.

```python
client = Neuphonic()  
  
ws = client.tts.AsyncWebsocketClient()  
audio_bytes = bytearray()  
  
async def on_message(message: APIResponse[TTSResponse]):  
    nonlocal audio_bytes  
    audio_bytes += message.data.audio  
    print(message.data.context_id) # here the context_id will be "1"  
  
ws.on(WebsocketEvents.MESSAGE, on_message)  
  
await ws.open()  
  
await ws.send(  
    {"text": "This is message one example.<STOP>", "context_id": "1" }  
)  
```

```javascript
import { createClient } from '@neuphonic/neuphonic-js';  
  
const client = createClient();  
  
const ws = await client.tts.websocketCb({  
  lang_code: 'en'  
});  
  
const msg = `Hello how are you?<STOP>`;  
  
let byteLen = 0;  
const chunks: Uint8Array[] = [];  
  
ws.onMessage((chunk) => {  
  byteLen += chunk.audio.byteLength;  
  chunks.push(chunk.audio);  
  console.log({ ...chunk, audio: chunk.audio.byteLength }); // here the context_id will be "1"  
});  
  
ws.send(msg, {  context_id: '1' });  
```

--------------------------------

### SSE Text-to-Speech Synthesis

Source: https://docs.neuphonic.com/

Use the SSE endpoint for single text-to-speech requests. Ensure your NEUPHONIC_API_KEY is set in your environment variables. The voice_id can be found using client.voices.list().

```typescript
import { writeFileSync } from 'fs';  
import { createClient, toWav } from '@neuphonic/neuphonic-js';  
import * as dotenv from 'dotenv';  
  
// Load environment variables from .env file  
dotenv.config();   
  
// Initialize Neuphonic client  
const client = createClient({ apiKey: process.env.NEUPHONIC_API_KEY || '' });  
  
// Text to synthesize  
const msg = `Hello how are you?`;  
  
// Create SSE TTS connection  
const sse = await client.tts.sse({  
  speed: 1.00,  
  lang_code: 'en',  
  voice_id: 'e564ba7e-aa8d-46a2-96a8-8dffedade48f', // choose a voice (client.voices.list() to view all)  
});  
  
// Send text and receive synthesized audio  
const res = await sse.send(msg);  
  
// Save audio to WAV buffer  
const wavData = Buffer.from(toWav(res.audio));  
  
// Write to output file  
writeFileSync('output.wav', wavData);
```

--------------------------------

### Make Outbound Call with Twilio

Source: https://docs.neuphonic.com/integrations/twilio

This Python script uses the Twilio REST API to initiate an outbound call. It requires TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, SERVER_BASE_URL, TO_NUMBER, and FROM_NUMBER to be set in the environment.

```python
import os
from twilio.rest import Client
from dotenv import load_dotenv

load_dotenv(override=True)

def make_outbound_call():
    client = Client(os.getenv('TWILIO_ACCOUNT_SID'), os.getenv('TWILIO_AUTH_TOKEN'))

    twiml_message = f'<?xml version="1.0" encoding="UTF-8"?><Response><Connect><Stream url="wss://{os.getenv("SERVER_BASE_URL")}/twilio/agent" /></Connect></Response>'

    call = client.calls.create(
        twiml=twiml_message,
        to=os.getenv('TO_NUMBER'),
        from_=os.getenv('FROM_NUMBER'),
        record=True,
    )

    print(call.sid)

if __name__ == '__main__':
    make_outbound_call()
```

--------------------------------

### WebSocket Response Structure

Source: https://docs.neuphonic.com/build-group/text-to-speech

The JSON schema for raw WebSocket responses containing audio data and metadata.

```json
{
  "data": {
    "audio": "<audio_data_string>",
    "text": "<input_text>",
    "sampling_rate": 24000 // Default value
  }
}
```

--------------------------------

### Delete a Cloned Voice using Python SDK

Source: https://docs.neuphonic.com/build-group/voice-cloning

Delete a cloned voice using its voice_id with the Python SDK. Ensure your API key is set as an environment variable.

```python
from pyneuphonic import Neuphonic
import os

# Ensure the API key is set in your environment
client = Neuphonic(api_key=os.environ.get('NEUPHONIC_API_KEY'))

# Delete using the voices `voice_id`
response = client.voices.delete(voice_id='<VOICE_ID>')

print(response.data)  # display response
```

--------------------------------

### WebSocket User Transcript Message Structure

Source: https://docs.neuphonic.com/how-to-use-agents

JSON structure for user transcript messages received from the server.

```json
{
    "type": "user_transcript",
    "text": "<TRANSCRIPTION>"
}
```
Tokens:
Raw