### JavaScript: Event Listeners for Start/Stop Recording
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Handles click events for start and stop buttons. The start button initiates audio recording setup and WebSocket connection. The stop button suspends the audio context and disconnects the audio processor.
```javascript
startButton.addEventListener('click', async () => {
packetPosition = 0;
const setup = await setupRecorder();
if (!setup) return;
connectWebSocket();
startButton.disabled = true;
stopButton.disabled = false;
statusDiv.textContent = 'Status: Recording and streaming...';
});
stopButton.addEventListener('click', async () => {
if (audioContext && audioContext.state === 'running') {
await audioContext.suspend();
if (audioProcessor) {
audioProcessor.disconnect();
}
}
});
```
--------------------------------
### HTML Structure for Wisprflow AI Streaming
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Provides the basic HTML structure for a web application that streams audio to the Wisprflow AI API. It includes JavaScript for initializing the AudioContext, setting up the WebSocket connection, handling audio processing via an AudioWorklet, and managing the UI elements for starting and stopping the stream. This structure is necessary for capturing microphone input and sending it to the API.
```html
Wisprflow AI Streaming
Wisprflow AI Audio Streaming
Status: Idle
```
--------------------------------
### WebSocket Authentication and Connection
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Establishes a WebSocket connection to the server and authenticates with an access token. Sends an authentication message with user context information.
```APIDOC
## POST /api/v1/dash/ws
### Description
Initiates a WebSocket connection for real-time audio streaming and authenticates the client using an API key and access token. The connection also sends context information for processing.
### Method
POST
### Endpoint
ws://localhost:8000/api/v1/dash/ws?api_key=Bearer%20
### Parameters
#### Query Parameters
- **api_key** (string) - Required - API key for authentication.
#### Request Body (Authentication message)
- **type** (string) - Required - The type of message, should be 'auth'.
- **access_token** (string) - Required - Access token for authentication.
- **context** (object) - Required - Contextual information for the AI service.
- **app** (object) - Required - Application details.
- **name** (string) - Required - Application name.
- **type** (string) - Required - Application type (e.g., "ai").
- **dictionary_context** (array) - Optional - Contextual dictionary information.
- **user_identifier** (string) - Required - Unique user identifier.
- **user_first_name** (string) - Required - User's first name.
- **user_last_name** (string) - Required - User's last name.
- **textbox_contents** (object) - Optional - Information about text box contents.
- **before_text** (string) - Optional - Text before the selection.
- **selected_text** (string) - Optional - Selected text.
- **after_text** (string) - Optional - Text after the selection.
- **screenshot** (null) - Optional - Screenshot data.
- **content_text** (null) - Optional - Content text.
- **content_html** (null) - Optional - Content HTML.
- **conversation** (null) - Optional - Conversation data.
- **language** (array) - Required - List of supported languages.
### Request Example
```json
{
"type": "auth",
"access_token": "",
"context": {
"app": {
"name": "Weather Forecast Chatbot",
"type": "ai"
},
"dictionary_context": [],
"user_identifier": "john_doe_1",
"user_first_name": "John",
"user_last_name": "Doe",
"textbox_contents": {
"before_text": "",
"selected_text": "",
"after_text": ""
},
"screenshot": null,
"content_text": null,
"content_html": null,
"conversation": null
},
"language": ["en"]
}
```
### Response
#### Success Response (200)
- **status** (string) - Status of the authentication process. Possible values: 'auth'
- **message** (object) - Information regarding the authentication process. Possible values depend on the status
#### Response Example
```json
{
"status": "auth"
}
```
```
--------------------------------
### WebSocket Message Handling
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Handles incoming WebSocket messages, including authentication status, informational messages, and text responses (transcripts).
```APIDOC
## WebSocket Messages
### Description
This section describes the types of messages the WebSocket server sends back to the client after successful authentication and during the audio streaming process.
### Method
N/A - This is a description of messages received.
### Endpoint
ws://localhost:8000/api/v1/dash/ws
### Response
#### Success Response (200)
- **status**: 'auth' - Indicates successful authentication.
- **status**: 'info' - Contains informational messages about the session (e.g., session_started, chunk_received). The message body varies based on the 'message' field.
- **status**: 'text' - Contains text responses, such as transcripts of the audio. The body.text field contains the transcript.
- **error**: If an error occurred, the response will contain an error message.
#### Response Example
```json
// Authentication Success
{
"status": "auth"
}
// Information Message
{
"status": "info",
"message": {
"event": "session_started"
}
}
// Text Response (Transcript)
{
"status": "text",
"body": {
"text": "This is a transcript."
}
}
// Error Message
{
"error": "Authentication failed"
}
```
```
--------------------------------
### JavaScript: WebSocket Connection and Authentication
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Establishes a WebSocket connection to the Wisprflow AI API and handles authentication using an access token. It sets up event listeners for connection status, incoming messages, and errors.
```javascript
function connectWebSocket() {
if (websocket) websocket.close();
websocket = new WebSocket('ws://localhost:8000/api/v1/dash/ws?api_key=Bearer%20');
websocket.onopen = () => {
statusDiv.textContent = 'Status: Connected to WebSocket';
websocket.send(JSON.stringify({
type: 'auth',
access_token: '',
context: {
app: {
name: "Weather Forecast Chatbot",
type: "ai"
},
dictionary_context: [],
user_identifier: "john_doe_1",
user_first_name: "John",
user_last_name: "Doe",
textbox_contents: {
before_text: "",
selected_text: "",
after_text: ""
},
screenshot: null,
content_text: null,
content_html: null,
conversation: null,
},
language: ['en'],
}));
};
websocket.onmessage = (event) => {
const message = JSON.parse(event.data);
console.log(`Received message: ${JSON.stringify(message)}`);
if (message.status === 'auth') {
statusDiv.textContent = 'Status: Authenticated, ready to stream';
} else if (message.status === 'info') {
// Handle info messages (session_started, chunk_received, etc.)
const info = message.message;
statusDiv.textContent = `Status: ${info.event}`;
} else if (message.status === 'text') {
// Handle text responses (transcripts)
if (message.body.text) {
transcriptDiv.textContent = `Transcript: ${message.body.text}`;
}
} else if (message.error) {
console.error('WebSocket error:', message.error);
statusDiv.textContent = `Error: ${message.error}`;
}
};
websocket.onclose = () => {
statusDiv.textContent = 'Status: WebSocket connection closed';
};
websocket.onerror = (error) => {
console.error('WebSocket error:', error);
statusDiv.textContent = 'Error: WebSocket encountered an error';
};
}
```
--------------------------------
### Wisprflow API WebSocket Response Examples
Source: https://api-docs.wisprflow.ai/websocket_api
Examples of JSON responses from the Wisprflow API via WebSocket, including authentication status, commit acknowledgments, partial transcriptions, and final transcriptions. These responses indicate the status of the API request and provide transcription details.
```json
{
"status": "auth"
}
```
```json
{
"status": "info",
"message": {
"event": "commit_received"
}
}
```
```json
{
"status": "text",
"position": 180,
"final": false,
"body": {
"text": "This is a partial transcription...",
"detected_language": "en"
}
}
```
```json
{
"status": "text",
"position": 205,
"final": false,
"body": {
"text": "This is a partial transcription... and this is the full text.",
"detected_language": "en"
}
}
```
--------------------------------
### HTML WebSocket Connection and Audio Capture
Source: https://api-docs.wisprflow.ai/websocket_quickstart
This HTML file demonstrates establishing a WebSocket connection, handling authentication, accessing the microphone, and processing audio for real-time streaming. It serves as the client-side interface for interacting with the WebSocket API. No external dependencies are required beyond a web browser.
```html
Wisprflow WebSocket Client
Wisprflow WebSocket Client
Status: Disconnected
Transcription:
```
--------------------------------
### Authentication Header Error Responses
Source: https://api-docs.wisprflow.ai/client_side_auth_basics
Provides examples of JSON error responses for issues related to the Authorization header. These include cases where the header is missing or does not start with the 'Bearer' prefix, typically resulting in a 401 Unauthorized status.
```json
{
"detail": "Authorization header is missing"
}
```
```json
{
"detail": "Authorization header must start with Bearer"
}
```
--------------------------------
### Start Transcription Session - JavaScript
Source: https://api-docs.wisprflow.ai/websocket_api
Sends a 'start' message to initiate a transcription session via the WebSocket. This message includes authentication details and optional contextual information to improve dictation accuracy. The server responds with an 'auth' message upon successful authentication.
```javascript
ws.send(JSON.stringify({
type: "auth",
access_token: "...",
language: ["en"], // If known beforehand, set to list of languages the user speaks
context: { // Optional contextual information used to improve dictation quality when applicable
app: {
name: "Weather Forecast Chatbot",
type: "ai" // one of {email, ai, other}
},
dictionary_context: [], // list of uncommon names or words relevant to the context
user_identifier: "john_doe_1",
user_first_name: "John",
user_last_name: "Doe",
textbox_contents: {
before_text: "", // text immediately before cursor position
selected_text: "", // text highlighted by cursor for replacement
after_text: "" // text immediately after cursor position
},
screenshot: null, // screenshor for when speaking about something on the screen
content_text: null, // plaintext content of the app user is dictating in
content_html: null, // HTML content of the app user is dictating in
conversation: { // chatbot style history of conversation (messaging or AI app)
id: "ai_chat_126", // conversation identifier
participants: ["John Doe", "AI Assistant"], // list of people the user might be addresing
messages: [
{
role: "user" // One of {user, human, assistant}
content: "How is the weather in SF today?"
},
{
role: "assistant"
content: "It's partly cloudy. Do you want to know the temperature?"
}
]
}
}
}));
```
--------------------------------
### HTML Structure for WebSocket Audio Streaming
Source: https://api-docs.wisprflow.ai/websocket_quickstart
Sets up the basic HTML document for the WebSocket audio streaming client. It includes a title, some basic styling, buttons to control streaming, and divs to display status and the transcript. It also includes a script tag for the client-side JavaScript logic.
```html
WebSocket Audio Stream
Audio Streaming via WebSocket
Status: Ready
Transcript will appear here...
```
--------------------------------
### Record Audio and Send to Wisprflow AI API (JavaScript)
Source: https://api-docs.wisprflow.ai/rest_api_quickstart
This snippet demonstrates how to request microphone access, record audio in chunks, convert WebM to WAV, encode to Base64, and send the audio data to the Wisprflow AI API. It handles the complete workflow from user interaction to API response, including status updates and error handling. Dependencies include browser's MediaRecorder API and `fetch`.
```javascript
const recordButton = document.getElementById('recordButton');
const statusDiv = document.getElementById('statusDiv');
const responseDiv = document.getElementById('responseDiv');
const audioPlayer = document.getElementById('audioPlayer');
let mediaRecorder;
let audioChunks = [];
let currentAudioBlob;
let currentBase64Audio;
// Convert WebM Blob to WAV Blob using a temporary AudioContext
async function convertWebMToWAV(webmBlob) {
const audioContext = new (window.AudioContext || window.webkitAudioContext)();
const response = await webmBlob.arrayBuffer();
const audioBuffer = await audioContext.decodeAudioData(response);
const numberOfChannels = audioBuffer.numberOfChannels;
const sampleRate = audioBuffer.sampleRate;
const bitDepth = 16;
const buffer = new ArrayBuffer(44 + audioBuffer.length * numberOfChannels * bitDepth / 8);
const view = new DataView(buffer);
// Write WAV header
function writeString(view, offset, string) {
for (let i = 0; i < string.length; i++) {
view.setUint8(offset + i, string.charCodeAt(i));
}
}
writeString(view, 0, 'RIFF');
view.setUint32(4, 36 + audioBuffer.length * numberOfChannels * bitDepth / 8, true);
writeString(view, 8, 'WAVE');
writeString(view, 12, 'fmt ');
view.setUint32(16, 16, true);
view.setUint16(20, 1, true);
view.setUint16(22, numberOfChannels, true);
view.setUint32(24, sampleRate, true);
view.setUint32(28, sampleRate * numberOfChannels * bitDepth / 8, true);
view.setUint16(32, numberOfChannels * bitDepth / 8, true);
view.setUint16(34, bitDepth, true);
writeString(view, 36, 'data');
view.setUint32(40, audioBuffer.length * numberOfChannels * bitDepth / 8, true);
// Write PCM samples
const channelData = audioBuffer.getChannelData(0);
let offset = 44;
for (let i = 0; i < audioBuffer.length; i++) {
const sample = channelData[i];
const intSample = sample < 0 ? sample * 0x8000 : sample * 0x7FFF;
view.setInt16(offset, intSample, true);
offset += 2;
}
return new Blob([view], { type: 'audio/wav' });
}
// Request microphone access and set up recorder
async function setupRecorder() {
try {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = (event) => {
audioChunks.push(event.data);
};
mediaRecorder.onstop = async () => {
const webmBlob = new Blob(audioChunks, { type: 'audio/webm' });
currentAudioBlob = await convertWebMToWAV(webmBlob);
currentBase64Audio = await blobToBase64(currentAudioBlob);
audioPlayer.src = URL.createObjectURL(currentAudioBlob);
await sendAudioToServer(currentAudioBlob);
audioChunks = [];
};
return true;
} catch (err) {
console.error('Error accessing microphone:', err);
statusDiv.textContent = 'Error: Could not access microphone';
return false;
}
}
// Convert Blob to Base64 string
function blobToBase64(blob) {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = () => {
resolve(reader.result.split(',')[1]);
};
reader.onerror = (error) => reject(error);
});
}
// Send audio to server
async function sendAudioToServer(audioBlob) {
try {
statusDiv.textContent = 'Status: Sending to server...';
responseDiv.textContent = '';
const startTime = performance.now();
const response = await fetch('https://platform-api.wisprflow.ai/api/v1/dash/api', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer ', // Replace with your actual API key
},
body: JSON.stringify({
audio: currentBase64Audio,
language: ["en"],
context: {
app: {
type: "email"
},
dictionary_context: [],
textbox_contents: {
before_text: "",
selected_text: "",
after_text: ""
},
// ... for a full list of available fields, see the "Request Schema" page
}
})
});
const apiDuration = (performance.now() - startTime) / 1000;
console.log(`API call took ${apiDuration}s`);
if (response.ok) {
const result = await response.json();
statusDiv.textContent = 'Status: Success!';
responseDiv.textContent = 'Server Response: ' + JSON.stringify(result, null, 2) + "\nAPI Duration: " + apiDuration;
} else {
throw new Error(`Server returned ${response.status}`);
}
} catch (err) {
console.error('Error sending audio:', err);
statusDiv.textContent = 'Status: Error sending audio to server';
responseDiv.textContent = 'Error: ' + err.message;
}
}
// Handle button click
recordButton.addEventListener('click', async () => {
if (!mediaRecorder) {
const setup = await setupRecorder();
if (!setup) return;
}
if (mediaRecorder.state === 'inactive') {
// Start recording
audioChunks = [];
mediaRecorder.start();
recordButton.textContent = 'Stop Recording';
statusDiv.textContent = 'Status: Recording...';
responseDiv.textContent = '';
} else {
// Stop recording
mediaRecorder.stop();
recordButton.textContent = 'Start Recording';
statusDiv.textContent = 'Status: Processing...';
}
});
```