### Dictation Lifecycle Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/dictation.md Demonstrates the complete lifecycle of a dictation session, from acquiring a microphone stream to starting the audio flow, handling transcripts, and stopping the session. ```typescript async function runDictationSession() { const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "token-xyz", }; const allTranscripts: TranscriptEventData[] = []; const handleTranscriptEvent = (msg: MessageEvent) => { try { const event = JSON.parse(msg.data); if (event.type === "transcript") { allTranscripts.push(event.data); // Log interim vs final if (event.data.isFinal) { console.log("✓ FINAL:", event.data.text); } else { console.log("… interim:", event.data.text); } } } catch (err) { console.error("Parse error:", err); } }; try { // Acquire microphone const stream = await getMicrophoneStream(); // Start dictation with default config const { recorderStarted, stop } = await startAudioFlow( stream, authCreds, handleTranscriptEvent ); if (!recorderStarted) { throw new Error("Failed to start recording"); } console.log("Dictation active. Speak now..."); // Cleanup after 60 seconds setTimeout(() => { console.log("Stopping dictation..."); stop(); stream.getAudioTracks().forEach((t) => t.stop()); console.log(`\nSession complete:`); console.log(`Total events: ${allTranscripts.length}`); allTranscripts.forEach((t) => { console.log(` "${t.text}" (final: ${t.isFinal})`); }); }, 60000); } catch (error) { console.error("Session error:", error); throw error; } } ``` -------------------------------- ### Dictation Configuration Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Example configuration object for dictation, specifying primary language, interim results, spoken and automatic punctuation, and custom voice commands. ```typescript { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "next section", action: "next_section", keywords: ["next", "section"], }, ], } ``` -------------------------------- ### TypeScript WebSocket Implementation Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md This TypeScript example demonstrates how to establish a WebSocket connection, send configuration, handle incoming transcript and fact messages, and end the stream. Ensure you have a `startMediaRecorder` function defined elsewhere. ```typescript const wsUrl = `wss://api.us.corti.app/audio-bridge/v2/interactions/sess-123/streams?tenant-name=acme&token=Bearer%20xyz123`; const ws = new WebSocket(wsUrl); ws.onopen = () => { // Send configuration ws.send(JSON.stringify({ type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: true, isMultichannel: false, participants: [{ channel: 0, role: "multiple" }], }, mode: { type: "facts", outputLocale: "en" }, }, })); }; ws.onmessage = (msg: MessageEvent) => { const data = JSON.parse(msg.data); if (data.type === "CONFIG_ACCEPTED") { startMediaRecorder(); // Begin sending audio } else if (data.type === "transcript") { console.log("Transcript:", data.data.transcript); } else if (data.type === "fact") { console.log("Fact:", data.data.text); } }; // Later: end the stream ws.send(JSON.stringify({ type: "end" })); ``` -------------------------------- ### Example of a Specific Command Configuration Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Illustrates a well-defined command with minimal and specific keywords to avoid false positives. ```typescript // Good: specific, minimal keywords { command: "create new note", action: "create_note", keywords: ["create", "note"], // Two keywords, specific } // Risky: too generic { command: "next action", action: "next", keywords: ["next"], // Single word, will match frequently } // Better: more specific { command: "go to next patient", action: "next_patient", keywords: ["next", "patient"], // Two keywords, specific combination } ``` -------------------------------- ### Robust WebSocket Connection Setup with Error Handling Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md Establishes a WebSocket connection, handling potential setup errors and defining error/close event handlers. Use this to ensure reliable connections and provide user feedback on failures. ```typescript let ws: WebSocket; try { const wsUrl = `wss://api.${authCreds.environment}.corti.app/...`; ws = new WebSocket(wsUrl); ws.onerror = (err: Event) => { console.error("WebSocket error:", err); showErrorUI("Connection failed. Please check your network and credentials."); }; ws.onclose = (event: CloseEvent) => { if (!event.wasClean) { console.error("Connection closed unexpectedly"); // Implement reconnection logic } }; } catch (error) { console.error("WebSocket setup failed:", error); showErrorUI("Failed to establish connection."); } ``` -------------------------------- ### Wrap Async Operations in Try/Catch Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md This example shows how to wrap asynchronous operations, like starting a call, in a try/catch block to handle potential errors gracefully and provide user feedback. It also re-throws the error for further handling. ```typescript async function startCall() { try { const stream = await getMicrophoneStream(); const { stop } = await startAudioFlow(stream, authCreds, interactionId, handleEvent); return { stop }; } catch (error) { console.error("Error starting call:", error); showErrorUI("Failed to start call. Please check permissions and try again."); throw error; // Re-throw for caller handling } } ``` -------------------------------- ### Interim Results Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Illustrates the step-by-step transcription process when interim results are enabled. ```text Without interimResults: - No feedback until phrase is complete - Lower latency - Larger transcription chunks With interimResults: true: 1. "the" 2. "the patient" 3. "the patient reports" 4. "The patient reports chest pain." (final, with punctuation) ``` -------------------------------- ### Start Virtual Consultation Flow Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md Initiates a virtual consultation by starting an audio flow with merged microphone and WebRTC streams. Handles incoming transcript and fact messages. Remember to clean up streams and the audio flow when done. ```typescript const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "token-xyz", }; const transcripts: TranscriptEventData[] = []; const facts: FactEventData[] = []; const handleMessages = (msg: MessageEvent) => { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data); } else if (parsed.type === "fact") { facts.push(parsed.data); } }; async function startVirtualConsultation() { const microphoneStream = await getMicrophoneStream(); const webRTCStream = new MediaStream(); // Get from peer connection const { stream, endStream } = mergeMediaStreams([ microphoneStream, webRTCStream, ]); const { stop } = await startAudioFlow( stream, authCreds, "consultation-456", handleMessages ); // Cleanup later stop(); endStream(); microphoneStream.getAudioTracks().forEach((t) => t.stop()); webRTCStream.getAudioTracks().forEach((t) => t.stop()); } ``` -------------------------------- ### Automatic Punctuation Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Demonstrates how raw speech input is transformed with automatic punctuation enabled. ```text Input (raw speech): "the patient reports chest pain and shortness of breath" With automaticPunctuation: true: "The patient reports chest pain and shortness of breath." ``` -------------------------------- ### startAudioFlow() Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-singlemicrophone.md Starts an audio flow by connecting a MediaStream to a WebSocket endpoint and sending a configuration. The flow begins once a CONFIG_ACCEPTED message is received, after which audio data is sent in 200ms chunks via a MediaRecorder. ```APIDOC ## startAudioFlow() ### Description Starts an audio flow by connecting a MediaStream to a WebSocket endpoint and sending a configuration. The flow begins once a CONFIG_ACCEPTED message is received, after which audio data is sent in 200ms chunks via a MediaRecorder. ### Signature ```typescript async function startAudioFlow( mediaStream: MediaStream, authCreds: AuthCreds, interactionId: string, handleEvent: (arg0: MessageEvent) => void, config?: Config ): Promise<{ recorderStarted: boolean; stop: () => void }> ``` ### Parameters #### Path Parameters - **mediaStream** (MediaStream) - Yes - The audio MediaStream to transmit to the WebSocket endpoint. Must contain at least one audio track. - **authCreds** (AuthCreds) - Yes - Authentication credentials object containing `environment`, `tenant`, and `token` properties for WebSocket connection. - **interactionId** (string) - Yes - The interaction identifier used in the WebSocket URL path. Represents the unique identifier for this audio stream session. - **handleEvent** ((arg0: MessageEvent) => void) - Yes - Callback function invoked when WebSocket messages are received. Receives the raw MessageEvent; consumer code must parse JSON. - **config** (Config) - No - Optional configuration object. If not provided, uses DEFAULT_CONFIG with diarization enabled, single channel, English language, and facts mode. ### Return **Type:** `Promise<{ recorderStarted: boolean; stop: () => void }> ` - `recorderStarted`: Boolean indicating whether the MediaRecorder has started recording and sending audio chunks - `stop`: Function that gracefully ends the audio flow by sending an end message, stopping the MediaRecorder, and closing the WebSocket connection ### Throws - Error: During WebSocket setup if the connection fails (logged to console) - Error: If JSON parsing of incoming messages fails (logged to console) ### Example ```typescript const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "auth-token-xyz", }; const transcripts: TranscriptEventData[] = []; const facts: FactEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { try { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data as TranscriptEventData); } else if (parsed.type === "fact") { facts.push(parsed.data as FactEventData); } } catch (err) { console.error("Failed to parse message:", err); } }; async function startCall() { const microphoneStream = await getMicrophoneStream(); const { recorderStarted, stop } = await startAudioFlow( microphoneStream, authCreds, "interaction-123", handleNewMessage ); console.log("Audio recording started:", recorderStarted); // Later, end the call stop(); microphoneStream.getAudioTracks().forEach((track) => track.stop()); } ``` ``` -------------------------------- ### Start Dictation with Configuration Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Initiates real-time dictation with support for spoken and automatic punctuation, and custom voice commands. Processes transcript events and logs the text. ```typescript import { DictationConfig, TranscriptEventData } from "./types"; const config: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "next section", action: "next_section", keywords: ["next", "section"], }, ], }; const transcripts: TranscriptEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data); const { text, isFinal } = parsed.data; console.log(`${isFinal ? "✓" : "…"}: ${text}`); } }; async function startDictation() { const microphoneStream = await getMicrophoneStream(); const { stop } = await startAudioFlow( microphoneStream, authCreds, handleNewMessage, config ); // Later... stop(); } ``` -------------------------------- ### Start Audio Flow to WebSocket Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md Initiates an audio data stream to a WebSocket endpoint for real-time processing. This function connects a MediaStream, sends configuration, and begins transmitting audio chunks once acknowledged. It returns a status indicating if the recorder started and a function to stop the flow. ```typescript async function startAudioFlow( mediaStream: MediaStream, authCreds: AuthCreds, interactionId: string, handleEvent: (arg0: MessageEvent) => void, config?: Config ): Promise<{ recorderStarted: boolean; stop: () => void }> ``` ``` -------------------------------- ### Default Dictation Configuration (Recommended) Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md A full-featured dictation configuration with all options enabled, including interim results, spoken and automatic punctuation, and a sample voice command. This is the recommended starting point. ```typescript const config: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "go to next section", action: "next_section", keywords: ["next", "section"], }, ], }; ``` -------------------------------- ### Spoken Punctuation Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Shows how spoken punctuation commands affect the transcription output. ```text With spokenPunctuation: true, and spoken: "period" "The patient reports chest pain." ``` -------------------------------- ### AuthCreds Usage Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Demonstrates how to define AuthCreds and construct a WebSocket URL using these credentials. ```typescript // Basic example const authCreds: AuthCreds = { environment: "us", tenant: "my-company", token: "abc123xyz789...", }; // Used in WebSocket connection const wsUrl = `wss://api.${authCreds.environment}.corti.app/audio-bridge/v2/transcribe?tenant-name=${authCreds.tenant}&token=Bearer%20${authCreds.token}`; ``` -------------------------------- ### Full WebSocket Transcription Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md This snippet shows a complete implementation for connecting to the Corti API, sending configuration, processing transcription results (both interim and final), and ending the session. ```typescript const wsUrl = `wss://api.us.corti.app/audio-bridge/v2/transcribe?tenant-name=acme&token=Bearer%20xyz123`; const ws = new WebSocket(wsUrl); ws.onopen = () => { ws.send(JSON.stringify({ type: "config", configuration: { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "next section", action: "next_section", keywords: ["next", "section"], }, ], }, })); }; ws.onmessage = (msg: MessageEvent) => { const data = JSON.parse(msg.data); if (data.type === "CONFIG_ACCEPTED") { startMediaRecorder(); // Begin sending audio } else if (data.type === "transcript") { const { text, isFinal } = data.data; if (isFinal) { console.log("✓ FINAL:", text); // Process final transcript } else { console.log("... interim:", text); // Update preview } } }; // Later: end transcription ws.send(JSON.stringify({ type: "end" })); ``` -------------------------------- ### AmbientScribe Config - Explicit Roles Mode Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Example configuration for AmbientScribe using explicit roles for multi-channel audio with assigned roles. ```typescript const config: Config = { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: false, isMultichannel: false, participants: [ { channel: 0, role: "doctor" }, { channel: 0, role: "patient" }, ], }, mode: { type: "facts", outputLocale: "en" }, }, }; ``` -------------------------------- ### TranscriptEventData Examples Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Illustrates interim and final transcript events, differentiating between potentially changing text and finalized transcription. ```typescript const interimEvent: TranscriptEventData = { text: "The patient reports...", rawTranscriptText: "the patient reports", isFinal: false, }; const finalEvent: TranscriptEventData = { text: "The patient reports chest pain.", rawTranscriptText: "the patient reports chest pain", isFinal: true, }; ``` -------------------------------- ### Start and Manage a Full Consultation Flow Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md Initiates a virtual consultation by capturing and merging audio streams, handling real-time transcripts and facts, and providing a cleanup function to end the session. Ensure you have valid authentication credentials and a consultation ID. ```typescript async function startConsultation() { try { // Capture local and remote audio const microphoneStream = await getMicrophoneStream(); const webRTCStream = new MediaStream(); // From WebRTC connection // Merge streams const { stream: mergedStream, endStream } = mergeMediaStreams([ microphoneStream, webRTCStream, ]); // Handle incoming transcripts and facts const transcripts: TranscriptEventData[] = []; const facts: FactEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data); } else if (parsed.type === "fact") { facts.push(parsed.data); } }; // Start audio flow const { stop } = await startAudioFlow( mergedStream, authCreds, "consultation-id", handleNewMessage ); // Cleanup function const endConsultation = () => { stop(); endStream(); microphoneStream.getAudioTracks().forEach((t) => t.stop()); webRTCStream.getAudioTracks().forEach((t) => t.stop()); console.log("Consultation ended. Transcripts:", transcripts.length, "Facts:", facts.length); }; return { endConsultation, transcripts, facts }; } catch (error) { console.error("Error starting consultation:", error); throw error; } } ``` -------------------------------- ### Start AmbientScribe with Virtual Consultations Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Merges microphone and WebRTC streams for doctor-patient scenarios using AmbientScribe. Manages audio flow and stream termination. ```typescript import { mergeMediaStreams, startAudioFlow } from "./virtualConsultations"; async function startConsultation() { const microphoneStream = await getMicrophoneStream(); const webRTCStream = new MediaStream(); // From peer connection const { stream, endStream } = mergeMediaStreams([ microphoneStream, webRTCStream, ]); const { stop } = await startAudioFlow( stream, authCreds, "consultation-456", handleMessages ); // Later... stop(); endStream(); } ``` -------------------------------- ### Start MediaRecorder After CONFIG_ACCEPTED Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md Prevents sending audio data before the server acknowledges configuration. This avoids potential data loss or stream rejection by ensuring the `CONFIG_ACCEPTED` message is received first. ```typescript let configAccepted = false; ws.onmessage = (msg: MessageEvent) => { const data = JSON.parse(msg.data); if (data.type === "CONFIG_ACCEPTED" && !recorderStarted) { configAccepted = true; recorderStarted = true; startMediaRecorder(); // Only start after config acceptance } }; function startMediaRecorder() { if (!configAccepted) { console.error("Cannot start recorder: CONFIG not accepted"); return; } mediaRecorder = new MediaRecorder(mediaStream); mediaRecorder.ondataavailable = (event: BlobEvent) => { if (ws.readyState === WebSocket.OPEN) { ws.send(event.data); } }; mediaRecorder.start(200); } ``` -------------------------------- ### AmbientScribe Config - Diarization Mode Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Example configuration for AmbientScribe using diarization mode for automatic speaker detection. ```typescript const config: Config = { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: true, isMultichannel: false, participants: [{ channel: 0, role: "multiple" }], }, mode: { type: "facts", outputLocale: "en" }, }, }; ``` -------------------------------- ### Start Audio Flow for Dictation Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/dictation.md Initiates an audio flow by connecting a MediaStream to a WebSocket endpoint for real-time transcription. This function requires authentication credentials and a callback for handling incoming messages. It returns a function to stop the dictation process. ```typescript async function startAudioFlow( mediaStream: MediaStream, authCreds: AuthCreds, handleEvent: (arg0: MessageEvent) => void, config?: DictationConfig ): Promise<{ recorderStarted: boolean; stop: () => void }> ``` ```typescript const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "auth-token-xyz", }; const transcripts: TranscriptEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { try { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { const event = parsed.data as TranscriptEventData; transcripts.push(event); console.log( `${event.isFinal ? "FINAL" : "interim"}: ${event.text}` ); } } catch (err) { console.error("Failed to parse message:", err); } }; async function startDictation() { try { const microphoneStream = await getMicrophoneStream(); const { recorderStarted, stop } = await startAudioFlow( microphoneStream, authCreds, handleNewMessage ); console.log("Dictation started:", recorderStarted); // Dictate for 30 seconds then stop setTimeout(() => { stop(); microphoneStream.getAudioTracks().forEach((t) => t.stop()); console.log("Dictation ended. Total transcripts:", transcripts.length); }, 30000); } catch (error) { console.error("Error starting dictation:", error); throw error; } } ``` -------------------------------- ### Default Dictation Configuration Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/dictation.md Defines the default settings for the dictation service, including language, interim results, punctuation handling, and sample commands. This configuration can be overridden when starting an audio flow. ```typescript const DEFAULT_CONFIG: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "go to next section", action: "next_section", keywords: ["next", "section"], }, ], }; ``` -------------------------------- ### Handle Media Devices API Not Supported Error Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md Catch and display a user-friendly message when the browser lacks navigator.mediaDevices API support. This is useful for guiding users to update their browser. ```typescript try { const stream = await getMicrophoneStream(); } catch (error) { if (error.message === "Media Devices API not supported in this browser") { console.error("Browser does not support microphone access"); showErrorUI("Your browser does not support audio input. Please use a modern browser."); } else { console.error("Microphone access failed:", error); showErrorUI("Unable to access microphone. Check permissions and try again."); } } ``` -------------------------------- ### Start Single Microphone AmbientScribe Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Initiates audio capture from a single microphone for real-time transcription and diarization. Handles incoming transcript and fact events. ```typescript import { AuthCreds, Config, TranscriptEventData, FactEventData } from "./types"; const authCreds: AuthCreds = { environment: "us", tenant: "your-tenant", token: "your-token", }; const transcripts: TranscriptEventData[] = []; const facts: FactEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data); } else if (parsed.type === "fact") { facts.push(parsed.data); } }; async function startCall() { const microphoneStream = await getMicrophoneStream(); const { stop } = await startAudioFlow( microphoneStream, authCreds, "interaction-123", handleNewMessage ); // Later... stop(); } ``` -------------------------------- ### Default Configuration for Single Microphone Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-singlemicrophone.md This is the default configuration object used when starting an audio flow without a custom configuration. It enables speaker diarization, sets the primary language to English, and configures the mode for fact extraction. ```typescript const DEFAULT_CONFIG: Config = { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: true, // Speaker diarization enabled isMultichannel: false, participants: [ { channel: 0, role: "multiple", // Diarization handles multiple speakers }, ], }, mode: { type: "facts", outputLocale: "en", }, }, }; ``` -------------------------------- ### Transcript Event Data Example Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Represents a real-time transcription event from the AmbientScribe API. Includes transcript text, timing information, finality status, and participant details. Received via WebSocket messages. ```typescript const event: TranscriptEventData = { id: "trx-123", start: 1500, duration: 3000, transcript: "The patient reports chest pain.", isFinal: true, participant: { channel: 0, role: "doctor", }, time: { start: 1500, end: 4500, }, }; ``` -------------------------------- ### Log WebSocket Errors with Context Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md This example shows how to log WebSocket errors, including a custom error code, timestamp, URL, and the error event itself. It also includes a placeholder for sending errors to a tracking service. ```typescript const ERROR_CODES = { AUDIO_NOT_SUPPORTED: "AUDIO_NOT_SUPPORTED", PERMISSION_DENIED: "PERMISSION_DENIED", WS_CONNECTION_FAILED: "WS_CONNECTION_FAILED", }; ws.onerror = (err: Event) => { const errorData = { code: ERROR_CODES.WS_CONNECTION_FAILED, timestamp: new Date().toISOString(), url: wsUrl, error: err.toString(), }; console.error("WebSocket Error:", errorData); // Send to error tracking service // logError(errorData); }; ``` -------------------------------- ### Start Audio Flow with AmbientScribe Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-singlemicrophone.md Initiates an audio stream to the Corti AI Platform WebSocket. This function connects a MediaStream, sends authentication, and processes incoming messages for transcription and fact extraction. Ensure you have valid AuthCreds and an interactionId. ```typescript async function startAudioFlow( mediaStream: MediaStream, authCreds: AuthCreds, interactionId: string, handleEvent: (arg0: MessageEvent) => void, config?: Config ): Promise<{ recorderStarted: boolean; stop: () => void }> ``` ```typescript const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "auth-token-xyz", }; const transcripts: TranscriptEventData[] = []; const facts: FactEventData[] = []; const handleNewMessage = (msg: MessageEvent) => { try { const parsed = JSON.parse(msg.data); if (parsed.type === "transcript") { transcripts.push(parsed.data as TranscriptEventData); } else if (parsed.type === "fact") { facts.push(parsed.data as FactEventData); } } catch (err) { console.error("Failed to parse message:", err); } }; async function startCall() { const microphoneStream = await getMicrophoneStream(); const { recorderStarted, stop } = await startAudioFlow( microphoneStream, authCreds, "interaction-123", handleNewMessage ); console.log("Audio recording started:", recorderStarted); // Later, end the call stop(); microphoneStream.getAudioTracks().forEach((track) => track.stop()); } ``` -------------------------------- ### Get Microphone Stream Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-singlemicrophone.md Retrieves the user's microphone MediaStream. Use this to get audio input from the default or a specific microphone. Handles browser API availability and user permissions. ```typescript const getMicrophoneStream = async (deviceId?: string): Promise ``` ```typescript // Request default microphone const stream = await getMicrophoneStream(); // Request specific microphone by device ID const stream = await getMicrophoneStream("abc123deviceId"); // Handle errors try { const stream = await getMicrophoneStream(); } catch (error) { console.error("Microphone access failed:", error); } ``` -------------------------------- ### startAudioFlow Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md Initiates an audio flow for a virtual consultation, handling real-time audio transmission and event callbacks. It returns a promise that resolves with the status of the recorder and a function to stop the audio flow. ```APIDOC ## startAudioFlow ### Description Initiates an audio flow for a virtual consultation, handling real-time audio transmission and event callbacks. It returns a promise that resolves with the status of the recorder and a function to stop the audio flow. ### Method POST (Implied, as it initiates a process) ### Endpoint (Not explicitly defined, but related to WebSocket connection details) ### Parameters #### Path Parameters None explicitly defined for this function call, but `interactionId` is used in the WebSocket URL. #### Query Parameters None explicitly defined for this function call, but `tenant` and `token` are used in the WebSocket URL. #### Request Body - **mediaStream** (MediaStream) - Required - The audio MediaStream to transmit. When using `mergeMediaStreams()`, pass the merged stream output. - **authCreds** (AuthCreds) - Required - Authentication credentials object containing `environment`, `tenant`, and `token`. - **interactionId** (string) - Required - The interaction identifier used in the WebSocket URL. Unique per consultation session. - **handleEvent** (function) - Required - Callback function invoked on WebSocket message arrival. Must handle JSON parsing. - **config** (Config) - Optional - Optional configuration. Uses `DEFAULT_CONFIG` if not provided. ### Request Example ```typescript const authCreds: AuthCreds = { environment: "us", tenant: "my-tenant", token: "token-xyz", }; const handleMessages = (msg: MessageEvent) => { const parsed = JSON.parse(msg.data); // Handle transcript or fact events }; async function startConsultation() { const microphoneStream = await getMicrophoneStream(); const webRTCStream = new MediaStream(); // Get from peer connection const { stream } = mergeMediaStreams([microphoneStream, webRTCStream]); const { stop } = await startAudioFlow( stream, authCreds, "consultation-456", handleMessages ); // Cleanup later // stop(); } ``` ### Response #### Success Response - **recorderStarted** (boolean) - Boolean indicating MediaRecorder state. - **stop** (function) - Function to gracefully end the audio flow and close WebSocket. #### Response Example ```json { "recorderStarted": true, "stop": "function" } ``` ### Throws - Error: During WebSocket initialization if connection fails (logged to console). - Error: If JSON parsing of messages fails (logged to console). ### WebSocket Connection Details - **URL Pattern:** `wss://api.{environment}.corti.app/audio-bridge/v2/interactions/{interactionId}/streams?tenant-name={tenant}&token=Bearer%20{token}` - **Audio Chunk Interval:** 200ms chunks - **Connection Lifecycle:** Waits for CONFIG_ACCEPTED before starting MediaRecorder ``` -------------------------------- ### Custom Voice Commands Configuration Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/dictation.md Illustrates how to define custom voice commands with specific actions and keywords, and how to pass this configuration to the `startAudioFlow` function. ```typescript const customConfig: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "create new note", action: "create_note", keywords: ["new", "note"], }, { command: "go to next patient", action: "next_patient", keywords: ["next", "patient"], }, { command: "save and exit", action: "save_exit", keywords: ["save", "exit"], }, ], }; // Pass to startAudioFlow const { stop } = await startAudioFlow( microphoneStream, authCreds, handleEvent, customConfig ); ``` -------------------------------- ### Default Configuration for Virtual Consultations Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md The default configuration for `startAudioFlow` disables diarization and explicitly assigns roles to participants. It is suitable for scenarios where speaker identity is known and enables fact extraction. ```typescript const DEFAULT_CONFIG: Config = { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: false, // No diarization; roles are explicit isMultichannel: false, participants: [ { channel: 0, role: "doctor", }, { channel: 0, role: "patient", }, ], }, mode: { type: "facts", outputLocale: "en", }, }, }; ``` -------------------------------- ### Initial Configuration Message Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md Send this JSON message upon connecting to the WebSocket to configure transcription and processing modes. The server will respond with CONFIG_ACCEPTED. ```json { "type": "config", "configuration": { "transcription": { "primaryLanguage": "en", "isDiarization": true, "isMultichannel": false, "participants": [ { "channel": 0, "role": "multiple" } ] }, "mode": { "type": "facts", "outputLocale": "en" } } } ``` -------------------------------- ### Configuration Accepted Response Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md This is the expected response from the server after sending the initial configuration message. The client must wait for this before sending audio data. ```json { "type": "CONFIG_ACCEPTED" } ``` -------------------------------- ### Initial Configuration Message Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md Send this JSON message upon connecting to configure transcription settings like language, interim results, and custom commands. Ensure the 'type' field is set to 'config'. ```json { "type": "config", "configuration": { "primaryLanguage": "en", "interimResults": true, "spokenPunctuation": true, "automaticPunctuation": true, "commands": [ { "command": "go to next section", "action": "next_section", "keywords": ["next", "section"] } ] } } ``` -------------------------------- ### Handle WebSocket Connection Errors Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md Logs WebSocket errors and displays a user-friendly error message. This snippet should be used within the WebSocket setup to catch connection-related issues. ```typescript ws.onerror = (err: Event) => { console.error("WebSocket encountered an error:", err); }; ``` -------------------------------- ### Handle AudioContext Creation Failure Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md This snippet demonstrates how to safely create an AudioContext and handle potential suspension, especially on iOS. It includes a fallback UI message if creation fails. ```typescript try { const audioContext = new AudioContext(); // Verify context state if (audioContext.state === "suspended") { // iOS and some browsers suspend AudioContext until user interaction document.addEventListener("click", () => { audioContext.resume(); }); } } catch (error) { console.error("AudioContext creation failed:", error); showErrorUI("Audio processing is not available in your browser."); } ``` -------------------------------- ### Get Microphone MediaStream Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md Retrieves the user's microphone audio stream. Optionally specify a device ID to select a particular microphone. This is useful for capturing the primary audio input for a consultation. ```typescript export const getMicrophoneStream = async (deviceId?: string): Promise ``` ```typescript try { const stream = await getMicrophoneStream(); console.log("Microphone access granted"); } catch (error) { console.error("Microphone access denied:", error); } ``` ``` -------------------------------- ### Handle Microphone Access Denied or Not Found Error Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md Differentiate between microphone permission denial, no device found, or security blocks. Provides specific user guidance for each scenario. ```typescript try { const stream = await getMicrophoneStream(); } catch (error) { if (error.name === "NotAllowedError") { showErrorUI("Microphone permission denied. Please grant permission in browser settings."); } else if (error.name === "NotFoundError") { showErrorUI("No microphone device found. Please connect a microphone."); } else if (error.name === "SecurityError") { showErrorUI("Microphone access blocked by security policy."); } else { console.error("Microphone error:", error); showErrorUI("Unable to access microphone."); } } ``` -------------------------------- ### Default Configuration (Source) Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/api-reference/ambientscribe-virtualconsultations.md This constant defines the default configuration object used by `startAudioFlow` when no custom configuration is provided. It specifies transcription settings and fact extraction mode. ```typescript const DEFAULT_CONFIG: Config = { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: false, isMultichannel: false, participants: [ { channel: 0, role: "doctor", }, { channel: 0, role: "patient", }, ], }, mode: { type: "facts", outputLocale: "en", }, }, }; ``` -------------------------------- ### Minimal Dictation Configuration (Interim Only) Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md A basic dictation configuration that disables interim results and custom commands, suitable for high-latency networks or simple transcription needs. It still enables automatic punctuation. ```typescript const config: DictationConfig = { primaryLanguage: "en", interimResults: false, spokenPunctuation: false, automaticPunctuation: true, commands: [], }; ``` -------------------------------- ### Configure Dictation Settings Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/types.md Defines the configuration for dictation, including language, interim results, punctuation, and custom commands. Used with the startAudioFlow() function. ```typescript const config: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "create new section", action: "new_section", keywords: ["create", "new", "section"], }, ], }; ``` -------------------------------- ### Spanish Language Dictation Configuration with Custom Commands Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md A dictation configuration localized for Spanish, enabling all features including interim results, punctuation, and custom Spanish voice commands. This is suitable for Spanish-speaking users. ```typescript const config: DictationConfig = { primaryLanguage: "es", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "ir a siguiente sección", action: "next_section", keywords: ["siguiente", "sección"], }, { command: "guardar nota", action: "save_note", keywords: ["guardar", "nota"], }, ], }; ``` -------------------------------- ### Corti AI Platform Architecture Overview Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Diagram illustrating the flow of audio data from a user application to the Corti AI Platform via WebSockets. ```text User Application ↓ ├─→ getMicrophoneStream() ──→ MediaStream │ ├─→ mergeMediaStreams() ──→ Merged MediaStream (Consultations only) │ └─→ startAudioFlow() ──→ WebSocket ├─ Send: Config, Audio Chunks, End └─ Receive: Transcripts, Facts ↓ Corti AI Platform ``` -------------------------------- ### Token Format for WebSocket URLs Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Shows the required format for passing authentication tokens in WebSocket URLs. ```text ?token=Bearer%20{token} ``` -------------------------------- ### Implement Graceful Degradation in Audio Flow Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md This function demonstrates graceful degradation by setting default configurations and handling non-critical errors, such as analytics failures, without stopping the main audio flow. It also includes input validation. ```typescript async function startAudioFlow(...): Promise<{ stop: () => void }> { // Validate inputs if (!mediaStream) throw new Error("No media stream provided"); // Set defaults if (!config) config = DEFAULT_CONFIG; // Continue even if non-critical operations fail try { // Optional: send analytics } catch (err) { console.warn("Analytics failed (non-critical):", err); } // Continue with main flow return { stop }; } ``` -------------------------------- ### Send Configuration to WebSocket Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md Send the configuration object to the WebSocket server after the connection is established. This configures transcription parameters. ```typescript ws.onopen = () => { ws.send(JSON.stringify({ type: "config", configuration: config, })); }; ``` -------------------------------- ### AmbientScribe Configuration with Diarization Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/README.md Configuration object for AmbientScribe enabling transcription with diarization and fact extraction. Specifies primary language and participant roles. ```typescript { type: "config", configuration: { transcription: { primaryLanguage: "en", isDiarization: true, isMultichannel: false, participants: [{ channel: 0, role: "multiple" }], }, mode: { type: "facts", outputLocale: "en" }, }, } ``` -------------------------------- ### AmbientScribe Configuration: Custom Language and Locale Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md Configuration to set a custom primary transcription language and output locale. Use for regional variations or non-English audio processing. ```typescript const config: Config = { type: "config", configuration: { transcription: { primaryLanguage: "es", // Spanish isDiarization: true, isMultichannel: false, participants: [ { channel: 0, role: "multiple", }, ], }, mode: { type: "facts", outputLocale: "es", // Spanish locale formatting }, }, }; ``` -------------------------------- ### WebSocket Connection Lifecycle Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md This outlines the sequence of messages exchanged between the client and server during a WebSocket connection for audio processing. It covers connection, configuration, audio streaming, and finalization. ```text 1. Client connects to WSS endpoint 2. Server connection opens (ws.onopen) 3. Client sends CONFIG message 4. Server responds with CONFIG_ACCEPTED 5. Client starts MediaRecorder and sends audio chunks 6. Server sends TRANSCRIPT and FACT messages as available 7. Client sends END message to finalize 8. Server closes connection ``` -------------------------------- ### Dictation WebSocket Endpoint URL Pattern Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md This is the base URL pattern for the dictation WebSocket endpoint. Replace placeholders like {environment}, {tenant}, and {token} with your specific values. ```text wss://api.{environment}.corti.app/audio-bridge/v2/transcribe?tenant-name={tenant}&token=Bearer%20{token} ``` -------------------------------- ### Advanced Dictation Configuration with Custom Commands Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/configuration.md A comprehensive dictation configuration enabling all features and including multiple application-specific voice commands. This is useful for medical dictation or multi-step transcription processes. ```typescript const config: DictationConfig = { primaryLanguage: "en", interimResults: true, spokenPunctuation: true, automaticPunctuation: true, commands: [ { command: "create new note", action: "create_note", keywords: ["create", "new", "note"], }, { command: "save and close", action: "save_close", keywords: ["save", "close"], }, { command: "go to next patient", action: "next_patient", keywords: ["next", "patient"], }, { command: "review facts", action: "review_facts", keywords: ["review", "facts"], }, ], }; ``` -------------------------------- ### Establish WebSocket Connection Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/endpoints.md Connect to the Corti API endpoint. Ensure your authentication credentials and environment are correctly configured in the URL. ```typescript const wsUrl = `wss://api.${authCreds.environment}.corti.app/audio-bridge/v2/transcribe?tenant-name=${authCreds.tenant}&token=Bearer%20${authCreds.token}`; const ws = new WebSocket(wsUrl); ws.onopen = () => console.log("Connected"); ws.onerror = (err) => console.error("Error:", err); ws.onclose = () => console.log("Closed"); ``` -------------------------------- ### Provide User-Friendly Error Messages Source: https://github.com/corticph/api-usage-examples/blob/main/_autodocs/errors.md This function maps specific error messages to user-friendly explanations for common issues like unsupported media devices or permission denials. It provides a default message for unhandled errors. ```typescript function getErrorMessage(error: Error): string { if (error.message.includes("Media Devices API not supported")) { return "Your browser does not support microphone input. Please use Chrome, Firefox, Safari, or Edge."; } if (error.message.includes("NotAllowedError")) { return "Microphone permission is required. Please grant permission in your browser settings."; } if (error.message.includes("does not have an audio track")) { return "Unable to access audio. Please ensure microphone or audio devices are properly connected."; } return "An error occurred. Please refresh the page and try again."; } showErrorUI(getErrorMessage(error)); ```