Fluid Inference

https://docs.fluidinference.com
Admin
Fluid Inference is a local audio AI platform for Apple devices providing speech-to-text, speaker...
Tokens:23,515
Snippets:230
Trust Score:5.1
Update:8 months ago
Show doc for...
Tokens:
Raw
### ASR CLI Commands - Fluid Inference

Source: https://docs.fluidinference.com/asr/getting-started

Provides examples of using the Fluid Inference command-line interface (CLI) for various ASR tasks. This includes basic transcription, specifying model versions, processing multiple files, and running benchmarks.

```bash
# Transcribe (multilingual)
swift run fluidaudio transcribe audio.wav

# English-only (better recall)
swift run fluidaudio transcribe audio.wav --model-version v2

# Multiple files in parallel
swift run fluidaudio multi-stream audio1.wav audio2.wav

# Benchmark on LibriSpeech
swift run fluidaudio asr-benchmark --subset test-clean --max-files 50
```

--------------------------------

### Swift Quick Start for Kokoro TTS Synthesis

Source: https://docs.fluidinference.com/tts/kokoro

Provides a basic Swift code example for initializing the TtSManager and synthesizing text to an audio file using Kokoro TTS. This is a straightforward way to integrate TTS into Swift applications.

```swift
import FluidAudioTTS

let manager = TtSManager()
try await manager.initialize()

let audioData = try await manager.synthesize(text: "Hello from FluidAudio!")
try audioData.write(to: URL(fileURLWithPath: "/tmp/demo.wav"))
```

--------------------------------

### Real-time Audio Capture and Diarization Setup (Swift)

Source: https://docs.fluidinference.com/diarization/streaming

Sets up a class for real-time audio capture and diarization. It initializes the audio engine, diarizer, and audio stream, then installs a tap on the input node to process audio buffers.

```swift
class RealTimeDiarizer {
    private let audioEngine = AVAudioEngine()
    private let diarizer: DiarizerManager
    private var audioStream: AudioStream

    init() async throws {
        let models = try await DiarizerModels.downloadIfNeeded()
        diarizer = DiarizerManager()
        diarizer.initialize(models: models)
        audioStream = AudioStream(
            chunkDuration: 5.0,
            chunkSkip: 3.0,
            streamStartTime: 0.0,
            chunkingStrategy: .useFixedSkip
        )
        audioStream.bind { [weak self] chunk, _ in
            Task {
                let result = try self?.diarizer.performCompleteDiarization(chunk)
                // Handle results
            }
        }
    }

    func startCapture() throws {
        let inputNode = audioEngine.inputNode
        let format = inputNode.outputFormat(forBus: 0)

        inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { 
            [weak self] buffer, _ in
            try? self?.audioStream.write(from: buffer)
        }

        audioEngine.prepare()
        try audioEngine.start()
    }
}
```

--------------------------------

### Install FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Instructions to build and install the FluidAudio command-line interface using Swift Package Manager. This involves navigating to the project directory and executing the build command.

```bash
cd FluidAudio
swift build -c release
# Binary at .build/release/fluidaudio
```

--------------------------------

### Run FluidAudio CLI Directly

Source: https://docs.fluidinference.com/reference/cli

Demonstrates how to run the FluidAudio CLI directly using 'swift run' followed by the command and its arguments. This is an alternative to building and installing the binary.

```bash
swift run fluidaudio <command>
```

--------------------------------

### Quick Start Synthesize and Save Audio with PocketTTS (Swift)

Source: https://docs.fluidinference.com/tts/pocket-tts

Demonstrates how to initialize the PocketTtsManager, synthesize text to audio data, and save the synthesized audio to a file using Swift. This code requires the FluidAudioTTS library.

```swift
import FluidAudioTTS

let manager = PocketTtsManager()
try await manager.initialize()

let audioData = try await manager.synthesize(text: "Hello, world!")

try await manager.synthesizeToFile(
    text: "Hello, world!",
    outputURL: URL(fileURLWithPath: "/tmp/output.wav")
)
```

--------------------------------

### Quick Start CLI for Kokoro TTS

Source: https://docs.fluidinference.com/tts/kokoro

Demonstrates how to use the FluidAudio command-line interface (CLI) to perform text-to-speech synthesis with Kokoro TTS. It specifies the output file and selects a voice.

```bash
swift run fluidaudio tts "Welcome to FluidAudio text to speech" \
  --output ~/Desktop/demo.wav \
  --voice af_heart
```

--------------------------------

### Example Finance and Technology Lexicon Entries

Source: https://docs.fluidinference.com/asr/custom-pronunciation

This text file provides example entries for a custom pronunciation lexicon, covering financial terms, technology names, and product names with their corresponding IPA phonetic representations.

```text
# Finance
NASDAQ=nˈæzdæk
EBITDA=iːbˈɪtdɑː

# Technology
NVIDIA=ɛnvˈɪdiə
Kubernetes=kuːbɚnˈɛtiːz

# Product Names
Kokoro=kəkˈɔɹO
FluidAudio=flˈuːɪd ˈɔːdioʊ
```

--------------------------------

### Install FluidAudio Package

Source: https://docs.fluidinference.com/quickstart

Add FluidAudio to your project using Swift Package Manager. This is the initial step to integrate FluidAudio's capabilities into your Swift applications.

```swift
dependencies: [
    .package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.9"),
]
```

--------------------------------

### Install FluidAudio for React Native / Expo

Source: https://docs.fluidinference.com/installation

This command installs the FluidAudio package for React Native and Expo projects using npm. It utilizes the official npm package for cross-platform audio capabilities.

```bash
npm install @fluidinference/react-native-fluidaudio
```

--------------------------------

### Manual Model Loading Examples in Swift

Source: https://docs.fluidinference.com/guides/manual-model-loading

Demonstrates how to manually load CoreML models for different FluidAudio modules (ASR, Diarization, VAD) using Swift. This is useful for offline or air-gapped deployments where automatic HuggingFace downloads are not possible. It shows the expected API calls and model configurations.

```swift
// Example: ASR
let models = try await AsrModels.load(
    from: URL(fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v3-coreml"),
    version: .v3
)

// Example: Diarization
let models = try await DiarizerModels.load(
    localSegmentationModel: segmentationURL,
    localEmbeddingModel: embeddingURL
)

// Example: VAD
let vadModel = try MLModel(contentsOf: modelURL, configuration: config)
let manager = VadManager(config: .default, vadModel: vadModel)
```

--------------------------------

### FluidAudio CLI Commands

Source: https://docs.fluidinference.com/diarization/getting-started

This example shows command-line interface (CLI) commands for using FluidAudio for audio processing and benchmarking. It includes commands for processing an audio file and running diarization benchmarks.

```bash
swift run fluidaudio process meeting.wav --output results.json --threshold 0.6
swift run fluidaudio diarization-benchmark --auto-download
```

--------------------------------

### Add FluidAudio to Swift Project via Swift Package Manager

Source: https://docs.fluidinference.com/installation

This snippet shows how to add the FluidAudio Swift package to your project's dependencies. It specifies the repository URL and the version range. You can choose between the core 'FluidAudio' product or the 'FluidAudioTTS' product which includes GPL dependencies.

```swift
dependencies: [
    .package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.9"),
]
```

```swift
// Core features only (no GPL dependencies):
.product(name: "FluidAudio", package: "FluidAudio")

// Add TTS support (includes GPL ESpeakNG):
.product(name: "FluidAudioTTS", package: "FluidAudio")
```

--------------------------------

### Add FluidAudio for Rust / Tauri

Source: https://docs.fluidinference.com/installation

This command adds the FluidAudio Rust crate to your project using Cargo. This enables integrating FluidAudio's functionalities within Rust or Tauri applications.

```bash
cargo add fluidaudio-rs
```

--------------------------------

### Batch Transcription with Parakeet Models (Swift)

Source: https://docs.fluidinference.com/asr/getting-started

Performs batch transcription of an audio file using Fluid Inference's Parakeet models. It requires downloading and loading the ASR models and initializing the ASR manager. The input is an audio file path, and the output is the transcribed text and its confidence score.

```swift
import FluidAudio

Task {
    let models = try await AsrModels.downloadAndLoad(version: .v3) // .v2 for English-only
    let asrManager = AsrManager(config: .default)
    try await asrManager.initialize(models: models)

    let samples = try AudioConverter().resampleAudioFile(
        path: "path/to/audio.wav"
    )
    let result = try await asrManager.transcribe(samples, source: .system)
    print("Transcription: \(result.text)")
    print("Confidence: \(result.confidence)")
}
```

```swift
let audioURL = URL(fileURLWithPath: "/path/to/audio.wav")
let result = try await asrManager.transcribe(audioURL, source: .system)
print(result.text)
```

```swift
let models = try await AsrModels.downloadAndLoad(version: .v2)
```

--------------------------------

### Run Benchmarks with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Provides examples for running various benchmarks using the FluidAudio CLI, including Automatic Speech Recognition (ASR), FLEURS multilingual, Diarization, and Voice Activity Detection (VAD) benchmarks. Options for subset selection, language specification, and automatic downloads are shown.

```bash
# ASR benchmark
fluidaudio asr-benchmark --subset test-clean --max-files 100

# FLEURS multilingual
fluidaudio fleurs-benchmark --languages en_us,fr_fr --samples 10

# Diarization benchmark
fluidaudio diarization-benchmark --auto-download
fluidaudio diarization-benchmark --single-file ES2004a --threshold 0.7

# VAD benchmark
fluidaudio vad-benchmark --num-files 40 --threshold 0.5
```

--------------------------------

### Known Speaker Recognition Setup (Swift)

Source: https://docs.fluidinference.com/diarization/getting-started

This Swift snippet illustrates how to set up known speaker recognition by pre-loading speaker profiles. It involves extracting embeddings from sample audio for known speakers and initializing the diarizer's speaker manager with these profiles.

```swift
let aliceAudio = loadAudioFile("alice_sample.wav")
let aliceEmbedding = try diarizer.extractEmbedding(aliceAudio)

let alice = Speaker(id: "Alice", name: "Alice", currentEmbedding: aliceEmbedding)
let bob = Speaker(id: "Bob", name: "Bob", currentEmbedding: bobEmbedding)
diarizer.speakerManager.initializeKnownSpeakers([alice, bob])

// Will use "Alice" instead of "Speaker_1" when matched
let result = try diarizer.performCompleteDiarization(audioSamples)
```

--------------------------------

### Add FluidAudio to Project via CocoaPods

Source: https://docs.fluidinference.com/installation

This snippet demonstrates how to add FluidAudio to your project using CocoaPods. It specifies the pod name and version constraint. This method is an alternative to Swift Package Manager for dependency management.

```ruby
pod 'FluidAudio', '~> 0.7.8'
```

--------------------------------

### Swift Coexistence of SSML and Markdown

Source: https://docs.fluidinference.com/tts/ssml

This Swift example shows how SSML tags can be used alongside Markdown syntax within the same text string for text-to-speech synthesis. It demonstrates using a `<phoneme>` tag for custom pronunciation and a Markdown link for another word.

```swift
let text = """
    <phoneme ph="kəˈkɔɹo">Kokoro</phoneme> and [Misaki](/mɪˈsɑːki/)
    """
```

--------------------------------

### Bash: CLI Commands for Offline Diarization

Source: https://docs.fluidinference.com/diarization/offline-pipeline

Provides examples of using the FluidAudio CLI for offline diarization tasks. Includes processing a single file, benchmarking on the AMI dataset, and processing with ground-truth RTTM files.

```bash
# Process a single file
swift run fluidaudio process meeting.wav --mode offline --threshold 0.6

# Benchmark on AMI dataset
swift run fluidaudio diarization-benchmark --mode offline \
  --dataset ami-sdm --threshold 0.6 --auto-download

# With ground-truth RTTM
swift run fluidaudio process meeting.wav --mode offline \
  --rttm ground_truth.rttm
```

--------------------------------

### Clone ASR Models using Git LFS

Source: https://docs.fluidinference.com/asr/manual-model-loading

This command clones the ASR model repository using Git Large File Storage (LFS). Ensure Git LFS is installed before running this command. It downloads all necessary model bundles and vocabulary files.

```bash
git lfs install
git clone https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v3-coreml
```

--------------------------------

### Transcribe Audio File using CLI

Source: https://docs.fluidinference.com/asr/streaming

Provides a command-line interface example using Swift's package manager to transcribe an audio file (`.wav`) with the Parakeet EOU model. The `--use-cache` flag can speed up subsequent runs.

```bash
swift run fluidaudio parakeet-eou --input audio.wav --use-cache
```

--------------------------------

### Voice Activity Detection with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Demonstrates the 'fluidaudio vad-analyze' command for Voice Activity Detection (VAD). Examples show offline segmentation, streaming mode with silence detection, and processing in both modes.

```bash
# Offline segmentation
fluidaudio vad-analyze audio.wav

# Streaming
fluidaudio vad-analyze audio.wav --streaming --min-silence-ms 300

# Both modes
fluidaudio vad-analyze audio.wav --mode both
```

--------------------------------

### Initialize Known Speakers

Source: https://docs.fluidinference.com/diarization/speaker-manager

Initializes the SpeakerManager with a predefined list of known speakers, each with an ID, name, and voice embedding. This allows the manager to recognize these speakers from the start.

```swift
let alice = Speaker(id: "alice", name: "Alice", currentEmbedding: aliceEmbedding)
let bob = Speaker(id: "bob", name: "Bob", currentEmbedding: bobEmbedding)
speakerManager.initializeKnownSpeakers([alice, bob])
```

--------------------------------

### Text-to-Speech with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Showcases the 'fluidaudio tts' command for converting text to speech. Examples include basic TTS, specifying output files, voice selection, and custom pronunciation using lexicons.

```bash
fluidaudio tts "Hello from FluidAudio" --output demo.wav --voice af_heart
fluidaudio tts "Custom pronunciation" --lexicon custom.txt --output out.wav
```

--------------------------------

### Transcribe Audio Files with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Examples of using the 'fluidaudio transcribe' command for batch and multi-stream audio transcription. It includes options for specifying model versions and handling multiple audio files in parallel.

```bash
# Batch transcription
fluidaudio transcribe audio.wav
fluidaudio transcribe audio.wav --model-version v2  # English-only

# Multi-stream parallel transcription
fluidaudio multi-stream audio1.wav audio2.wav

# Streaming transcription (Parakeet EOU)
fluidaudio parakeet-eou --input audio.wav --use-cache
```

--------------------------------

### Configure Diarizer Manager (Swift)

Source: https://docs.fluidinference.com/diarization/getting-started

This code example shows how to configure the DiarizerManager with custom settings in Swift. It allows adjusting parameters such as clustering threshold, minimum speech duration, minimum silence gap, minimum active frames, and debug mode.

```swift
let config = DiarizerConfig(
    clusteringThreshold: 0.7,      // Speaker separation (0.0-1.0)
    minSpeechDuration: 1.0,         // Minimum segment duration (seconds)
    minSilenceGap: 0.5,             // Minimum silence between speakers
    minActiveFramesCount: 10.0,     // Minimum active frames
    debugMode: false
)
let diarizer = DiarizerManager(config: config)
```

--------------------------------

### Define Custom Vocabulary with Aliases (Swift)

Source: https://docs.fluidinference.com/asr/custom-vocabulary

This code example shows how to configure a custom vocabulary that includes aliases for specific terms. When a match is found using either the canonical term or one of its aliases, the canonical form is used in the final transcription output.

```swift
let vocabulary = CustomVocabularyContext(terms: [
    CustomVocabularyTerm(
        text: "Hagen-Dazs",
        aliases: ["Haagen-Dazs", "Hagen-Das", "Hagen Daz"]
    ),
    CustomVocabularyTerm(
        text: "macOS",
        aliases: ["Mac OS", "Mac O S", "Macos"]
    ),
])
```

--------------------------------

### Define Custom Pronunciations in Lexicon File

Source: https://docs.fluidinference.com/asr/custom-pronunciation

This text file defines custom pronunciations for words using IPA phonemes. Each line contains a word followed by its phonetic representation, separated by an equals sign. Comments start with '#'.

```text
# This is a comment
kokoro=kəkˈɔɹO
NASDAQ=nˈæzdæk
UN=junˈaɪtᵻd nˈeɪʃənz
```

--------------------------------

### Offline Audio Segmentation with Silero VAD v6 (Swift)

Source: https://docs.fluidinference.com/vad/getting-started

Segments speech from an audio file using Silero VAD v6. It takes audio samples as input and returns a list of speech segments with start and end times. Requires the FluidAudio library.

```swift
import FluidAudio

let manager = try await VadManager(
    config: VadConfig(defaultThreshold: 0.75)
)

let samples = try AudioConverter().resampleAudioFile(
    URL(fileURLWithPath: "audio.wav")
)

var segmentation = VadSegmentationConfig.default
segmentation.minSpeechDuration = 0.25
segmentation.minSilenceDuration = 0.4
segmentation.speechPadding = 0.12

let segments = try await manager.segmentSpeech(samples, config: segmentation)
for (index, segment) in segments.enumerated() {
    print(String(format: "Segment %02d: %.2f-%.2fs", index + 1, segment.startTime, segment.endTime))
}
```

--------------------------------

### Create Model Directory and Initialize Dependencies (Bash)

Source: https://docs.fluidinference.com/mobius/converting-models

This snippet demonstrates how to create the necessary directory structure for a new model and initialize Python dependencies using `uv`. It includes creating the directory, initializing the environment, and adding coremltools and torch as dependencies.

```bash
mkdir -p models/stt/my-new-model/coreml
cd models/stt/my-new-model/coreml

# Initialize with uv
uv init
uv add coremltools torch

# Write your conversion script
# ... convert-my-model.py
```

--------------------------------

### Benchmark Streaming ASR using CLI

Source: https://docs.fluidinference.com/asr/streaming

Demonstrates how to run benchmarks for the Parakeet EOU ASR model from the command line. This allows testing performance with different chunk sizes and limiting the number of files processed.

```bash
swift run fluidaudio parakeet-eou --benchmark --chunk-size 160 --max-files 100 --use-cache
```

--------------------------------

### Run VAD Analysis and Benchmarks via CLI (Bash)

Source: https://docs.fluidinference.com/vad/getting-started

Command-line interface commands for performing voice activity detection analysis and benchmarks. Supports offline and streaming modes for analysis, and configurable benchmarking.

```bash
# Offline segmentation
swift run fluidaudio vad-analyze audio.wav

# Streaming mode
swift run fluidaudio vad-analyze audio.wav --streaming --min-silence-ms 300

# Both modes
swift run fluidaudio vad-analyze audio.wav --mode both

# Benchmark
swift run fluidaudio vad-benchmark --num-files 50 --threshold 0.3
```

--------------------------------

### Manage Datasets with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Illustrates how to use the 'fluidaudio download' command to fetch various datasets, including AMI-SDM, LibriSpeech test sets, and VAD datasets. This command simplifies data acquisition for testing and development.

```bash
fluidaudio download --dataset ami-sdm
fluidaudio download --dataset librispeech-test-clean
fluidaudio download --dataset librispeech-test-other
fluidaudio download --dataset vad
```

--------------------------------

### Run Sortformer Benchmark CLI

Source: https://docs.fluidinference.com/diarization/sortformer

This command executes the Sortformer benchmark using the Swift Package Manager. It enables NVIDIA's high-latency configuration, utilizes Hugging Face models, and automatically downloads necessary assets. This is useful for testing Sortformer's performance in a streaming context.

```bash
swift run fluidaudio sortformer-benchmark \
  --nvidia-high-latency --hf --auto-download
```

--------------------------------

### Initialize and Transcribe with Custom Vocabulary (Swift)

Source: https://docs.fluidinference.com/asr/custom-vocabulary

This snippet demonstrates how to initialize the AsrManager, download and load CTC models, create a CtcKeywordSpotter, define a custom vocabulary with terms, and then transcribe audio using this vocabulary. The output text will reflect the recognized custom terms.

```swift
let asrManager = try await AsrManager.shared
let ctcModels = try await CtcModels.downloadAndLoad()
let ctcSpotter = CtcKeywordSpotter(models: ctcModels)

let vocabulary = CustomVocabularyContext(terms: [
    CustomVocabularyTerm(text: "NVIDIA"),
    CustomVocabularyTerm(text: "TensorRT"),
])

let result = try await asrManager.transcribe(
    audioSamples,
    customVocabulary: vocabulary
)
// result.text: "NVIDIA announced TensorRT optimizations"
```

--------------------------------

### Swift Imports for FluidAudio and FluidAudioTTS

Source: https://docs.fluidinference.com/tts/kokoro

Demonstrates the necessary import statements in Swift to utilize the core FluidAudio functionalities (like ASR, diarization, VAD) and the specific FluidAudioTTS module for text-to-speech synthesis.

```swift
import FluidAudio       // Core (ASR, diarization, VAD)
import FluidAudioTTS    // TTS features
```

--------------------------------

### Load ASR Models from Local Directory in Swift

Source: https://docs.fluidinference.com/asr/manual-model-loading

This Swift code snippet demonstrates how to load ASR models from a specified local directory. It requires the `FluidAudio` framework and assumes the model bundles and vocabulary are correctly placed. The function returns an initialized `AsrManager` ready for use.

```swift
import FluidAudio

let repoDirectory = URL(
    fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v3-coreml",
    isDirectory: true
)

let models = try await AsrModels.load(
    from: repoDirectory,
    configuration: AsrModels.defaultConfiguration(),
    version: .v3
)

let asrManager = AsrManager()
try await asrManager.initialize(models: models)
```

--------------------------------

### Quantize CoreML Model for Size and Speed

Source: https://docs.fluidinference.com/mobius/converting-models

This script performs quantization on CoreML models to reduce their size and potentially improve latency, especially on Apple Neural Engine (ANE). It sweeps through various quantization strategies and reports on the trade-offs between size reduction, quality impact, and performance.

```bash
uv run python quantize_coreml.py \
  --input-dir ./build \
  --output-root ./build_quantized \
  --compute-units ALL --runs 10
```

--------------------------------

### Clone and Convert Parakeet STT Model using möbius

Source: https://docs.fluidinference.com/mobius/getting-started

This snippet demonstrates how to clone the möbius repository, set up the environment, convert a Parakeet STT model from N Nemo format to CoreML, and validate its parity. It requires the möbius repository and a local path to the .nemo model file.

```bash
# Clone
git clone https://github.com/FluidInference/mobius.git
cd mobius

# Pick a model
cd models/stt/parakeet-tdt-v3-0.6b/coreml

# Set up environment
uv sync

# Convert
uv run python convert-parakeet.py convert \
  --nemo-path /path/to/parakeet-tdt-0.6b-v3.nemo \
  --output-dir parakeet_coreml

# Validate parity
uv run python compare-components.py compare \
  --output-dir parakeet_coreml \
  --runs 10 --warmup 3
```

--------------------------------

### Run FluidAudio Benchmarks (Bash)

Source: https://docs.fluidinference.com/reference/benchmarks

A collection of bash commands to execute various FluidAudio benchmarks. These commands cover transcription for multiple languages, custom vocabulary ASR, streaming ASR, Voice Activity Detection (VAD), offline and streaming diarization, Sortformer, and text-to-speech (TTS).

```bash
swift run -c release fluidaudio fleurs-benchmark --languages all --samples all
swift run -c release fluidaudio asr-benchmark --max-files all
swift run -c release fluidaudio ctc-earnings-benchmark --auto-download
swift run -c release fluidaudio parakeet-eou --benchmark --chunk-size 320 --use-cache
swift run -c release fluidaudio vad-benchmark --dataset voices-subset --all-files --threshold 0.85
swift run -c release fluidaudio diarization-benchmark --mode offline --auto-download
swift run -c release fluidaudio diarization-benchmark --mode streaming \
  --dataset ami-sdm --threshold 0.8 --chunk-seconds 5.0 --overlap-seconds 0.0
swift run -c release fluidaudio sortformer-benchmark --nvidia-high-latency --hf --auto-download
swift run -c release fluidaudio tts --benchmark
```

--------------------------------

### Switch ASR Model Versions in Swift

Source: https://docs.fluidinference.com/asr/manual-model-loading

This Swift code shows how to load a different version of the ASR models from an alternative directory. It's useful for managing multiple language models or versions. The `AsrModels.load` function is used with a different repository path and version identifier.

```swift
let englishRepo = URL(fileURLWithPath: "/opt/models/parakeet-tdt-0.6b-v2-coreml")
let englishModels = try await AsrModels.load(from: englishRepo, version: .v2)
```

--------------------------------

### Swift Package Manager Configuration for FluidAudio

Source: https://docs.fluidinference.com/tts/kokoro

Shows how to add the FluidAudio library, including TTS features, as a dependency in your Swift project using the Package.swift file. This enables the use of FluidAudio's TTS capabilities in your application.

```swift
dependencies: [
    .package(url: "https://github.com/FluidInference/FluidAudio.git", from: "0.7.7"),
],
targets: [
    .target(
        name: "YourTarget",
        dependencies: [
            .product(name: "FluidAudioWithTTS", package: "FluidAudio")
        ]
    )
]
```

--------------------------------

### Swift Detailed Synthesis with Chunk Metadata

Source: https://docs.fluidinference.com/tts/kokoro

Illustrates how to synthesize text with Kokoro TTS and retrieve detailed chunk metadata, including variant, token count, and text for each chunk. This is useful for analyzing the synthesis process or implementing chunk-based audio processing.

```swift
let detailed = try await manager.synthesizeDetailed(
    text: "FluidAudio can report chunk splits for you.",
    variantPreference: .fifteenSecond
)

for chunk in detailed.chunks {
    print("Chunk #\(chunk.index) -> variant: \(chunk.variant), tokens: \(chunk.tokenCount)")
    print("  text: \(chunk.text)")
}
```

--------------------------------

### Initialize and Use Streaming ASR Manager in Swift

Source: https://docs.fluidinference.com/asr/streaming

Demonstrates how to initialize `StreamingEouAsrManager`, load models, process audio buffers incrementally, retrieve the final transcript, and reset the manager for a new utterance. This is the primary workflow for real-time transcription.

```swift
let manager = StreamingEouAsrManager(chunkSize: .ms160, eouDebounceMs: 1280)
try await manager.loadModels(modelDir: modelsURL)

// Process audio incrementally
_ = try await manager.process(audioBuffer: buffer1)
_ = try await manager.process(audioBuffer: buffer2)

// Get final transcript
let transcript = try await manager.finish()

// Reset for next utterance
await manager.reset()
```

--------------------------------

### Configure SpeakerManager

Source: https://docs.fluidinference.com/diarization/speaker-manager

Initializes the SpeakerManager with specific thresholds for speaker matching, embedding updates, and minimum speech durations. These parameters control how speakers are identified, updated, and created.

```swift
let speakerManager = SpeakerManager(
    speakerThreshold: 0.65,           // Max cosine distance for speaker match
    embeddingThreshold: 0.45,         // Max distance for embedding updates
    minSpeechDuration: 1.0,           // Min seconds to create new speaker
    minEmbeddingUpdateDuration: 2.0   // Min seconds to update embeddings
)
```

--------------------------------

### Export PyTorch Model to CoreML (.mlpackage)

Source: https://docs.fluidinference.com/mobius/converting-models

This script converts a PyTorch, NeMo, or ONNX model to the CoreML .mlpackage format. It requires fixed input shapes and uses the 'coremltools' library. The process involves loading the model, tracing it with specified input shapes, converting it, and saving the output.

```bash
cd models/stt/parakeet-tdt-v3-0.6b/coreml
uv sync
uv run python convert-parakeet.py convert \
  --nemo-path /path/to/model.nemo \
  --output-dir ./build
```

--------------------------------

### Validate CoreML Model Parity and Latency

Source: https://docs.fluidinference.com/mobius/converting-models

This script compares the numerical output and measures the latency of both the original PyTorch model and its CoreML conversion. It runs the models side-by-side with identical inputs and generates detailed reports including numerical differences, latency comparisons, and plots.

```bash
uv run python compare-components.py compare \
  --output-dir ./build \
  --model-id nvidia/parakeet-tdt-0.6b-v3 \
  --runs 10 --warmup 3
```

--------------------------------

### Configure Proxy Settings via Environment Variable

Source: https://docs.fluidinference.com/configuration

Configure proxy settings for applications running behind a corporate firewall by setting the `https_proxy` environment variable. This allows traffic to be routed through a specified proxy server. Supports both unauthenticated and authenticated proxies.

```bash
export https_proxy=http://proxy.company.com:8080

# Or for authenticated proxies:
export https_proxy=http://user:password@proxy.company.com:8080
```

--------------------------------

### Perform Diarization with FluidAudio CLI

Source: https://docs.fluidinference.com/reference/cli

Illustrates the 'fluidaudio process' command for audio diarization. It covers processing a single file, using offline mode, and incorporating ground-truth data for evaluation.

```bash
# Process a file
fluidaudio process meeting.wav --output results.json --threshold 0.6

# Offline mode
fluidaudio process meeting.wav --mode offline --threshold 0.6

# With ground-truth
fluidaudio process meeting.wav --rttm ground_truth.rttm
```

--------------------------------

### Model Registry Configuration

Source: https://docs.fluidinference.com/reference/api

Instructions on how to override the default HuggingFace URL for the Model Registry, either programmatically or via environment variables.

```APIDOC
## Model Registry

Override the default HuggingFace URL:

```swift
ModelRegistry.baseURL = "https://your-mirror.example.com"
```

Or via environment: `REGISTRY_URL=...`
```

--------------------------------

### Manual Core ML Model Loading (Swift)

Source: https://docs.fluidinference.com/diarization/getting-started

This Swift code demonstrates how to manually load Core ML models for offline deployment of the speaker diarization pipeline. It specifies the file paths for the segmentation and embedding models.

```swift
let basePath = "/opt/models/speaker-diarization-coreml"
let segmentation = URL(fileURLWithPath: basePath)
    .appendingPathComponent("pyannote_segmentation.mlmodelc")
let embedding = URL(fileURLWithPath: basePath)
    .appendingPathComponent("wespeaker_v2.mlmodelc")

let models = try await DiarizerModels.load(
    localSegmentationModel: segmentation,
    localEmbeddingModel: embedding
)
```

--------------------------------

### PocketTtsManager API

Source: https://docs.fluidinference.com/reference/api

API reference for the PocketTtsManager, including initialization, audio data synthesis, and direct file synthesis.

```APIDOC
## PocketTtsManager

| Method                              | Description                        |
| ----------------------------------- | ---------------------------------- |
| `initialize()`                      | Download and load PocketTTS models |
| `synthesize(text:)`                 | Generate audio Data                |
| `synthesizeToFile(text:outputURL:)` | Generate directly to file          |
```

--------------------------------

### Transcribe Audio with FluidAudio

Source: https://docs.fluidinference.com/quickstart

Transcribe audio files using FluidAudio's Automatic Speech Recognition (ASR) capabilities. This requires downloading and loading ASR models and initializing the AsrManager.

```swift
import FluidAudio

Task {
    let models = try await AsrModels.downloadAndLoad(version: .v3)
    let asrManager = AsrManager(config: .default)
    try await asrManager.initialize(models: models)

    let audioURL = URL(fileURLWithPath: "/path/to/audio.wav")
    let result = try await asrManager.transcribe(audioURL, source: .system)
    print("Transcription: \(result.text)")
}
```

--------------------------------

### Swift: Real-time Voice Activity Detection with Streaming

Source: https://docs.fluidinference.com/vad/streaming

Demonstrates real-time voice activity detection using FluidAudio's streaming capabilities. It initializes a VadManager, processes audio chunks from a microphone, and handles speech start/end events. This method is suitable for continuous audio streams where precise timing of speech segments is required.

```swift
import FluidAudio

let manager = try await VadManager()
var state = await manager.makeStreamState()

for chunk in microphoneChunks {
    let result = try await manager.processStreamingChunk(
        chunk,
        state: state,
        config: .default,
        returnSeconds: true,
        timeResolution: 2
    )

    state = result.state
    print(String(format: "Probability: %.3f", result.probability))

    if let event = result.event {
        switch event.kind {
        case .speechStart:
            print("Speech began at \(event.time ?? 0) s")
        case .speechEnd:
            print("Speech ended at \(event.time ?? 0) s")
        }
    }
}
```

--------------------------------

### Manually Load Core ML VAD Model (Swift)

Source: https://docs.fluidinference.com/vad/getting-started

Loads the Silero VAD Core ML model manually for offline environments. Allows specifying the model path and configuring compute units (e.g., CPU only).

```swift
let modelURL = URL(
    fileURLWithPath: "/opt/models/silero-vad-coreml/silero-vad-unified-256ms-v6.0.0.mlmodelc",
    isDirectory: true
)
var configuration = MLModelConfiguration()
configuration.computeUnits = .cpuOnly
let vadModel = try MLModel(contentsOf: modelURL, configuration: configuration)
let manager = VadManager(config: .default, vadModel: vadModel)
```

--------------------------------

### Perform Diarization with FluidAudio (Swift)

Source: https://docs.fluidinference.com/diarization/getting-started

This snippet demonstrates how to perform complete speaker diarization on an audio file using the FluidAudio library in Swift. It includes downloading necessary models, initializing the diarizer, processing audio samples, and iterating through the results to print speaker segments.

```swift
import FluidAudio

let models = try await DiarizerModels.downloadIfNeeded()
let diarizer = DiarizerManager()
diarizer.initialize(models: models)

let samples = try AudioConverter().resampleAudioFile(
    URL(fileURLWithPath: "meeting.wav")
)
let result = try diarizer.performCompleteDiarization(samples)

for segment in result.segments {
    print("Speaker \(segment.speakerId): \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}
```

--------------------------------

### FluidAudio CLI Usage

Source: https://docs.fluidinference.com/quickstart

Command-line interface commands for FluidAudio. This allows for quick execution of transcription, diarization, and text-to-speech tasks directly from the terminal.

```bash
# Transcribe
swift run fluidaudio transcribe audio.wav

# Diarize
swift run fluidaudio process meeting.wav --mode offline --threshold 0.6

# TTS
swift run fluidaudio tts "Hello from FluidAudio" --output out.wav
```

--------------------------------

### AsrModels API

Source: https://docs.fluidinference.com/reference/api

API reference for managing ASR models, including downloading, loading, and checking for existing model bundles.

```APIDOC
## AsrModels

| Method                      | Description                  |
| --------------------------- | ---------------------------- |
| `downloadAndLoad(version:)` | Download and compile models  |
| `load(from:version:)`       | Load from staged directory   |
| `modelsExist(at:)`          | Check if bundles are present |
```

--------------------------------

### Audio Format

Source: https://docs.fluidinference.com/reference/api

Details on the expected audio format and how to use the AudioConverter for normalization.

```APIDOC
## Audio Format

All pipelines expect **16 kHz mono Float32** samples. Use `AudioConverter` to normalize input:

```swift
let converter = AudioConverter()
let samples = try converter.resampleAudioFile(path: "audio.wav")
```
```

--------------------------------

### Swift API for Custom Pronunciation

Source: https://docs.fluidinference.com/asr/custom-pronunciation

This Swift code illustrates how to load a custom lexicon, initialize the TTS manager with it, synthesize speech, and update the lexicon at runtime. It utilizes the TtsCustomLexicon and TtSManager classes.

```swift
let lexicon = try TtsCustomLexicon.load(from: fileURL)

let manager = TtSManager(customLexicon: lexicon)
try await manager.initialize()
let audio = try await manager.synthesize(text: "Welcome to Kokoro TTS")

// Update at runtime
manager.setCustomLexicon(newLexicon)
```

--------------------------------

### Configure Streaming ASR Manager in Swift

Source: https://docs.fluidinference.com/asr/streaming

Shows how to configure the `StreamingEouAsrManager` with custom `chunkSize` and `eouDebounceMs` values. The `chunkSize` affects processing granularity, while `eouDebounceMs` sets the minimum silence duration before an end-of-utterance is detected.

```swift
let manager = StreamingEouAsrManager(
    chunkSize: .ms320,        // .ms160, .ms320, or .ms1600
    eouDebounceMs: 1280       // Minimum silence before EOU triggers
)
```

--------------------------------

### SSML Say-as Tag for Content Interpretation

Source: https://docs.fluidinference.com/tts/ssml

The `<say-as>` tag instructs the speech synthesizer on how to interpret and pronounce the enclosed content. The `interpret-as` attribute specifies the type of content, such as cardinal numbers, dates, or telephone numbers.

```xml
<say-as interpret-as="cardinal">123</say-as>
```

--------------------------------

### Process Chunk-Level Speech Probabilities (Swift)

Source: https://docs.fluidinference.com/vad/getting-started

Processes audio samples to obtain chunk-level speech probabilities. This provides a continuous measure of speech presence over time. Useful for fine-grained analysis or custom triggering.

```swift
let results = try await manager.process(samples)
for (index, chunk) in results.enumerated() {
    print(String(format: "Chunk %02d: prob=%.3f", index, chunk.probability))
}
```

--------------------------------

### TtSManager (Kokoro) API

Source: https://docs.fluidinference.com/reference/api

API reference for the TtSManager (Kokoro), covering model initialization and audio synthesis methods.

```APIDOC
## TtSManager (Kokoro)

| Method                      | Description                     |
| --------------------------- | ------------------------------- |
| `initialize()`              | Download and load Kokoro models |
| `synthesize(text:voice:)`   | Generate audio Data             |
| `synthesizeDetailed(text:)` | Generate with chunk metadata    |
```

--------------------------------

### Initialize and Process Audio Stream for Diarization (Swift)

Source: https://docs.fluidinference.com/diarization/streaming

Initializes the DiarizerManager and an AudioStream to process audio chunks in real-time. It binds a handler to the stream to perform diarization on each chunk and iterate through the results.

```swift
let diarizer = DiarizerManager()
diarizer.initialize(models: models)

var stream = AudioStream(
    chunkDuration: 5.0,
    chunkSkip: 2.0,
    streamStartTime: 0.0,
    chunkingStrategy: .useMostRecent
)

stream.bind { chunk, time in
    let results = try diarizer.performCompleteDiarization(chunk, atTime: time)
    for segment in results.segments {
        handleSpeakerSegment(segment)
    }
}

for audioSamples in audioStream {
    try stream.write(from: audioSamples)
}
```

--------------------------------

### Manage Speakers (Upsert, Merge, Remove)

Source: https://docs.fluidinference.com/diarization/speaker-manager

Provides methods to manage the speaker database. This includes adding or updating speakers (upsert), merging one speaker into another, removing individual speakers, removing inactive speakers based on a time threshold, and managing speaker permanence.

```swift
// Upsert
speakerManager.upsertSpeaker(speaker)

// Merge speakers
speakerManager.mergeSpeaker("1", into: "alice", mergedName: "Alice")

// Remove
speakerManager.removeSpeaker("1")

// Remove inactive
speakerManager.removeSpeakersInactive(for: 10.0)

// Permanent speakers
speakerManager.makeSpeakerPermanent("alice")
speakerManager.revokePermanence(from: "alice")
```

--------------------------------

### Swift: Process Audio File using Memory-Mapped Streaming

Source: https://docs.fluidinference.com/diarization/offline-pipeline

Shows how to process large audio files efficiently using memory-mapped streaming with the OfflineDiarizerManager. This approach is suitable for files that may not fit entirely into memory.

```swift
let url = URL(fileURLWithPath: "meeting.wav")
let result = try await manager.process(url)
```

--------------------------------

### CLI Usage for Custom Pronunciation

Source: https://docs.fluidinference.com/asr/custom-pronunciation

This bash command demonstrates how to use the FluidAudio TTS CLI to synthesize speech with a custom lexicon file. It specifies the input text, the lexicon file, and the output audio file.

```bash
swift run fluidaudio tts "The NASDAQ index rose today" \
  --lexicon custom.txt --output output.wav
```

--------------------------------

### Swift: Process Audio File with Offline Diarizer

Source: https://docs.fluidinference.com/diarization/offline-pipeline

Demonstrates how to initialize the OfflineDiarizerManager, prepare models, process an audio file (meeting.wav), and iterate through the resulting speaker segments. Requires macOS 14 / iOS 17 or later.

```swift
import FluidAudio

let config = OfflineDiarizerConfig()
let manager = OfflineDiarizerManager(config: config)
try await manager.prepareModels()

let samples = try AudioConverter().resampleAudioFile(path: "meeting.wav")
let result = try await manager.process(audio: samples)

for segment in result.segments {
    print("\(segment.speakerId) \(segment.startTimeSeconds)s - \(segment.endTimeSeconds)s")
}
```

--------------------------------

### Override Model Registry URL with Environment Variables

Source: https://docs.fluidinference.com/configuration

Set the model registry URL using environment variables for command-line interface (CLI) tools or testing. This method is recommended for non-programmatic configurations. The `REGISTRY_URL` or `MODEL_REGISTRY_URL` environment variables can be used.

```bash
export REGISTRY_URL=https://your-mirror.example.com
swift run fluidaudio transcribe audio.wav

# Or use the alias
export MODEL_REGISTRY_URL=https://models.internal.corp
```

--------------------------------

### Define VadSegmentationConfig Structure (Swift)

Source: https://docs.fluidinference.com/vad/segmentation-config

Defines the structure for Voice Activity Detection (VAD) segmentation configuration in Swift. It includes parameters to control speech and silence durations, padding, and thresholds for accurate audio segmentation.

```swift
public struct VadSegmentationConfig {
    var minSpeechDuration: TimeInterval    // Default: 0.15s
    var minSilenceDuration: TimeInterval   // Default: 0.75s
    var maxSpeechDuration: TimeInterval    // Default: 14s
    var speechPadding: TimeInterval        // Default: 0.1s
    var silenceThresholdForSplit: Float    // Default: 0.3
    var negativeThreshold: Float?          // Default: nil (auto)
    var negativeThresholdOffset: Float     // Default: 0.15
    var minSilenceAtMaxSpeech: TimeInterval // Default: 0.098s
    var useMaxPossibleSilenceAtMaxSpeech: Bool // Default: true
}
```

--------------------------------

### AsrManager API

Source: https://docs.fluidinference.com/reference/api

API reference for the AsrManager, including model initialization and transcription methods for both samples and file URLs.

```APIDOC
## AsrManager

| Method                  | Description                |
| ----------------------- | -------------------------- |
| `initialize(models:)`   | Load ASR models            |
| `transcribe(_:source:)` | Transcribe Float32 samples |
| `transcribe(_:source:)` | Transcribe from file URL   |
```

--------------------------------

### Swift Usage of SSML for TTS Synthesis

Source: https://docs.fluidinference.com/tts/ssml

This Swift code snippet demonstrates how to use the FluidAudioTTS library to synthesize speech from text containing SSML tags. It initializes the TTS manager, defines text with `<say-as>` tags for cardinal numbers and telephone numbers, and then synthesizes the audio.

```swift
import FluidAudioTTS

let ttsManager = TtSManager()
try await ttsManager.initialize()

let text = """
    The price is <say-as interpret-as="cardinal">42</say-as> dollars.
    Call us at <say-as interpret-as="telephone">555-1234</say-as>.
    """
let audio = try await ttsManager.synthesize(text: text, voice: .afHeart)
```

--------------------------------

### Override Model Registry URL (Swift)

Source: https://docs.fluidinference.com/reference/api

Allows overriding the default HuggingFace URL for the Model Registry. This is useful for using a custom mirror or local registry. The URL can also be set via the REGISTRY_URL environment variable.

```swift
ModelRegistry.baseURL = "https://your-mirror.example.com"
```

--------------------------------

### SSML Phoneme Tag for Custom Pronunciation

Source: https://docs.fluidinference.com/tts/ssml

The `<phoneme>` tag allows for custom pronunciation using phonetic alphabets like IPA. It takes an `alphabet` attribute and a `ph` attribute for the phonetic transcription. The content within the tag is the text to be pronounced.

```xml
<phoneme alphabet="ipa" ph="kəˈkɔɹo">Kokoro</phoneme>
```

--------------------------------

### Synthesize Speech with FluidAudioTTS

Source: https://docs.fluidinference.com/quickstart

Generate spoken audio from text using FluidAudio's Text-to-Speech (TTS) capabilities. This requires initializing the TtSManager and then synthesizing the desired text.

```swift
import FluidAudioTTS

Task {
    let manager = TtSManager()
    try await manager.initialize()
    let audio = try await manager.synthesize(text: "Hello from FluidAudio!")
    try audio.write(to: URL(fileURLWithPath: "/tmp/demo.wav"))
}
```

--------------------------------

### Extract Audio Clips from Speech Segments (Swift)

Source: https://docs.fluidinference.com/vad/getting-started

Extracts buffered audio clips corresponding to detected speech segments. This is useful for feeding directly into ASR systems. Requires a VadManager instance and processed audio samples.

```swift
let clips = try await manager.segmentSpeechAudio(samples, config: segmentation)
print("Extracted \(clips.count) buffered segments ready for ASR")
```

--------------------------------

### Convert Audio Buffer to 16 kHz Mono Float32 (Swift)

Source: https://docs.fluidinference.com/guides/audio-conversion

Converts an AVAudioPCMBuffer to a 16 kHz mono Float32 array using FluidAudio. This is useful for real-time processing where audio is received in chunks. The converter is stateless and can be reused.

```swift
let converter = AudioConverter()

func processChunk(_ pcmBuffer: AVAudioPCMBuffer) async throws {
    let samples = try converter.resampleBuffer(pcmBuffer)
    // Feed samples to ASR/VAD/diarization
}
```

--------------------------------

### DiarizerManager API

Source: https://docs.fluidinference.com/reference/api

API reference for the DiarizerManager, including initialization, diarization processing, and cleanup.

```APIDOC
## DiarizerManager

| Method                                      | Description                       |
| ------------------------------------------- | --------------------------------- |
| `initialize(models:)`                       | Initialize with Core ML models    |
| `performCompleteDiarization(_:sampleRate:)` | Process audio and return segments |
| `cleanup()`                                 | Release resources                 |
```

--------------------------------

### Override Model Registry URL in Swift

Source: https://docs.fluidinference.com/configuration

Programmatically override the default HuggingFace model registry URL in Swift applications. This is useful for using custom mirrors, local servers, or air-gapped environments. It requires the FluidAudio library.

```swift
import FluidAudio

ModelRegistry.baseURL = "https://your-mirror.example.com"

let diarizer = DiarizerManager()
```

--------------------------------

### Normalize Audio Samples with AudioConverter (Swift)

Source: https://docs.fluidinference.com/reference/api

Resamples audio files to the expected 16 kHz mono Float32 format using the AudioConverter class. This is a common requirement for FluidAudio pipelines. Ensure the input audio file path is correct.

```swift
let converter = AudioConverter()
let samples = try converter.resampleAudioFile(path: "audio.wav")
```

--------------------------------

### Merging Custom Lexicons in Swift

Source: https://docs.fluidinference.com/asr/custom-pronunciation

This Swift code snippet shows how to merge two custom lexicon objects into a single one. This is useful for combining different sets of custom pronunciations.

```swift
let combined = baseLexicon.merged(with: domainLexicon)
```
Tokens:
Raw