### Build TTS App Example Source: https://github.com/depasqualeorg/mlx-swift-audio/blob/main/README.md Build the Text-to-Speech (TTS) example application using this command. This requires the project path to the example. ```sh xcodebuild -project 'examples/TTS App/TTS App.xcodeproj' -scheme 'TTS App' -destination 'platform=macOS' build ``` -------------------------------- ### AudioFileWriter Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates saving raw PCM samples to WAV or CAF files using AudioFileWriter. Allows specifying a custom directory and file format. ```swift import MLXAudio func audioFileWriterExample() throws { let samples: [Float] = [0.0, 0.1, -0.1] // PCM float32 samples let sampleRate = 24000 // Save to the app's caches directory (default) let url = try AudioFileWriter.save( samples: samples, sampleRate: sampleRate, filename: "output" ) print("WAV saved to \(url.path)") // Save to a custom directory as CAF let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let cafURL = try AudioFileWriter.save( samples: samples, sampleRate: sampleRate, to: documentsURL, filename: "output", format: .caf ) print("CAF saved to \(cafURL.path)") } ``` -------------------------------- ### Swift Package Manager Installation Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Add the MLXAudio package to your project using Swift Package Manager. Link MLXAudio and optionally the Kokoro library target. ```swift // Package.swift (for Swift packages) .package(url: "https://github.com/DePasqualeOrg/mlx-swift-audio", branch: "main") // Then add to your target: .product(name: "MLXAudio", package: "mlx-swift-audio"), // GPLv3 Kokoro engine (optional): .product(name: "Kokoro", package: "mlx-swift-audio"), ``` -------------------------------- ### AudioFilePlayer Programmatic Usage Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates programmatic control of the AudioFilePlayer, including loading audio, playback, seeking, and stopping. Useful for non-SwiftUI contexts. ```swift import MLXAudio @MainActor func audioFilePlayerExample() { let player = AudioFilePlayer() player.loadAudio(from: URL(fileURLWithPath: "/audio/speech.wav")) player.play() // player.duration — total seconds // player.currentTime — current position // player.isPlaying — Observable state for SwiftUI player.seek(to: 5.0) player.togglePlayPause() player.stop() } ``` -------------------------------- ### Kokoro TTS Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to use the Kokoro TTS engine for multilingual synthesis with typed voices, speed control, and streaming. Note that Kokoro is in a separate GPLv3 library. ```swift import MLXAudio import Kokoro // separate GPLv3 library @MainActor func kokoroExample() async throws { let engine = KokoroEngine() try await engine.load() // Typed voice enum – autocomplete covers all 50+ voices try await engine.say("Hello from American English.", voice: .afHeart) try await engine.say("Bonjour!", voice: .ffSiwis) // French try await engine.say("こんにちは!", voice: .jfAlpha) // Japanese try await engine.say("你好!", voice: .zfXiaobei) // Chinese // Speed control (unique to Kokoro) let audio = try await engine.generate("Fast speech.", voice: .amEcho, speed: 1.4) await engine.play(audio) // Streaming let stream = engine.generateStreaming("Sentence one. Sentence two.", voice: .bfEmma) for try await chunk in stream { print("chunk \(chunk.samples.count) @ \(chunk.sampleRate) Hz") } } ``` -------------------------------- ### Build MLX Audio Library Source: https://github.com/depasqualeorg/mlx-swift-audio/blob/main/README.md Use this command to build the main MLX Audio library. Ensure you have Xcode installed. ```sh xcodebuild -scheme mlx-audio -destination 'platform=macOS' build ``` -------------------------------- ### Fun-ASR STT Engine Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates using the Fun-ASR engine for transcription, translation, and token-level streaming. Requires loading the engine before use. ```swift import MLXAudio @MainActor func funASRExample() async throws { // Standard transcription engine let engine = STT.funASR(modelType: .nano, quantization: .q4) try await engine.load() let audioURL = URL(fileURLWithPath: "/speech.wav") // --- Transcription (auto language detection) --- let result = try await engine.transcribe(audioURL) print(result.text) // --- With language hint --- let chinese = try await engine.transcribe(audioURL, language: .chinese) print(chinese.text) // --- Token streaming (unique to Fun-ASR) --- let stream = try await engine.transcribeStreaming(audioURL) for try await token in stream { print(token, terminator: "") } print() // --- Translation (use MLT variant for best results) --- let mltEngine = STT.funASR(modelType: .mltNano, quantization: .q4) try await mltEngine.load() let translation = try await mltEngine.translate(audioURL, targetLanguage: .english) print("Translation: \(translation.text)") // --- Translation streaming --- let translateStream = try await mltEngine.transcribeStreaming( audioURL, task: .translate, targetLanguage: .english ) for try await token in translateStream { print(token, terminator: "") } } ``` -------------------------------- ### AudioResult Handling Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to handle the AudioResult enum, which can contain either PCM samples or a file URL. Provides accessors for duration, playback speed, sample rate, and file URL. ```swift import MLXAudio @MainActor func audioResultExample() async throws { let engine = TTS.orpheus() try await engine.load() let audio = try await engine.generate("Hello world.", voice: .dan) switch audio { case let .samples(data, sampleRate, processingTime): print("samples: \(data.count), rate: \(sampleRate) Hz, time: \(processingTime)s") print("duration: \(Double(data.count) / Double(sampleRate))s") case let .file(url, processingTime): print("saved to: \(url.path), time: \(processingTime)s") } // Convenience accessors (work on both cases) print(audio.duration ?? "n/a") // seconds print(audio.realTimeFactor ?? "n/a") // < 1.0 = faster than real-time print(audio.sampleRate ?? "n/a") // Hz (nil for .file case) print(audio.fileURL ?? "n/a") // nil for .samples case // Play through the engine's internal player await engine.play(audio) } ``` -------------------------------- ### OuteTTS Engine Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Illustrates the OuteTTS engine, which uses JSON-based speaker profiles for speech synthesis. Examples include using the bundled default speaker and loading custom speaker profiles from JSON files for customized voice characteristics. Supports both immediate generation and streaming. ```swift import MLXAudio @MainActor func outeTTSExample() async throws { let engine = TTS.outetts() try await engine.load() // Use bundled default speaker try await engine.say("Using the default voice.") // Load a custom speaker profile from a JSON file let profileURL = Bundle.main.url(forResource: "my_speaker", withExtension: "json")! let speaker = try await OuteTTSSpeakerProfile.load(from: profileURL.path) let audio = try await engine.generate("Custom voice synthesis.", speaker: speaker) print("Duration: \(audio.duration ?? 0)s, RTF: \(audio.realTimeFactor ?? 0)") await engine.play(audio) // Streaming with custom speaker let resultStream = engine.generateStreaming("Streaming with custom profile.", speaker: speaker) for try await chunk in resultStream { print("Received \(chunk.samples.count) samples") } } ``` -------------------------------- ### CosyVoice 3 TTS Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates CosyVoice 3 TTS engine for cross-lingual, zero-shot, instruct mode, token-level streaming, and voice conversion. Ensure the reference audio files exist at the specified paths. ```swift import MLXAudio @MainActor func cosyVoice3Example() async throws { let engine = TTS.cosyVoice3() try await engine.load() // Cross-lingual — safe default, transcript not required let speaker = try await engine.prepareSpeaker(from: URL(fileURLWithPath: "/ref.wav")) try await engine.say("Cross-lingual synthesis.", speaker: speaker) // Zero-shot — pass the EXACT transcript of the reference clip let zeroShotSpeaker = try await engine.prepareSpeaker( from: URL(fileURLWithPath: "/ref.wav"), transcription: "Verbatim text from the reference audio clip." ) try await engine.say("Tighter voice alignment.", speaker: zeroShotSpeaker) // Instruct mode try await engine.say( "Welcome to the show.", speaker: speaker, instruction: "Read with warm enthusiasm" ) // Token-level streaming (lower latency than sentence) let tokenStream = engine.generateStreaming( "Low latency token streaming output.", speaker: speaker, granularity: .token ) for try await chunk in tokenStream { print("token chunk: \(chunk.samples.count) samples") } // Voice conversion let converted = try await engine.convertVoice( from: URL(fileURLWithPath: "/source.wav"), to: speaker ) await engine.play(converted) } ``` -------------------------------- ### Whisper STT Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the Whisper STT engine for transcription, translation, and language detection. Supports various model sizes and quantization levels. Ensure the audio file exists at the specified path. ```swift import MLXAudio @MainActor func whisperExample() async throws { // Large Turbo: best quality/speed balance (809M params) let engine = STT.whisper(model: .largeTurbo, quantization: .q4) try await engine.load { p in print("Load: \(Int(p.fractionCompleted*100))%") } let audioURL = URL(fileURLWithPath: "/speech.wav") // --- Transcription (auto language detection) --- let result = try await engine.transcribe(audioURL) print(result.text) print("Language: \(result.language), RTF: \(result.realTimeFactor)") // --- Transcription with forced language --- let spanish = try await engine.transcribe(audioURL, language: .spanish) print(spanish.text) // --- Segment timestamps --- let withSegments = try await engine.transcribe(audioURL, timestamps: .segment) for seg in withSegments.segments { print("[\(seg.start)–\(seg.end)] \(seg.text) noSpeech=\(seg.noSpeechProb)") } // --- Translation to English --- let translation = try await engine.translate(audioURL) print("English: \(translation.text)") // --- Language detection only --- let (language, confidence) = try await engine.detectLanguage(audioURL) print("\(language.displayName) (\(confidence))") // e.g. "French (0.97)" // --- From MLXArray (e.g. live microphone buffer at 16 kHz) --- let rawAudio = MLXArray([Float](repeating: 0, count: 16000)) // 1s silence example let liveResult = try await engine.transcribe(rawAudio) print(liveResult.text) } ``` -------------------------------- ### Chatterbox TTS Voice Cloning and Multi-Speaker Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates cloning voices from reference audio clips using Chatterbox TTS. Shows how to prepare reference audio once for efficient multi-speaker use and how to switch between voices. Includes streaming generation and preparation from raw audio samples. ```swift import MLXAudio @MainActor func chatterboxExample() async throws { let engine = TTS.chatterbox() // Optional: choose quantization at init time // let engine = ChatterboxEngine(quantization: .q8) try await engine.load() // Tune generation parameters engine.exaggeration = 0.3 // emotion expressiveness (0–1) engine.cfgWeight = 0.5 // classifier-free guidance engine.temperature = 0.8 engine.repetitionPenalty = 1.2 // Prepare reference audio once (expensive; reuse the result) let speakerA = try await engine.prepareReferenceAudio( from: URL(fileURLWithPath: "/path/to/alice.wav"), exaggeration: 0.3 ) let speakerB = try await engine.prepareReferenceAudio( from: URL(fileURLWithPath: "/path/to/bob.wav") ) // Switch between voices at zero cost try await engine.say("Hello, I'm Alice.", referenceAudio: speakerA) try await engine.say("And I'm Bob.", referenceAudio: speakerB) // Streaming generation let stream = engine.generateStreaming("Chunk by chunk with Alice's voice.", referenceAudio: speakerA) for try await chunk in stream { print("chunk rtf: \(chunk.processingTime)") } // Prepare from raw samples instead of a file let rawSamples: [Float] = [] // your PCM data let speakerC = try await engine.prepareReferenceAudio(fromSamples: rawSamples, sampleRate: 24000) try await engine.say("From raw samples.", referenceAudio: speakerC) } ``` -------------------------------- ### AudioFilePlayer SwiftUI View Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt An @Observable audio player suitable for SwiftUI UIs, featuring playback controls and progress tracking. Loads audio from a URL on appear. ```swift import MLXAudio import SwiftUI @MainActor struct PlaybackView: View { @State private var player = AudioFilePlayer() var body: some View { VStack { Text(player.isPlaying ? "Playing" : "Stopped") Slider(value: Binding( get: { player.currentTime }, set: { player.seek(to: $0) } ), in: 0...max(player.duration, 1)) HStack { Button("Play") { player.play() } Button("Pause") { player.pause() } Button("Stop") { player.stop() } } } .onAppear { let url = URL(fileURLWithPath: "/audio/output.wav") player.loadAudio(from: url) } } } ``` -------------------------------- ### Orpheus TTS Engine Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to use the Orpheus TTS engine for synthesizing speech with inline emotion tags and different voices. Adjust temperature and topP for synthesis quality. Audio can be played immediately or generated for export. ```swift import MLXAudio @MainActor func orpheusExample() async throws { let engine = TTS.orpheus() try await engine.load() // Adjust sampling parameters (optional) engine.temperature = 0.6 engine.topP = 0.8 // Synthesise and play immediately using a typed voice enum try await engine.say("Hello! That's hilarious.", voice: .tara) // Generate without playing (useful for export) let audio = try await engine.generate("A calm, measured reading.", voice: .leo) // audio: AudioResult.samples(data: [Float], sampleRate: 24000, processingTime: ...) print("RTF: \(audio.realTimeFactor ?? 0)") // < 1.0 = faster than real-time // Save the result to a WAV file if let url = engine.lastGeneratedAudioURL { print("Saved to \(url.path)") } // Streaming playback (sentence-by-sentence) try await engine.sayStreaming( "This sentence streams first. Then this one follows.", voice: .zoe ) // Available voices: .tara, .leah, .jess, .leo, .dan, .mia, .zac, .zoe } ``` -------------------------------- ### OuteTTS Engine Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to use the OuteTTS engine, which synthesizes speech using JSON-based speaker profiles. It includes examples for using the default speaker and loading custom profiles. ```APIDOC ## OuteTTS — `OuteTTSEngine` OuteTTS generates speech using JSON-based speaker profiles that encode reference audio characteristics. A default speaker is bundled; custom profiles are loaded from JSON files. ```swift import MLXAudio @MainActor func outeTTSExample() async throws { let engine = TTS.outetts() try await engine.load() // Use bundled default speaker try await engine.say("Using the default voice.") // Load a custom speaker profile from a JSON file let profileURL = Bundle.main.url(forResource: "my_speaker", withExtension: "json")! let speaker = try await OuteTTSSpeakerProfile.load(from: profileURL.path) let audio = try await engine.generate("Custom voice synthesis.", speaker: speaker) print("Duration: \(audio.duration ?? 0)s, RTF: \(audio.realTimeFactor ?? 0)") await engine.play(audio) // Streaming with custom speaker let resultStream = engine.generateStreaming("Streaming with custom profile.", speaker: speaker) for try await chunk in resultStream { print("Received \(chunk.samples.count) samples") } } ``` ``` -------------------------------- ### Marvis TTS Engine Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the Marvis TTS engine, optimized for low-latency conversational speech with frame-by-frame streaming. Configure model variant, quality level, and streaming interval for desired output. Supports both streaming and batch generation. ```swift import MLXAudio @MainActor func marvisExample() async throws { let engine = TTS.marvis() // Choose model variant (100M default, 250M for higher quality) engine.modelVariant = .model250m_v0_2_6bit engine.qualityLevel = .high // .low / .medium / .high / .maximum engine.streamingInterval = 0.08 // seconds between frame chunks try await engine.load() // Streaming playback (frame-by-frame — lowest latency of all engines) try await engine.sayStreaming("This plays continuously as frames arrive.", voice: .conversationalA) // Batch generation let audio = try await engine.generate("Short utterance.", voice: .conversationalB) await engine.play(audio) // Manual chunk loop let stream = engine.generateStreaming("Chunk by chunk.", voice: .conversationalA) for try await chunk in stream { print("chunk: \(chunk.samples.count) samples @ \(chunk.sampleRate) Hz") } } ``` -------------------------------- ### CosyVoice 2 TTS Inference Modes and Voice Conversion Example Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Illustrates various modes of CosyVoice 2 TTS, including cross-lingual, zero-shot, and instruct modes. Demonstrates streaming generation and voice conversion from source audio to a target speaker, including a two-step process for UI workflows. ```swift import MLXAudio @MainActor func cosyVoice2Example() async throws { let engine = TTS.cosyVoice2() try await engine.load() // --- Cross-lingual mode (default) --- // Auto-transcribes reference audio; works across languages let speaker = try await engine.prepareSpeaker(from: URL(fileURLWithPath: "/ref.wav")) try await engine.say("Hello in the cloned voice.", speaker: speaker) // --- Zero-shot mode --- // Provide exact transcript of the reference clip for tighter alignment let zeroShotSpeaker = try await engine.prepareSpeaker( from: URL(fileURLWithPath: "/ref.wav"), transcription: "The exact words spoken in the reference clip." ) try await engine.say("Better aligned output.", speaker: zeroShotSpeaker) // --- Instruct mode (style control) --- try await engine.say( "Breaking news from the capital.", speaker: speaker, instruction: "Speak in a serious news anchor tone." ) // --- Streaming --- let stream = engine.generateStreaming("First sentence. Second sentence.", speaker: speaker) for try await chunk in stream { print("\(chunk.samples.count) samples") } // --- Voice conversion --- // Convert source audio to sound like the target speaker let converted: AudioResult = try await engine.convertVoice( from: URL(fileURLWithPath: "/source_audio.wav"), to: speaker ) await engine.play(converted) // Two-step voice conversion (for UI workflows where source is loaded separately) try await engine.prepareSourceAudio(from: URL(fileURLWithPath: "/source_audio.wav")) let result = try await engine.generateVoiceConversion(speaker: speaker) await engine.play(result) } ``` -------------------------------- ### TTSEngine Protocol Demonstration Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the lifecycle, streaming granularities, playback, and resource management methods defined by the TTSEngine protocol. Models are downloaded on first use. ```swift import MLXAudio @MainActor func demonstrateTTSEngineProtocol() async { let engine: any TTSEngine = TTS.orpheus() // --- Lifecycle --- // Load with progress reporting (downloads model on first run) try? await engine.load { progress in print("Download: \(Int(progress.fractionCompleted * 100))%") } // or simply: try? await engine.load() print(engine.isLoaded) // true print(engine.isGenerating) // false while idle print(engine.isPlaying) // false while idle // --- Streaming granularities --- print(engine.supportedStreamingGranularities) // e.g. [.sentence] print(engine.defaultStreamingGranularity) // .sentence // --- Playback --- // Play a previously generated AudioResult through the engine's internal player // (each engine maintains its own AVAudioEngine / AVPlayer) // let audio = try await someEngine.generate("Hello") // await engine.play(audio) // --- Resource management --- await engine.stop() // cancel in-flight generation + playback await engine.unload() // free GPU weights; preserve expensive pre-computed data try? await engine.cleanup() // full release of all resources } ``` -------------------------------- ### STTEngine Protocol Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the lifecycle and basic usage of an STT engine conforming to the STTEngine protocol. It shows loading, checking states, and unloading the engine. ```APIDOC ## `STTEngine` Protocol All STT engines conform to `STTEngine`, providing the same lifecycle interface as `TTSEngine`. ```swift import MLXAudio @MainActor func demonstrateSTTEngineProtocol() async { let engine: any STTEngine = STT.whisper(model: .base) try? await engine.load { progress in print("Loading \(Int(progress.fractionCompleted * 100))%") } print(engine.isLoaded) // true print(engine.isTranscribing) // false while idle print(engine.transcriptionTime) // seconds taken by last transcription await engine.stop() // interrupt in-flight transcription await engine.unload() // free GPU weights try? await engine.cleanup() // full resource release } ``` ``` -------------------------------- ### Orpheus TTS Engine Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Example usage of the Orpheus TTS engine for generating expressive English speech. It covers loading, setting parameters, synthesizing speech with different voices, and streaming. ```APIDOC ## Orpheus TTS — `OrpheusEngine` Orpheus generates expressive English speech from 8 typed voices and supports inline emotion tags such as ``, ``, ``, ``, ``, ``, ``, and ``. ```swift import MLXAudio @MainActor func orpheusExample() async throws { let engine = TTS.orpheus() try await engine.load() // Adjust sampling parameters (optional) engine.temperature = 0.6 engine.topP = 0.8 // Synthesise and play immediately using a typed voice enum try await engine.say("Hello! That's hilarious.", voice: .tara) // Generate without playing (useful for export) let audio = try await engine.generate("A calm, measured reading.", voice: .leo) // audio: AudioResult.samples(data: [Float], sampleRate: 24000, processingTime: ...) print("RTF: \(audio.realTimeFactor ?? 0)") // < 1.0 = faster than real-time // Save the result to a WAV file if let url = engine.lastGeneratedAudioURL { print("Saved to \(url.path)") } // Streaming playback (sentence-by-sentence) try await engine.sayStreaming( "This sentence streams first. Then this one follows.", voice: .zoe ) // Available voices: .tara, .leah, .jess, .leo, .dan, .mia, .zac, .zoe } ``` ``` -------------------------------- ### AudioFileWriter Usage Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to use AudioFileWriter to save raw PCM samples to WAV or CAF files on disk, including saving to default cache directory or a custom location. ```APIDOC ## `AudioFileWriter` `AudioFileWriter` saves raw PCM samples to WAV or CAF files on disk. ```swift import MLXAudio func audioFileWriterExample() throws { let samples: [Float] = [0.0, 0.1, -0.1] // PCM float32 samples let sampleRate = 24000 // Save to the app's caches directory (default) let url = try AudioFileWriter.save( samples: samples, sampleRate: sampleRate, filename: "output" ) print("WAV saved to \(url.path)") // Save to a custom directory as CAF let documentsURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] let cafURL = try AudioFileWriter.save( samples: samples, sampleRate: sampleRate, to: documentsURL, filename: "output", format: .caf ) print("CAF saved to \(cafURL.path)") } ``` ``` -------------------------------- ### STTEngine Protocol Usage Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the lifecycle of an STT engine conforming to the STTEngine protocol. Ensure the engine is loaded before transcription and unloaded/cleaned up when no longer needed. ```swift import MLXAudio @MainActor func demonstrateSTTEngineProtocol() async { let engine: any STTEngine = STT.whisper(model: .base) try? await engine.load { progress in print("Loading \(Int(progress.fractionCompleted * 100))%") } print(engine.isLoaded) // true print(engine.isTranscribing) // false while idle print(engine.transcriptionTime) // seconds taken by last transcription await engine.stop() // interrupt in-flight transcription await engine.unload() // free GPU weights try? await engine.cleanup() // full resource release } ``` -------------------------------- ### Inspect and Use Streaming Granularity for TTS Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Shows how to inspect an TTS engine's supported streaming granularities and use a specific granularity (e.g., `.token`) for generating audio streams. Lower latency granularities may produce less natural chunks. ```swift import MLXAudio @MainActor func streamingGranularityExample() async throws { let engine = TTS.cosyVoice3() try await engine.load() // Inspect what this engine supports print(engine.supportedStreamingGranularities) // CosyVoice3: [.sentence, .token] print(engine.defaultStreamingGranularity) // .sentence // StreamingGranularity properties let g = StreamingGranularity.token print(g.description) // "Token-by-token" print(g.shortDescription) // "Low latency" print(g.relativeLatency) // 1 (fastest) // Pass granularity to the engine (engine-specific parameter) let speaker = try await engine.prepareSpeaker(from: URL(fileURLWithPath: "/ref.wav")) let stream = engine.generateStreaming("Fast start.", speaker: speaker, granularity: .token) for try await chunk in stream { print("token chunk: \(chunk.samples.count) @ \(chunk.sampleRate) Hz, t=\(chunk.processingTime)s") } } ``` -------------------------------- ### CosyVoice3 TTS with Zero-Shot and Cross-Lingual Guidance Source: https://github.com/depasqualeorg/mlx-swift-audio/blob/main/README.md Implement CosyVoice3 for advanced zero-shot prompt handling. Prepare speakers using cross-lingual mode by default or with explicit transcriptions for tighter semantic alignment with <=30s reference clips. ```swift // CosyVoice3 - stricter zero-shot prompt handling let cosyVoice3 = TTS.cosyVoice3() try await cosyVoice3.load() // Cross-lingual is the safe default when you only have a reference clip. let crossLingualSpeaker = try await cosyVoice3.prepareSpeaker(from: audioFileURL) try await cosyVoice3.say("Speaking with your voice.", speaker: crossLingualSpeaker) // For zero-shot, pass an explicit transcript for the exact <=30s reference clip. let zeroShotSpeaker = try await cosyVoice3.prepareSpeaker( from: audioFileURL, transcription: "Exact transcript of the clipped reference audio." ) try await cosyVoice3.say("Speaking with tighter semantic alignment.", speaker: zeroShotSpeaker) ``` -------------------------------- ### Fun-ASR STT Engine Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates how to use the FunASREngine for standard transcription, transcription with language hints, token-level streaming, and translation. ```APIDOC ## Fun-ASR STT — `FunASREngine` Fun-ASR combines a SenseVoice encoder with a Qwen3 LLM decoder for high-quality multilingual transcription and translation. It is the only STT engine that supports **token-level streaming** (`transcribeStreaming`). ```swift import MLXAudio @MainActor func funASRExample() async throws { // Standard transcription engine let engine = STT.funASR(modelType: .nano, quantization: .q4) try await engine.load() let audioURL = URL(fileURLWithPath: "/speech.wav") // --- Transcription (auto language detection) --- let result = try await engine.transcribe(audioURL) print(result.text) // --- With language hint --- let chinese = try await engine.transcribe(audioURL, language: .chinese) print(chinese.text) // --- Token streaming (unique to Fun-ASR) --- let stream = try await engine.transcribeStreaming(audioURL) for try await token in stream { print(token, terminator: "") } print() // --- Translation (use MLT variant for best results) --- let mltEngine = STT.funASR(modelType: .mltNano, quantization: .q4) try await mltEngine.load() let translation = try await mltEngine.translate(audioURL, targetLanguage: .english) print("Translation: \(translation.text)") // --- Translation streaming --- let translateStream = try await mltEngine.transcribeStreaming( audioURL, task: .translate, targetLanguage: .english ) for try await token in translateStream { print(token, terminator: "") } } ``` ``` -------------------------------- ### AudioFilePlayer for Playback Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Illustrates how to use AudioFilePlayer, an Observable class suitable for SwiftUI, to load, play, pause, stop, and seek audio files. ```APIDOC ## `AudioFilePlayer` `AudioFilePlayer` is an `@Observable` file-based audio player with progress tracking, suitable for SwiftUI playback UIs. ```swift import MLXAudio import SwiftUI @MainActor struct PlaybackView: View { @State private var player = AudioFilePlayer() var body: some View { VStack { Text(player.isPlaying ? "Playing" : "Stopped") Slider(value: Binding( get: { player.currentTime }, set: { player.seek(to: $0) } ), in: 0...max(player.duration, 1)) HStack { Button("Play") { player.play() } Button("Pause") { player.pause() } Button("Stop") { player.stop() } } } .onAppear { let url = URL(fileURLWithPath: "/audio/output.wav") player.loadAudio(from: url) } } } // Programmatic usage: @MainActor func audioFilePlayerExample() { let player = AudioFilePlayer() player.loadAudio(from: URL(fileURLWithPath: "/audio/speech.wav")) player.play() // player.duration — total seconds // player.currentTime — current position // player.isPlaying — Observable state for SwiftUI player.seek(to: 5.0) player.togglePlayPause() player.stop() } ``` ``` -------------------------------- ### AudioResult Handling Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Explains the AudioResult enum, which is the return type for audio generation calls, and demonstrates how to handle both in-memory samples and saved file URLs. ```APIDOC ## `AudioResult` `AudioResult` is the return type of all `generate()` calls. It is a value-type enum that carries either in-memory PCM samples or a saved file URL. ```swift import MLXAudio @MainActor func audioResultExample() async throws { let engine = TTS.orpheus() try await engine.load() let audio = try await engine.generate("Hello world.", voice: .dan) switch audio { case let .samples(data, sampleRate, processingTime): print("samples: \(data.count), rate: \(sampleRate) Hz, time: \(processingTime)s") print("duration: \(Double(data.count) / Double(sampleRate))s") case let .file(url, processingTime): print("saved to: \(url.path), time: \(processingTime)s") } // Convenience accessors (work on both cases) print(audio.duration ?? "n/a") // seconds print(audio.realTimeFactor ?? "n/a") // < 1.0 = faster than real-time print(audio.sampleRate ?? "n/a") // Hz (nil for .file case) print(audio.fileURL ?? "n/a") // nil for .samples case // Play through the engine's internal player await engine.play(audio) } ``` ``` -------------------------------- ### Handle TTS and STT Errors Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates how to catch and handle specific `TTSError` and `STTError` types, providing localized descriptions and recovery suggestions. Ensure models are loaded before use. ```swift import MLXAudio @MainActor func errorHandlingExample() async { let engine = TTS.chatterbox() do { // Forgetting to call load() first _ = try await engine.generate("Test") } catch let error as TTSError { switch error { case .modelNotLoaded: print(error.errorDescription!) print(error.recoverySuggestion!) case let .modelLoadFailed(underlying): print("Download failed: \(underlying)") case .insufficientMemory: print("Not enough GPU memory — try a smaller model or q4 quantization") case let .invalidArgument(msg): print("Bad input: \(msg)") case let .unsupportedStreamingGranularity(requested, supported): print("\(requested) not supported; use one of \(supported)") default: print(error.localizedDescription) } } // STTError has a parallel structure do { let sttEngine = STT.funASR() _ = try await sttEngine.transcribe(URL(fileURLWithPath: "/audio.wav")) } catch let error as STTError { print(error.localizedDescription) } } ``` -------------------------------- ### TTSEngine Protocol Usage Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt The TTSEngine protocol defines common operations for all TTS engines, including lifecycle management, state observation, and playback control. Engines conforming to this protocol are observable on the main actor. ```APIDOC ## `TTSEngine` Protocol All TTS engines conform to `TTSEngine`. The protocol defines lifecycle, state observation, and playback helpers that are common across every engine. ```swift import MLXAudio @MainActor func demonstrateTTSEngineProtocol() async { let engine: any TTSEngine = TTS.orpheus() // --- Lifecycle --- // Load with progress reporting (downloads model on first run) try? await engine.load { progress in print("Download: \(Int(progress.fractionCompleted * 100))%") } // or simply: try? await engine.load() print(engine.isLoaded) // true print(engine.isGenerating) // false while idle print(engine.isPlaying) // false while idle // --- Streaming granularities --- print(engine.supportedStreamingGranularities) // e.g. [.sentence] print(engine.defaultStreamingGranularity) // .sentence // --- Playback --- // Play a previously generated AudioResult through the engine's internal player // (each engine maintains its own AVAudioEngine / AVPlayer) // let audio = try await someEngine.generate("Hello") // await engine.play(audio) // --- Resource management --- await engine.stop() // cancel in-flight generation + playback await engine.unload() // free GPU weights; preserve expensive pre-computed data try? await engine.cleanup() // full release of all resources } ``` ``` -------------------------------- ### Marvis TTS Engine Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Demonstrates the Marvis TTS engine, a conversational engine with low latency due to frame-by-frame streaming. It allows adjustment of model variants and quality levels. ```APIDOC ## Marvis TTS — `MarvisEngine` Marvis is a conversational TTS engine built on the CSM architecture. It streams audio frame-by-frame (the only engine with `.frame` granularity), providing very low latency first-audio onset. Quality is adjustable via codebook count. ```swift import MLXAudio @MainActor func marvisExample() async throws { let engine = TTS.marvis() // Choose model variant (100M default, 250M for higher quality) engine.modelVariant = .model250m_v0_2_6bit engine.qualityLevel = .high // .low / .medium / .high / .maximum engine.streamingInterval = 0.08 // seconds between frame chunks try await engine.load() // Streaming playback (frame-by-frame — lowest latency of all engines) try await engine.sayStreaming("This plays continuously as frames arrive.", voice: .conversationalA) // Batch generation let audio = try await engine.generate("Short utterance.", voice: .conversationalB) await engine.play(audio) // Manual chunk loop let stream = engine.generateStreaming("Chunk by chunk.", voice: .conversationalA) for try await chunk in stream { print("chunk: \(chunk.samples.count) samples @ \(chunk.sampleRate) Hz") } } ``` ``` -------------------------------- ### TTSError and STTError Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Illustrates how to handle specific errors thrown by Text-to-Speech (TTS) and Speech-to-Text (STT) operations in the MLX Audio library. ```APIDOC ## `errorHandlingExample()` ### Description This function demonstrates how to catch and handle `TTSError` and `STTError` exceptions. It shows how to inspect error codes, access localized descriptions, failure reasons, and recovery suggestions. ### Method `errorHandlingExample()` ### Parameters None ### Request Example ```swift import MLXAudio @MainActor func errorHandlingExample() async { let engine = TTS.chatterbox() do { // Forgetting to call load() first _ = try await engine.generate("Test") } catch let error as TTSError { switch error { case .modelNotLoaded: print(error.errorDescription!) // "Model not loaded. Call load() first." print(error.recoverySuggestion!) // "Call the load() method before…" case let .modelLoadFailed(underlying): print("Download failed: \(underlying)") case .insufficientMemory: print("Not enough GPU memory — try a smaller model or q4 quantization") case let .invalidArgument(msg): print("Bad input: \(msg)") case let .unsupportedStreamingGranularity(requested, supported): print("\(requested) not supported; use one of \(supported)") default: print(error.localizedDescription) } } // STTError has a parallel structure do { let sttEngine = STT.funASR() _ = try await sttEngine.transcribe(URL(fileURLWithPath: "/audio.wav")) } catch let error as STTError { print(error.localizedDescription) } } ``` ### Response Prints detailed error messages and recovery suggestions to the console based on the type of `TTSError` or `STTError` caught. ``` -------------------------------- ### Transcribe Audio with Word Timestamps Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Transcribes an audio file and retrieves full text, language, duration, and segment-level details including word-level timestamps when `.word` granularity is enabled. Requires the audio file to be accessible. ```swift import MLXAudio @MainActor func transcriptionResultExample() async throws { let engine = STT.whisper(model: .base) try await engine.load() let result = try await engine.transcribe( URL(fileURLWithPath: "/interview.wav"), timestamps: .word // enables Word-level data ) print(result.text) // full transcript print("Language: \(result.language)") // ISO 639-1 code, e.g. "en" print("Duration: \(result.duration)s") print("RTF: \(result.realTimeFactor)") // < 1.0 = faster than real-time print("Processing: \(result.processingTime)s") // Segment-level detail for seg in result.segments { print("[\(seg.start)–\(seg.end)] \(seg.text)") print(" avgLogProb=\(seg.avgLogProb) noSpeechProb=\(seg.noSpeechProb)") // Word-level timestamps (when timestamps: .word) if let words = seg.words { for w in words { print(" '\(w.word)' [\(w.start)–\(w.end)] p=\(w.probability)") } } } } ``` -------------------------------- ### STT Factory Usage Source: https://context7.com/depasqualeorg/mlx-swift-audio/llms.txt Instantiate STT engines using the STT factory enum. Supports Whisper with various model sizes and quantization, and Fun-ASR with different model types. ```swift import MLXAudio // Whisper – choose model size and quantization let whisperBase: WhisperEngine = STT.whisper(model: .base) // 4-bit, 74M params let whisperLargeTurbo: WhisperEngine = STT.whisper(model: .largeTurbo, quantization: .q8) let whisperLargeFull: WhisperEngine = STT.whisper(model: .large, quantization: .fp16) // Fun-ASR – LLM-based (SenseVoice + Qwen3) let funASRDefault: FunASREngine = STT.funASR() // nano, 4-bit let funASRMlt: FunASREngine = STT.funASR(modelType: .mltNano, quantization: .q4) // MLT for translation ``` -------------------------------- ### CosyVoice2 TTS with Zero-Shot and Cross-Lingual Modes Source: https://github.com/depasqualeorg/mlx-swift-audio/blob/main/README.md Utilize CosyVoice2 for voice matching. Load the model, prepare a speaker profile from audio, and generate speech. Supports custom instructions for speaking style and voice conversion. ```swift import MLXAudio // CosyVoice2 - voice matching with zero-shot and cross-lingual modes let cosyVoice = TTS.cosyVoice2() try await cosyVoice.load() let speaker = try await cosyVoice.prepareSpeaker(from: audioFileURL) try await cosyVoice.say("Speaking with your voice.", speaker: speaker) // With style instructions try await cosyVoice.say("This is exciting news!", speaker: speaker, instruction: "Speak with enthusiasm") // Voice conversion - transform audio to sound like the speaker let converted = try await cosyVoice.convertVoice(from: sourceAudioURL, to: speaker) ``` -------------------------------- ### OuteTTS TTS with Custom Voices from Reference Audio Source: https://github.com/depasqualeorg/mlx-swift-audio/blob/main/README.md Integrate OuteTTS to create custom voices using reference audio. Load the model and speaker profiles from JSON files to generate speech. ```swift // OuteTTS - custom voices from reference audio let outetts = TTS.outetts() try await outetts.load() let speaker = try await OuteTTSSpeakerProfile.load(from: "speaker.json") try await outetts.say("Using reference audio.", speaker: speaker) ```