TheStage AI

https://docs.thestage.ai/
Admin
TheStage AI provides an inference acceleration and optimization platform that enables developers to...
Tokens:321,704
Snippets:1,999
Trust Score:6.7
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install Modal and setup

Source: https://docs.thestage.ai/tutorials/source/modal_thestage.html

Install the Modal CLI and authenticate with your account.

```bash
pip install modal
modal setup

```

--------------------------------

### Clone and run the AppleSDK example

Source: https://docs.thestage.ai/tutorials/source/apple_sdk_on_device_small_llms.html

Clone the AppleSDK repository, navigate to the tutor_small_llms example, set the TS_API_TOKEN environment variable, and run the Swift example in release mode. The PACK variable selects the model (lfm230, lfm350, qwen06, gemma1b) and GUIDE selects a subset of guides 1-5. The first run downloads the pack; later runs start from cache.

```bash
git clone https://github.com/TheStageAI/AppleSDK.git
cd AppleSDK/examples/tutor_small_llms
export TS_API_TOKEN=th_…

PACK=qwen06 GUIDE=1,2,3,4,5 swift run -c release
# PACK: lfm230 | lfm350 | qwen06 | gemma1b   GUIDE: any subset of 1..5

```

--------------------------------

### Run a Flutter example: setup, secrets, and build

Source: https://docs.thestage.ai/android_sdk/docs/source/get_started.html

Commands to set up the host environment, copy API key secrets, and build/run a Flutter example on a connected Snapdragon device. Run ./scripts/setup.sh once, then edit the example's secrets.json, and finally use flutter run with --dart-define-from-file.

```bash
# 1. One-time host setup: symlink the AARs into the plugin and
#    bootstrap each example's secrets.json. Idempotent.
./scripts/setup.sh

# 2. Drop your API keys into the example you want to run.
cp examples/tts_front_stream/secrets.example.json \
   examples/tts_front_stream/secrets.json
$EDITOR examples/tts_front_stream/secrets.json

# 3. Build & run on a connected Snapdragon device.
cd examples/tts_front_stream
flutter pub get
flutter run --release \
    --dart-define-from-file=secrets.json \
    -d <YOUR_DEVICE_ID>
```

--------------------------------

### Fully offline setup in Swift

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.voice_agent.html

Initializes the SDK, loads a local language model, and starts a voice agent with local LLM provider. Subscribe to transcripts and LLM deltas before calling start().

```Swift
import TheStageSDK

try await TheStageAI.shared.initialize(api_token: "your-api-token")

// 1. Load the language model under a handle
try await TheStageAI.shared.start_model(
    model_name: "llm",
    engines_path: "TheStageAI/Qwen3-0.6B"
)

// 2. Point the agent at that handle
let llm = TSLocalLLMProvider(
    model_path: "llm",
    // weather, time, search — or []
    tools: DefaultTools.live,
    system_prompt: DefaultTools.voice_system_prompt
)
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)
config.tts_voice = "dave"

// 3. Subscribe, then start
let agent = TSVoiceAgent(config: config)
Task {
    for await text in agent.transcripts.recv() { chat.addUser(text) }
}
Task {
    for await delta in agent.llm_deltas.recv() { chat.appendAssistant(delta) }
}

// listening from here
try await agent.start()

```

--------------------------------

### Swift Quick Start: Configure and Start TSVoiceAgent

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.audio_nodes.html

Swift example showing how to create a TSVoiceAgent with VAD, STT, TTS, and LLM, set audio sample rate and echo cancellation method, then start the agent. Note that audio changes apply on the next start() call.

```swift
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)

// match your AVAudioSession
config.audio.sample_rate_out = 48_000
// iOS default; .NONE on Mac speakers
config.audio.aec_method = .VPIO

let agent = TSVoiceAgent(config: config)
try await agent.start()

```

--------------------------------

### Flutter Quick Start: Configure and Start TSVoiceAgent

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.audio_nodes.html

Flutter example showing how to create a TSVoiceAgent and start it with a configuration map including VAD, STT, TTS, LLM, sample rate, and echo cancellation method. Note that audio changes apply on the next start() call.

```dart
final agent = TSVoiceAgent();
await agent.start(config: {
  'vad': 'TheStageAI/silero-vad',
  'stt': 'TheStageAI/thewhisper-large-v3-turbo',
  'tts': 'TheStageAI/neutts-nano-multilingual',
  'llm_provider': 'local', 'llm_model': 'llm',

  // match your audio session
  'sample_rate_out': 48000,
  // 'vpio' | 'neural' | 'none'
  'aec_method': 'vpio',
});

```

--------------------------------

### Fully offline setup in Flutter

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.voice_agent.html

Initializes the Flutter SDK, loads a local language model, and starts a voice agent with local LLM provider. Subscribes to transcripts and LLM deltas before starting.

```Flutter
import 'package:thestage_apple_sdk/thestage_apple_sdk.dart';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');

// 1. Load the language model under a handle
await TheStageFlutterSDK.start_model(
  model_name: 'llm',
  engines_path: 'TheStageAI/Qwen3-0.6B',
);

// 2 + 3. Subscribe, then start with the same handle
final agent = TSVoiceAgent();
agent.transcripts.listen(chat.addUser);
agent.llm_deltas.listen(chat.appendAssistant);

await agent.start(config: {
  'vad': 'TheStageAI/silero-vad',
  'stt': 'TheStageAI/thewhisper-large-v3-turbo',
  'tts': 'TheStageAI/neutts-nano-multilingual',
  'tts_voice': 'dave',
  'llm_provider': 'local',
  // the start_model handle
  'llm_model': 'llm',
  // none | voice | web | phone
  'llm_tools': 'voice',
// listening from here
});

```

--------------------------------

### Flutter: Complete first-run example with progress subscription

Source: https://docs.thestage.ai/android_sdk/docs/source/get_started.html

Shows a full first-run flow: initialize the SDK, subscribe to progress events for a specific model, start the model, and run inference. Subscribe before start_model to capture download/extract/load phases.

```dart
import 'package:thestage_android_sdk/thestage_android_sdk.dart';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');

// Subscribe to load progress for any model_name.
TheStageFlutterSDK.on_progress.listen((event) {
  if (event['model_name'] != 'stt') return;
  print('[stt] ${event['phase']} ${(event['progress'] as double) * 100}%');
});

await TheStageFlutterSDK.start_model(
  model_name: 'stt',
  engines_path: 'TheStageAI/thewhisper-large-v3-turbo',
);

final result = await TheStageFlutterSDK.infer(
  model_name: 'stt',
  input_json: {
    'audio': pcm_16k_mono,   // Float32List, mono, [-1, 1]
    'language': 'en',
  },
);
print(result[0]['transcription']);
```

--------------------------------

### Test setup by printing available models

Source: https://docs.thestage.ai/elastic_models/docs/source/index.html

Run this Python script to verify your installation. It imports elastic_models and prints a table of available models and their supported sizes on various GPUs.

```python
import elastic_models

elastic_models.print_available_models()

```

--------------------------------

### Install OpenAI client

Source: https://docs.thestage.ai/tutorials/source/modal_thestage.html

Install the OpenAI Python client.

```bash
pip install openai

```

--------------------------------

### Flutter: Configure player sample rate and stream overlap for correct playback

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.tts.html

Starts the TTS model and configures the audio player to use the sample_rate from the result to avoid pitch/speed issues. Uses TTSStreamConfig(overlap_frames: 3) to smooth chunk seams. Includes commented-out hard-coded 44100 example and streaming setup.

```Flutter
await TheStageFlutterSDK.start_model(
  model_name: 'tts',
  engines_path: 'TheStageAI/neutts-nano-multilingual',
  config: {'voice_id': 'dave', 'language': 'english'},
);

// ✅ player follows the result
final rate = rows[0]['sample_rate'] as int;
final player = TSAudioPlayer(sampleRate: rate);

// ❌ TSAudioPlayer(sampleRate: 44100) for a 24 kHz clip → slow and deep

// Clicks at seams while streaming:
final speech = await TTSStream.open(
  model_name: 'tts',
  stream_config: const TTSStreamConfig(overlap_frames: 3),
);

```

--------------------------------

### Configure Modal image with startup script

Source: https://docs.thestage.ai/tutorials/source/modal_thestage.html

Build the Modal image from the registry, add local files, and set entrypoint.

```python
image = modal.Image.from_registry(
    IMG,
    add_python="3.11"
)\
    .env(ENVS)\
    .add_local_file("modal_start.sh", "/usr/local/bin/startup.sh", copy=True)\
    .add_local_file("supervisord.conf", "/etc/supervisor/supervisord.conf", copy=True)\
    .run_commands("chmod +x /usr/local/bin/startup.sh")\
    .entrypoint(["/usr/local/bin/startup.sh"])

```

--------------------------------

### Start session log in Swift

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.logging.html

Starts a session log to a temporary file with debug level and console tee. Call end_session_log() to close the file and get its URL.

```swift
import TheStageSDK

let llm = try await TSLLM(engines_path: "TheStageAI/Qwen3-0.6B")
let session = LLMChatEngine(
    llm: llm).chat_session(system_prompt: "You are a helpful assistant.",
    memory: .SLIDING(max_turns: 10)
)

let url = FileManager.default.temporaryDirectory
    .appendingPathComponent("thestage-session.log")
try TheStageAI.start_session_log(to: url, level: .debug, tee_console: true)

// ... run the app ...

// attach this file to a ticket
let path = TheStageAI.end_session_log()

```

--------------------------------

### Basic LSQ usage: setup, calibration, and training

Source: https://docs.thestage.ai/qlip/docs/source/qlip_algorithms_api.html

Shows the full LSQ workflow: setup with 4-bit weights and 8-bit activations, calibration on representative data, optimizer setup, and a standard training loop. Requires quantization-aware training with access to training data.

```python
from qlip_algorithms.quantization import LSQ
import torch
import torch.optim as optim

# Setup LSQ with 4-bit weights and 8-bit activations
handle, quantized_model, param_groups = LSQ.setup_model(
    model=model,
    weights_scheme={'scheme': 'int', 'bits': 4, 'symmetric': True},
    activations_scheme={'scheme': 'uint', 'bits': 8, 'symmetric': False},
    calibration_iterations=32
)
# Run calibration on representative data to estimate initial quantization parameters
for batch in calibration_dataloader:
    quantized_model(batch)

# Setup optimizer with quantization parameter groups
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# Training loop with LSQ
for epoch in range(num_epochs):
    for batch in train_dataloader:
        optimizer.zero_grad()
        output = quantized_model(batch['input'])
        loss = criterion(output, batch['target'])
        loss.backward()
        optimizer.step()

```

--------------------------------

### Run the example app from AppleSDK checkout

Source: https://docs.thestage.ai/tutorials/source/apple_sdk_tts_language_switching.html

Commands to set up and run the TutorTTS example app from an AppleSDK checkout (tag 1.3.0+). Requires setting TS_API_TOKEN from https://app.thestage.ai. Simulator is not supported; use a physical iPhone.

```bash
cd examples/tutor_tts
cp Secrets.xcconfig.example Secrets.xcconfig
# set TS_API_TOKEN=… from https://app.thestage.ai

xcodegen generate
open TutorTTS.xcodeproj
# Xcode → Team → Run (Release) on a physical iPhone

```

--------------------------------

### Segment a recording (Flutter)

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.vad.html

Starts the VAD model and calls infer with 'extract_segments': true to get speech regions. Prints each segment's start and end times in seconds.

```Flutter
await TheStageFlutterSDK.start_model(
  model_name: 'vad',
  engines_path: 'TheStageAI/silero-vad',
);

final rows = await TheStageFlutterSDK.infer(
  model_name: 'vad',
  input_json: {
    // Float32List, 16 kHz mono
    'audio': recording,
    'extract_segments': true,
  },
);
for (final r in rows) {
  // sample indices
  final start = r['start'] as int, end = r['end'] as int;
  print('${start / 16000}s – ${end / 16000}s');
}

```

--------------------------------

### PostTrainingQuantization.setup_modules

Source: https://docs.thestage.ai/qlip/docs/source/qlip_algorithms_api.html

Static method to set up post-training quantization for a specific list of modules with fine-grained control.

```APIDOC
## PostTrainingQuantization.setup_modules

### Description
Sets up post-training quantization for a specific list of modules with fine-grained control.

### Method
Static method

### Signature
`PostTrainingQuantization.setup_modules(modules, weights_scheme=QuantScheme('int', 8, True), weights_granularity='per-channel', activations_scheme=QuantScheme('int', 8, False), activations_granularity='per-batch', placement='input', quantization_mode='static', observer=None, observer_kwargs={}, calibration_iterations=np.inf, train_weights_offset=False, train_activations_offset=False, weights_scale_offset_dtype=torch.float32, activations_scale_offset_dtype=torch.float32)`

### Parameters
- **modules** (list) - Required - List of torch.nn.Module instances to quantize
- **weights_scheme** (Union[QuantScheme, Dict[str, Any]]) - Optional - Quantization scheme for weights (data type, bit-width, symmetric/asymmetric)
- **weights_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Quantization granularity for weights ("per-tensor", "per-channel", "per-token")
- **activations_scheme** (Union[QuantScheme, Dict[str, Any]]) - Optional - Quantization scheme for activations
- **activations_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Quantization granularity for activations
- **placement** (Union[str, QuantPlacement]) - Optional - Determines where quantizers are placed "input" quantizes input activations of modules, "output" quantizes output activations, "attention" sets quantizers for quantized attention kernels
- **quantization_mode** (str) - Optional - Quantization mode ("static" for calibration-based, "dynamic" for runtime)
- **observer** (Observer) - Optional - Observer class for collecting activation statistics (defaults to BoundsObserver)
- **observer_kwargs** (dict) - Optional - Additional parameters for the observer (e.g., percentile settings)
- **calibration_iterations** (int) - Optional - Maximum number of calibration batches for collecting statistics
- **train_weights_offset** (bool) - Optional - Whether weight zero-points are trainable
- **train_activations_offset** (bool) - Optional - Whether activation zero-points are trainable
- **weights_scale_offset_dtype** (torch.dtype) - Optional - Data type for weight quantization parameters
- **activations_scale_offset_dtype** (torch.dtype) - Optional - Data type for activation quantization parameters

### Returns
- **QuantizationManager** - Quantization manager used to quantize the modules
```

--------------------------------

### PostTrainingQuantization.setup_model

Source: https://docs.thestage.ai/qlip/docs/source/qlip_algorithms_api.html

Static method to set up post-training quantization for an entire model.

```APIDOC
## PostTrainingQuantization.setup_model

### Description
Sets up post-training quantization for an entire model by automatically identifying and quantizing specified module types.

### Method
Static method

### Signature
`PostTrainingQuantization.setup_model(model, weights_scheme=QuantScheme('int', 8, True), weights_granularity='per-channel', activations_scheme=QuantScheme('int', 8, True), activations_granularity='per-batch', modules_types=(torch.nn.Conv2d, torch.nn.Linear), except_layers=[], placement='input', attention_cls=None, quantization_mode='static', observer=None, observer_kwargs={}, calibration_iterations=np.inf, train_weights_offset=False, train_activations_offset=False, inplace=True, weights_scale_offset_dtype=torch.float32, activations_scale_offset_dtype=torch.float32)`

### Parameters
- **model** (torch.nn.Module) - Required - PyTorch model to quantize
- **weights_scheme** (Union[QuantScheme, Dict[str, Any]]) - Optional - Quantization scheme for weights (data type, bit-width, symmetric/asymmetric)
- **weights_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Quantization granularity for weights ("per-tensor", "per-channel", "per-token")
- **activations_scheme** (Union[QuantScheme, Dict[str, Any]]) - Optional - Quantization scheme for activations
- **activations_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Quantization granularity for activations
- **modules_types** (tuple) - Optional - Types of modules to quantize automatically
- **except_layers** (list) - Optional - List of layer names to exclude from quantization
- **placement** (Union[str, QuantPlacement]) - Optional - Determines where quantizers are placed "input" quantizes input activations of modules, "output" quantizes output activations, "attention" sets quantizers for quantized attention kernels
- **attention_cls** (str) - Optional - Name of attention module class for specialized quantization (e.g., 'MultiheadAttention')
- **quantization_mode** (str) - Optional - Quantization mode ("static" for calibration-based, "dynamic" for runtime)
- **observer** (Observer) - Optional - Observer class for collecting activation statistics (defaults to BoundsObserver)
- **observer_kwargs** (dict) - Optional - Additional parameters for the observer (e.g., percentile settings)
- **calibration_iterations** (int) - Optional - Maximum number of calibration batches for collecting statistics
- **train_weights_offset** (bool) - Optional - Whether weight zero-points are trainable
- **train_activations_offset** (bool) - Optional - Whether activation zero-points are trainable
- **inplace** (bool) - Optional - Whether to modify the original model or create a copy
- **weights_scale_offset_dtype** (torch.dtype) - Optional - Data type for weight quantization parameters
- **activations_scale_offset_dtype** (torch.dtype) - Optional - Data type for activation quantization parameters

### Returns
- **Tuple[torch.nn.Module, QuantizationManager, Dict[str, List]]** - Tuple containing the quantized model, quantization manager, and parameter groups for optimizers
```

--------------------------------

### Install thestage and set access token

Source: https://docs.thestage.ai/elastic_models/docs/source/index.html

Install the thestage package and configure your access token. Replace <YOUR_ACCESS_TOKEN> with your actual token.

```bash
pip install thestage
thestage config set --access-token <YOUR_ACCESS_TOKEN>

```

--------------------------------

### Create rented instance with thestage CLI

Source: https://docs.thestage.ai/platform/src/thestage-ai-cli.html

Rents a new GPU instance. Use --force-yes to skip the confirmation prompt. Options: --name/-n, --provider/-p (Amazon, Nebius), --gpu/-g (A100, A10G, H100, L40S, T4), --num-gpu/-ng, --disk-size/-ds.

```bash
thestage instance rented create \
    --name my-gpu-server \
    --provider Nebius \
    --gpu H100 \
    --num-gpu 1 \
    --disk-size 200

# Skip confirmation prompt
thestage instance rented create -n my-server -p Amazon -g A100 -ng 1 -ds 100 --force-yes

```

--------------------------------

### Start TTS model with load progress handler (Kotlin)

Source: https://docs.thestage.ai/android_sdk/docs/source/android_sdk.tts.html

Kotlin example showing how to start the TTS model with an on_load_progress handler that reports progress through four phases (downloading, extracting, loading, ready) with a monotonic fraction in 0.0..1.0.

```kotlin
TheStageAI.start_model(
    model_name = "tts",
    engines_path = "TheStageAI/neutts-multilingual",
    config = mapOf("voice_id" to "paul"),
    on_load_progress = { p ->
        println("[${p.model}] ${p.phase} ${(p.fraction * 100).toInt()}%")
    }
)

```

--------------------------------

### LLM.infer_stream(prompt:config:)

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.streaming.html

Starts an LLM inference stream that yields tokens as they are generated. Use this to get streaming text output from the LLM.

```APIDOC
## LLM.infer_stream(prompt:config:)

### Description
Starts an LLM inference stream that yields tokens as they are generated. Use this to get streaming text output from the LLM.

### Method
infer_stream

### Endpoint
LLM.infer_stream(prompt:config:)

### Parameters
#### Parameters
- **prompt** (String) - Required - The input prompt for the LLM.
- **config** (GenerationConfig) - Required - Configuration for generation, e.g., enable_thinking.

### Returns
AsyncStream<LLMStreamChunk> - A stream of LLM stream chunks.

### Example
```swift
for await token in llm.infer_stream(prompt: question, config: config) {
    if !token.is_final { speech.send(token.text) }
}
```
```

--------------------------------

### Start TTS model with load progress listener (Flutter)

Source: https://docs.thestage.ai/android_sdk/docs/source/android_sdk.tts.html

Flutter example showing how to listen to load progress events and start the TTS model. The on_progress stream provides phase and progress values; the start_model call includes voice_id and language configuration.

```dart
TheStageFlutterSDK.on_progress.listen((event) {
  if (event['model_name'] != 'tts') return;
  final phase    = event['phase']    as String?;   // downloading | extracting | loading | ready
  final fraction = event['progress'] as double?;   // 0.0 ... 1.0, monotonic
  print('[tts] $phase ${(fraction ?? 0) * 100}%');
});

await TheStageFlutterSDK.start_model(
  model_name: 'tts',
  engines_path: 'TheStageAI/neutts-multilingual',
  config: {'voice_id': 'paul', 'language': 'english'},
);

```

--------------------------------

### Prepare Qwen3 voice pack commands

Source: https://docs.thestage.ai/tutorials/source/apple_sdk_tts_language_switching.html

Commands to set up a virtual environment, install Qwen3 requirements, and run prepare_qwen3_voice_pack.py to create a voice pack. Note: uses PyPI and Hugging Face only, not TheStage Models.

```bash
cd examples/tools/prepare_voice_packs
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-qwen3.txt   # Qwen3 packs

# One pack per language for the tutor loop
python prepare_qwen3_voice_pack.py \
  --ref-audio ./tutor_en.wav \
  --ref-text "We still have time to practice clear speech." \
  --language english \
  --name tutor_en \
  --out-dir ./VoicePacks/tutor_en
```

--------------------------------

### Complete example file tree

Source: https://docs.thestage.ai/android_sdk/docs/source/android_sdk.ai_packs.html

Shows the project structure for the complete example with a single on-demand model (whisper). The asset files are named with the pattern thestageai_models_whisper#group_<soc>/engines.zip for each device group.

```text
my-app/
├── settings.gradle.kts
├── android/                         # (Flutter app's android/)
│   ├── gradle.properties
│   ├── app/
│   │   ├── build.gradle.kts
│   │   ├── device_targeting_config.xml
│   │   └── src/main/kotlin/.../MainActivity.kt
│   └── thestageai_models_whisper/
│       ├── build.gradle.kts
│       └── src/main/assets/
│           ├── thestageai_models_whisper#group_sm8550/engines.zip
│           ├── thestageai_models_whisper#group_sm8650/engines.zip
│           ├── thestageai_models_whisper#group_sm8750/engines.zip
│           ├── thestageai_models_whisper#group_sm8850/engines.zip
│           └── thestageai_models_whisper#group_other/engines.zip
└── lib/main.dart

```

--------------------------------

### Swift: Start model, enrol, and verify speaker embedding

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.speaker_embedding.html

Starts the speaker_id model with TheStageAI/redimnet2, loads a 2-second 16 kHz WAV, enrols to get a 192-d embedding, then verifies a new clip and returns cosine similarity. Accept if similarity > 0.75.

```swift
try await TheStageAI.shared.start_model(
    model_name: "speaker_id",
    engines_path: "TheStageAI/redimnet2"
)

// 2 s of the user's voice
let pcm16k = try AudioIO.load_wav(
    path: enrolClipURL.path,
    target_sample_rate: 16_000
)
// Enrol: 2 s of the user's voice → 192-d vector
let enrolled = try TheStageAI.shared.infer(
    model_name: "speaker_id",
    input_json: ["audio": pcm16k]
)
let reference = enrolled[0]["embedding"] as! [Double]

let newPcm16k = try AudioIO.load_wav(
    path: checkClipURL.path,
    target_sample_rate: 16_000
)
// Verify: new audio + reference → similarity
let check = try TheStageAI.shared.infer(
    model_name: "speaker_id",
    input_json: ["audio": newPcm16k, "embedding": reference]
)
// accept if > 0.75
let similarity = check[0]["similarity"] as! Double

```

--------------------------------

### Cleanup VAD model in Flutter

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.vad.html

Starts and stops the VAD model using TheStageFlutterSDK. The start_model call is included to show the required setup before stopping.

```dart
await TheStageFlutterSDK.start_model(
  model_name: 'vad',
  engines_path: 'TheStageAI/silero-vad',
);

await TheStageFlutterSDK.stop_model(model_name: 'vad');

```

--------------------------------

### Live captions with ASREngine (Swift)

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.asr.html

Sets up live captions where the SDK owns the microphone. Configures VAD and STT models, subscribes to turns, and starts the engine. Note: subscribe before start() to avoid missing events.

```swift
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo"
)
config.asr_generation = ASRGenerationConfig(
    language: "en",
    timestamps: .WORD
)

let engine = ASREngine(config: config)
let captions = Task {
    for await turn in engine.turns.recv() {
        committedLabel.text = turn.committed
        hypothesisLabel.text =
            turn.end_of_turn ? "" : turn.hypothesis
    }
}

// the SDK now owns the microphone
try await engine.start()
// ... later
await engine.stop()
await captions.value

```

--------------------------------

### Flutter: Configure TSVoiceAgent start map

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.voice_agent.html

Flutter configuration example for TSVoiceAgent.start(). Uses a map with keys for VAD, STT, TTS, LLM, reply, voice, turn taking, barge-in, audio, and auto-listen. Changes marked 'live' can be applied at runtime; others require stop() then start().

```dart
// agent as in Quick start; agentConfig is its model map
final agent = TSVoiceAgent();

await agent.start(config: {
  'vad': vadRepo, 'stt': sttRepo, 'tts': ttsRepo,
  'llm_provider': 'local', 'llm_model': 'llm',

  // Reply
  'system_prompt': "You are Acme's assistant. Be brief.",
  'max_tokens': 256,
  'chat_memory_max_turns': 10,

  // Voice
  // or tts_voice_dir / tts_language
  'tts_voice': 'dave',

  // Turn taking
  'silence_timeout_ms': 608,
  'turn_end_mode': 'dnn',
  'turn_detector': 'TheStageAI/smart-turn-v3',

  // Barge-in
  'interrupt_mode': 'vad',
  'interrupt_min_speech_ms': 600,

  // Audio
  'sample_rate_out': 48000,
  'aec_method': 'vpio',

  'auto_listen': false,
});

```

--------------------------------

### Expected output: number of quantized modules

Source: https://docs.thestage.ai/tutorials/source/quantization_tutorial.html

Shows the expected output from the setup_model call, indicating 225 quantized modules.

```text
Number of quantized modules: 225
```

--------------------------------

### Flutter: Start model, enrol, and verify speaker embedding

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.speaker_embedding.html

Starts the speaker_id model with TheStageAI/redimnet2, reads a headerless float32 PCM file (16 kHz mono, 2 s), enrols to get a 192-d embedding, then verifies a new clip and returns cosine similarity. Accept if similarity > 0.75.

```dart
await TheStageFlutterSDK.start_model(
  model_name: 'speaker_id',
  engines_path: 'TheStageAI/redimnet2',
);

// Enrol — pcm16k: Float32List, 16 kHz mono, 2 s (here a headerless float32 file)
final enrolBytes = await File('/path/to/enrol_16k.pcm').readAsBytes();
final pcm16k = Float32List.view(enrolBytes.buffer);
final enrolled = await TheStageFlutterSDK.infer(
  model_name: 'speaker_id',
  input_json: {'audio': pcm16k},
);
final reference = (enrolled[0]['embedding'] as List).cast<double>();

// Verify
final checkBytes = await File('/path/to/check_16k.pcm').readAsBytes();
final newPcm16k = Float32List.view(checkBytes.buffer);
final check = await TheStageFlutterSDK.infer(
  model_name: 'speaker_id',
  input_json: {'audio': newPcm16k, 'embedding': reference},
);
// accept if > 0.75
final similarity = check[0]['similarity'] as double;

```

--------------------------------

### Flutter: Tool calling with TheStageFlutterSDK and manual tool execution

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.llm.html

Flutter example showing how to start the model, define tools, and handle a stream where the stream ends at a tool call. The developer must execute the tool, append the result, and call again. The example uses the 'order_status' tool and a recursive turn() function.

```dart
await TheStageFlutterSDK.start_model(
  model_name: 'llm',
  engines_path: 'TheStageAI/Qwen3-0.6B',
);

// Flutter runs the tool itself: the stream ends at the tool call,
// you execute it, append the result, and ask again.
final tools = [
  {
    'name': 'order_status',
    'description': 'Look up the delivery status of an order by its number.',
    'parameters': {
      'type': 'object',
      'properties': {
        'order_id': {'type': 'string', 'description': 'Order number'},
      },
      'required': ['order_id'],
    },
  },
];
final history = [
  {'role': 'user', 'content': 'Where is my order 4471?'},
];

Future<void> turn() async {
  final stream = TheStageFlutterSDK.infer_stream(
    model_name: 'llm',
    input_json: {'messages': history, 'tools': tools, 'enable_thinking': false},
  );
  await for (final chunk in stream) {
    switch (chunk['kind']) {
      case 'text_delta':
        bubble.value += chunk['delta'] as String? ?? '';
      case 'tool_call':
        final name = chunk['name'] as String;
        final args = chunk['arguments'] as Map;
        final result = await orders.statusJson(args['order_id']);
        history.add({'role': 'tool', 'content': result});
        // continue the reply
        await turn();
        return;
      case 'final':
        return;
    }
  }
}
await turn();
```

--------------------------------

### setup_modules

Source: https://docs.thestage.ai/qlip.core/docs/source/generated/qlip.quantization.QuantizationManager.html

Attaches quantization parameters and sets quantization hooks for the given modules.

```APIDOC
## setup_modules

### Description
Attaches quantization parameters and sets quantization hooks for the given modules.

### Method
setup_modules

### Signature
setup_modules(modules, *, weights_scheme=int32, weights_granularity='per-channel', activations_scheme=int32, activations_granularity='per-batch', placement='input', quantization_mode='static', calibration_iterations=1, train_activations_scale=False, train_activations_offset=False, train_weights_scale=False, train_weights_offset=False, observer=None, observer_kwargs={}, weights_scale_offset_dtype=torch.float32, activations_scale_offset_dtype=torch.float32, **kwargs)

### Parameters
- **modules** (list[torch.nn.Module]) - Required - List of modules to setup.
- **weights_scheme** (QuantScheme) - Optional - Weights scheme.
- **weights_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Default is per-channel.
- **activations_scheme** (QuantScheme) - Optional - Activations scheme.
- **activations_granularity** (Union[str, QuantGranularity, Dict[str, Any]]) - Optional - Possible values: per-batch, per-sample or per-token. Default is per-batch.
- **placement** (Union[str, QuantPlacement]) - Optional - Three possible values: input, output or attention. Quantize input / output activations or attention. Default is input.
- **quantization_mode** (Union[str, QuantizationMode]) - Optional - Two possible values: static or dynamic. Default is static.
- **calibration_iterations** (size_t) - Optional - The number of batches to initialize scale and offset parameters for activations quantizer. Ignored if quantization_mode='dynamic'. Default is 1.
- **train_activations_scale** (bool) - Optional - Sets requires_grad attribute for activations scale parameters.
- **train_activations_offset** (bool) - Optional - Sets requires_grad attribute for activations offset parameters.
- **train_weights_scale** (bool) - Optional - Sets requires_grad attribute for weights scale parameters.
- **train_weights_offset** (bool) - Optional - Sets requires_grad attribute for weights offset parameters.
- **observer** (qlip.observers.IObserver) - Optional - Implemented observer interface. Default is MinMaxObserver.
- **observer_kwargs** (dict) - Optional - Additional keyword arguments for observer constructor.
- **weights_scale_offset_dtype** (torch.dtype) - Optional - Data type for weights scale and offset parameters. Default is torch.float32.
- **activations_scale_offset_dtype** (torch.dtype) - Optional - Data type for activations scale and offset parameters. Default is torch.float32.

### Returns
None
```

--------------------------------

### setup_model

Source: https://docs.thestage.ai/qlip.core/docs/source/generated/qlip.quantization.QuantizationManager.html

Abstract method for wrapping a model. Should be implemented in derived classes.

```APIDOC
## setup_model

### Description
Abstract method for wrapping a model. This method should be implemented in derived classes to define the specific wrapping behavior for a model.

### Method
instance

### Signature
setup_model(model)

### Parameters
- **model** (`Any`) - Required - The model to be wrapped.

### Returns
- **Any** - The wrapped model.
```

--------------------------------

### Run modal serve

Source: https://docs.thestage.ai/tutorials/source/modal_thestage.html

Run the modal app for the first time to cache model weights.

```bash
modal serve modal_serving.py

```

--------------------------------

### Flutter: JSON path for TTS

Source: https://docs.thestage.ai/android_sdk/docs/source/android_sdk.tts.html

Flutter example using the JSON path. Initializes the SDK, starts the 'tts' model, and infers. The result contains audio as Float32List and sample_rate as int (24000).

```dart
import 'package:thestage_android_sdk/thestage_android_sdk.dart';
import 'dart:typed_data';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');

await TheStageFlutterSDK.start_model(
  model_name: 'tts',
  engines_path: 'TheStageAI/neutts-multilingual',
  config: {'voice_id': 'paul', 'language': 'english'},
);

final result = await TheStageFlutterSDK.infer(
  model_name: 'tts',
  input_json: {'text': 'Hello, world!'},
);
final audio       = result[0]['audio']       as Float32List;
final sampleRate  = result[0]['sample_rate'] as int; // 24000

```

--------------------------------

### Swift: Extract meeting from WhatsApp export with LLM

Source: https://docs.thestage.ai/tutorials/source/apple_sdk_on_device_small_llms.html

Uses a system prompt to instruct the model to copy day and time strings exactly as said, and decodes the JSON response. The example includes a few-shot example and strips export prefixes to avoid mistaking send times for meeting times. The resolved start date is computed with resolve_said_datetime.

```swift
let system = """
Read a chat thread and report the meeting they agreed on. Do not \
compute dates and do not convert anything.

Reply with this JSON object only — no markdown, no extra keys:
{"title": string, "day_said": string, "start_said": string, \
"end_said": string or null, "location": string or null}

title — short Title Case name: the activity plus who. Not small talk \
from earlier lines. Not a sentence.
day_said — copy the day exactly as the thread says it: "14/08", \
"tomorrow", "friday". Do not turn it into a date.
start_said — the agreed start time as they wrote it, 24-hour clock. \
If they revised it, use the time they settled on.
end_said — the time someone said they have to leave, or null if nobody said.
location — the place they named, or null.
"""

let example_input = """
Ana Kovac: coffee tue 19/08 10:30 at the station? i leave 11:00
You: yes
"""
let example_output =
    "{\"title\":\"Coffee With Ana\",\"day_said\":\"19/08\",\"start_said\":\"10:30\",\"end_said\":\"11:00\",\"location\":\"the station\"}"

let export = """
[13/08/2026, 12:04:11] Marco Rossi: yo u around fri
[13/08/2026, 12:05:03] You: yeah till sat
[13/08/2026, 12:07:44] Marco Rossi: lunch 14/08 13:00 at the termini place? i gotta run 14:30 so dont be late
[13/08/2026, 12:08:02] You: ok see you there
"""

let result = try await engine.infer(
    messages: [.user(example_input), .assistant(example_output),
               .user(strip_export_prefixes(export))],
    system_prompt: system, tools: [],
    config: greedy_config(llm, max_new_tokens: 128)
)

struct Meeting: Decodable {
    let title: String; let day_said: String; let start_said: String
    let end_said: String?; let location: String?
}
let meeting = try JSONDecoder().decode(Meeting.self, from: Data(result.text.utf8))
guard let start = resolve_said_datetime(day_said: meeting.day_said,
                                        start_said: meeting.start_said,
                                        today: today, calendar: calendar)
else {
    // Unknown phrase: show "\(meeting.day_said) \(meeting.start_said)" and ask
    return
}
// start == 2026-08-14 13:00
```

--------------------------------

### Import tutorial utilities and load demo image

Source: https://docs.thestage.ai/tutorials/source/text2image_evaluation_tutorial.html

Imports utility functions for loading a demo image, applying deformations, building metric demo grids, plotting image grids, building annotated collages, and downloading cats and dogs dataset. Loads the base demo image with run_internet=True.

```python
import torch
from tutorial_utils import (
    load_demo_image, apply_deformations,
    build_metric_demo_grid, plot_image_grid,
    build_annotated_collage,
    download_cats_and_dogs
)

base_image = load_demo_image(run_internet=True)
base_image

```

--------------------------------

### One answer with infer() in Flutter

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.llm.html

Initializes the Flutter SDK, starts the model, and calls infer(model_name:input_json:) to get a single answer. The input JSON includes prompt, system_prompt, max_new_tokens, and enable_thinking.

```Flutter
import 'package:thestage_apple_sdk/thestage_apple_sdk.dart';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');
await TheStageFlutterSDK.start_model(
  model_name: 'llm',
  engines_path: 'TheStageAI/Qwen3-0.6B',
);

final rows = await TheStageFlutterSDK.infer(
  model_name: 'llm',
  input_json: {
    'prompt': 'Give me a two-line haiku about the ocean.',
    'system_prompt': 'You are a concise assistant.',
    'max_new_tokens': 128,
    'enable_thinking': false,
  },
);
print(rows[0]['text']);

```

--------------------------------

### Verify voice at login (Swift)

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.speaker_embedding.html

Loads a recorded phrase and the stored reference embedding, then calls TheStageAI.shared.infer with both audio and embedding to get a similarity score. The threshold 0.8 is a starting point; pick your own threshold on your data.

```swift
// the phrase just recorded, and the vector stored at enrolment
let phrasePcm: [Float] = try AudioIO.load_wav(
    path: recorder.fileURL.path,
    target_sample_rate: 16_000
)
let storedReference: [Double] = keychain.load("owner_voice")

let check = try TheStageAI.shared.infer(
    model_name: "speaker_id",
    // 2 s, 16 kHz mono
    input_json: ["audio": phrasePcm,
                 "embedding": storedReference])
let ok = (check[0]["similarity"] as! Double) > 0.8
```

--------------------------------

### Swift: Stream LLM reply to TTS

Source: https://docs.thestage.ai/apple_sdk/docs/source/apple_sdk.tts.html

Swift example using NeuTTSMultilingualPipeline, AudioStreamPlayer, TSLLM, and LLMChatEngine to stream text deltas to TTS. Start the consumer before sending tokens, send only text_delta, and close the stream when done.

```Swift
let tts = try await NeuTTSMultilingualPipeline(
    engines_path: "TheStageAI/neutts-nano-multilingual",
    voice_id: "dave",
    language: "english"
)
let player = AudioStreamPlayer(config: AudioStreamConfig(sample_rate: 24_000))
player.start()
let llm = try await TSLLM(engines_path: "TheStageAI/Qwen3-0.6B")
let session = LLMChatEngine(
    llm: llm).chat_session(system_prompt: "You are a helpful assistant.",
    memory: .SLIDING(max_turns: 10)
)
var llmConfig = llm.generation_defaults
llmConfig.enable_thinking = false
// the user's question (typed, or an ASR transcript)
let question = "How long does the battery last?"

let speech = tts.open_stream()
let consumer = Task {
    for await chunk in speech.output {
        if let pcm = chunk.audio { player.enqueue(pcm) }
    }
}

for await event in try session.infer_stream(
    user_request: question, config: llmConfig
) {
    // only text_delta
    if case .text_delta(let t) = event { speech.send(t) }
}
speech.close()
await consumer.value

```

--------------------------------

### Setup and Imports for Flux Caching Tutorial

Source: https://docs.thestage.ai/tutorials/source/flux_caching_tutorial.html

Imports required libraries and applies compatibility patches for cache-dit with older diffusers versions. Must be run before importing cache-dit.

```python
import os
import sys
import torch
import time
import numpy as np
from pathlib import Path
import matplotlib.pyplot as plt

# Add current directory to path for importing utils
sys.path.insert(0, os.path.dirname(os.path.abspath('__file__')))

from flux_caching_tutorial_utils import (
    setup_diffusers_compatibility, set_seed,
    visualize_comparison, create_performance_charts,
    print_performance_summary, save_results,
)

# Setup compatibility patches BEFORE importing cache-dit
setup_diffusers_compatibility()

# Now import elastic models and cache-dit
from elastic_models.diffusers import (
    DiffusionPipeline as ElasticDiffusionPipeline
)
import cache_dit
from cache_dit import (
    BasicCacheConfig, BlockAdapter, ForwardPattern
)

# Suppress warnings
import warnings
warnings.filterwarnings('ignore')

print("✓ All imports complete")
```
Tokens:
Raw