TensorSharp

https://tensorsharp.ai
Admin
TensorSharp is a native .NET LLM inference engine for GGUF models with CLI, web chat, and Ollama-...
Tokens:30,105
Snippets:342
Trust Score:4.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### Quick start server deployment

Source: https://tensorsharp.ai/server.html

Commands to clone, build, and run the server on Linux with NVIDIA CUDA support.

```bash
git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda
```

--------------------------------

### Verify .NET SDK Installation

Source: https://tensorsharp.ai/getting-started.html

Run this command in a terminal to confirm that the .NET 10 SDK is correctly installed and accessible.

```bash
dotnet --list-sdks
```

--------------------------------

### Start the TensorSharp server

Source: https://tensorsharp.ai/index.html

Launches the web server interface for API access and browser-based chat.

```bash
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 128
# open http://localhost:5000/index.html
```

--------------------------------

### Start OpenAI-compatible API server

Source: https://tensorsharp.ai/index.html

Launches a local server providing an OpenAI-compatible API endpoint.

```bash
dotnet run --project TensorSharp.Server -- \
--model model.gguf --backend cpu
```

--------------------------------

### Example JSON Configuration File

Source: https://tensorsharp.ai/server.html

A sample configuration file demonstrating variable definition, backend selection, and auto-download settings for model files.

```json
{
  "variables": { "modelRoot": "C:/models", "repo": "https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main" },
  "backend": "ggml_cuda",
  "max-tokens": 4096,
  "continuous-batching": true,
  "stop": ["</s>", "<|eot|>"],
  "model":  { "path": "${modelRoot}/gemma-4-E4B-it-Q8_0.gguf", "urls": [ "${repo}/gemma-4-E4B-it-Q8_0.gguf" ] },
  "mmproj": { "path": "${modelRoot}/gemma-4-E4B-mmproj-F16.gguf", "urls": [ "${repo}/mmproj-F16.gguf" ] }
}
```

--------------------------------

### Run Qwen-Image-Edit via CLI and Server

Source: https://tensorsharp.ai/models.html

Execute the model using the CLI for single image processing or start the server for ongoing requests. Ensure all model paths are correctly mapped to the downloaded files.

```bash
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
    --prompt "Make the sky a dramatic sunset." --output edited.png \
    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --qwen-image-mmproj models/mmproj-BF16.gguf \
    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
    --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --qwen-image-mmproj models/mmproj-BF16.gguf \
    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
    --backend ggml_cuda
```

--------------------------------

### GET /

Source: https://tensorsharp.ai/api-reference.html

Returns a plain liveness response for the server.

```APIDOC
## GET /

### Description
Plain liveness response.

### Method
GET

### Endpoint
/
```

--------------------------------

### Run interactive chat REPL

Source: https://tensorsharp.ai/getting-started.html

Starts an interactive session for turn-by-turn messaging.

```bash
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf -i --max-tokens 128 --backend ggml_cuda
```

--------------------------------

### SSE Terminal Frame Example

Source: https://tensorsharp.ai/http-api.html

Example of the final SSE frame returned upon completion of a chat request.

```text
data: {"done":true,"tokenCount":187,"elapsed":2.143,"tokPerSec":87.23,"promptTokens":512,"kvReusedTokens":420,"kvReusePercent":82.0}
```

--------------------------------

### JSONL Batch Format

Source: https://tensorsharp.ai/cli.html

Example structure for input files used in batch processing.

```json
{"id": "q1", "messages": [{"role": "user", "content": "What is 2+3?"}], "max_tokens": 50}
{"id": "q2", "messages": [{"role": "user", "content": "Write a haiku."}], "max_tokens": 100, "temperature": 0.8}
```

--------------------------------

### Configure Redis for Distributed State

Source: https://tensorsharp.ai/distributed.html

Examples for configuring Redis as a shared backend for KV cache and response storage in TensorSharp server processes.

```bash
# Both tiers on one Redis
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --redis-url localhost:6379

# KV cache only, 12-hour TTL
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --paged-kv-redis-url localhost:6379 --paged-kv-redis-ttl 720
```

--------------------------------

### Build the full solution

Source: https://tensorsharp.ai/getting-started.html

Compiles the entire solution including native bridges for GPU support.

```bash
dotnet build TensorSharp.slnx -c Release
```

--------------------------------

### Run Server with Configuration File

Source: https://tensorsharp.ai/server.html

Execute the server using a configuration file, with optional command-line overrides for specific parameters.

```bash
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basic.json
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basic.json --backend ggml_cpu
```

--------------------------------

### GET /api/tags

Source: https://tensorsharp.ai/http-api.html

Lists the available models.

```APIDOC
## GET /api/tags

### Description
Lists available models.

### Method
GET

### Endpoint
/api/tags
```

--------------------------------

### Run pre-built server binaries

Source: https://tensorsharp.ai/server.html

Execute the server binary directly for various hardware backends and multimodal configurations.

```bash
# Apple Silicon / Metal
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal

# NVIDIA / GGML CUDA
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_cuda

# AMD, Intel, or NVIDIA / Vulkan; inspect device indices first
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --list-gpus
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_vulkan --gpu-device 1

# Multimodal: the projector is always explicit
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --mmproj ./models/mmproj.gguf --backend ggml_cuda
```

--------------------------------

### Text Inference Commands

Source: https://tensorsharp.ai/cli.html

Commands for running text inference on different hardware backends and managing multi-GPU setups.

```bash
# Text inference (macOS)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_metal

# Text inference on Windows/Linux + NVIDIA GPU
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_cuda

# Text inference on any Vulkan GPU (AMD / Intel / NVIDIA)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_vulkan

# Multi-GPU host: list the visible Vulkan devices, then pick one by index
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- --list-gpus
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 200 \
    --backend ggml_vulkan --gpu-device 1

# Interactive turn-by-turn chat (REPL) with KV-cache reuse and slash commands
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal --interactive
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal -i \
    --system "You are a terse assistant." --temperature 0.7 --top-p 0.9 --think

# DSpark block speculative decoding (DeepSeek V4): --model is the FIRST shard of
# the split GGUF, --draft-model the separate DSpark drafter. Needs greedy sampling.
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <DeepSeek-V4-Flash-...-00001-of-00005.gguf> --backend ggml_cuda \
    --draft-model <DSpark-drafter.gguf> --input prompt.txt --max-tokens 200 --temperature 0
```

--------------------------------

### Run Qwen 3 Models

Source: https://tensorsharp.ai/models.html

Commands to download and execute Qwen 3 models with thinking and tool support.

```bash
hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda
```

--------------------------------

### Build with Managed-CPU backend

Source: https://tensorsharp.ai/getting-started.html

Use this configuration to skip native toolchain builds and rely on the managed CPU backend.

```bash
-p:TensorSharpSkipGgmlNative=true -p:TensorSharpSkipMlxNative=true -- --backend cpu
```

--------------------------------

### List and Show Models

Source: https://tensorsharp.ai/http-api.html

Retrieve a list of available models or show details for a specific model.

```bash
curl http://localhost:5000/api/tags

curl -X POST http://localhost:5000/api/show \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen3-4B-Q8_0.gguf"}'
```

--------------------------------

### Run TensorSharp CLI on Windows PowerShell with NVIDIA

Source: https://tensorsharp.ai/cli.html

Configures environment variables, builds the project, and executes a prompt using a GGUF model.

```powershell
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl.exe --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
Set-Content prompt.txt "Explain local inference in one paragraph."
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda
```

--------------------------------

### Run Qwen 3.5 / 3.6 Models

Source: https://tensorsharp.ai/models.html

Commands for multimodal Qwen 3.5/3.6 models requiring mmproj files.

```bash
hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda
```

--------------------------------

### Download Qwen-Image-Edit Model Weights

Source: https://tensorsharp.ai/models.html

Use these commands to download the required GGUF and safetensors files for the Qwen-Image-Edit model into the local models directory.

```bash
hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models
```

--------------------------------

### ModelBase.Create

Source: https://tensorsharp.ai/api-reference.html

Initializes a new model instance from a GGUF file path with a specified backend.

```APIDOC
## static ModelBase Create(string ggufPath, BackendType backend)

### Description
Creates a model instance by auto-detecting the architecture from the provided GGUF metadata.

### Parameters
- **ggufPath** (string) - Required - Path to the GGUF model file.
- **backend** (BackendType) - Required - The backend to use (Cpu, GgmlCpu, GgmlMetal, GgmlCuda, Cuda, Mlx).
```

--------------------------------

### Run Gemma 4 E4B Q8_0 on Linux with NVIDIA

Source: https://tensorsharp.ai/models.html

Downloads the model, builds the project with CUDA support, and executes inference via CLI or Server.

```bash
hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
echo "Explain why local inference is useful." > prompt.txt

# Text needs no mmproj.
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda
```

--------------------------------

### CLI Configuration via JSON

Source: https://tensorsharp.ai/cli.html

Use a configuration file to define CLI parameters, with command-line arguments taking precedence.

```bash
# Use the file, but override the backend for this run
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --config config/cli-basic.json --backend ggml_cpu
```

--------------------------------

### Clone the TensorSharp repository

Source: https://tensorsharp.ai/index.html

Initializes the local environment by cloning the source code.

```bash
git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
```

--------------------------------

### ModelBase.Sample

Source: https://tensorsharp.ai/api-reference.html

Applies sampling configuration to logits to select the next token.

```APIDOC
## int Sample(float[] logits, SamplingConfig config, IList<int> generated = null)

### Description
Applies penalties and sampling logic based on the provided configuration.
```

--------------------------------

### Reasoning and Sampling Commands

Source: https://tensorsharp.ai/cli.html

Commands for enabling reasoning modes, tool calling, and configuring sampling parameters.

```bash
# Thinking / reasoning mode
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 400 --backend ggml_metal --think

# Tool calling
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 300 --backend ggml_metal \
    --tools tools.json

# With sampling parameters (the CLI defaults to greedy decoding)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 200 --backend ggml_metal \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.2 --seed 42
```

--------------------------------

### Run TensorSharp CLI on Linux with NVIDIA

Source: https://tensorsharp.ai/cli.html

Builds the project with CUDA support and executes a prompt using a GGUF model.

```bash
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
echo "Explain local inference in one paragraph." > prompt.txt
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda
```

--------------------------------

### Build GGML Native Libraries on Windows

Source: https://tensorsharp.ai/backends.html

Use the PowerShell script to build the GGML native library on Windows with specific hardware acceleration flags.

```powershell
# Windows — CPU only, force CUDA on, force Vulkan off
.\build-windows.ps1 --no-cuda
.\build-windows.ps1 --cuda
.\build-windows.ps1 --no-vulkan
```

--------------------------------

### Configure DSpark in CLI and Server

Source: https://tensorsharp.ai/advanced.html

Run the CLI or Server with the --draft-model flag to enable speculative decoding. The server additionally requires the --mtp-spec flag.

```bash
# CLI — every single-sequence path (--input, --multi-turn-jsonl, --interactive) uses it
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model DeepSeek-V4-Flash-...-00001-of-00005.gguf \
    --backend ggml_cuda --draft-model DSpark-drafter-Q2K-Q8-0731.gguf \
    --input prompt.txt --max-tokens 200 --temperature 0

# Server — same drafter, plus --mtp-spec
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model DeepSeek-V4-Flash-...-00001-of-00005.gguf \
    --backend ggml_cuda --tp 4 --mtp-spec --draft-model DSpark-drafter-Q2K-Q8-0731.gguf
```

--------------------------------

### Configure Distributed Tensor Parallelism

Source: https://tensorsharp.ai/distributed.html

Set up multi-node clusters by specifying node IDs and peer lists. Ensure all nodes use the same peer list and unique node IDs.

```bash
# 2 nodes x 2 GPUs each = global TP degree 4.
# Node 0:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — same peer list, different node ID:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"
```

```bash
# Node 0 — server / driver:
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --tp 2 --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — CLI worker:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"
```

--------------------------------

### Build specific applications

Source: https://tensorsharp.ai/getting-started.html

Builds either the CLI or Server application individually.

```bash
# Console application
dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release

# Web application
dotnet build TensorSharp.Server/TensorSharp.Server.csproj -c Release
```

--------------------------------

### Build MLX Native Library on macOS

Source: https://tensorsharp.ai/backends.html

Execute the helper script to fetch and build the libmlxc dependency for the MLX backend.

```bash
bash TensorSharp.Backends.MLX/build-native-macos.sh
```

--------------------------------

### Load and generate text with TensorSharp

Source: https://tensorsharp.ai/code-api.html

Demonstrates the full pipeline of loading a GGUF model, configuring sampling parameters, and performing iterative token generation.

```csharp
using System;
using System.Collections.Generic;
using System.Linq;
using TensorSharp.Models;
using TensorSharp.Runtime;

// 1. Load any supported GGUF — the architecture is detected from metadata.
//    Pick the BackendType your build supports: GgmlCuda, GgmlMetal, GgmlVulkan, or GgmlCpu.
using var model = ModelBase.Create("gemma-4-E4B-it-Q8_0.gguf", BackendType.GgmlCuda);

// 2. Configure sampling (defaults match Ollama: temp 0.8, top_k 40, top_p 0.9).
var sampling = new SamplingConfig { Temperature = 0.7f, TopP = 0.9f, TopK = 40 };

// 3. Tokenize the prompt.
var tokens = model.Tokenizer
    .Encode("Explain mixture-of-experts in one sentence.", addSpecial: true)
    .ToList();
var generated = new List<int>();

// 4. Prefill the complete prompt once.
float[] logits = model.Forward(tokens.ToArray());

// 5. Decode one new token at a time. Forward() owns the KV-cache position,
// so after prefill pass only the newly sampled token, not the full history.
for (int step = 0; step < 200; step++)
{
    int next = model.Sample(logits, sampling, generated); // applies penalties + sampling
    if (model.Tokenizer.IsEos(next)) break;
    generated.Add(next);
    logits = model.Forward(new[] { next });
}

// 6. Detokenize the result.
Console.WriteLine(model.Tokenizer.Decode(generated));
```

--------------------------------

### Run Gemma 3 Multimodal Inference

Source: https://tensorsharp.ai/models.html

Downloads the model and projection weights, then executes text-image inference via CLI or Server.

```bash
hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda
```

--------------------------------

### Run one-shot generation via CLI

Source: https://tensorsharp.ai/index.html

Executes a single prompt using a specified GGUF model and backend.

```bash
dotnet run --project TensorSharp.Cli -- \
--model gemma-4-E4B-it-Q8_0.gguf \
--input prompt.txt \
--backend ggml_cuda
```

--------------------------------

### Run xUnit tests

Source: https://tensorsharp.ai/benchmarks.html

Executes the InferenceWeb unit tests using the .NET CLI.

```bash
dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csproj
```

--------------------------------

### Build GGML with CUDA via .NET CLI

Source: https://tensorsharp.ai/backends.html

Enable CUDA support directly through the dotnet build command.

```bash
TENSORSHARP_GGML_ENABLE_CUDA=ON dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release
```

--------------------------------

### Advanced Configuration with Variables and Auto-download

Source: https://tensorsharp.ai/cli.html

Define shared variables and auto-download model files using a JSON configuration.

```json
{
  "variables": { "modelRoot": "C:/models", "hf": "https://huggingface.co" },
  "backend": "ggml_cuda",
  "max-tokens": 256,
  "temperature": 0.7,
  "model": {
    "path": "${modelRoot}/Qwen3.5-9B-Q8_0.gguf",
    "urls": [ "${hf}/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q8_0.gguf" ]
  }
}
```

--------------------------------

### Chat completions using OpenAI SDK

Source: https://tensorsharp.ai/http-api.html

Initializes the OpenAI client to communicate with the TensorSharp API and performs a standard chat completion request.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:5000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="Qwen3-4B-Q8_0.gguf",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 2+3?"}
    ],
    max_tokens=50, temperature=0.7
)
print(response.choices[0].message.content)
```

--------------------------------

### Run Nemotron-H Models

Source: https://tensorsharp.ai/models.html

Commands for running Nemotron-H reasoning models.

```bash
hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
    --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda
```

--------------------------------

### Run DiffusionGemma and Inspect Prompts

Source: https://tensorsharp.ai/cli.html

Execute text-diffusion generation or inspect tokenization without running inference.

```bash
# DiffusionGemma text-diffusion generation
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <diffusion-gemma.gguf> --input prompt.txt --backend ggml_metal \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0

# Inspect the rendered prompt and tokenization without running inference
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --dump-prompt
```

--------------------------------

### Run GPT OSS Models

Source: https://tensorsharp.ai/models.html

Commands for running GPT OSS models with thinking enabled.

```bash
hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda
```

--------------------------------

### Run DiffusionGemma Models

Source: https://tensorsharp.ai/models.html

Commands for running DiffusionGemma models with specific diffusion parameters.

```bash
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda
```

--------------------------------

### Batch Processing and Benchmarking

Source: https://tensorsharp.ai/cli.html

Perform batch inference, multi-turn chat simulations, or throughput benchmarks.

```bash
# Batch processing (JSONL)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input-jsonl requests.jsonl \
    --output results.txt --backend ggml_metal

# Multi-turn chat simulation with KV-cache reuse (mirrors the web UI behavior)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --multi-turn-jsonl chat.jsonl \
    --backend ggml_metal --max-tokens 200

# Throughput benchmark: best-of-N prefill and decode timing
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal \
    --benchmark --bench-prefill 256 --bench-decode 128 --bench-runs 3
```

--------------------------------

### CLI Flags - TensorSharp.Cli

Source: https://tensorsharp.ai/api-reference.html

A comprehensive list of command-line flags used to configure and run the TensorSharp CLI tool.

```APIDOC
## CLI Flags - TensorSharp.Cli

### Description
These flags are used to configure the behavior of the `TensorSharp.Cli` executable, including model paths, hardware backends, sampling parameters, and benchmarking options.

### Usage
`TensorSharp.Cli --model <path> [options]`

### Available Flags
- **--model** (path) - Required - Path to a GGUF model file.
- **--input** (path) - Optional - Text file containing the user prompt.
- **--input-jsonl** (path) - Optional - JSONL file with batch requests.
- **--multi-turn-jsonl** (path) - Optional - JSONL for multi-turn chat simulation.
- **--output** (path) - Optional - Write generated text to this file.
- **--image / --video / --audio** (path) - Optional - Media for vision / video / audio inference.
- **--pdf** (path) - Optional - PDF document input.
- **--mmproj** (path) - Optional - Multimodal projector GGUF.
- **--max-tokens** (N) - Optional - Maximum tokens to generate (default 100).
- **--backend** (type) - Optional - `cpu`, `cuda`, `mlx`, `ggml_cpu`, `ggml_metal`, `ggml_cuda`, `ggml_vulkan`.
- **--gpu-device** (N) - Optional - Vulkan device index.
- **--list-gpus** (flag) - Optional - List visible Vulkan devices and exit.
- **--kv-cache-dtype** (type) - Optional - KV cache precision: `f32`, `f16`, `q8_0`, `q4_0`.
- **--interactive / -i** (flag) - Optional - Start the interactive REPL.
- **--system / --system-file** (text/path) - Optional - Seed the system prompt.
- **--think** (flag) - Optional - Enable thinking / reasoning mode.
- **--tools** (path) - Optional - JSON file with tool / function definitions.
- **--draft-model** (path) - Optional - Speculative-decoding drafter GGUF.
- **--spec-draft-n-max / --spec-draft-conf-min** (N/p) - Optional - Speculative decoding controls.
- **--temperature / --top-k / --top-p / --min-p** (values) - Optional - Sampling controls.
- **--repeat-penalty / --presence-penalty / --frequency-penalty** (values) - Optional - Penalties (1.0 / 0 = off).
- **--seed / --stop** (N/string) - Optional - Random seed or stop sequence.
- **--dump-prompt** (flag) - Optional - Render prompt + tokenization and exit.
- **--diffusion-steps / --diffusion-seed / --diffusion-blocks** (N) - Optional - DiffusionGemma generation controls.
- **--prompt** (text) - Optional - Qwen-Image-Edit edit instruction.
- **--cfg** (F) - Optional - Qwen-Image-Edit true-CFG guidance scale.
- **--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj** (path) - Optional - Override Qwen-Image-Edit companion GGUFs.
- **--qwen-image-lora** (path) - Optional - Lightning distillation LoRA.
- **--paged-kv-quant-bits** (0|2|4|8) - Optional - TurboQuant paged-KV block compression.
- **--benchmark / --bench-prefill / --bench-decode / --bench-runs** (flags) - Optional - Synthetic throughput benchmark.
- **--bench-kvcache / --bench-kv-turns** (N) - Optional - Multi-turn KV-cache reuse benchmark.
- **--warmup-runs** (N) - Optional - Throw-away forward passes before timing.
- **--test / --test-templates** (dir) - Optional - Built-in tokenizer/template tests.
- **--tp** (N) - Optional - Tensor parallelism degree.
- **--tp-node-id / --tp-peers** (N/list) - Optional - Multi-node distributed TP settings.
- **--log-level / --log-dir / --log-file / --log-console** (values) - Optional - Logger settings.
```

--------------------------------

### Download the Gemma 4 model

Source: https://tensorsharp.ai/index.html

Downloads the specified GGUF model file into the models directory.

```bash
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
```

--------------------------------

### Edit an image with Qwen-Image-Edit

Source: https://tensorsharp.ai/code-api.html

Load the Qwen-Image model, configure parameters, and perform an image edit operation.

```csharp
using TensorSharp.Models;
using TensorSharp.Models.QwenImage;
using TensorSharp.Runtime;

// Load the MMDiT GGUF (architecture = qwen_image). The VAE + Qwen2.5-VL
// companions are resolved from the same directory (or the TS_QWEN_IMAGE_* env vars).
using var model = (QwenImageModel)ModelBase.Create("qwen-image-edit-DiT-Q4_K_M.gguf", BackendType.GgmlCuda);

// Load the input image (PNG/JPEG decoded to an RgbImage).
RgbImage input = ImageIO.Load("input.png");

var p = new QwenImageParams
{
    Steps = 30,               // FlowMatch-Euler denoising steps
    CfgScale = 4.0f,          // true-CFG guidance; <= 1 disables the negative pass
    NegativePrompt = " ",     // used only when CfgScale > 1
    Seed = 0,
    TargetArea = 1024 * 1024, // ~1 MP; aspect follows the input (dims snapped to /16)
    // Width = 0, Height = 0, // pin an explicit output size to bypass the VRAM clamp
};

RgbImage output = model.EditImage("Make the sky a dramatic sunset.", input, p);
ImageIO.SavePng("edited.png", output);
Console.WriteLine($"Saved {output.Width}x{output.Height} edited image.");
```

--------------------------------

### Run Mistral 3 Models

Source: https://tensorsharp.ai/models.html

Commands for running multimodal Mistral 3 models with mmproj support.

```bash
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda
```

--------------------------------

### Configure server-wide sampling parameters

Source: https://tensorsharp.ai/server.html

Override default sampling settings by passing arguments directly to the server executable.

```bash
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.1 \
    --presence-penalty 0.0 --frequency-penalty 0.0 --seed 42 \
    --stop "</s>" --stop "<|endoftext|>"
```

--------------------------------

### Configure Local Tensor Parallelism via JSON

Source: https://tensorsharp.ai/distributed.html

Define parallelism settings in a configuration file instead of using CLI flags.

```json
{ "model": "model.gguf", "backend": "cuda", "tp": 2 }
```

--------------------------------

### Run inference via CLI

Source: https://tensorsharp.ai/index.html

Executes the model using the CUDA backend with a provided prompt file.

```bash
echo "What is TensorSharp? Answer briefly." > prompt.txt
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda
```
Tokens:
Raw