TensorSharp

https://github.com/zhongkaifu/tensorsharp
Admin
TensorSharp is a native .NET LLM inference engine for GGUF models that enables high-performance...
Tokens:1,065,433
Snippets:6,237
Trust Score:8.7
License:BSD-3-Clause
Update:1 week ago
Show doc for...
Tokens:
Raw
### Start reference server and generate reference outputs

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/eng/validation/bonsai2.md

Starts the publisher's server with the exact model under test, F16 K/V caches, and a bounded context. The example uses four 2,048-token slots. Then runs the reference script to generate golden outputs.

```sh
artifacts/bonsai2/llama.cpp-prism/build/bin/llama-server \
  -m /path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 \
  -c 8192 -np 4 -fa on -ctk f16 -ctv f16 -b 512 -ub 512 \
  --host 127.0.0.1 --port 8083 --jinja
```

```sh
python3 eng/validation/bonsai2-bench.py reference \
  --url http://127.0.0.1:8083 --tokens 32 \
  --golden docs/validation/bonsai2/pq2-golden.json \
  --output docs/validation/bonsai2/prism-pq2-reference.json
```

--------------------------------

### Install TensorSharp and run with tensor parallelism

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/getting_started.md

Install the CUDA toolkit, then set up the environment, clone the repo, build, and run the CLI with --tp 2 on the cuda or ggml_cuda backend. The example includes a comment about setting LD_LIBRARY_PATH for RunPod Ubuntu 24.04 and adding the dotnet backports PPA for older Ubuntu.

```bash
# On RunPod's Ubuntu 24.04 images, point the loader at the CUDA compat libraries first:
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/compat:$LD_LIBRARY_PATH
# On older Ubuntu releases the .NET 10 SDK comes from the backports PPA:
add-apt-repository ppa:dotnet/backports

apt update && apt install dotnet-sdk-10.0
git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
mkdir models
wget "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -O models/gemma-4-E4B-it-Q8_0.gguf
bash TensorSharp.GGML.Native/build-linux.sh
dotnet build -c Release

# 2 GPUs in one process
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend cuda --interactive --max-tokens 20000 --tp 2

# Same thing on the GGML CUDA backend (add TENSORSHARP_TP_DEVICES=0,2 to pick GPUs)
TensorSharp.Cli/bin/TensorSharp.Cli --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda --interactive --max-tokens 20000 --tp 2
```

--------------------------------

### Skills field in GET /api/models

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/TensorSharp.Server.Host/API_EXAMPLES.md

Shows the skills field returned by GET /api/models, indicating whether skills are enabled, installable, and script execution is allowed.

```json
"skills": { "enabled": true, "installable": true, "allowScripts": false, "count": 7 }
```

--------------------------------

### Start server with configuration file (auto-downloads model)

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/USAGE.md

Reads all settings from a reusable JSON file. The model is auto-downloaded on first run. The second command overrides the backend to CPU.

```bash
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json
```

```bash
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json --backend ggml_cpu
```

--------------------------------

### Skill object JSON structure

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/agent_skills.md

Example of a skill object returned by the API. origin is discovered for scanned skills and installed for uploaded ones; only installed skills can be deleted. /api/skills wraps the list with enabled, installable, skills, and errors.

```json
{
  "id": "pdf",
  "object": "skill",
  "name": "pdf",
  "description": "Extract text and tables from PDF files...",
  "license": "Apache-2.0",
  "compatibility": "Requires python3 with pypdf installed.",
  "files": [
    {"path": "scripts/extract_tables.py", "bytes": 4021, "kind": "script", "text": true},
    {"path": "references/forms.md", "bytes": 18233, "kind": "reference", "text": true}
  ],
  "bytes": 41288,
  "origin": "installed",
  "warnings": [],
  "modified": "2026-08-29T12:00:00Z"
}
```

--------------------------------

### Launch CLI: set TENSORSHARP_MODELS, build, generate and edit with qwen-image-2.1

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/models/qwenimage21.md

Setup commands for launching the CLI. Set TENSORSHARP_MODELS to choose where models are stored (default resolves to ../models relative to config/). Build the CLI and server host projects, then run text-to-image or image editing with the qwen-image-2.1 config. Use --backend ggml_cuda on NVIDIA, --backend cpu for pure C#.

```bash
export TENSORSHARP_MODELS="$PWD/../models"
```

```bash
dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release

```

```bash
dotnet build TensorSharp.Server.Host/TensorSharp.Server.Host.csproj -c Release

```

```bash
dotnet run --project TensorSharp.Cli -c Release --no-build -- \
  --config config/qwen-image-2.1.json \
  --prompt 'A small orange cat beside a blue ceramic vase, soft daylight, detailed photograph' \
  --width 2048 --height 2048 --diffusion-steps 40 --cfg 1 \
  --diffusion-seed 42 --output generated.png
```

```bash
dotnet run --project TensorSharp.Cli -c Release --no-build -- \
  --config config/qwen-image-2.1.json \
  --image generated.png \
  --prompt 'Change the blue vase to a red vase. Preserve the cat, lighting and composition.' \
  --width 2048 --height 2048 --diffusion-steps 40 --cfg 1 \
  --diffusion-seed 42 --output edited.png
```

--------------------------------

### Start MiniMax-H3 video generation with Ref2VA checkpoint

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/website/server.html

Command-line invocation to start the TensorSharp.Server host with the MiniMax-H3 Ref2VA checkpoint, using the same geometry as the FL2VA example. This checkpoint uses identity/appearance references into a new scene.

```bash
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model ./models/minimax_h3_ref2va_pruned-Q4_K.gguf --backend ggml_cuda \
    --video-width 640 --video-height 384 --video-steps 20 --video-frames 22
```

--------------------------------

### Build and run Qwen-Image-2.1 with config/qwen-image-2.1.json

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/website/models-image.html

Builds the CLI and server host, then runs text-to-image, image editing, and the server with the ready-made config. The config selects ggml_metal; append --backend ggml_cuda on NVIDIA, or use --backend cpu for pure C#.

```bash
dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release
dotnet build TensorSharp.Server.Host/TensorSharp.Server.Host.csproj -c Release

# Text-to-image
dotnet run --project TensorSharp.Cli -c Release --no-build -- \
  --config config/qwen-image-2.1.json \
  --prompt 'A small orange cat beside a blue ceramic vase, soft daylight, detailed photograph' \
  --width 2048 --height 2048 --diffusion-steps 40 --cfg 1 \
  --diffusion-seed 42 --output generated.png

# Image editing
dotnet run --project TensorSharp.Cli -c Release --no-build -- \
  --config config/qwen-image-2.1.json \
  --image generated.png \
  --prompt 'Change the blue vase to a red vase. Preserve the cat, lighting and composition.' \
  --width 2048 --height 2048 --diffusion-steps 40 --cfg 1 \
  --diffusion-seed 42 --output edited.png

# Server + Web UI
dotnet run --project TensorSharp.Server.Host -c Release --no-build -- \
  --config config/qwen-image-2.1.json --host 127.0.0.1 --port 5000
```

--------------------------------

### Start agentic session with code execution tools

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/website/agentic.html

Run the TensorSharp CLI with the pdf skill and code execution enabled, disabling command networking and package installation.

```bash
dotnet run --project TensorSharp.Cli -c Release -- \
  --model models/model.gguf --backend ggml_metal \
  --skills-dir ./skills --skill pdf --code-exec -i
```

--------------------------------

### C# example: Create KnowledgeStore, import, search, and read

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/knowledge-system.md

Creates a KnowledgeStore with built-in extractor, imports a markdown file, searches for 'launch requirements', and reads the top hit with context. Requires an absolute path to the library and a markdown file. The core project has no dependency on TensorAgent or a running inference model.

```csharp
using TensorSharp.Knowledge;

var options = new KnowledgeOptions();
using var store = new KnowledgeStore(
    "/absolute/path/to/library", new BuiltInKnowledgeExtractor(options), options: options);
var collection = await store.CreateCollectionAsync("Project research");
await using var input = File.OpenRead("notes.md");
var document = await store.ImportAsync(collection.Id, "notes.md", input, "text/markdown");
var hits = await store.SearchAsync("launch requirements", new KnowledgeSearchOptions
{
    CollectionIds = new[] { collection.Id }, Limit = 6, MaxCharacters = 8000
});
if (hits.Count > 0)
{
    var context = store.Read(hits[0].Reference, maxCharacters: 8000,
        adjacentChunks: 1, collectionIds: new[] { collection.Id });
}
```

--------------------------------

### Print full option reference with --help

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/getting_started.md

Both the CLI and server print their full option reference when started with no arguments or with --help. This shows the description, default, range, and an example per flag.

```bash
dotnet run --project TensorSharp.Cli -c Release -- --help
dotnet run --project TensorSharp.Server.Host -c Release -- --help
```

--------------------------------

### Start server with Qwen-Image-2.1 LoRA plug-in and request image generation

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/TensorSharp.Server.Host/API_EXAMPLES.md

Starts the server with a step-distillation LoRA plug-in, making its recipe (8 steps, CFG 1) the default for every image request. Weights download and are hash-checked on first use. The curl request omits steps and cfg to use the plug-in's recipe.

```bash
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/qwen-image-2.1.json \
  --lora config/lora/qwen-image-2.1-pruna-8step.json
```

```bash
curl --fail-with-body http://localhost:5000/api/image-generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"A cat beside a blue vase, soft daylight","width":1024,"height":1024,"seed":42}'
```

--------------------------------

### Run TensorSharp.Cli with ggml_cuda backend

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/models/deepseek4.md

Starts a chat session using the ggml CUDA backend, pointing --model at the first shard of an IQ4_XS quantized model. Requires 2x80 GB GPUs for this example.

```bash
# point --model at the FIRST shard; this IQ4_XS example needs 2x80 GB GPUs
TensorSharp.Cli --model DeepSeek-V4-Flash-UD-IQ4_XS-00001-of-00004.gguf \
    --backend ggml_cuda --layer-split 2 --chat
```

--------------------------------

### Launch TensorSharp.Server.Host for Qwen-Image 2.1

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/docs/models/qwenimage21.md

Launch the TensorSharp.Server.Host with the Qwen-Image 2.1 configuration. Open http://127.0.0.1:5000 afterwards.

```bash
dotnet run --project TensorSharp.Server.Host -c Release --no-build -- \
  --config config/qwen-image-2.1.json --host 127.0.0.1 --port 5000
```

--------------------------------

### Send a chat completion request with curl

Source: https://github.com/zhongkaifu/tensorsharp/blob/main/TensorSharp.Server.Host/API_EXAMPLES.md

Example curl command to send a chat completion request to the running server at localhost:5000. The server must be started first. The response will be a JSON object with the model's reply.

```bash
curl -s http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Reply with one short hello."}],"max_tokens":32}'
```
Tokens:
Raw