### Quickstart: GigaChat3-10B-A1.8B-base with vLLM Server

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Instructions on how to start a vLLM server for the GigaChat3-10B-A1.8B-base model and an example cURL request to the completion endpoint.

```APIDOC
## Quickstart: GigaChat3-10B-A1.8B-base using `vLLM`

### Description
This guide explains how to deploy the `ai-sage/GigaChat3-10B-A1.8B-base` model using vLLM and provides an example API request.

### Start vLLM Server
Execute the following command in your terminal:
```bash
vllm serve ai-sage/GigaChat3-10B-A1.8B-base \
  --disable-sliding-window \
  --dtype "auto" \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1, "disable_padded_drafter_batch": false}'
```

### Request Example
Send a POST request to the vLLM API endpoint:
```bash
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GigaChat3-10B-A1.8B-base",
    "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
    "max_tokens": 400,
    "temperature": 0
  }'
```

### Response Example
```json
{
  "id": "cmpl-xxxxxxxxxxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1677652377,
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "choices": [
    {
      "text": "<The generated text from the model>",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 400,
    "total_tokens": 415
  }
}
```
```

--------------------------------

### Quickstart: GigaChat3-10B-A1.8B-base with Transformers

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

A quickstart guide demonstrating how to use the GigaChat3-10B-A1.8B-base model with the Hugging Face transformers library for text generation.

```APIDOC
## Quickstart: GigaChat3-10B-A1.8B-base using `transformers`

### Description
This guide provides a Python code snippet to load and use the `ai-sage/GigaChat3-10B-A1.8B-base` model with the `transformers` library for text generation.

### Method
1.  Initialize Tokenizer and Model:
    ```python
    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

    model_name = "ai-sage/GigaChat3-10B-A1.8B-base"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )
    model.generation_config = GenerationConfig.from_pretrained(model_name)
    ```

2.  Generate Text:
    ```python
    prompt = "Ниже я написал подробное доказательство теоремы о неподвижной точке:"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            inputs.input_ids,
            attention_mask=inputs.attention_mask,
            max_new_tokens=400,
        )

    result = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:],
        skip_special_tokens=False,
    )
    print(result)
    ```

### Request Example
```json
{
  "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:"
}
```

### Response Example
```json
{
  "generated_text": "<The generated text from the model>"
}
```
```

--------------------------------

### Serve GigaChat3-10B-A1.8B-base with vLLM

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

This section provides bash commands to start a vLLM server for the GigaChat3-10B-A1.8B-base model and an example `curl` command to make a completion request to the server. It configures the server to use the MTP speculative decoding method. Ensure vLLM is installed and accessible.

```bash
vllm serve ai-sage/GigaChat3-10B-A1.8B-base \
  --disable-sliding-window \
  --dtype "auto" \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1, "disable_padded_drafter_batch": false}'
```

```bash
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GigaChat3-10B-A1.8B-base",
    "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
    "max_tokens": 400,
    "temperature": 0
  }'
```

--------------------------------

### Launch SGLang Server with NEXTN Speculative Algorithm

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Starts a SGLang inference server for the GigaChat3-10B model, enabling the NEXTN speculative decoding algorithm for optimized performance and memory allocation. Requires SGLang to be installed.

```bash
# Start SGLang server with NEXTN algorithm
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-10B-A1.8B-base \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype auto \
  --mem-fraction-static 0.88 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1

# Server will start on http://0.0.0.0:30000
# Expected output: SGLang server initialized with model loaded
```

--------------------------------

### Start SGLang Server for Gigachat3

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

This section details how to launch the SGLang server with the Gigachat3 model. It includes configuration options for the model path, host, port, data type, memory fraction, and speculative decoding.

```APIDOC
## Start SGLang Server

### Description
Launches the SGLang server to serve the Gigachat3 model. Allows configuration of various server and model parameters.

### Method
CLI Command

### Endpoint
N/A (Server startup)

### Parameters
#### Command Line Arguments
- **`--model-path`** (string) - Required - Path to the model weights (e.g., `ai-sage/GigaChat3-10B-A1.8B-base`).
- **`--host`** (string) - Optional - Host address to bind the server to (default: `0.0.0.0`).
- **`--port`** (integer) - Optional - Port number to listen on (default: `30000`).
- **`--dtype`** (string) - Optional - Data type for model loading (e.g., `auto`, `float16`, `bfloat16`).
- **`--mem-fraction-static`** (float) - Optional - Fraction of static memory to use for the model (e.g., `0.88`).
- **`--speculative-algorithm`** (string) - Optional - Algorithm for speculative decoding (e.g., `NEXTN`).
- **`--speculative-num-steps`** (integer) - Optional - Number of steps for speculative decoding.
- **`--speculative-eagle-topk`** (integer) - Optional - Top-k value for Eagle speculative decoding.

### Request Example
```bash
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-10B-A1.8B-base \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype auto \
  --mem-fraction-static 0.88 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1
```
```

--------------------------------

### Starting SGLang Server with NEXTN Speculative Algorithm

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Launch GigaChat3-10B using SGLang with optimized memory allocation and the NEXTN speculative decoding algorithm.

```APIDOC
## Starting SGLang Server with NEXTN Speculative Algorithm

### Description
Launch GigaChat3-10B using SGLang with optimized memory allocation and speculative decoding.

### Method
Command Line

### Endpoint
N/A (Server Initialization)

### Parameters
- `--model-path` (string) - Required - Path to the model (e.g., `ai-sage/GigaChat3-10B-A1.8B-base`).
- `--host` (string) - Host address for the server (e.g., `0.0.0.0`).
- `--port` (integer) - Port for the server (e.g., `30000`).
- `--dtype` (string) - Data type for the model (e.g., `auto`).
- `--mem-fraction-static` (number) - Fraction of memory to allocate for static cache.
- `--speculative-algorithm` (string) - The speculative decoding algorithm to use (e.g., `NEXTN`).
- `--speculative-num-steps` (integer) - Number of speculative steps.
- `--speculative-eagle-topk` (integer) - Top-k value for the Eagle speculative algorithm.

### Request Example
```bash
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-10B-A1.8B-base \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype auto \
  --mem-fraction-static 0.88 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1
```

### Response
#### Success Response (Server Start)
- **Server Status** (string) - Indicates the SGLang server has been initialized and the model is loaded. The server will be available at http://0.0.0.0:30000.
```

--------------------------------

### Start SGLang Server for GigaChat3

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Launches an SGLang server to host the GigaChat3 model. This command specifies the model path, host, port, and various server configurations including speculative decoding parameters. Ensure the model path is correct and sufficient resources are available.

```bash
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-10B-A1.8B-base \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype auto \
  --mem-fraction-static 0.88 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1
```

--------------------------------

### Benchmark GigaChat3 with lm-eval and SGLang Server

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Executes benchmark evaluations for GigaChat3 models using the lm-eval harness after starting an SGLang server. Requires installing lm-eval and sglang, setting an environment variable, and configuring model arguments for the SGLang API.

```bash
# Install dependencies
# pip install lm-eval[api]==0.4.9.1
# pip install sglang[all]==0.5.5

export HF_ALLOW_CODE_EVAL=1

# Start SGLang server first (10B model example)
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-10B-A1.8B-base \
  --host 127.0.0.1 \
  --port 30000 \
  --tp 1 \
  --dp 8 \
  --dtype bfloat16 \
  --mem-fraction-static 0.7 \
  --trust-remote-code \
  --allow-auto-truncate \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2

# Run MMLU Pro benchmark
python -m lm_eval \
  --model sglang-generate \
  --output_path ./results \
  --batch_size 16 \
  --model_args base_url=http://127.0.0.1:30000/generate,num_concurrent=16,tokenized_requests=True,max_length=131072,tokenizer=ai-sage/GigaChat3-10B-A1.8B-base \
  --trust_remote_code \
  --confirm_run_unsafe_code \
  --num_fewshot 5 \
  --tasks mmlu_pro
```

--------------------------------

### Run GigaChat3-10B-A1.8B-base with Transformers

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

This Python script demonstrates how to load and use the GigaChat3-10B-A1.8B-base model with the Hugging Face transformers library. It sets up the tokenizer and model, prepares a prompt, generates text, and decodes the output. Ensure you have the `transformers` and `torch` libraries installed.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

model_name = "ai-sage/GigaChat3-10B-A1.8B-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.generation_config = GenerationConfig.from_pretrained(model_name)

prompt = "Ниже я написал подробное доказательство теоремы о неподвижной точке:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        inputs.input_ids,
        attention_mask=inputs.attention_mask,
        max_new_tokens=400,
    )

result = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False,
)
print(result)
```

--------------------------------

### Make Completion Requests to vLLM Server

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Sends HTTP POST requests to a running vLLM inference server for text completion. This example uses a prompt to generate a continuation. Requires a running vLLM server and curl.

```bash
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GigaChat3-10B-A1.8B-base",
    "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
    "max_tokens": 400,
    "temperature": 0
  }'

# Expected response:
# {
#   "id": "cmpl-
    ...",
#   "object": "text_completion",
#   "created": 1704355200,
#   "model": "ai-sage/GigaChat3-10B-A1.8B-base",
#   "choices": [{
#     "text": "Continuation of mathematical proof...",
#     "index": 0,
#     "logprobs": null,
#     "finish_reason": "length"
#   }],
#   "usage": {
#     "prompt_tokens": 15,
#     "completion_tokens": 400,
#     "total_tokens": 415
#   }
# }
```

--------------------------------

### Deploy GigaChat3-10B with vLLM Server using MTP

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Deploys the GigaChat3-10B model using vLLM with Multi-Token Prediction (MTP) speculative decoding enabled for optimized throughput. This command starts a local inference server. Requires vLLM to be installed.

```bash
# Start vLLM server with MTP speculative decoding enabled
vllm serve ai-sage/GigaChat3-10B-A1.8B-base \
  --disable-sliding-window \
  --dtype "auto" \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1, "disable_padded_drafter_batch": false}'

# Server will start on http://localhost:8000
# Expected output: vLLM server running and accepting requests
```

--------------------------------

### Example cURL Request to SGLang Server

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Demonstrates how to send a completion request to a running SGLang server. This cURL command targets the completions endpoint with a JSON payload specifying the model, prompt, maximum tokens, and temperature. Ensure the server is running and accessible.

```bash
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GigaChat3-10B-A1.8B-base",
    "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
    "max_tokens": 400,
    "temperature": 0
  }'
```

--------------------------------

### Starting vLLM Server with MTP Speculative Decoding

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Deploy GigaChat3-10B model using vLLM with Multi-Token Prediction (MTP) for optimized throughput and faster inference.

```APIDOC
## Starting vLLM Server with MTP Speculative Decoding

### Description
Deploy GigaChat3-10B model using vLLM with Multi-Token Prediction for optimized throughput.

### Method
Command Line

### Endpoint
N/A (Server Initialization)

### Parameters
- `--disable-sliding-window` (flag) - Disable sliding window attention.
- `--dtype` (string) - Data type for the model (e.g., "auto").
- `--speculative-config` (JSON string) - Configuration for speculative decoding. Example: `'{"method": "mtp", "num_speculative_tokens": 1, "disable_padded_drafter_batch": false}'`.

### Request Example
```bash
vllm serve ai-sage/GigaChat3-10B-A1.8B-base \
  --disable-sliding-window \
  --dtype "auto" \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1, "disable_padded_drafter_batch": false}'
```

### Response
#### Success Response (Server Start)
- **Server Status** (string) - Indicates the vLLM server is running and accepting requests on http://localhost:8000.
```

--------------------------------

### Make Completion Requests to SGLang Server

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Sends HTTP POST requests to a running SGLang inference server for text generation. This example requests a continuation of a given prompt. Requires a running SGLang server and curl.

```bash
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GigaChat3-10B-A1.8B-base",
    "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
    "max_tokens": 400,
    "temperature": 0
  }'

# Expected response format:
# {
#   "id": "...",
#   "object": "text_completion",
#   "created": 1704355200,
#   "model": "ai-sage/GigaChat3-10B-A1.8B-base",
#   "choices": [{
#     "text": "Generated mathematical proof continuation...",
#     "index": 0,

```

--------------------------------

### Verify Model Metrics with lm-eval and sglang

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

This snippet demonstrates how to verify model metrics using the `lm-eval` and `sglang` libraries. It includes instructions for setting up the environment, launching the sglang server, and running a specific benchmark task (MMLU Pro). Ensure correct versions of libraries and model paths are specified.

```shell
# lm-eval[api]==0.4.9.1
# sglang[all]==0.5.5
# or 
# vllm==0.11.2

export HF_ALLOW_CODE_EVAL=1

# sglang server up

# 10B
python -m sglang.launch_server --model-path <path_to_model> --host 127.0.0.1 --port 30000 --tp 1 --dp 8 --dtype bfloat16 --mem-fraction-static 0.7 --trust-remote-code --allow-auto-truncate --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2

# mmlu pro check
python -m lm_eval --model sglang-generate --output_path <path_to_model> --batch_size 16 --model_args base_url=http://127.0.0.1:30000/generate,num_concurrent=16,tokenized_requests=True,max_length=131072,tokenizer=<path_to_model> --trust_remote_code --confirm_run_unsafe_code --num_fewshot 5 --tasks mmlu_pro
```

--------------------------------

### GigaChat3 Ultra (702B-A36B) - Deployment Overview

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Information about the GigaChat3 Ultra model, suitable for cluster and on-premise deployments, highlighting supported inference engines and optimization features.

```APIDOC
## GigaChat3 Ultra (702B-A36B)

### Description
`GigaChat3 Ultra Preview` is designed for cluster and on-prem scenarios with serious infrastructure, supporting popular inference engines and advanced features like BF16/FP8 modes and MLA/MTP for KV-cache optimization.

### Inference Engines
* vLLM
* SGLang
* LMDeploy
* TensorRT-LLM
* Other frameworks

### Features
* BF16 and FP8 modes supported (FP8 requires a separate build and GPU configuration).
* MLA and MTP reduce KV-cache size and speed up generation.
* Using a proxy/gateway layer is recommended for integration with external services, tools, and agent frameworks.

### Related Guides
* DeepSeek-V3: [https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file#6-how-to-run-locally](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file#6-how-to-run-locally)
* Kimi-K2-Instruct: [https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md)
```

--------------------------------

### GigaChat 3 Ultra Preview - Model Verification

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

This section details how to verify the model metrics for the GigaChat 3 Ultra Preview model using lm-eval and sglang.

```APIDOC
## GigaChat 3 Ultra Preview - Model Verification

### Description
Instructions on how to verify the performance metrics of the GigaChat 3 Ultra Preview model using evaluation frameworks like lm-eval and sglang.

### Method
Shell commands for setting up and running model evaluations.

### Endpoint
N/A (Local execution)

### Parameters
N/A

### Request Example
```shell
# lm-eval[api]==0.4.9.1
# sglang[all]==0.5.5
# or 
# vllm==0.11.2

export HF_ALLOW_CODE_EVAL=1

# sglang server up

# 700B
python -m sglang.launch_server --model-path <path_to_model> --host 127.0.0.1 --port 30000 --nnodes 2 --node-rank <0/1> --tp 16 --ep 16 --dtype auto --mem-fraction-static 0.7 --trust-remote-code --allow-auto-truncate --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 --dist-init-addr <master_node_ip>:50000

# mmlu pro check
python -m lm_eval --model sglang-generate --output_path <path_to_model> --batch_size 16 --model_args "base_url=[http://127.0.0.1:30000/generate,num_concurrent=16,tokenized_requests=True,max_length=131072,tokenizer=<path_to_model>]" --trust_remote_code --confirm_run_unsafe_code --num_fewshot 5 --tasks mmlu_pro
```

### Response
N/A (Output depends on evaluation results)

### Response Example
N/A
```

--------------------------------

### POST /v1/completions

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Generates text completions for a given prompt using the Gigachat3 model. This endpoint allows specifying the model, prompt, maximum tokens, and temperature for controlling the generation.

```APIDOC
## POST /v1/completions

### Description
Generates text completions based on a provided prompt using the specified model. You can control the output length and randomness.

### Method
POST

### Endpoint
`/v1/completions`

### Parameters
#### Request Body
- **`model`** (string) - Required - The name or path of the model to use (e.g., `ai-sage/GigaChat3-10B-A1.8B-base`).
- **`prompt`** (string) - Required - The input text prompt for generation.
- **`max_tokens`** (integer) - Optional - The maximum number of tokens to generate in the completion (default: depends on server configuration).
- **`temperature`** (float) - Optional - Controls the randomness of the output. Lower values make the output more deterministic (default: depends on server configuration).

### Request Example
```json
{
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
  "max_tokens": 400,
  "temperature": 0
}
```

### Response
#### Success Response (200)
- **`choices`** (array) - An array of completion choices.
  - **`text`** (string) - The generated text completion.
  - **`index`** (integer) - The index of the choice.
  - **`finish_reason`** (string) - The reason the generation finished (e.g., `stop`, `length`).
- **`id`** (string) - A unique identifier for the request.
- **`model`** (string) - The model used for generation.
- **`usage`** (object) - Information about token usage.
  - **`prompt_tokens`** (integer) - Number of tokens in the prompt.
  - **`completion_tokens`** (integer) - Number of tokens in the completion.
  - **`total_tokens`** (integer) - Total tokens used.

#### Response Example
```json
{
  "id": "cmpl-xxxxxxxxxxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1677652721,
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "choices": [
    {
      "text": " \n\n... (generated text) ...",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 23,
    "completion_tokens": 400,
    "total_tokens": 423
  }
}
```
```

--------------------------------

### Loading GigaChat3-10B Base Model with Transformers

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Demonstrates how to load and run inference with the GigaChat3-10B-A1.8B base model using the Hugging Face transformers library.

```APIDOC
## Loading GigaChat3-10B Base Model with Transformers

### Description
Load and run inference with the GigaChat3-10B-A1.8B base model using the Hugging Face transformers library.

### Method
Python Script

### Endpoint
N/A (Local Execution)

### Parameters
None

### Request Example
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

# Load model and tokenizer
model_name = "ai-sage/GigaChat3-10B-A1.8B-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.generation_config = GenerationConfig.from_pretrained(model_name)

# Prepare input
prompt = "Ниже я написал подробное доказательство теоремы о неподвижной точке:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        inputs.input_ids,
        attention_mask=inputs.attention_mask,
        max_new_tokens=400,
    )

# Decode output
result = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False,
)
print(result)
```

### Response
#### Success Response (Output)
- **result** (string) - The generated text continuation.
```

--------------------------------

### Deploy GigaChat3 Ultra 702B on Multi-Node Cluster with SGLang

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Deploys the GigaChat3 Ultra 702B model across a cluster of nodes using tensor and expert parallelism with SGLang. Requires specifying model path, network configuration, and parallelism settings. Intended for distributed inference.

```bash
# Node 0 (master node)
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-702B-A36B-preview \
  --host 127.0.0.1 \
  --port 30000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp 16 \
  --ep 16 \
  --dtype auto \
  --mem-fraction-static 0.7 \
  --trust-remote-code \
  --allow-auto-truncate \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --dist-init-addr <master_node_ip>:50000

# Node 1 (worker node)
python -m sglang.launch_server \
  --model-path ai-sage/GigaChat3-702B-A36B-preview \
  --host 127.0.0.1 \
  --port 30000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp 16 \
  --ep 16 \
  --dtype auto \
  --mem-fraction-static 0.7 \
  --trust-remote-code \
  --allow-auto-truncate \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --dist-init-addr <master_node_ip>:50000
```

--------------------------------

### Load GigaChat3-10B Instruct Model for Dialogue with Transformers

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Loads the GigaChat3-10B Instruct model using the Hugging Face Transformers library for conversational tasks. It sets up the tokenizer, model, and generation configuration, then processes a chat template to generate a response to a user query.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

# Load instruct model
model_name = "ai-sage/GigaChat3-10B-A1.8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
model.generation_config = GenerationConfig.from_pretrained(model_name)

# Prepare instruction prompt
messages = [
    {"role": "user", "content": "Explain quantum entanglement in simple terms"}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        inputs.input_ids,
        attention_mask=inputs.attention_mask,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.9
    )

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)
```

--------------------------------

### GigaChat3 Lightning (10B-A1.8B) - Performance

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Details on the performance of the GigaChat3 Lightning model, emphasizing its inference speed and quality compared to other models, with benchmark data.

```APIDOC
## GigaChat3 Lightning (10B-A1.8B)

### Description
`GigaChat3-10B-A1.8B` offers exceptional inference speed, delivering throughput comparable to much smaller dense models while maintaining high quality, especially in MTP mode. This makes it a cost-effective alternative to larger models.

### Performance Benchmarks
Measurements were conducted using vLLM v0.11.0, dtype bfloat16, with `batch_size=1`. Benchmark code available at: [https://gist.github.com/ajpqs/ce941aa6f0f48ef36a65cb87a2a1d726](https://gist.github.com/ajpqs/ce941aa6f0f48ef36a65cb87a2a1d726).

| Model                          | request_throughput | output_throughput | total_token_throughput | mean_ttft_ms |
| ------------------------------ | ------------------ | ----------------- | ---------------------- | ------------ |
| `Qwen3-1.7B`                   | 1.689              | 357.308           | 726.093                | 11.824       |
| `mtp-GigaChat3-10B-A1.8B-base` | 1.533              | 333.620           | 678.894                | 26.345       |
| `GigaChat3-10B-A1.8B-base`     | 1.077              | 234.363           | 476.912                | 31.053       |
| `Qwen3-4B`                     | 0.978              | 206.849           | 420.341                | 14.947       |
| `Qwen3-8B`                     | 0.664              | 140.432           | 285.375                | 16.663       |
| `YandexGPT-5-Lite-8B-pretrain` | 0.641              | 147.305           | 300.269                | 16.711       |

### Comparison
In terms of speed and inference cost, the 10B parameter model can be treated as an alternative to 3–4B dense models, and in MTP mode, it compares favorably even to smaller ones.
```

--------------------------------

### Fine-tune GigaChat3 Ultra with BF16 Checkpoint using Transformers Trainer

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Enables custom fine-tuning of the GigaChat3 Ultra model using a BF16 checkpoint. It involves loading the model and tokenizer, preparing a dataset, configuring Hugging Face `TrainingArguments`, and initiating the `Trainer` for the fine-tuning process.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset
import torch

# Load BF16 checkpoint for fine-tuning
model_name = "ai-sage/GigaChat3-702B-A36B-preview-bf16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Load custom dataset
dataset = load_dataset("your_dataset_name")

# Configure training
training_args = TrainingArguments(
    output_dir="./gigachat3-ultra-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=32,
    learning_rate=1e-5,
    bf16=True,
    logging_steps=10,
    save_steps=100,
    save_total_limit=2
)

# Initialize trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    tokenizer=tokenizer
)

# Start fine-tuning
trainer.train()
```

--------------------------------

### POST /v1/completions (vLLM Server)

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Send HTTP requests to the vLLM inference server for text completion using the GigaChat3-10B model.

```APIDOC
## POST /v1/completions (vLLM Server)

### Description
Send HTTP requests to the vLLM inference server for text completion.

### Method
POST

### Endpoint
`http://localhost:8000/v1/completions`

### Parameters
#### Request Body
- **model** (string) - Required - The name of the model to use (e.g., `ai-sage/GigaChat3-10B-A1.8B-base`).
- **prompt** (string) - Required - The input text prompt for generation.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **temperature** (number) - Optional - Controls randomness in generation (0 for deterministic).

### Request Example
```json
{
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
  "max_tokens": 400,
  "temperature": 0
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **object** (string) - Type of object returned (e.g., `text_completion`).
- **created** (integer) - Timestamp of creation.
- **model** (string) - The model used for completion.
- **choices** (array) - List of completion choices.
  - **text** (string) - The generated text.
  - **index** (integer) - Index of the choice.
  - **logprobs** (null) - Placeholder for log probabilities.
  - **finish_reason** (string) - Reason for stopping generation (e.g., `length`).
- **usage** (object) - Token usage statistics.
  - **prompt_tokens** (integer) - Number of tokens in the prompt.
  - **completion_tokens** (integer) - Number of tokens generated.
  - **total_tokens** (integer) - Total tokens used.

#### Response Example
```json
{
  "id": "cmpl-...",
  "object": "text_completion",
  "created": 1704355200,
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "choices": [{
    "text": "Continuation of mathematical proof...",
    "index": 0,
    "logprobs": null,
    "finish_reason": "length"
  }],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 400,
    "total_tokens": 415
  }
}
```
```

--------------------------------

### POST /v1/completions (SGLang Server)

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Send HTTP requests to the SGLang inference server for text generation using the GigaChat3-10B model.

```APIDOC
## POST /v1/completions (SGLang Server)

### Description
Send HTTP requests to the SGLang inference server for text generation.

### Method
POST

### Endpoint
`http://localhost:30000/v1/completions`

### Parameters
#### Request Body
- **model** (string) - Required - The name of the model to use (e.g., `ai-sage/GigaChat3-10B-A1.8B-base`).
- **prompt** (string) - Required - The input text prompt for generation.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **temperature** (number) - Optional - Controls randomness in generation (0 for deterministic).

### Request Example
```json
{
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "prompt": "Ниже я написал подробное доказательство теоремы о неподвижной точке:",
  "max_tokens": 400,
  "temperature": 0
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **object** (string) - Type of object returned (e.g., `text_completion`).
- **created** (integer) - Timestamp of creation.
- **model** (string) - The model used for completion.
- **choices** (array) - List of completion choices.
  - **text** (string) - The generated text.
  - **index** (integer) - Index of the choice.
- **usage** (object) - Token usage statistics.
  - **prompt_tokens** (integer) - Number of tokens in the prompt.
  - **completion_tokens** (integer) - Number of tokens generated.
  - **total_tokens** (integer) - Total tokens used.

#### Response Example
```json
{
  "id": "...",
  "object": "text_completion",
  "created": 1704355200,
  "model": "ai-sage/GigaChat3-10B-A1.8B-base",
  "choices": [{
    "text": "Generated mathematical proof continuation...",
    "index": 0
  }],
  "usage": {
    "prompt_tokens": 15,
    "completion_tokens": 400,
    "total_tokens": 415
  }
}
```
```

--------------------------------

### Load GigaChat3-10B Model with Hugging Face Transformers

Source: https://context7.com/salute-developers/gigachat3/llms.txt

Loads the GigaChat3-10B-A1.8B base model and tokenizer using the Hugging Face transformers library. It prepares input and generates a response, demonstrating basic inference capabilities. Requires the 'transformers' and 'torch' libraries.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

# Load model and tokenizer
model_name = "ai-sage/GigaChat3-10B-A1.8B-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.generation_config = GenerationConfig.from_pretrained(model_name)

# Prepare input
prompt = "Ниже я написал подробное доказательство теоремы о неподвижной точке:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        inputs.input_ids,
        attention_mask=inputs.attention_mask,
        max_new_tokens=400,
    )

# Decode output
result = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=False,
)
print(result)
# Expected: Continuation of the mathematical proof about fixed-point theorem
```

--------------------------------

### GigaChat3 Shared Model Properties

Source: https://github.com/salute-developers/gigachat3/blob/main/README.md

Common properties and characteristics shared by GigaChat3 models, including their reasoning capabilities, MTP support, MLA feature, training methodology, compatibility, and licensing.

```APIDOC
## Shared Model Properties

### Description
This section outlines the shared characteristics across GigaChat3 models, covering their functional capabilities, technical implementations, and licensing terms.

### Properties
*   **Reasoning**: Not specialized reasoning models, but support a basic level of reasoning.
*   **MTP (Multiple Tokens Per Step)**: Capable of predicting multiple tokens per step.
*   **MLA (Memory-efficient Layer Attention)**: Reduces KV-cache size and memory requirements.
*   **Training**: Trained from scratch without initialization from third-party weights.
*   **Compatibility**: Compatible with Hugging Face, vLLM, SGLang, LMDeploy, and standard inference/fine-tuning pipelines.
*   **License**: Released under the MIT license, permitting commercial use.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.