### Install DeepSeek-Infer Demo Requirements

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Navigate to the inference directory and install the project's Python dependencies from the requirements.txt file.

```shell
cd DeepSeek-V3/inference
pip install -r requirements.txt
```

--------------------------------

### FP8 Quantization Configuration Example

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README_WEIGHTS.md

This JSON snippet shows an example configuration for FP8 quantization, specifying the activation scheme, format, and weight block size.

```json
"quantization_config": {
  "activation_scheme": "dynamic",
  "fmt": "e4m3",
  "quant_method": "fp8",
  "weight_block_size": [128, 128]
}
```

--------------------------------

### Install DeepSeek-Infer Demo Dependencies

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Install the required Python packages for the DeepSeek-Infer Demo. It's recommended to use a virtual environment.

```pip-requirements
torch==2.4.1
triton==3.0.0
transformers==4.46.3
safetensors==0.4.5
```

--------------------------------

### Interactive chat on 2 nodes with 8 GPUs each (16 total)

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Launch an interactive chat session for distributed inference across multiple nodes and GPUs. Ensure environment variables RANK, MASTER_ADDR are set correctly. This command assumes a 2-node setup with 8 GPUs per node.

```bash
# Interactive chat on 2 nodes with 8 GPUs each (16 total)
torchrun \
    --nnodes 2 \
    --nproc-per-node 8 \
    --node-rank $RANK \
    --master-addr $MASTER_ADDR \
    --master-port 29500 \
    generate.py \
    --ckpt-path /path/to/DeepSeek-V3-Demo \
    --config configs/config_671B.json \
    --interactive \
    --temperature 0.7 \
    --max-new-tokens 200
```

--------------------------------

### Batch inference from file

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Perform batch inference by reading prompts from a file. This command is configured for a multi-node, multi-GPU setup. Adjust --nnodes and --nproc-per-node based on your cluster configuration.

```bash
# Batch inference from file
torchrun \
    --nnodes 2 \
    --nproc-per-node 8 \
    --node-rank $RANK \
    --master-addr $MASTER_ADDR \
    generate.py \
    --ckpt-path /path/to/DeepSeek-V3-Demo \
    --config configs/config_671B.json \
    --input-file prompts.txt \
    --temperature 0.2 \
    --max-new-tokens 500
```

--------------------------------

### DeepSeek API Chat Completion using cURL

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Provides a cURL command for making a chat completion request to the DeepSeek API. This example demonstrates the necessary headers and JSON payload for direct API interaction.

```bash
curl https://api.deepseek.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPSEEK_API_KEY" \
  -d '{
    "model": "deepseek-chat",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7
  }'
```

--------------------------------

### DeepSeek API Chat Completion with OpenAI Client

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Demonstrates how to initialize the OpenAI client to interact with the DeepSeek API for chat completions. Includes setting the API key and base URL, and making a standard completion request.

```python
from openai import OpenAI

# Initialize client with DeepSeek API
client = OpenAI(
    api_key="your-deepseek-api-key",
    base_url="https://api.deepseek.com"
)

# Chat completion
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the Mixture-of-Experts architecture."}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)
```

--------------------------------

### Initialize and Use Mixture-of-Experts (MoE) Layer

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Initializes the MoE layer with custom arguments and demonstrates a forward pass. It also shows how to inspect the gating mechanism for weights and expert indices.

```python
from model import MoE, Gate, Expert, ModelArgs
import torch

args = ModelArgs(
    dim=7168,
    moe_inter_dim=2048,
    n_routed_experts=256,
    n_shared_experts=1,
    n_activated_experts=8,
    n_expert_groups=8,
    n_limited_groups=4,
    route_scale=2.5,
    score_func="sigmoid"
)

# Initialize MoE layer
moe = MoE(args).cuda()

# Forward pass
x = torch.randn(2, 128, args.dim, device="cuda", dtype=torch.bfloat16)
output = moe(x)
print(f"MoE output shape: {output.shape}")  # torch.Size([2, 128, 7168])

# Examine gating mechanism
gate = Gate(args).cuda()
flat_x = x.view(-1, args.dim)
weights, indices = gate(flat_x)
print(f"Expert weights: {weights.shape}")   # torch.Size([256, 8])
print(f"Selected experts: {indices.shape}") # torch.Size([256, 8])
print(f"Top experts for first token: {indices[0].tolist()}")
```

--------------------------------

### Run DeepSeek-V3 in Interactive Mode

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Launch the DeepSeek-V3 model for interactive chat using the DeepSeek-Infer Demo. This command requires specifying the number of nodes, processes per node, rank, master address, checkpoint path, config file, and generation parameters.

```shell
torchrun --nnodes 2 --nproc-per-node 8 --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path /path/to/DeepSeek-V3-Demo --config configs/config_671B.json --interactive --temperature 0.7 --max-new-tokens 200
```

--------------------------------

### Initialize and Use Multi-Head Latent Attention (MLA)

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Initializes the MLA module with specified arguments and demonstrates forward passes for both prefill and incremental decoding. Ensure CUDA is available and necessary arguments are configured.

```python
from model import MLA, ModelArgs, precompute_freqs_cis
import torch

args = ModelArgs(
    dim=7168,
    n_heads=128,
    kv_lora_rank=512,
    q_lora_rank=1536,
    qk_nope_head_dim=128,
    qk_rope_head_dim=64,
    v_head_dim=128,
    max_batch_size=4,
    max_seq_len=4096
)

# Initialize attention layer
mla = MLA(args).cuda()

# Prepare inputs
batch_size, seq_len = 2, 128
x = torch.randn(batch_size, seq_len, args.dim, device="cuda", dtype=torch.bfloat16)
freqs_cis = precompute_freqs_cis(args).cuda()

# Forward pass (prefill)
output = mla(x, start_pos=0, freqs_cis=freqs_cis[:seq_len], mask=None)
print(f"Output shape: {output.shape}")  # torch.Size([2, 128, 7168])

# Incremental decoding (single token)
new_token = torch.randn(batch_size, 1, args.dim, device="cuda", dtype=torch.bfloat16)
output = mla(new_token, start_pos=seq_len, freqs_cis=freqs_cis[seq_len:seq_len+1], mask=None)
print(f"Decode output: {output.shape}")  # torch.Size([2, 1, 7168])
```

--------------------------------

### Run DeepSeek-V3 for Batch Inference

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Perform batch inference on a given file using the DeepSeek-Infer Demo. Similar to interactive mode, this requires specifying distributed training parameters and the input file path.

```shell
torchrun --nnodes 2 --nproc-per-node 8 --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path /path/to/DeepSeek-V3-Demo --config configs/config_671B.json --input-file $FILE
```

--------------------------------

### Convert Hugging Face Weights for Demo

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Convert downloaded Hugging Face model weights into the format required by the DeepSeek-Infer Demo. Specify the paths for input and output, and configure expert and model parallelism.

```shell
python convert.py --hf-ckpt-path /path/to/DeepSeek-V3 --save-path /path/to/DeepSeek-V3-Demo --n-experts 256 --model-parallel 16
```

--------------------------------

### Configure and Initialize DeepSeek-V3 Model

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Use ModelArgs to define architecture hyperparameters and initialize the Transformer model on GPU. Ensure the device is set to CUDA and the default dtype is configured before instantiation.

```python
from model import ModelArgs, Transformer
import torch

# Configure the 671B parameter model
args = ModelArgs(
    vocab_size=129280,
    dim=7168,
    inter_dim=18432,
    moe_inter_dim=2048,
    n_layers=61,
    n_dense_layers=3,
    n_heads=128,
    n_routed_experts=256,
    n_shared_experts=1,
    n_activated_experts=8,
    n_expert_groups=8,
    n_limited_groups=4,
    route_scale=2.5,
    score_func="sigmoid",
    q_lora_rank=1536,
    kv_lora_rank=512,
    qk_nope_head_dim=128,
    qk_rope_head_dim=64,
    v_head_dim=128,
    dtype="fp8",              # Use FP8 precision
    max_batch_size=8,
    max_seq_len=16384
)

# Initialize the model
torch.set_default_dtype(torch.bfloat16)
torch.set_default_device("cuda")
model = Transformer(args)

# Forward pass with token IDs
tokens = torch.randint(0, args.vocab_size, (2, 128))
logits = model(tokens)
print(f"Output shape: {logits.size()}")  # Output shape: torch.Size([2, 129280])
```

--------------------------------

### Convert HuggingFace weights to 16-way model parallel format

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Use this script to convert HuggingFace model weights to a 16-way model parallel format. Specify the input HuggingFace checkpoint path, the desired save path, the number of experts, and the model parallel degree.

```bash
python convert.py \
    --hf-ckpt-path /path/to/DeepSeek-V3 \
    --save-path /path/to/DeepSeek-V3-Demo \
    --n-experts 256 \
    --model-parallel 16
```

```bash
python convert.py \
    --hf-ckpt-path /path/to/DeepSeek-V3 \
    --save-path /path/to/DeepSeek-V3-Demo-8GPU \
    --n-experts 256 \
    --model-parallel 8
```

--------------------------------

### FP8 quantization kernels for activations

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

This Python code demonstrates the use of optimized Triton kernels for quantizing activations to FP8 with block-wise scaling. Ensure PyTorch and CUDA are set up correctly.

```python
from kernel import act_quant, weight_dequant, fp8_gemm
import torch

torch.set_default_dtype(torch.bfloat16)
torch.set_default_device("cuda")

# Quantize activations to FP8 with block-wise scaling
activations = torch.randn(4, 2048, 7168)  # batch, seq, hidden
fp8_acts, scales = act_quant(activations, block_size=128)
print(f"Quantized dtype: {fp8_acts.dtype}")  # torch.float8_e4m3fn
print(f"Scales shape: {scales.shape}")       # torch.Size([4, 2048, 56])
```

--------------------------------

### DeepSeek API Streaming Chat Completion with OpenAI Client

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Shows how to enable streaming responses from the DeepSeek API using the OpenAI client. This is useful for real-time interaction and displaying content as it's generated.

```python
from openai import OpenAI

# Initialize client with DeepSeek API
client = OpenAI(
    api_key="your-deepseek-api-key",
    base_url="https://api.deepseek.com"
)

# Streaming response
stream = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

--------------------------------

### Clone DeepSeek-V3 Repository

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Clone the official DeepSeek-V3 GitHub repository to access the demo code and scripts.

```shell
git clone https://github.com/deepseek-ai/DeepSeek-V3.git
```

--------------------------------

### Convert FP8 Weights to BF16

Source: https://github.com/deepseek-ai/deepseek-v3/blob/main/README.md

Use this script to convert FP8 model weights to BF16 format if needed for experimentation. Ensure you have the correct paths for input and output weights.

```shell
cd inference
python fp8_cast_bf16.py --input-fp8-hf-path /path/to/fp8_weights --output-bf16-hf-path /path/to/bf16_weights
```

--------------------------------

### FP8 matrix multiplication (GEMM) with scaling

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

This Python code demonstrates FP8 matrix multiplication (GEMM) using optimized Triton kernels, including handling of scaling factors for both input matrices. Ensure PyTorch and CUDA are configured.

```python
# FP8 matrix multiplication with scaling
a = torch.randn(32, 7168, dtype=torch.float8_e4m3fn)
a_scale = torch.ones(32, 56, dtype=torch.float32)
b = torch.randn(18432, 7168, dtype=torch.float8_e4m3fn)
b_scale = torch.ones(144, 56, dtype=torch.float32)

result = fp8_gemm(a, a_scale, b, b_scale)
print(f"GEMM result shape: {result.shape}")  # torch.Size([32, 18432])
```

--------------------------------

### Programmatic weight conversion with dequantization

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

This Python function allows programmatic conversion of FP8 weights to BF16, including dequantization. It requires the 'fp8_cast_bf16' module and optionally the 'kernel' module for manual dequantization.

```python
# Programmatic weight conversion with dequantization
from fp8_cast_bf16 import main as convert_fp8_to_bf16
from kernel import weight_dequant
import torch

# Full conversion
convert_fp8_to_bf16(
    fp8_path="/path/to/fp8_weights",
    bf16_path="/path/to/bf16_weights"
)

# Manual dequantization of individual tensors
fp8_weight = torch.randn(4096, 4096, dtype=torch.float8_e4m3fn, device="cuda")
scale_inv = torch.ones(32, 32, dtype=torch.float32, device="cuda")  # 128x128 blocks

bf16_weight = weight_dequant(fp8_weight, scale_inv, block_size=128)
print(f"Converted shape: {bf16_weight.shape}, dtype: {bf16_weight.dtype}")
```

--------------------------------

### Convert FP8 weights to BF16 format

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Use this script to dequantize FP8 weights to BF16 format. This is useful for frameworks that do not support FP8 inference. Specify the input FP8 weights path and the output BF16 weights path.

```bash
# Convert FP8 weights to BF16
python fp8_cast_bf16.py \
    --input-fp8-hf-path /path/to/fp8_weights \
    --output-bf16-hf-path /path/to/bf16_weights
```

--------------------------------

### FP8 dequantization kernels for weights

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

This Python code shows how to dequantize FP8 weights back to BF16 using optimized Triton kernels. It requires the 'kernel' module and PyTorch with CUDA support.

```python
# Dequantize FP8 weights back to BF16
fp8_weights = torch.randn(18432, 7168, dtype=torch.float8_e4m3fn)
weight_scales = torch.ones(144, 56, dtype=torch.float32)  # ceil(18432/128), ceil(7168/128)
bf16_weights = weight_dequant(fp8_weights, weight_scales, block_size=128)
print(f"Dequantized dtype: {bf16_weights.dtype}")  # torch.bfloat16
```

--------------------------------

### Generate Text Completions

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Perform autoregressive generation using the generate function with support for chat templates and batch processing. Requires a pre-loaded model and tokenizer.

```python
from generate import generate, sample
from model import Transformer, ModelArgs
from transformers import AutoTokenizer
import torch
import json

# Load model configuration
with open("configs/config_671B.json") as f:
    args = ModelArgs(**json.load(f))

# Initialize model and tokenizer
torch.set_default_dtype(torch.bfloat16)
model = Transformer(args)
tokenizer = AutoTokenizer.from_pretrained("/path/to/DeepSeek-V3-Demo")

# Single prompt generation
prompt = "Explain quantum computing in simple terms:"
messages = [{"role": "user", "content": prompt}]
prompt_tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

completion_tokens = generate(
    model=model,
    prompt_tokens=[prompt_tokens],
    max_new_tokens=200,
    eos_id=tokenizer.eos_token_id,
    temperature=0.7
)

completion = tokenizer.decode(completion_tokens[0], skip_special_tokens=True)
print(f"Response: {completion}")

# Batch generation with multiple prompts
prompts = [
    "What is machine learning?",
    "Explain neural networks briefly.",
    "What is deep learning?"
]
batch_tokens = [
    tokenizer.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True)
    for p in prompts
]

batch_completions = generate(
    model=model,
    prompt_tokens=batch_tokens,
    max_new_tokens=150,
    eos_id=tokenizer.eos_token_id,
    temperature=0.5
)

for prompt, tokens in zip(prompts, batch_completions):
    print(f"Q: {prompt}")
    print(f"A: {tokenizer.decode(tokens, skip_special_tokens=True)}\n")
```

--------------------------------

### Programmatic usage of weight conversion

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

This Python function provides programmatic control over the weight conversion process. Ensure the 'convert' module is available and pass the necessary paths and parameters.

```python
from convert import main as convert_weights

convert_weights(
    hf_ckpt_path="/path/to/DeepSeek-V3",
    save_path="/path/to/DeepSeek-V3-Demo",
    n_experts=256,
    mp=16  # Number of model parallel shards
)
# Output files: model0-mp16.safetensors, model1-mp16.safetensors, ...
```

--------------------------------

### Single node with 8 GPUs inference

Source: https://context7.com/deepseek-ai/deepseek-v3/llms.txt

Run inference on a single node with multiple GPUs using torchrun. This command is suitable for testing or smaller-scale deployments. Ensure the --standalone flag is used for single-node execution.

```bash
# Single node with 8 GPUs
torchrun \
    --standalone \
    --nproc-per-node 8 \
    generate.py \
    --ckpt-path /path/to/DeepSeek-V3-Demo \
    --config configs/config_671B.json \
    --interactive
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.