### Install and Serve Model with vLLM

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/57b5051efa88e73f86be1e323c517628ac03272f

Install vLLM via pip and start the server. This enables an OpenAI-compatible API for the model.

```bash
pip install vllm
vllm serve "Nanbeige/Nanbeige4.1-3B"
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]'

```

--------------------------------

### Install and Serve Model with vLLM

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/28f4b2e2142194291ac8339791513585eb2c2ef2

Install vLLM and start a server to serve the model. The server provides an OpenAI-compatible API for chat completions.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.1-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		] 
	}'
```

--------------------------------

### Install and Serve Model with vLLM

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/special_tokens_map.json

Install vLLM via pip and start the server. Use curl to interact with the OpenAI-compatible API.

```bash
pip install vllm
vllm serve "Nanbeige/Nanbeige4.1-3B"
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ "model": "Nanbeige/Nanbeige4.1-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'
```

--------------------------------

### Install and Serve Nanbeige/Nanbeige4.1-3B with vLLM

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/generation_config.json

Installs vLLM and starts a server for the model. The server provides an OpenAI-compatible API for chat completions.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.1-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ 
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Install and Serve Model with SGLang

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/special_tokens_map.json

Install SGLang via pip and start the server. Interact with the OpenAI-compatible API using curl.

```bash
pip install sglang
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ "model": "Nanbeige/Nanbeige4.1-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'
```

--------------------------------

### Serve Model with vLLM

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/main/figures

Install vLLM and start a server to serve the model. This provides an OpenAI-compatible API endpoint. Ensure vLLM is installed via pip.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.1-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]'

```

--------------------------------

### Serve Model with SGLang

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/main/figures

Install SGLang and start a server for the model. This provides an OpenAI-compatible API. Ensure SGLang is installed via pip.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]'

```

--------------------------------

### Install and Serve Model with SGLang

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/57b5051efa88e73f86be1e323c517628ac03272f

Install SGLang via pip and start the server. This provides an OpenAI-compatible API. Ensure Python 3 is used.

```bash
pip install sglang
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]'

```

--------------------------------

### Install and Serve Nanbeige/Nanbeige4.1-3B with SGLang

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/generation_config.json

Installs SGLang and starts a server for the model. The server provides an OpenAI-compatible API for chat completions.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ 
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Serve Model with vLLM and Use OpenAI-Compatible API

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/config.json

Install vLLM and start the server. Then, use curl to send requests to the chat completions endpoint. Ensure vLLM is installed.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.1-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ "model": "Nanbeige/Nanbeige4.1-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'
```

--------------------------------

### Serve Model with SGLang and Use OpenAI-Compatible API

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/config.json

Install SGLang and start the server. Then, use curl to send requests to the chat completions endpoint. Ensure SGLang is installed.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{ "model": "Nanbeige/Nanbeige4.1-3B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'
```

--------------------------------

### Serve Model with vLLM and Use OpenAI-Compatible API

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/main/model-00001-of-00002.safetensors

Install vLLM, start the server, and interact using curl. Assumes OpenAI-compatible API endpoint.

```bash
pip install vllm
vllm serve "Nanbeige/Nanbeige4.1-3B"
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Serve Model with SGLang and Use OpenAI-Compatible API

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commits/main/model-00001-of-00002.safetensors

Install SGLang, start the server, and interact using curl. Assumes OpenAI-compatible API endpoint.

```bash
pip install sglang
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Install and Run with llmpm

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/discussions/41

Use the llmpm tool to easily install, serve, and manage Hugging Face models. Requires pip installation of llmpm.

```bash
pip install llmpm
llmpm install Nanbeige/Nanbeige4.1-3B
llmpm serve Nanbeige/Nanbeige4.1-3B
```

--------------------------------

### Serve Model with SGLang

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/model-00001-of-00002.safetensors

Install SGLang and start a server for model interaction. Use 'curl' to send requests to the OpenAI-compatible API. Requires Python 3.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Serve Nanbeige/Nanbeige4.1-3B with SGLang and Query via OpenAI-compatible API

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commit/d02c34c2f13f0b85174202eb4ce115ec2104ac9d

Install SGLang and start a server to host the model. Use curl to send chat completion requests to the local endpoint. Ensure necessary ports and volumes are mapped for Docker.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.1-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

```bash
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.1-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.1-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
```

--------------------------------

### Nanbeige4.1-3B Inference Quickstart

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commit/d02c34c2f13f0b85174202eb4ce115ec2104ac9d

Use this code to perform inference with the Nanbeige4.1-3B model for chat scenarios. Ensure you have the transformers library installed and the model downloaded.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
'Nanbeige/Nanbeige4.1-3B',
use_fast=False,
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
'Nanbeige/Nanbeige4.1-3B',
torch_dtype='auto',
device_map='auto',
trust_remote_code=True
)
messages = [
{'role': 'user', 'content': 'Which number is bigger, 9.11 or 9.8?'}
]
prompt = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=False
)
input_ids = tokenizer(prompt, add_special_tokens=False, return_tensors='pt').input_ids
output_ids = model.generate(input_ids.to('cuda'), eos_token_id=166101)
resp = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)
```

--------------------------------

### Run llama.cpp server with ROCm

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/discussions/35

This command starts a llama.cpp server using ROCm for GPU acceleration. It mounts a local models directory and configures server parameters like context size and temperature. Ensure llama.cpp and ROCm are correctly set up.

```bash
docker run -d --name llama-server --restart unless-stopped --privileged --network=host --device=/dev/kfd --device=/dev/dri --group-add video --group-add render --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --ipc=host --shm-size 16G -v ~/models:/data docker.io/mixa3607/llama.cpp-gfx906:full-b7091-rocm-6.3.3 --server -m /data/nanbeige4.1-3b-q8_0.gguf --host 0.0.0.0 --port 8080 -c 8192 -n 4096 --temp 0.6 --top-p 0.95 --top-k 0 -ngl 999
```

--------------------------------

### Run SGLang Server with Docker

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blame/main/.gitattributes

Launch the SGLang server using Docker, mounting necessary volumes and setting environment variables. Ensure you have a Hugging Face token if required.

```bash
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.1-3B" \
        --host 0.0.0.0 \
        --port 30000
```

--------------------------------

### Load Model for Chat Scenario with Transformers

Source: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/commit/16b326a87f38ae4696b2757749ff3177060d5dd4

Load tokenizer and model for chat interactions. Ensure transformers library is installed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
'Nanbeige/Nanbeige4.1-3B',
)
model = AutoModelForCausalLM.from_pretrained(
'Nanbeige/Nanbeige4.1-3B',
)
```