Jetson AI Lab Models

https://www.jetson-ai-lab.com/models
Admin
Jetson AI Lab Models is a curated collection of generative AI models optimized to run on NVIDIA...
Tokens:14,443
Snippets:128
Trust Score:9.4
Update:9 months ago
Show doc for...
Tokens:
Raw
### Install and Run Gemma 3 1B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/gemma3-1b

This snippet shows how to install Ollama and run the Gemma 3 1B model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a stable internet connection for the installation script.

```bash
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run gemma3:1b
```

--------------------------------

### Run Qwen3 VL 4B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-vl-4b

This command deploys the Qwen3 VL 4B Instruct AWQ 4-bit model using vLLM on a Jetson Orin device. It requires Docker to be installed and the NVIDIA container runtime enabled. The command pulls the latest vLLM image for Jetson Orin and starts the model server.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve cpatonn/Qwen3-VL-4B-Instruct-AWQ-4bit
```

--------------------------------

### Install and Run Llama 3.1 70B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/llama3-1-70b

This snippet shows how to install Ollama and then run the Llama 3.1 70B model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a stable internet connection for the download.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run llama3.1:70b
```

--------------------------------

### Run Qwen3 VL 4B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-vl-4b

This command deploys the Qwen3 VL 4B Instruct AWQ 4-bit model using vLLM on a Jetson Thor device. It requires Docker to be installed and the NVIDIA container runtime enabled. The command pulls the latest vLLM image for Jetson Thor and starts the model server.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve cpatonn/Qwen3-VL-4B-Instruct-AWQ-4bit
```

--------------------------------

### Running FunctionGemma with llama.cpp on Jetson (Docker)

Source: https://www.jetson-ai-lab.com/models/functiongemma

Provides the Docker run command to start the llama.cpp server with FunctionGemma on Jetson Orin. This command specifies the NVIDIA runtime, network host, the llama_cpp image, and arguments for the llama-server to load the FunctionGemma model.

```bash
sudo docker run -it --rm --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin llama-server --jinja -fa on -hf ggml-org/functiongemma-270m-it-GGUF --alias functiongemma
```

--------------------------------

### Install and Run Ministral 3 3B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/ministral3-3b-reasoning

This snippet shows how to install Ollama and run the Ministral 3 3B model. Ollama simplifies the process of downloading and running large language models locally. This is suitable for users who want a quick setup on their Jetson device.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run ministral-3:3b
```

--------------------------------

### Install and Run Ministral 3 8B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/ministral3-8b-reasoning

This snippet shows how to install Ollama and then run the Ministral 3 8B model. Ollama simplifies the process of downloading and running large language models locally.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run ministral-3:8b
```

--------------------------------

### Install and Run Gemma 3 270M with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/gemma3-270m

This snippet details the installation of Ollama and the subsequent command to run the Gemma 3 270M model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a compatible Jetson environment.

```bash
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run gemma3:270m
```

--------------------------------

### Install and Run Gemma 3 12B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/gemma3-12b

This snippet shows how to install Ollama and then run the Gemma 3 12B model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a stable internet connection for the download.

```bash
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run gemma3:12b
```

--------------------------------

### Install and Run Gemma 3 27B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/gemma3-27b

This snippet shows how to install Ollama and then run the Gemma 3 27B model. Ollama simplifies the process of downloading and running large language models locally. This is suitable for Jetson Orin and Jetson Thor.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run gemma3:27b
```

--------------------------------

### Run Nemotron 3 Nano 30B-A3B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/nemotron-nano-30b-a3b

This command deploys the NVIDIA Nemotron 3 Nano 30B-A3B model using vLLM on a Jetson Thor platform. It downloads the reasoning parser script and starts the vLLM server with specific configurations for tool choice, reasoning parser, and KV cache data type.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor bash -c "wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8/resolve/main/nano_v3_reasoning_parser.py && vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser-plugin nano_v3_reasoning_parser.py --reasoning-parser nano_v3 --kv-cache-dtype fp8"
```

--------------------------------

### Install and Run Llama 3.1 8B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/llama3-1-8b

This snippet shows how to install Ollama and then run the Llama 3.1 8B model. Ollama simplifies the process of downloading and running large language models locally. This is suitable for general use on Jetson devices.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run llama3.1:8b
```

--------------------------------

### Install and Run Ministral 3 14B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/ministral3-14b-reasoning

This snippet shows how to install Ollama and then run the Ministral 3 14B model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a stable internet connection for the download.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run ministral-3:14b
```

--------------------------------

### Install and Run Gemma 3 4B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/gemma3-4b

This snippet shows how to install Ollama and run the Gemma 3 4B model. Ollama simplifies the process of downloading and running large language models locally. Ensure you have a stable internet connection for the download.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run gemma3:4b
```

--------------------------------

### Parallel Tool Calling with FunctionGemma (cURL)

Source: https://www.jetson-ai-lab.com/models/functiongemma

Illustrates how to enable and utilize parallel tool calling with the FunctionGemma model. This example shows the addition of the 'parallel_tool_calls: true' flag in the request payload for handling multiple commands simultaneously.

```bash
curl http://localhost:8080/v1/chat/completions -d '{
    "model": "functiongemma",
    "parallel_tool_calls": true,
    "messages": [
        {"role": "user", "content": "Turn on the living room lights and set the temperature to 70"}
    ],
    "tools": [...] 
}'
```

--------------------------------

### Install and Run Llama 3.2 3B with Ollama on Jetson

Source: https://www.jetson-ai-lab.com/models/llama3-2-3b

This snippet shows how to install Ollama and then run the Llama 3.2 3B model. Ollama simplifies the process of downloading and running large language models locally. This is suitable for Jetson Orin and Jetson Thor devices.

```shell
# Installation
curl -fsSL https://ollama.ai/install.sh | sh

# Run Command
ollama run llama3.2:3b
```

--------------------------------

### Run Nemotron Nano 12B VL with Docker

Source: https://www.jetson-ai-lab.com/models/nemotron-nano-12b-vl

This command demonstrates how to run the Nemotron Nano 12B VL model using Docker on a Jetson Thor platform. It pulls the latest image and serves the specified model. Ensure you have Docker and the NVIDIA Container Toolkit installed.

```shell
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD
```

--------------------------------

### Run Llama 3.1 70B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/llama3-1-70b

This command deploys the Llama 3.1 70B Instruct model using vLLM on a Jetson Orin device. It utilizes a Docker container and requires the NVIDIA Container Toolkit to be installed. The command pulls the latest vLLM image for Jetson Orin and serves the specified quantized model.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Meta-Llama-3.1-70B-Instruct-quantized.w4a16
```

--------------------------------

### Run Ministral 3 8B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/ministral3-8b-reasoning

This command shows how to serve the Ministral 3 8B Reasoning model using vLLM on a Jetson Thor device. This setup leverages NVIDIA's containerized vLLM for optimized inference.

```shell
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-8B-Reasoning-2512
```

--------------------------------

### Run Llama 3.1 70B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/llama3-1-70b

This command deploys the Llama 3.1 70B Instruct model using vLLM on a Jetson Thor device. It utilizes a Docker container and requires the NVIDIA Container Toolkit to be installed. The command pulls the latest vLLM image for Jetson Thor and serves the specified quantized model.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Meta-Llama-3.1-70B-Instruct-quantized.w4a16
```

--------------------------------

### Run Gemma 3 4B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/gemma3-4b

This command deploys the Gemma 3 4B model using vLLM on a Jetson Orin device. vLLM is an efficient inference engine for LLMs. This requires Docker and NVIDIA Container Runtime to be installed.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/gemma-3-4b-it-quantized.w4a16
```

--------------------------------

### Run Gemma 3 4B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/gemma3-4b

This command deploys the Gemma 3 4B model using vLLM on a Jetson Thor device. vLLM is an efficient inference engine for LLMs. This requires Docker and NVIDIA Container Runtime to be installed.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/gemma-3-4b-it-quantized.w4a16
```

--------------------------------

### Run Gemma 3 12B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/gemma3-12b

This command deploys the Gemma 3 12B model using vLLM on a Jetson Orin device. vLLM is an efficient inference engine for LLMs. This command assumes you have the necessary NVIDIA container runtime and Docker installed.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/gemma-3-12b-it-quantized.w4a16
```

--------------------------------

### Run Ministral 3 14B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/ministral3-14b-reasoning

This command deploys the Ministral 3 14B Reasoning model using vLLM on a Jetson Thor device. It leverages Docker for a reproducible setup and requires the NVIDIA container runtime for GPU access. This is suitable for more powerful Jetson configurations.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-14B-Reasoning-2512
```

--------------------------------

### Run Llama 3.2 3B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/llama3-2-3b

This command deploys the Llama 3.2 3B model using the vLLM inference engine within a Docker container specifically for Jetson Thor. This setup leverages vLLM's performance optimizations for LLM inference.

```docker
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve espressor/meta-llama.Llama-3.2-3B-Instruct_W4A16
```

--------------------------------

### Run Llama 3.1 8B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/llama3-1-8b

This command shows how to deploy the Llama 3.1 8B model using vLLM on Jetson Thor devices. It utilizes a specific Docker image optimized for Jetson Thor, ensuring efficient inference. This setup is ideal for high-performance inference tasks.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16
```

--------------------------------

### Run Qwen3 32B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-32b

This command deploys the Qwen3 32B model quantized to W4A16 precision on a Jetson Orin device using the vLLM inference engine. It requires Docker and the NVIDIA Container Runtime. The model is served for inference.

```docker
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Qwen3-32B-quantized.w4a16
```

--------------------------------

### Run Ministral 3 3B Instruct on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-3b-instruct

This command executes the Ministral 3 3B Instruct model using vLLM on a Jetson Orin device. It requires Docker and pulls the latest Jetson Orin compatible vLLM image. The model is served directly after container startup.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-3B-Instruct-2512
```

--------------------------------

### Run Ministral 3 8B Instruct on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-8b-instruct

This command deploys the Ministral 3 8B Instruct model using vLLM on a Jetson Orin device. It requires Docker and the NVIDIA Container Runtime. The model is served via a network host connection.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-8B-Instruct-2512
```

--------------------------------

### Run Ministral 3 8B Instruct on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-8b-instruct

This command deploys the Ministral 3 8B Instruct model using vLLM on a Jetson Thor device. It requires Docker and the NVIDIA Container Runtime. The model is served via a network host connection.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-8B-Instruct-2512
```

--------------------------------

### Run Qwen3 32B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-32b

This command deploys the Qwen3 32B model quantized to W4A16 precision on a Jetson Thor device using the vLLM inference engine. It requires Docker and the NVIDIA Container Runtime. The model is served for inference.

```docker
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Qwen3-32B-quantized.w4a16
```

--------------------------------

### Run Ministral 3 3B Instruct on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-3b-instruct

This command deploys the Ministral 3 3B Instruct model using vLLM on a Jetson Thor device. It utilizes a Docker container with the latest Jetson Thor vLLM image. The model is accessible after the container is running.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-3B-Instruct-2512
```

--------------------------------

### Run Ministral 3 14B Instruct on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-14b-instruct

This command deploys the Ministral 3 14B Instruct model using vLLM on a Jetson Orin device. It requires Docker and the NVIDIA container runtime. The model is pulled from the specified container registry.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-14B-Instruct-2512
```

--------------------------------

### Deploy Qwen3 30B-A3B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-30b-a3b

This command deploys the Qwen3 30B-A3B model using vLLM on a Jetson Orin device. It requires Docker and the specified vLLM container image. The command pulls the latest image and serves the quantized model.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Qwen3-30B-A3B-quantized.w4a16
```

--------------------------------

### Run Qwen3 VL 8B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-vl-8b

This command deploys the Qwen3 VL 8B Instruct AWQ 4-bit model using vLLM on a Jetson Orin device. It utilizes a Docker container with NVIDIA's runtime and network configuration.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve cpatonn/Qwen3-VL-8B-Instruct-AWQ-4bit
```

--------------------------------

### Run Ministral 3 14B Instruct on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/ministral3-14b-instruct

This command deploys the Ministral 3 14B Instruct model using vLLM on a Jetson Thor device. It requires Docker and the NVIDIA container runtime. The model is pulled from the specified container registry.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-14B-Instruct-2512
```

--------------------------------

### Run Qwen3 4B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-4b

This command deploys the Qwen3 4B model using vLLM on a Jetson Orin device. It utilizes a Docker container, ensuring a consistent environment. The model is served from the RedHatAI/Qwen3-4B-quantized.w4a16 repository.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Qwen3-4B-quantized.w4a16
```

--------------------------------

### Deploy Qwen3 30B-A3B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-30b-a3b

This command deploys the Qwen3 30B-A3B model using vLLM on a Jetson Thor device. It requires Docker and the specified vLLM container image. The command pulls the latest image and serves the quantized model.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Qwen3-30B-A3B-quantized.w4a16
```

--------------------------------

### Run Qwen3 4B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-4b

This command deploys the Qwen3 4B model using vLLM on a Jetson Thor device. It leverages a Docker container for environment consistency and serves the model from the RedHatAI/Qwen3-4B-quantized.w4a16 repository.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Qwen3-4B-quantized.w4a16
```

--------------------------------

### Run Qwen3 VL 8B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-vl-8b

This command deploys the Qwen3 VL 8B Instruct AWQ 4-bit model using vLLM on a Jetson Thor device. It leverages a Docker container with NVIDIA's runtime and network configuration.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve cpatonn/Qwen3-VL-8B-Instruct-AWQ-4bit
```

--------------------------------

### Run Nemotron Nano 9B v2 with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/nemotron-nano-9b-v2

This command deploys the NVIDIA Nemotron Nano 9B v2 model using vLLM on a Jetson Thor platform. It requires Docker and pulls the latest vLLM image for Jetson Thor. The model is served using its NVFP4 quantized version.

```shell
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4
```

--------------------------------

### Basic Function Calling with FunctionGemma (cURL)

Source: https://www.jetson-ai-lab.com/models/functiongemma

Demonstrates how to perform a basic function call using the FunctionGemma model via cURL. It includes system and user messages, along with a tool definition for 'get_current_weather'. The output is expected to be a structured function call.

```bash
curl http://localhost:8080/v1/chat/completions -d '{
    "model": "functiongemma",
    "messages": [
        {"role": "system", "content": "You are a chatbot that uses tools/functions. Dont overthink things."},
        {"role": "user", "content": "What is the weather in Istanbul?"}
    ],
    "tools": [{
        "type":"function",
        "function":{
            "name":"get_current_weather",
            "description":"Get the current weather in a given location",
            "parameters":{
                "type":"object",
                "properties":{
                    "location":{
                        "type":"string",
                        "description":"The city and country/state, e.g. `San Francisco, CA`, or `Paris, France`"
                    }
                },
                "required":["location"]
            }
        }
    }]
}'
```

--------------------------------

### Deploy Qwen3 8B on Jetson Orin with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-8b

This command deploys the Qwen3 8B model quantized to W4A16 precision using the vLLM inference engine on a Jetson Orin device. It utilizes a Docker container and requires NVIDIA's container runtime.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Qwen3-8B-quantized.w4a16
```

--------------------------------

### Deploy Gemma 3 1B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/gemma3-1b

This command deploys the Gemma 3 1B model using the vLLM inference engine specifically for the Jetson Orin platform. It utilizes Docker for containerization, ensuring a consistent environment. The `--runtime=nvidia` flag is crucial for GPU acceleration.

```bash
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve google/gemma-3-1b-it
```

--------------------------------

### Run Ministral 3 3B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/ministral3-3b-reasoning

This command demonstrates how to serve the Ministral 3 3B Reasoning model using vLLM on a Jetson Orin platform. vLLM is an efficient inference engine for large language models, optimized for high throughput. This method is recommended for performance-critical applications.

```bash
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-3B-Reasoning-2512
```

--------------------------------

### Run Gemma 3 270M with vLLM Container on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/gemma3-270m

This command demonstrates how to deploy the Gemma 3 270M model using a vLLM container specifically for the Jetson Orin platform. It utilizes Docker with NVIDIA's runtime for GPU acceleration. This method is suitable for environments where containerization is preferred.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve google/gemma-3-270m-it
```

--------------------------------

### Deploy Qwen3 8B on Jetson Thor with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-8b

This command deploys the Qwen3 8B model quantized to W4A16 precision using the vLLM inference engine on a Jetson Thor device. It utilizes a Docker container and requires NVIDIA's container runtime.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/Qwen3-8B-quantized.w4a16
```

--------------------------------

### Run Gemma 3 27B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/gemma3-27b

This command deploys the Gemma 3 27B model using the vLLM inference engine on a Jetson Orin device. It requires Docker and pulls the necessary container image. The model is served for inference.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/gemma-3-27b-it-quantized.w4a16
```

--------------------------------

### Run Ministral 3 3B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/ministral3-3b-reasoning

This command shows how to deploy the Ministral 3 3B Reasoning model using vLLM on a Jetson Thor platform. Similar to the Jetson Orin deployment, this leverages vLLM for efficient inference. Ensure your Jetson Thor environment is correctly set up with Docker and NVIDIA drivers.

```bash
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve mistralai/Ministral-3-3B-Reasoning-2512
```

--------------------------------

### Deploy Gemma 3 1B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/gemma3-1b

This command deploys the Gemma 3 1B model using the vLLM inference engine tailored for the Jetson Thor platform. It leverages Docker for environment consistency and requires the NVIDIA runtime for GPU utilization. This is suitable for users with Jetson Thor hardware.

```bash
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve google/gemma-3-1b-it
```

--------------------------------

### Run Ministral 3 8B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/ministral3-8b-reasoning

This command demonstrates how to serve the Ministral 3 8B Reasoning model using vLLM on a Jetson Orin device. vLLM is an efficient inference engine for large language models.

```shell
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-8B-Reasoning-2512
```

--------------------------------

### Run Ministral 3 14B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/ministral3-14b-reasoning

This command deploys the Ministral 3 14B Reasoning model using vLLM on a Jetson Orin device. It utilizes Docker for containerization, ensuring a consistent environment. The `--runtime=nvidia` flag is crucial for GPU acceleration.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve mistralai/Ministral-3-14B-Reasoning-2512
```

--------------------------------

### Run Gemma 3 270M with vLLM Container on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/gemma3-270m

This snippet shows the command to run the Gemma 3 270M model using a vLLM container on the Jetson Thor platform. Similar to the Orin deployment, it leverages Docker and NVIDIA's runtime for efficient inference on the Jetson hardware. This is ideal for users who need a containerized solution for Jetson Thor.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve google/gemma-3-270m-it
```

--------------------------------

### Run Gemma 3 27B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/gemma3-27b

This command deploys the Gemma 3 27B model using the vLLM inference engine on a Jetson Thor device. It requires Docker and pulls the necessary container image. The model is served for inference.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/gemma-3-27b-it-quantized.w4a16
```

--------------------------------

### Run Gemma 3 12B with vLLM on Jetson Thor

Source: https://www.jetson-ai-lab.com/models/gemma3-12b

This command deploys the Gemma 3 12B model using vLLM on a Jetson Thor device. vLLM is optimized for high-throughput LLM serving. This command requires the NVIDIA container runtime and Docker to be set up.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor vllm serve RedHatAI/gemma-3-12b-it-quantized.w4a16
```

--------------------------------

### Run Llama 3.1 8B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/llama3-1-8b

This command demonstrates how to run the Llama 3.1 8B model using the vLLM inference engine specifically configured for Jetson Orin. vLLM is known for its high-throughput LLM serving capabilities. Ensure you have the correct Docker image for Jetson Orin.

```shell
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16
```

--------------------------------

### Run Llama 3.2 3B with vLLM on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/llama3-2-3b

This command deploys the Llama 3.2 3B model using the vLLM inference engine within a Docker container specifically for Jetson Orin. vLLM is known for its high-throughput LLM serving capabilities.

```docker
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve espressor/meta-llama.Llama-3.2-3B-Instruct_W4A16
```
Tokens:
Raw