Jetson AI Lab

https://www.jetson-ai-lab.com/
Admin
Jetson AI Lab is NVIDIA's comprehensive resource platform for deploying generative AI models and...
Tokens:118,094
Snippets:1,573
Trust Score:9.4
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install OpenClaw and Ollama on Jetson Orin Nano

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

A single command to install Ollama, pull the Qwen 3.5 2B model, install OpenClaw, configure it for low memory, start the gateway, and send a test message. This is the fastest way to get started.

```bash
curl -fsSL https://raw.githubusercontent.com/NVIDIA-AI-IOT/jetson-ai-lab/main/public/code-samples/openclaw-orin-nano/setup-openclaw-orin-nano.py | python3
```

--------------------------------

### Start NVIDIA RIVA SDK

Source: https://www.jetson-ai-lab.com/tutorials/multi-modal-ai-studio

Initialize and start the NVIDIA RIVA SDK for ASR and TTS services. Ensure RIVA is installed and initialized according to the NVIDIA Riva setup instructions.

```bash
cd riva_quickstart_arm64_v2.24.0
bash riva_start.sh
```

--------------------------------

### Serve Qwen3.5 35B-A3B with vLLM on Jetson Thor (alternative)

Source: https://www.jetson-ai-lab.com/models/qwen3-5-35b-a3b

Alternative command to serve the Qwen3.5 35B-A3B model using vLLM on a Jetson Thor module. This command uses the NVFP4 precision model and includes reasoning and tool-call parsers, similar to the first example but potentially for a different setup.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve Kbenkhaled/Qwen3.5-35B-A3B-NVFP4 \
    --gpu-memory-utilization 0.8 --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

--------------------------------

### Install Live VLM WebUI with Docker

Source: https://www.jetson-ai-lab.com/tutorials/cosmos-reason2-vlm

Clone the repository and use the provided script to start the Live VLM WebUI in a Docker container.

```bash
git clone https://github.com/nvidia-ai-iot/live-vlm-webui.git
cd live-vlm-webui
./scripts/start_container.sh
```

--------------------------------

### Clone and Setup jetson-containers

Source: https://www.jetson-ai-lab.com/tutorial_nanoowl.html

Clone the jetson-containers repository and execute the installation script. This is a prerequisite for running NanoOWL.

```bash
git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh
```

--------------------------------

### Install Node.js and OpenClaw

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Install Node.js version 22+ and then install OpenClaw globally using npm. Verify installations by checking their versions.

```bash
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
node --version   # v22.x.x or higher
```

```bash
sudo npm install -g openclaw@latest
openclaw --version
```

--------------------------------

### Quick Hello World Example for Function Calling

Source: https://www.jetson-ai-lab.com/models/functiongemma

A basic cURL command demonstrating a function calling scenario. It includes system and user messages, and defines a tool for getting weather information.

```bash
curl http://localhost:8080/v1/chat/completions -d '{
    "model": "functiongemma",
    "messages": [
        {"role": "system", "content": "You are a chatbot that uses tools/functions. Dont overthink things."},
        {"role": "user", "content": "What is the weather in Istanbul?"}
    ],
    "tools": [{
        "type":"function",
        "function":{
            "name":"get_current_weather",
            "description":"Get the current weather in a given location",
            "parameters":{
                "type":"object",
                "properties":{
                    "location":{
                        "type":"string",
                        "description":"The city and country/state, e.g. `San Francisco, CA`, or `Paris, France`"
                    }
                },
                "required":["location"]
            }
        }
    }]
}'
```

--------------------------------

### Launch Multi-modal AI Studio

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/conversation-ai-multi-modal-ai-studio

Starts the Multi-modal AI Studio application, configuring the ASR, TTS, and LLM backend servers. Ensure the ports and model names match your setup.

```bash
multi-modal-ai-studio --port 8092 \
  --asr-server localhost:50051 \
  --tts-server localhost:50051 \
  --llm-api-base http://localhost:8010/v1 \
  --llm-model nvidia/cosmos-reason2-8b-fp8
```

--------------------------------

### Clone and Install Multi-Modal AI Studio

Source: https://www.jetson-ai-lab.com/tutorials/multi-modal-ai-studio

Clone the Multi-Modal AI Studio repository, set up a virtual environment, and install the application. This prepares the Studio for launch.

```bash
git clone https://github.com/NVIDIA-AI-IOT/multi_modal_ai_studio.git ~/multi_modal_ai_studio
cd ~/multi_modal_ai_studio

python3 -m venv .venv
source .venv/bin/activate

pip install -e .
```

--------------------------------

### Launch vLLM Server

Source: https://www.jetson-ai-lab.com/tutorials/workshop-gtc-dc-2025

Starts the vLLM inference server for a specified model. Startup may take a few minutes.

```bash
vllm serve openai/gpt-oss-120b
```

--------------------------------

### Install nvidia-container Package

Source: https://www.jetson-ai-lab.com/tips_ssd-docker.html

Install the `nvidia-container` package using apt. This step can be skipped if using an NVIDIA-supplied SD card image.

```bash
sudo apt update
sudo apt install -y nvidia-container
```

--------------------------------

### Example Prompts for Voice + Vision

Source: https://www.jetson-ai-lab.com/tutorials/multi-modal-ai-studio

Try these example prompts to test the multi-modal capabilities of the AI Studio, requiring both speech and vision input.

```text
What object am I holding?
```

```text
Describe what changed in the scene.
```

```text
Is there a person visible? Answer yes or no.
```

--------------------------------

### Install and Configure NGC CLI

Source: https://www.jetson-ai-lab.com/models/cosmos-reason2-8b

Installs the NGC CLI for downloading models from NVIDIA's NGC registry. Ensure you have an NGC account and API key.

```bash
wget -O ngccli_arm64.zip https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.13.0/files/ngccli_arm64.zip
unzip ngccli_arm64.zip && chmod u+x ngc-cli/ngc
export PATH="$PATH:$(pwd)/ngc-cli"
ngc config set
```

--------------------------------

### NemoClaw Onboarding - Accept License

Source: https://www.jetson-ai-lab.com/tutorials/nemoclaw

Provides the 'yes' input to accept the license agreement during the NemoClaw installation and onboarding process.

```bash
yes
```

--------------------------------

### Start vLLM Server with Cosmos-Reason2

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/vision-ai-live-vlm-webui

This command starts the vLLM inference server with the Cosmos-Reason2 model, making it available for API requests.

```APIDOC
## Start vLLM Server with Cosmos-Reason2

### Description
This command starts the vLLM inference server with the Cosmos-Reason2 model, making it available for API requests.

### Method
`docker run`

### Endpoint
N/A (Docker command)

### Parameters
#### Environment Variables
- **MODEL_PATH** (string) - Required - Path to the model weights.

#### Docker Arguments
- `--rm` - Automatically remove the container when it exits.
- `--runtime=nvidia` - Use the NVIDIA container runtime.
- `--network host` - Use the host's network stack.
- `-v ${HOME}/models/cosmos-reason2-8b-fp8:/models/cosmos-reason2-8b:ro` - Mount the model directory.
- `-v ${HOME}/.cache/vllm:/root/.cache/vllm` - Mount the vLLM cache directory.

#### Docker Image
`ghcr.io/nvidia-ai-iot/vllm:0.14.0-r38.3-arm64-sbsa-cu130-24.04`

#### vLLM Serve Arguments
- `vllm serve /models/cosmos-reason2-8b` - Path to the model to serve.
- `--served-model-name nvidia/cosmos-reason2-8b-fp8` - Name to serve the model as.
- `--max-model-len 8192` - Maximum sequence length.
- `--gpu-memory-utilization 0.7` - Fraction of GPU memory to use.
- `--reasoning-parser qwen3` - Specifies the reasoning parser.
- `--media-io-kwargs '{"video": {"num_frames": -1}}'` - Arguments for media I/O, enabling video frame processing.
- `--enable-prefix-caching` - Enable prefix caching for faster inference.
- `--port 8000` - Port to run the server on.

### Request Example
```bash
export MODEL_PATH="${HOME}/.cache/huggingface/hub/cosmos-reason2-8b_v1208-fp8-static-kv8"

sudo docker run -it --rm --runtime=nvidia --network host \
  -v ${HOME}/models/cosmos-reason2-8b-fp8:/models/cosmos-reason2-8b:ro \
  -v ${HOME}/.cache/vllm:/root/.cache/vllm \
  ghcr.io/nvidia-ai-iot/vllm:0.14.0-r38.3-arm64-sbsa-cu130-24.04 \
  vllm serve /models/cosmos-reason2-8b \
    --served-model-name nvidia/cosmos-reason2-8b-fp8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 \
    --media-io-kwargs '{"video": {"num_frames": -1}}' \
    --enable-prefix-caching \
    --port 8000
```

### Response
#### Success Response
Server starts and indicates it is running on http://0.0.0.0:8000.
```
INFO:     Started server process
INFO:     Uvicorn running on http://0.0.0.0:8000
```
```

--------------------------------

### Install SDK Manager on Ubuntu 20.04

Source: https://www.jetson-ai-lab.com/tutorials/initial-setup-sdk-manager

Use these commands to add the NVIDIA CUDA repository and install the SDK Manager on Ubuntu 20.04.

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install sdkmanager
```

--------------------------------

### Custom Dataset Example

Source: https://www.jetson-ai-lab.com/tutorials/finetune-on-jetson

Prepare your dataset in JSON Lines format with 'instruction', 'input', and 'output' fields. This structure is used for supervised fine-tuning.

```json
[
  {
    "instruction": "Classify the sentiment of this review.",
    "input": "The battery life is amazing and the screen is crystal clear.",
    "output": "Positive"
  },
  {
    "instruction": "Extract the part number from this text.",
    "input": "Please ship 50 units of PN-4820-X to warehouse B.",
    "output": "PN-4820-X"
  },
  {
    "instruction": "Translate the following to Spanish.",
    "input": "The system is operating normally.",
    "output": "El sistema está funcionando normalmente."
  }
]
```

--------------------------------

### Serve FP16 Model (Baseline)

Source: https://www.jetson-ai-lab.com/tutorials/workshop-gtc-dc-2025

Starts the vLLM server with a model using FP16 precision. This serves as the baseline for performance and quality comparisons.

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct
```

--------------------------------

### Verify TensorRT-Edge LLM Build

Source: https://www.jetson-ai-lab.com/tutorials/tensorrt-edge-llm

Run help commands for the LLM build and inference examples to verify the build was successful.

```bash
cd ~/TensorRT-Edge-LLM
./build/examples/llm/llm_build --help
./build/examples/llm/llm_inference --help
```

--------------------------------

### Ollama CLI Usage

Source: https://www.jetson-ai-lab.com/tutorials/ollama

Run this command to interact with the Ollama client after installation.

```bash
ollama
```

--------------------------------

### Install TensorRT-Edge LLM Inside Container (x86)

Source: https://www.jetson-ai-lab.com/tutorials/tensorrt-edge-llm

After entering the container, clone the repository, initialize submodules, create a virtual environment, and install the Edge-LLM package and its dependencies.

```bash
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git submodule update --init --recursive

python3 -m venv venv
source venv/bin/activate
pip3 install .
```

--------------------------------

### Clone and Start Live VLM WebUI Container

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/vision-ai-live-vlm-webui

Clone the Live VLM WebUI repository and start the Docker container. Ensure vLLM is running in a separate terminal before executing this script.

```bash
git clone https://github.com/NVIDIA-AI-IOT/live-vlm-webui.git ~/live-vlm-webui
cd ~/live-vlm-webui
./scripts/start_container.sh
```

--------------------------------

### Activate Environment and Serve Model

Source: https://www.jetson-ai-lab.com/tutorials/cosmos-reason2-vlm

Inside the container, activate the virtual environment and use the `vllm serve` command to start the model server. Configure parameters like max model length, media I/O, reasoning parser, and GPU memory utilization.

```bash
cd /opt/ 
source venv/bin/activate

vllm serve /models/cosmos-reason2 \
  --max-model-len 8192 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.8
```

--------------------------------

### Serve Nemotron3 Nano 4B with llama.cpp

Source: https://www.jetson-ai-lab.com/models/nemotron3-nano-4b

Use this command to start the llama.cpp server for the Nemotron3 Nano 4B model. Ensure the Docker image is pulled and the necessary volumes are mounted.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-thor \
  llama-server \
    --hf-repo nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF \
    --hf-file NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
    --ctx-size 8196 \
    --alias my_model \
    --n-gpu-layers 999
```

--------------------------------

### Clone and Set Up Multi-modal AI Studio

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/conversation-ai-multi-modal-ai-studio

Clones the Multi-modal AI Studio repository, creates a virtual environment, and installs the package in development mode. Activate the virtual environment before proceeding.

```bash
git clone https://github.com/NVIDIA-AI-IOT/multi_modal_ai_studio.git ~/multi_modal_ai_studio
cd ~/multi_modal_ai_studio

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

# Install in development mode
pip install -e .
```

--------------------------------

### Set Jetson to MAXN Mode

Source: https://www.jetson-ai-lab.com/tutorials/genai-benchmarking

Ensures the Jetson is in its highest performance mode for benchmarking. This is a prerequisite before starting the setup.

```bash
sudo nvpmodel -m 0
```

--------------------------------

### Serve MiniMax M2.7 with llama.cpp

Source: https://www.jetson-ai-lab.com/models/minimax-m2-7

Use this command to start the llama.cpp server for the MiniMax M2.7 model. Ensure you have the correct Docker image and model file path. Adjust --port and --alias as needed.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-thor \
  llama-server \
    --hf-repo unsloth/MiniMax-M2.7-GGUF \
    --hf-file UD-IQ4_XS/MiniMax-M2.7-UD-IQ4_XS-00001-of-00004.gguf \
    --ctx-size 4096 \
    --port 8080 \
    --alias my_model \
    --n-gpu-layers 999
```

--------------------------------

### Install and Verify OpenClaw

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Install the latest version of OpenClaw globally using npm and verify the installation by checking the installed version.

```bash
sudo npm install -g openclaw@latest
openclaw --version
```

--------------------------------

### Run OpenClaw Onboarding Wizard

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Initiate the OpenClaw onboarding wizard to configure model providers, gateways, and channels. The `--skip-daemon` flag is used to avoid potential issues with systemd on headless sessions.

```bash
openclaw onboard --skip-daemon
```

--------------------------------

### Install NemoClaw

Source: https://www.jetson-ai-lab.com/tutorials/nemoclaw

Installs NemoClaw by downloading and executing an installation script. This process may require administrator privileges to enable kernel modules.

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
```

--------------------------------

### Start vLLM with Cosmos-Reason2

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/conversation-ai-multi-modal-ai-studio

Launches the vLLM service with the Cosmos-Reason2 model, exposing it on a different port than Riva to avoid conflicts. Ensure the model path and cache directories are correctly set.

```bash
sudo sysctl -w vm.drop_caches=3

sudo docker run -it --rm --runtime=nvidia --network host \
  -v ~/models/cosmos-reason2-8b:/models/cosmos-reason2-8b:ro \
  -v ${HOME}/.cache/vllm:/root/.cache/vllm \
  ghcr.io/nvidia-ai-iot/vllm:0.14.0-r38.3-arm64-sbsa-cu130-24.04 \
  vllm serve /models/cosmos-reason2-8b \
    --served-model-name nvidia/cosmos-reason2-8b-fp8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 \
    --media-io-kwargs '{"video": {"num_frames": -1}}' \
    --enable-prefix-caching \
    --port 8010
```

--------------------------------

### Install QSPI Updater Package

Source: https://www.jetson-ai-lab.com/tutorials/initial-setup-jetson-orin-nano

Installs the necessary package to update the Quad SPI (QSPI) flash memory. An internet connection is required for this installation.

```bash
sudo apt-get install nvidia-l4t-jetson-orin-nano-qspi-updater
```

--------------------------------

### Run QLoRA Fine-tuning

Source: https://www.jetson-ai-lab.com/tutorials/finetune-on-jetson

Execute the QLoRA fine-tuning script. This command starts the fine-tuning process for the Qwen3.5 27B model using QLoRA, saving the adapters to the specified output directory.

```bash
python qlora_finetuning.py --output_dir ./qlora_adapter
```

--------------------------------

### Install Missing Python Packages

Source: https://www.jetson-ai-lab.com/tutorials/openpi_on_thor

Installs the 'chex' and 'toolz' Python packages without their dependencies to avoid conflicts with existing JAX/NumPy installations. The Dockerfile already includes these.

```bash
pip install chex --no-deps && pip install toolz --no-deps
```

--------------------------------

### Install or Update jetson-stats for GPU Monitoring

Source: https://www.jetson-ai-lab.com/tutorials/live-vlm-webui

Ensure the `jetson-stats` library is installed and up-to-date on your Jetson device to enable GPU and CPU monitoring. A reboot is required after installation.

```bash
sudo pip3 install -U jetson-stats
sudo reboot
```

--------------------------------

### Start vLLM Server with Cosmos-Reason2

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/vision-ai-live-vlm-webui

Run the vLLM inference engine with the Cosmos-Reason2 model. This command mounts the model weights, configures GPU memory utilization, and sets up the server to listen on port 8000. Ensure the model path and Docker image are correct for your environment.

```bash
export MODEL_PATH="${HOME}/.cache/huggingface/hub/cosmos-reason2-8b_v1208-fp8-static-kv8"

sudo docker run -it --rm --runtime=nvidia --network host \
  -v ${HOME}/models/cosmos-reason2-8b-fp8:/models/cosmos-reason2-8b:ro \
  -v ${HOME}/.cache/vllm:/root/.cache/vllm \
  ghcr.io/nvidia-ai-iot/vllm:0.14.0-r38.3-arm64-sbsa-cu130-24.04 \
  vllm serve /models/cosmos-reason2-8b \
    --served-model-name nvidia/cosmos-reason2-8b-fp8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 \
    --media-io-kwargs '{"video": {"num_frames": -1}}' \
    --enable-prefix-caching \
    --port 8000
```

--------------------------------

### Install Node.js 22+

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Install Node.js version 22 or later using the NodeSource repository. This includes updating the package list and installing the nodejs package, followed by a version check.

```bash
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
node --version
```

--------------------------------

### Install Docker and nvidia-container on JetPack 6.x

Source: https://www.jetson-ai-lab.com/tips_ssd-docker.html

For JetPack 6.x users flashing with SDK Manager, manually install Docker and configure the nvidia-container runtime after installing `nvidia-container` via apt.

```bash
sudo apt update
sudo apt install -y nvidia-container curl
curl https://get.docker.com | sh && sudo systemctl --now enable docker
sudo nvidia-ctk runtime configure --runtime=docker
```

--------------------------------

### Serve Qwen3.5 9B Model with Docker

Source: https://www.jetson-ai-lab.com/models/qwen3-5-9b

Use this command to serve the Qwen3.5 9B model with vLLM on Jetson Thor. Ensure Docker is set up and the correct Jetson host is specified.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve Kbenkhaled/Qwen3.5-9B-NVFP4 \
    --gpu-memory-utilization 0.8 \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
```

--------------------------------

### Verify Ollama Installation

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/intro-ai-microservice

Check if Ollama is installed and running on your Jetson device.

```bash
ollama --version
```

--------------------------------

### Install or Update curl

Source: https://www.jetson-ai-lab.com/tutorials/nemoclaw

Installs or updates the curl utility, which is required for downloading scripts.

```bash
sudo apt update && sudo apt install -y curl
```

--------------------------------

### Serve Qwen3.6 35B-A3B with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-6-35b-a3b

Command to serve the Qwen3.6-35B-A3B model using vLLM with specific configurations for reasoning and tool calling. Ensure the correct Docker image and model name are used.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  vllm/vllm-openai:nightly-aarch64 \
  bash -c "pip install -q 'vllm[audio]' && vllm serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 \
    --gpu-memory-utilization 0.8 \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder"
```

--------------------------------

### Install Ollama on Windows

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/intro-ai-microservice

Use this PowerShell command to install Ollama on Windows systems.

```powershell
irm https://ollama.com/install.ps1 | iex
```

--------------------------------

### Install ONNX Slim

Source: https://www.jetson-ai-lab.com/tutorials/openpi_on_thor

Installs the 'onnxslim' Python package. This is typically already included in the Dockerfile.

```bash
pip install onnxslim
```

--------------------------------

### Serve Qwen3.5 0.8B Model with Docker

Source: https://www.jetson-ai-lab.com/models/qwen3-5-0-8b

Use this command to serve the Qwen3.5 0.8B model using Docker on a Jetson Thor system with vLLM. Ensure the correct Docker image and model name are used.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve Qwen/Qwen3.5-0.8B \
    --gpu-memory-utilization 0.8 \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
```

--------------------------------

### Start OpenCLAW Gateway

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Starts the OpenCLAW gateway in the background and redirects its output to a log file.

```bash
nohup openclaw gateway run > /tmp/openclaw-gateway.log 2>&1 &

```

--------------------------------

### Install jetson-stats from GitHub for Jetson Thor

Source: https://www.jetson-ai-lab.com/tutorials/live-vlm-webui

For Jetson Thor devices, install `jetson-stats` directly from its GitHub repository using pip, enable its system service, and reboot. This is a workaround for potential compatibility issues with standard installations.

```bash
sudo pip3 install --break-system-packages git+https://github.com/rbonghi/jetson_stats.git
sudo jtop --install-service
sudo reboot
```

--------------------------------

### Install OpenClaw

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Use this command to install or update OpenClaw to the latest version if the 'openclaw' command is not found.

```bash
sudo npm install -g openclaw@latest
```

--------------------------------

### Serve Qwen3.6 27B Model with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-6-27b

Command to serve the Qwen3.6-27B model using vLLM with specific configurations for reasoning and tool choice. Ensure you have the correct Jetson module and sufficient memory.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  vllm/vllm-openai:nightly-aarch64 \
  bash -c "pip install -q 'vllm[audio]' && vllm serve sakamakismile/Qwen3.6-27B-NVFP4 \
    --gpu-memory-utilization 0.8 \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder"
```

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host \
  vllm/vllm-openai:nightly-aarch4 \
  bash -c "pip install -q 'vllm[audio]' && vllm serve sakamakismile/Qwen3.6-27B-NVFP4 \
    --gpu-memory-utilization 0.8 --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder"
```

--------------------------------

### Install aiohttp Module

Source: https://www.jetson-ai-lab.com/tutorial_nanoowl.html

Install the aiohttp Python package, which may be required by the tree prediction demo.

```python
pip install aiohttp
```

--------------------------------

### Install Ollama on Linux/macOS

Source: https://www.jetson-ai-lab.com/tutorials/gtc26/intro-ai-microservice

Use this command to install Ollama on Linux or macOS systems if it's not already present.

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

--------------------------------

### NemoClaw Installation Complete

Source: https://www.jetson-ai-lab.com/tutorials/nemoclaw

Indicates the successful completion of the NemoClaw installation, providing commands to source the environment and connect to the assistant.

```bash
[INFO]  === Installation complete ===

NemoClaw

Your OpenClaw Sandbox is live.

Next:
$ source /home/jetson/.bashrc
$ nemoclaw my-assistant connect
sandbox@my-assistant$ openclaw tui
```

--------------------------------

### Install jtop for Jetson Monitoring

Source: https://www.jetson-ai-lab.com/tutorials/hackathon-guide

Install the jetson-stats package to monitor Jetson resource utilization in real-time. This requires pip3.

```bash
sudo apt install -y python3-pip
sudo pip3 install -U jetson-stats
```

--------------------------------

### Serve Cosmos Reason2 VLM Inside Container (Orin Super Nano)

Source: https://www.jetson-ai-lab.com/tutorials/cosmos-reason2-vlm

Activates the environment and starts the vLLM server with aggressive memory optimization flags for the Orin Super Nano. Key flags like --max-model-len and --gpu-memory-utilization are crucial.

```bash
cd /opt/
source venv/bin/activate

vllm serve /models/cosmos-reason2 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --enforce-eager \
  --max-model-len 768 \
  --max-num-batched-tokens 768 \
  --gpu-memory-utilization 0.52 \
  --max-num-seqs 1 \
  --enable-chunked-prefill \
  --limit-mm-per-prompt '{"image":1}'
```

--------------------------------

### Serve Qwen3 4B Model with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-4b

Command to serve the Qwen3 4B model using Docker and vLLM. Ensure the correct Jetson module and GPU memory utilization are specified.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve RedHatAI/Qwen3-4B-quantized.w4a16 --gpu-memory-utilization 0.8
```

--------------------------------

### Serve Qwen3.5 4B Model with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-5-4b

Command to serve the Qwen3.5 4B model using Docker with vLLM on Jetson Thor. Ensure the correct module and engine are selected.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit \
    --gpu-memory-utilization 0.8 \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
```

--------------------------------

### Serve Cosmos Reason2 VLM Inside Container (Jetson Orin)

Source: https://www.jetson-ai-lab.com/tutorials/cosmos-reason2-vlm

Activates the environment and starts the vLLM server for the Cosmos Reason2 model. Adjust parameters like max-model-len and gpu-memory-utilization as needed.

```bash
cd /opt/
source venv/bin/activate
vllm serve /models/cosmos-reason2 \
  --max-model-len 8192 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.8
```

--------------------------------

### Install SDK Manager on Ubuntu 22.04

Source: https://www.jetson-ai-lab.com/tutorials/initial-setup-sdk-manager

Use these commands to add the NVIDIA CUDA repository and install the SDK Manager on Ubuntu 22.04.

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install sdkmanager
```

--------------------------------

### Start OpenClaw Gateway

Source: https://www.jetson-ai-lab.com/tutorials/openclaw

Run the OpenClaw gateway as a user service using systemd-run. This command ensures the gateway starts automatically on boot.

```bash
systemd-run --user --unit=openclaw-gateway openclaw gateway run
```

--------------------------------

### Serve Qwen3.6 35B-A3B AWQ-4bit on Jetson Orin

Source: https://www.jetson-ai-lab.com/models/qwen3-6-35b-a3b

Docker command to serve the AWQ-4bit quantized version of Qwen3.6 35B-A3B on Jetson Orin devices. Includes specific parameters for model length and reasoning.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin \
  vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --gpu-memory-utilization 0.8 --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --max-model-len 4096
```

--------------------------------

### Serve Nemotron Nano 9B v2 with Docker and vLLM

Source: https://www.jetson-ai-lab.com/models/nemotron-nano-9b-v2

Use this command to serve the model. Ensure you have the correct Jetson module and Docker image specified.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4 --gpu-memory-utilization 0.8
```

--------------------------------

### Install Python Dependencies

Source: https://www.jetson-ai-lab.com/tutorials/finetune-on-jetson

Installs necessary Python libraries for fine-tuning within the launched container. These include TRL, datasets, accelerate, PEFT, and bitsandbytes.

```bash
pip install trl datasets accelerate peft bitsandbytes
```

--------------------------------

### Serve Qwen3 32B Model with vLLM

Source: https://www.jetson-ai-lab.com/models/qwen3-32b

Command to serve the Qwen3 32B model using Docker with vLLM on a Jetson Thor module. Ensure the correct Docker image and model path are used.

```bash
sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve RedHatAI/Qwen3-32B-quantized.w4a16 --gpu-memory-utilization 0.8
```

--------------------------------

### Launch PyTorch Container

Source: https://www.jetson-ai-lab.com/tutorials/finetune-on-jetson

Launches the PyTorch container, mounting the current directory and Hugging Face cache. Navigate to your desired training directory before running.

```bash
cd ~/Desktop/train/finetune

docker run --runtime nvidia -it --rm --ipc=host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v $(pwd):/workspace \
  -w /workspace \
  nvcr.io/nvidia/pytorch:25.11-py3
```

--------------------------------

### Install Ollama and Download Model

Source: https://www.jetson-ai-lab.com/tutorials/live-vlm-webui

Installs Ollama and downloads a recommended VLM model. Use the specific version command for Jetson Thor with JetPack 7.0.

```bash
# Ollama installer for Linux/Mac
curl -fsSL https://ollama.com/install.sh | sh

# Download recommended model (lightweight)
ollama pull gemma3:4b

# Or other vision models
# ollama pull llama3.2-vision:11b
# ollama pull qwen2.5-vl:7b
```

```bash
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.12.9 sh
```

--------------------------------

### Serve Qwen3.5 27B with vLLM on Jetson Thor (Quantized)

Source: https://www.jetson-ai-lab.com/models/qwen3-5-27b

Command to serve the Qwen3.5 27B model using vLLM on a Jetson AGX Thor Developer Kit or T5000 module, utilizing a quantized model.

```bash
sudo docker run -it --rm --pull always --runtime=nvidia --network host \
  ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
  vllm serve Kbenkhaled/Qwen3.5-27B-NVFP4 \
    --gpu-memory-utilization 0.8 --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder
```
Tokens:
Raw