### Example: Install Model with Custom Name (gpt-3.5-turbo)

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

An example demonstrating how to install a model using a specific configuration file but assigning it a different name, 'gpt-3.5-turbo', using the `name` parameter.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
      "url": "github:mudler/LocalAI/gallery/gpt4all-j.yaml",
      "name": "gpt-3.5-turbo"
   }'  
```

--------------------------------

### Install Host Dependencies

Source: https://github.com/mudler/localai/blob/master/docs/content/integrations.md

Install the required host dependencies for the realtime voice assistant example. This command requires sudo privileges.

```bash
sudo bash setup.sh
```

--------------------------------

### Clone Realtime Voice Assistant Example

Source: https://github.com/mudler/localai/blob/master/docs/content/integrations.md

Clone the example repository to get started with the realtime voice assistant. Navigate into the cloned directory.

```bash
git clone https://github.com/mudler/LocalAI-examples.git
cd LocalAI-examples/realtime
```

--------------------------------

### SAM3 Backend Setup

Source: https://github.com/mudler/localai/blob/master/docs/content/features/object-detection.md

Manual configuration example for the SAM3 backend, specifying model and thread parameters.

```APIDOC
## Manual Configuration

```yaml
name: sam3
backend: sam3-cpp
parameters:
  model: edgetam_q4_0.ggml
  threads: 4
known_usecases:
  - detection
```
```

--------------------------------

### Install a Gallery Model via CLI

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Use the local-ai CLI to install a specific model from the gallery without starting the service.

```bash
local-ai models install hermes-2-theta-llama-3-8b
```

--------------------------------

### Generate Video with Starting Image

Source: https://github.com/mudler/localai/blob/master/docs/content/features/video-generation.md

Generate a video using a starting image as a reference. This allows for guided generation, incorporating elements from the provided image. Includes parameters for seed, guidance scale, and inference steps.

```bash
curl http://localhost:8080/video \
  -H "Content-Type: application/json" \
  -d '{
    "model": "video-model",
    "prompt": "A timelapse of flowers blooming",
    "start_image": "https://example.com/flowers.jpg",
    "num_frames": 24,
    "fps": 12,
    "seed": 42,
    "cfg_scale": 7.5,
    "step": 30
  }'
```

--------------------------------

### Complete Reasoning Configuration Example

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/model-configuration.md

A full example demonstrating various reasoning settings for a model.

```yaml
name: deepseek-model
backend: llama-cpp
parameters:
  model: deepseek.gguf

reasoning:
  disable: false
  disable_reasoning_tag_prefill: false
  strip_reasoning_only: false
```

--------------------------------

### Run LocalAI CPU Image with Docker or Podman

Source: https://github.com/mudler/localai/blob/master/docs/content/installation/containers.md

The fastest way to get started is with the CPU image. This command starts LocalAI and makes the API available at http://localhost:8080. You will need to install models separately.

```bash
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
# Or with Podman:
podman run -p 8080:8080 --name local-ai -ti localai/localai:latest
```

--------------------------------

### Install Specific Model (Hermes-2-Pro-Mistral)

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Example of installing a specific model, hermes-2-pro-mistral, by providing its configuration URL. This demonstrates the usage of the `config_url` parameter.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "config_url": "https://raw.githubusercontent.com/mudler/LocalAI/v2.25.0/embedded/models/hermes-2-pro-mistral.yaml"
   }' 
```

--------------------------------

### RF-DETR Native Backend Setup

Source: https://github.com/mudler/localai/blob/master/docs/content/features/object-detection.md

Instructions for installing and running the RF-DETR native backend, including using pre-defined model gallery entries.

```APIDOC
## Install RF-DETR Backend

```bash
local-ai backends install rfdetr-cpp
```

## Using Model Gallery

### Detection Variants

```bash
local-ai run rfdetr-cpp-nano
local-ai run rfdetr-cpp-small
local-ai run rfdetr-cpp-base
local-ai run rfdetr-cpp-medium
local-ai run rfdetr-cpp-large
```

### Segmentation Variants

```bash
local-ai run rfdetr-cpp-seg-nano
local-ai run rfdetr-cpp-seg-small
local-ai run rfdetr-cpp-seg-medium
local-ai run rfdetr-cpp-seg-large
local-ai run rfdetr-cpp-seg-xlarge
local-ai run rfdetr-cpp-seg-2xlarge
```
```

--------------------------------

### Install and Run RF-DETR Model

Source: https://github.com/mudler/localai/blob/master/docs/content/features/object-detection.md

Use the local-ai run command to easily install and start the rfdetr-base model. This is the recommended method for getting started.

```bash
# Install and run the rfdetr-base model
local-ai run rfdetr-base
```

--------------------------------

### List, Install, and Run Models via CLI

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Use the local-ai CLI to manage models from the gallery. This includes listing available models, installing a specific model, and starting LocalAI with a gallery model.

```bash
# List available models
local-ai models list

# Install a specific model
local-ai models install llama-3.2-1b-instruct:q4_k_m

# Start LocalAI with a model from the gallery
local-ai run llama-3.2-1b-instruct:q4_k_m
```

--------------------------------

### Get API Guide for Specific Instruction

Source: https://github.com/mudler/localai/blob/master/docs/content/features/api-discovery.md

Retrieve a markdown guide for a specific instruction area, useful for LLMs and humans.

```bash
curl http://localhost:8080/api/instructions/config-management
```

--------------------------------

### Install Model using API

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/advanced-usage.md

Use the `/models/apply` API endpoint to install models programmatically. This example shows how to install the `lunademo` model using its definition URL.

```bash
curl --location 'http://localhost:8080/models/apply' \
--header 'Content-Type: application/json' \
--data-raw '{ 
    "id": "TheBloke/Luna-AI-Llama2-Uncensored-GGML/luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin",
    "name": "lunademo"
}'
```

--------------------------------

### Install Protobuf Go Plugins

Source: https://github.com/mudler/localai/blob/master/docs/content/installation/build.md

Installs the Go plugins for protobuf compilation, required for gRPC services.

```bash
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
```

```bash
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
```

--------------------------------

### Install and Run a Specific Backend

Source: https://github.com/mudler/localai/blob/master/backend/python/README.md

Navigate to a backend's directory and use `make` or `bash` scripts to install dependencies and run the backend.

```bash
# Navigate to a specific backend
cd backend/python/transformers

# Install dependencies
make transformers
# or
bash install.sh

# Run the backend
make run
# or
bash run.sh
```

--------------------------------

### Example: Full Watchdog and Request Simulation

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/vram-management.md

Demonstrates a full watchdog configuration with example API calls to simulate model usage.

```bash
LOCALAI_WATCHDOG_IDLE=true \
LOCALAI_WATCHDOG_IDLE_TIMEOUT=10m \
LOCALAI_WATCHDOG_BUSY=true \
LOCALAI_WATCHDOG_BUSY_TIMEOUT=5m \
./local-ai

```

```bash
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-a", ...}'

```

```bash
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-b", ...}'

```

--------------------------------

### LocalAI CLI Command Completion Example

Source: https://github.com/mudler/localai/blob/master/docs/content/reference/shell-completion.md

After installation, typing `local-ai` followed by a tab will display available top-level commands.

```bash
$ local-ai <TAB>
run              backends         completion       explorer         models
federated        sound-generation transcript       tts              util
```

--------------------------------

### Use Unified Build System Commands

Source: https://github.com/mudler/localai/blob/master/backend/python/README.md

Source `libbackend.sh` to access consistent commands for installing requirements, starting the backend, and running tests across different backends.

```bash
# Source the library in your backend script
source $(dirname $0)/../common/libbackend.sh

# Install requirements (automatically handles hardware detection)
installRequirements

# Start the backend server
startBackend $@

# Run tests
runUnittests
```

--------------------------------

### Comprehensive Model Configuration Example

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/model-configuration.md

A detailed example combining various model parameters, system prompts, templates, and feature flags.

```yaml
name: my-llm-model
description: A high-performance LLM model
backend: llama-stable

parameters:
  model: my-model.gguf
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 2048

context_size: 4096
threads: 8
f16: true
gpu_layers: 35

system_prompt: "You are a helpful AI assistant."

template:
  chat: |
    {{.System}}
    {{range .Messages}}
    {{if eq .Role "user"}}User: {{.Content}}
    {{else if eq .Role "assistant"}}Assistant: {{.Content}}
    {{end}}
    {{end}}
    Assistant:

roles:
  user: "User:"
  assistant: "Assistant:"
  system: "System:"

stopwords:
  - "\n\nUser:"
  - "\n\nHuman:"

prompt_cache_path: "cache/my-model"
prompt_cache_all: true

function:
  grammar:
    parallel_calls: true
    mixed_mode: false

feature_flags:
  experimental_feature: true
```

--------------------------------

### Install turboquant Backend

Source: https://github.com/mudler/localai/blob/master/docs/content/features/text-generation.md

Install the turboquant backend using the local-ai CLI. Specific flavors for different hardware are available.

```bash
local-ai backends install turboquant
```

--------------------------------

### Download Example Audio File

Source: https://github.com/mudler/localai/blob/master/docs/content/features/audio-to-text.md

Download an example audio file using wget for testing the transcription endpoint.

```bash
wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg
```

--------------------------------

### Build and Run LocalAI on Mac

Source: https://github.com/mudler/localai/blob/master/docs/content/installation/build.md

Builds LocalAI, downloads a model, sets up prompt templates, installs a backend, and starts the server on macOS.

```bash
brew install abseil cmake go grpc protobuf wget protoc-gen-go protoc-gen-go-grpc

git clone https://github.com/go-skynet/LocalAI.git

cd LocalAI

make build

wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf -O models/phi-2.Q2_K

cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/phi-2.Q2_K.tmpl

./local-ai backends install llama-cpp

./local-ai --models-path=./models/ --debug=true
```

--------------------------------

### Install Fish Completion

Source: https://github.com/mudler/localai/blob/master/docs/content/reference/shell-completion.md

Pipe the completion script output to `source` for temporary use or save it to `~/.config/fish/completions/` for permanent installation.

```fish
local-ai completion fish | source
```

```fish
local-ai completion fish > ~/.config/fish/completions/local-ai.fish
```

--------------------------------

### Install RF-DETR Native Backend

Source: https://github.com/mudler/localai/blob/master/docs/content/features/object-detection.md

Installs the rfdetr-cpp backend using the local-ai CLI.

```bash
local-ai backends install rfdetr-cpp
```

--------------------------------

### Install Model from Gallery

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Install a model from a gallery repository by providing its identifier to the `/models/apply` endpoint. The repository name is optional.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "id": "localai@bert-embeddings"
   }'
```

--------------------------------

### Start Distributed Mode with Docker Compose

Source: https://github.com/mudler/localai/blob/master/docs/content/features/distributed-mode.md

Quickly set up a local distributed mode environment including PostgreSQL, NATS, a LocalAI frontend, and a worker node. This command automatically handles backend installation and model loading for inference requests.

```bash
docker compose -f docker-compose.distributed.yaml up
```

--------------------------------

### Install Dependencies on Debian

Source: https://github.com/mudler/localai/blob/master/docs/content/installation/build.md

Installs Go, make, and protobuf compiler for LocalAI on Debian-based systems.

```bash
apt install golang make protobuf-compiler-grpc
```

--------------------------------

### Install Zsh Completion

Source: https://github.com/mudler/localai/blob/master/docs/content/reference/shell-completion.md

Configure your `~/.zshrc` to source the completion script or install it to a directory in your `$fpath`.

```zsh
source <(local-ai completion zsh)
```

```zsh
local-ai completion zsh > "${fpath[1]}/_local-ai"
```

```zsh
autoload -Uz compinit
compinit
```

--------------------------------

### Manage LocalAI Backends

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/troubleshooting.md

List installed backends and install a missing one, such as llama-cpp, if a model fails to load.

```bash
local-ai backends list
```

```bash
local-ai backends install llama-cpp
```

--------------------------------

### Install Llama-CPP Backend

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Install the llama-cpp backend if it's not already present. This is often required for loading specific model types.

```bash
local-ai backends install llama-cpp  # if needed
```

--------------------------------

### Install LocalAI with Helm Chart

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/kubernetes.md

Steps to add the LocalAI Helm repository, update it, fetch default values, and install the chart.

```bash
helm repo add go-skynet https://go-skynet.github.io/helm-charts/
```

```bash
helm repo update
```

```bash
helm show values go-skynet/local-ai > values.yaml
```

```bash
helm install local-ai go-skynet/local-ai -f values.yaml
```

--------------------------------

### Run Models via Various URI Schemes

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Examples demonstrating how to run models using different URI schemes including file paths, Hugging Face repositories, OCI registries, and configuration file URLs.

```bash
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```

--------------------------------

### Install Model with Base Configuration and Additional Files

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Install a model using the 'base' configuration and provide an URL to LocalAI. This is useful when the model is not in the gallery. The `files` parameter allows specifying additional files to download.

```bash
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "url": "github:mudler/LocalAI/gallery/base.yaml@master",
     "name": "model-name",
     "files": [
        {
            "uri": "<additional_file_url>",
            "sha256": "<additional_file_hash>",
            "filename": "model"
        }
     ]
   }'
```

--------------------------------

### Example Fine-Tuning API Request

Source: https://github.com/mudler/localai/blob/master/docs/content/features/fine-tuning.md

Demonstrates how to initiate a fine-tuning job via the API, specifying model, training parameters, dataset, and reward functions.

```bash
curl -X POST http://localhost:8080/api/fine-tuning/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "backend": "trl",
    "training_method": "grpo",
    "training_type": "lora",
    "dataset_source": "my-reasoning-dataset",
    "num_epochs": 1,
    "batch_size": 2,
    "learning_rate": 5e-6,
    "reward_functions": [
      {"type": "builtin", "name": "reasoning_accuracy_reward"},
      {"type": "builtin", "name": "format_reward"},
      {"type": "builtin", "name": "length_reward", "params": {"target_length": "200"}},
      {"type": "inline", "name": "think_presence", "code": "return [1.0 if \"<think>\" in c else 0.0 for c in completions]"}
    ],
    "extra_options": {
      "num_generations": "4",
      "max_completion_length": "256"
    }
  }'
```

--------------------------------

### Install Models via CLI

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/quickstart.md

Automatically install specified models when starting LocalAI from the CLI. Supports various model sources like GGUF, Ollama, and remote URLs.

```bash
local-ai run llama-3.2-1b-instruct:q4_k_m
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```

--------------------------------

### Install Model from Configuration URL

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Use this command to install a model by providing a URL to its configuration file. LocalAI will download the model files and set up the configuration.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "config_url": "<MODEL_CONFIG_FILE_URL>"
   }' 
```

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "id": "<GALLERY>@<MODEL_NAME>"
   }' 
```

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "url": "<MODEL_CONFIG_FILE_URL>"
   }' 
```

--------------------------------

### Install Backend API Request

Source: https://github.com/mudler/localai/blob/master/core/http/views/backends.html

Initiates the installation of a backend by sending a POST request to the API. It updates the local state to reflect the backend's processing status and job ID.

```javascript
async installBackend(backendId) {
  try {
    const response = await fetch(`/api/backends/install/${encodeURIComponent(backendId)}`, {
      method: 'POST'
    });
    const data = await response.json();
    if (data.jobID) {
      const backend = this.backends.find(b => b.id === backendId);
      if (backend) {
        backend.processing = true;
        backend.jobID = data.jobID;
        backend.isDeletion = false;
      }
    }
  } catch (error) {
    console.error('Error installing backend:', error);
    alert('Failed to start installation');
  }
}
```

--------------------------------

### VAD API Response Example

Source: https://github.com/mudler/localai/blob/master/docs/content/features/voice-activity-detection.md

The response contains a list of detected speech segments, each with a start and end time in seconds.

```json
{
  "segments": [
    {
      "start": 0.5,
      "end": 2.3
    },
    {
      "start": 3.1,
      "end": 5.8
    }
  ]
}
```

--------------------------------

### Example: Loading Models with Max Active Backends

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/vram-management.md

Demonstrates loading models sequentially while respecting the maximum active backends limit. The first request loads 'model-a'.

```bash
# Allow 2 active backends
LOCALAI_MAX_ACTIVE_BACKENDS=2 ./local-ai

# First request - model-a is loaded (1 active)
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-a", ...}'
```

--------------------------------

### Run Realtime Voice Assistant

Source: https://github.com/mudler/localai/blob/master/docs/content/integrations.md

Execute the script to start the realtime voice assistant application after all setup steps are completed.

```bash
bash run.sh
```

--------------------------------

### Get Model Job State

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Retrieves the current status of a model installation job using its unique job ID.

```APIDOC
## GET /models/jobs/{JOB_ID}

### Description
Fetches the status of a previously initiated model installation job.

### Method
GET

### Endpoint
/models/jobs/{JOB_ID}

### Parameters
#### Path Parameters
- **JOB_ID** (string) - Required - The unique identifier of the model installation job.

### Response
#### Success Response (200)
- **error** (any) - Contains error information if the job failed, otherwise null.
- **processed** (boolean) - Indicates if the job has completed.
- **message** (string) - A status message, e.g., "completed".

#### Response Example
```json
{
  "error": null,
  "processed": true,
  "message": "completed"
}
```
```

--------------------------------

### Build with Authentication Support (Make)

Source: https://github.com/mudler/localai/blob/master/docs/content/features/authentication.md

Build the project from source with CGO enabled for SQLite support and the 'auth' build tag. This command uses 'make'.

```bash
GO_TAGS=auth make build
```

--------------------------------

### Get Model Job State via Curl

Source: https://github.com/mudler/localai/blob/master/docs/content/features/model-gallery.md

Retrieve the status of a model installation job using its unique job ID.

```bash
curl http://localhost:8080/models/jobs/<JOB_ID>
```

--------------------------------

### Get LocalAI Version and Enable Debug Logging

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/troubleshooting.md

Check the installed LocalAI version or enable debug logging for more detailed output during runtime.

```bash
local-ai --version
```

```bash
DEBUG=true local-ai run
```

```bash
local-ai run --log-level=debug
```

--------------------------------

### ds4 Worker Setup Command

Source: https://github.com/mudler/localai/blob/master/docs/content/features/distributed-mode.md

Start a ds4 worker by specifying its role, model path, layer range, and the coordinator's host and port.

```bash
local-ai worker ds4-distributed -- \
  --role worker \
  --model /models/ds4flash.gguf \
  --layers 20:output \
  --coordinator <coordinator-host> 1234
```

--------------------------------

### Run LocalAI with Docker

Source: https://github.com/mudler/localai/blob/master/docs/content/_index.md

This command starts the LocalAI service using Docker, exposing the API on port 8080. It's the recommended installation method for most users.

```bash
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
```

--------------------------------

### Start Fine-tuning Process

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/fine-tuning.md

Launches the model fine-tuning process using Axolotl and Accelerate. Requires a properly configured axolotl.yaml file.

```bash
accelerate launch -m axolotl.cli.train axolotl.yaml
```

--------------------------------

### Install and Run Model from Hugging Face via CLI

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Install and run a model directly from a Hugging Face repository using the local-ai CLI. The format specifies the repository and optionally a model file.

```bash
# Install and run a model from Hugging Face
local-ai run huggingface://TheBloke/phi-2-GGUF
```

--------------------------------

### Get System Information

Source: https://github.com/mudler/localai/blob/master/docs/content/reference/system-info.md

Retrieves a list of available backends and currently loaded models from the LocalAI instance. Use this endpoint to understand the capabilities and loaded resources of your LocalAI setup.

```bash
curl http://localhost:8080/system
```

```json
{
  "backends": [
    "llama-cpp",
    "huggingface",
    "diffusers",
    "whisper"
  ],
  "loaded_models": [
    {
      "id": "my-llama-model"
    },
    {
      "id": "whisper-1"
    }
  ]
}
```

--------------------------------

### Pre-install Backends on LocalAI Startup

Source: https://github.com/mudler/localai/blob/master/docs/content/features/backends.md

Specify backends to be pre-installed when LocalAI starts by setting the LOCALAI_EXTERNAL_BACKENDS environment variable.

```bash
export LOCALAI_EXTERNAL_BACKENDS="llm-backend,diffusion-backend"
local-ai run
```

--------------------------------

### Run LocalAI with NVIDIA GPU (CUDA 12) using Docker

Source: https://github.com/mudler/localai/blob/master/README.md

Starts a LocalAI container with NVIDIA GPU support for CUDA 12. This is an alternative if CUDA 13 is not compatible with your setup.

```bash
# CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
```

--------------------------------

### Docker Compose for LocalAI with MCP and Docker

Source: https://github.com/mudler/localai/blob/master/docs/content/features/mcp.md

Configuration for Docker Compose to run LocalAI, enabling it to interact with Docker itself for MCP server commands. This setup installs Docker within the LocalAI container.

```yaml
services:
  local-ai:
    image: localai/localai:latest
    #image: localai/localai:latest-gpu-nvidia-cuda-13
    #image: localai/localai:latest-gpu-nvidia-cuda-12
    container_name: local-ai
    restart: always
    entrypoint: [ "/bin/bash" ]
    command: >
     -c "apt-get update &&
         apt-get install -y docker.io &&
         /entrypoint.sh"
    environment:
      - DEBUG=true
      - LOCALAI_WATCHDOG_IDLE=true
      - LOCALAI_WATCHDOG_BUSY=true
      - LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m
      - LOCALAI_WATCHDOG_BUSY_TIMEOUT=15m
      - LOCALAI_API_KEY=my-beautiful-api-key
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_TLS_VERIFY=1
      - DOCKER_CERT_PATH=/certs/client
    ports:
      - "8080:8080"
    volumes:
      - /data/models:/models
      - /data/backends:/backends
      - certs:/certs:ro
    # uncomment for nvidia
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - capabilities: [gpu]
    #           device_ids: ['7']
    # runtime: nvidia

  docker:
    image: docker:dind
    privileged: true
    container_name: docker
    volumes:
      - certs:/certs
    healthcheck:
      test: ["CMD", "docker", "info"]
      interval: 10s
      timeout: 5s
volumes:
  certs:

```

--------------------------------

### Set up vLLM Backend Environment

Source: https://github.com/mudler/localai/blob/master/docs/content/advanced/advanced-usage.md

Command to prepare the Python environment for the vLLM backend. This is a prerequisite for registering the vLLM backend.

```bash
make -C backend/python/vllm
```

--------------------------------

### Run LocalAI with NVIDIA GPU (CUDA 13) using Docker

Source: https://github.com/mudler/localai/blob/master/README.md

Starts a LocalAI container with NVIDIA GPU support for CUDA 13. Ensure your Docker installation is configured to recognize your NVIDIA GPUs.

```bash
# CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13
```

--------------------------------

### Worked Example: 2-Node Kimi-K2.6 Follower Command

Source: https://github.com/mudler/localai/blob/master/docs/content/features/distributed-mode.md

Command to run on the follower node (10.0.0.2) for a 2-node Kimi-K2.6 deployment. It specifies the model, topology, start rank, master address, and registration details.

```bash
# On 10.0.0.2 (follower)
local-ai p2p-worker vllm moonshotai/Kimi-K2.6-Instruct \
  --data-parallel-size 8 --data-parallel-size-local 4 --start-rank 4 \
  --master-addr 10.0.0.1 --master-port 32100 \
  --register-to http://10.0.0.1:8080 --registration-token changeme
```

--------------------------------

### Run Docker Container

Source: https://github.com/mudler/localai/blob/master/docs/README.md

Starts the Docker container for the docsy-example project. The website will be accessible at http://localhost:1313.

```bash
docker-compose up
```

--------------------------------

### Manual Rank 0 and Worker Startup

Source: https://github.com/mudler/localai/blob/master/docs/content/features/mlx-distributed.md

Manually starts rank 0 as an external gRPC backend and then starts a worker. LocalAI is then started pointing to the external rank 0.

```bash
# On Mac A: start rank 0 manually
local-ai worker mlx-distributed --hostfile hosts.json --rank 0 --addr 192.168.1.10:50051

# On Mac B: start rank 1
local-ai worker mlx-distributed --hostfile hosts.json --rank 1

# On any machine: start LocalAI pointing at rank 0
local-ai run --external-grpc-backends "mlx-distributed:192.168.1.10:50051"
```

--------------------------------

### Example Model Configuration

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/troubleshooting.md

Illustrates a typical YAML configuration for a model, specifying its name, backend, and parameters like model file, context size, and threads.

```yaml
# Example model config
name: my-model
backend: llama-cpp
parameters:
  model: my-model.gguf  # Relative to models directory
  context_size: 2048
  threads: 4  # Should match physical CPU cores
```

--------------------------------

### Build Backend Dependencies

Source: https://github.com/mudler/localai/blob/master/backend/python/README.md

Install the necessary dependencies for a specific backend using the `make <backend-name>` command.

```bash
# Install dependencies
make <backend-name>

# Clean build artifacts
make clean
```

--------------------------------

### Build with Authentication Support (Go Build)

Source: https://github.com/mudler/localai/blob/master/docs/content/features/authentication.md

Build the project from source with CGO enabled for SQLite support and the 'auth' build tag. This command uses 'go build' directly.

```bash
go build -tags auth ./...
```

--------------------------------

### Install Bash Completion

Source: https://github.com/mudler/localai/blob/master/docs/content/reference/shell-completion.md

Add the completion script to your `~/.bashrc` for user-specific installation or to `/etc/bash_completion.d/` for system-wide availability.

```bash
source <(local-ai completion bash)
```

```bash
local-ai completion bash > /etc/bash_completion.d/local-ai
```

--------------------------------

### Run LocalAI from Binary

Source: https://github.com/mudler/localai/blob/master/docs/content/getting-started/models.md

Starts the LocalAI service using a downloaded binary, specifying the models path.

```bash
# With binary
local-ai --models-path ./models
```

--------------------------------

### Install Model API Call

Source: https://github.com/mudler/localai/blob/master/core/http/views/models.html

Initiates the installation of a model by sending a POST request to the /api/models/install endpoint. Handles success by updating model state and job ID, or logs errors.

```javascript
async installModel(modelId) { try { const response = await fetch(`/api/models/install/${encodeURIComponent(modelId)}`, { method: 'POST' }); const data = await response.json(); if (data.jobID) { const model = this.models.find(m => m.id === modelId); if (model) { model.processing = true; model.jobID = data.jobID; model.isDeletion = false; } } } catch (error) { console.error('Error installing model:', error); this.addNotification('Failed to start installation', 'error'); } }
```

--------------------------------

### Install Dependencies on macOS

Source: https://github.com/mudler/localai/blob/master/docs/content/installation/build.md

Installs necessary build tools and libraries for LocalAI on macOS using Homebrew.

```bash
brew install go protobuf protoc-gen-go protoc-gen-go-grpc wget
```

--------------------------------

### Basic Object Detection Example

Source: https://github.com/mudler/localai/blob/master/docs/content/features/object-detection.md

Example of performing basic object detection using a specified model.

```APIDOC
## Basic Object Detection

```bash
curl -X POST http://localhost:8080/v1/detection \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rfdetr-base",
    "image": "https://example.com/image.jpg"
  }'
```
```