LocalAI

https://localai.io/docs/
Admin
LocalAI is an open-source, OpenAI-compatible AI runtime that enables users to run LLMs, audio,...
Tokens:582,569
Snippets:4,627
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install gpt-3.5-turbo example

Source: https://localai.io/docs/models/index.html

An example of installing a model and renaming it to gpt-3.5-turbo.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
      "url": "github:mudler/LocalAI/gallery/gpt4all-j.yaml",
      "name": "gpt-3.5-turbo"
   }'
```

--------------------------------

### Get an instruction guide in markdown

Source: https://localai.io/docs/features/api-discovery/index.print.html

Retrieves a markdown guide for a specific instruction set by name.

```bash
curl http://localhost:8080/api/instructions/config-management
```

--------------------------------

### Install a vision model from the gallery

Source: https://localai.io/docs/features/gpt-vision/index.html

Use the CLI to install a vision-capable model from the gallery. The example uses `moondream2-20250414`, a small vision model. This command is also shown in the Setup section.

```bash
local-ai run moondream2-20250414
```

--------------------------------

### Install hermes-2-pro-mistral model

Source: https://localai.io/docs/models/index.html

A specific example of installing the hermes-2-pro-mistral model using its configuration URL.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "config_url": "https://raw.githubusercontent.com/mudler/LocalAI/v2.25.0/embedded/models/hermes-2-pro-mistral.yaml"
   }'
```

--------------------------------

### GET /api/instructions/:name

Source: https://localai.io/docs/features/api-discovery/index.print.html

Fetches a detailed instruction guide for a specific area. By default returns a markdown guide; with ?format=json returns a raw OpenAPI fragment.

```APIDOC
## GET /api/instructions/:name

### Description
Returns a markdown guide for a specific instruction area. Optionally, add ?format=json to get a raw OpenAPI fragment (filtered Swagger spec with only relevant paths and definitions).

### Method
GET

### Endpoint
/api/instructions/:name

### Parameters
#### Path Parameters
- **name** (string) - Required - The instruction name (e.g., config-management)

#### Query Parameters
- **format** (string) - Optional - Set to 'json' to receive an OpenAPI fragment instead of markdown

### Request Example
curl http://localhost:8080/api/instructions/config-management

### Response
#### Success Response (200)
Returns a markdown guide by default, or an OpenAPI fragment when ?format=json is used.

#### Response Example
(No explicit response example provided in the source.)
```

--------------------------------

### Get an instruction guide as OpenAPI fragment

Source: https://localai.io/docs/features/api-discovery/index.print.html

Retrieves a raw OpenAPI fragment for a specific instruction set by appending the ?format=json query parameter.

```bash
curl http://localhost:8080/api/instructions/config-management?format=json
```

--------------------------------

### Build LocalAI on Mac (M1/M2/M3)

Source: https://localai.io/docs/basics/build/index.html

Complete build and run example for Mac (M1, M2, M3) without Docker. Installs prerequisites via brew, clones the repo, builds, downloads a model, installs a backend, and runs the server with a test request.

```bash
brew install abseil cmake go grpc protobuf wget protoc-gen-go protoc-gen-go-grpc

git clone https://github.com/go-skynet/LocalAI.git

cd LocalAI

make build

wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf -O models/phi-2.Q2_K

cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/phi-2.Q2_K.tmpl

./local-ai backends install llama-cpp

./local-ai --models-path=./models/ --debug=true

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "phi-2.Q2_K",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'
```

--------------------------------

### Complete example with reasoning configuration

Source: https://localai.io/docs/advanced/model-configuration/index.html

Shows a full model configuration with all reasoning fields set to their defaults.

```yaml
name: deepseek-model
backend: llama-cpp
parameters:
  model: deepseek.gguf

reasoning:
  disable: false
  disable_reasoning_tag_prefill: false
  strip_reasoning_only: false
```

--------------------------------

### Function calling with curl

Source: https://localai.io/docs/basics/try/index.html

Sends a chat completion request that defines a function tool for getting the weather and sets tool_choice to auto. The example assumes the qwen3-4b model is installed.

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [
      {
        "role": "user",
        "content": "What is the weather like in Boston?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"]
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

```

--------------------------------

### Example Watchdog Deployment and API Usage

Source: https://localai.io/docs/advanced/vram-management/index.html

A complete example showing LocalAI startup with both watchdogs enabled, followed by example API requests to models.

```bash
LOCALAI_WATCHDOG_IDLE=true \
LOCALAI_WATCHDOG_IDLE_TIMEOUT=10m \
LOCALAI_WATCHDOG_BUSY=true \
LOCALAI_WATCHDOG_BUSY_TIMEOUT=5m \
./local-ai

curl http://localhost:8080/v1/chat/completions -d '{"model": "model-a", ...}'
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-b", ...}'
```

--------------------------------

### Install host dependencies with setup.sh

Source: https://localai.io/docs/integrations/index.html

Installs the required host dependencies. Requires sudo privileges.

```bash
sudo bash setup.sh
```

--------------------------------

### Run phi-2 model with Docker using gist config URL

Source: https://localai.io/docs/getting-started/customize-model

Starts the phi-2 model using the LocalAI container image, passing the URL of the model configuration file as an argument. The example uses a specific gist URL from the quickstart.

```bash
docker run -p 8080:8080 localai/localai:v4.9.0 https://gist.githubusercontent.com/mudler/ad601a0488b497b69ec549150d9edd18/raw/a8a8869ef1bb7e3830bf5c0bae29a0cce991ff8d/phi-2.yaml
```

--------------------------------

### Engine Configuration Example

Source: https://localai.io/docs/features/text-generation

A sample configuration for the qwen35-a3b model using the vllm-cpp backend, demonstrating KV cache sizing and scheduler settings.

```yaml
name: qwen35-a3b
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3.5-A3B"
context_size: 16384
template:
  use_tokenizer_template: true
engine_args:
  # KV cache sizing: num_blocks * block_size tokens of cache.
  block_size: 32
  num_blocks: 1024
  # Concurrency and the per-step chunked-prefill token budget.
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  # Automatic prefix caching. Omit to keep the model's own default
  # (on for dense models, off for hybrid / attention-free ones).
  enable_prefix_caching: true
  # Scheduler admission order: fcfs (default), priority, or lpm
  # (cache-aware longest-prefix-match; needs prefix caching to have any effect).
  scheduling_policy: lpm
```

--------------------------------

### Check and install backend

Source: https://localai.io/docs/getting-started/models/index.print.html

List installed backends and install llama-cpp if needed.

```bash
local-ai backends list
local-ai backends install llama-cpp  # if needed
```

--------------------------------

### Install audio-transform model with local-ai run

Source: https://localai.io/docs/features/audio-transform/index.print.html

Installs the audio-transform model `localvqe-v1.3-4.8m` from the gallery before using the endpoint.

```bash
local-ai run localvqe-v1.3-4.8m
```

--------------------------------

### Install a model via CLI

Source: https://localai.io/docs/getting-started/models/index.print.html

Use this command to install only the specified model without starting the runtime.

```bash
local-ai models install hermes-2-theta-llama-3-8b
```

--------------------------------

### Run models via URI examples

Source: https://localai.io/docs/getting-started/models/index.print.html

Shows multiple URI examples for running models from Hugging Face, Ollama, HTTP(S) configuration files, and OCI registries. Note that `file://` URLs must point to files within the models directory.

```bash
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```

--------------------------------

### Get model job state with curl

Source: https://localai.io/docs/models/index.html

Sends a GET request to /models/jobs/<JOB_ID> to retrieve the state of a model installation job.

```bash
curl http://localhost:8080/models/jobs/<JOB_ID>
```

--------------------------------

### Cluster object in GET /api/resources and GET /api/models

Source: https://localai.io/docs/features/distributed-mode

Example JSON response showing the additional 'cluster' object returned by GET /api/resources and GET /api/models in distributed mode. The object is absent in single-node mode or when the registry cannot be read.

```json
{
  "cluster": {
    "enabled": true,
    "node_id": "a1b2c3",
    "node_name": "dgx-01",
    "total_memory": 85899345920,
    "is_gpu": true,
    "node_count": 4
  }
}
```

--------------------------------

### Install longcat-video model

Source: https://localai.io/docs/features/video-generation/index.html

Use the CLI to install the longcat-video model from the gallery.

```bash
local-ai run longcat-video
```

--------------------------------

### GET /api/auth/oidc/login

Source: https://localai.io/docs/features/authentication/index.html

Starts OIDC login. No authentication required.

```APIDOC
## GET /api/auth/oidc/login

### Description
Starts OIDC login. No authentication required.

### Method
GET

### Endpoint
/api/auth/oidc/login

### Parameters
#### Query Parameters
- **invite_code** (string) - Optional - Invite code to be used during the OAuth flow.

### Response
#### Success Response (302)
Redirects to the OIDC provider's authorization page.

#### Response Example
```
HTTP/1.1 302 Found
Location: https://provider.example.com/authorize?...
```
```

--------------------------------

### Install qwen3-4b model via CLI

Source: https://localai.io/docs/getting-started/first-agent/index.html

Run this command to install the qwen3-4b model, a small CPU-friendly Qwen3 model that supports tool calling. This is required before creating an agent.

```bash
local-ai run qwen3-4b
```

--------------------------------

### Run flux.1-dev-ggml model with local-ai run

Source: https://localai.io/docs/features/image-generation/index.html

Starts LocalAI with the flux.1-dev-ggml model from the gallery. This is a quick way to get started with the stablediffusion-ggml backend.

```bash
local-ai run flux.1-dev-ggml
```

--------------------------------

### vllm.cpp Backend Setup

Source: https://localai.io/docs/features/text-generation

Basic configuration for the vllm.cpp backend.

```yaml
name: vllm-cpp
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3-4B"
context_size: 8192
template:
  use_tokenizer_template: true
```

--------------------------------

### Install and run rfdetr-base model

Source: https://localai.io/docs/features/object-detection/index.print.html

Installs and runs the rfdetr-base model from the LocalAI model gallery. This is the recommended setup for the Python-based RF-DETR backend.

```bash
# Install and run the rfdetr-base model
local-ai run rfdetr-base
```

--------------------------------

### Example options for memory and device placement

Source: https://localai.io/docs/features/image-generation/index.html

Shows how to mmap diffusion weights from disk while keeping the text encoder on the CPU. Uses params_backend and keep_clip_on_cpu options.

```yaml
options:
- "diffusion_model"
- "sampler:euler"
- "params_backend:diffusion=disk"
- "keep_clip_on_cpu:true"
```

--------------------------------

### Job queued response

Source: https://localai.io/docs/models/index.html

Example JSON response indicating the job is queued because another installation is still running. Installations are processed one at a time.

```json
{"error":null,"processed":false,"message":"queued","phase":"queued"}
```

--------------------------------

### Install a model from gallery with /models/apply

Source: https://localai.io/docs/models/index.print.html

Install a model by sending a POST request to /models/apply with the model id in the JSON body. The repository prefix is optional; if omitted, LocalAI searches all repositories and the first match wins.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "id": "localai@bert-embeddings"
   }'  
```

--------------------------------

### GET /api/auth/github/login

Source: https://localai.io/docs/features/authentication/index.html

Starts the GitHub OAuth flow. No authentication required.

```APIDOC
## GET /api/auth/github/login

### Description
Starts the GitHub OAuth flow. No authentication required.

### Method
GET

### Endpoint
/api/auth/github/login

### Parameters
#### Query Parameters
- **invite_code** (string) - Optional - Invite code to be used during the OAuth flow.

### Response
#### Success Response (302)
Redirects to the GitHub authorization page.

#### Response Example
```
HTTP/1.1 302 Found
Location: https://github.com/login/oauth/authorize?...
```
```

--------------------------------

### Install audio-cpp backend via CLI

Source: https://localai.io/docs/features/audio-cpp/index.html

Use the LocalAI CLI to install the audio-cpp backend.

```bash
local-ai backends install audio-cpp
```

--------------------------------

### Install model from gallery

Source: https://localai.io/docs/models/index.html

Install a model by sending a POST request to the /models/apply endpoint with the model identifier.

```bash
LOCALAI=http://localhost:8080
curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "id": "localai@bert-embeddings"
   }'
```
Tokens:
Raw