### Install llmaz From Source via Make

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/installation.md

Clones the llmaz Git repository, changes into the directory, creates the 'llmaz-system' namespace, switches context to it, and runs the 'make helm-install' command to build and install from source.

```cmd
git clone https://github.com/inftyai/llmaz.git && cd llmaz
kubectl create ns llmaz-system && kubens llmaz-system
make helm-install
```

--------------------------------

### Install Released llmaz Version via Helm

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/installation.md

Installs a specific released version of llmaz using the Helm OCI chart from Docker Hub into the 'llmaz-system' namespace. The '--create-namespace' flag ensures the namespace exists.

```cmd
helm install llmaz oci://registry-1.docker.io/inftyai/llmaz --namespace llmaz-system --create-namespace --version 0.0.9
```

--------------------------------

### Install Kubernetes Metric Server (cmd)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/hpa/README.md

This command installs the Kubernetes Metric Server from its official release URL. The Metric Server is required for HPA to collect resource utilization data for scaling decisions.

```cmd
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```

--------------------------------

### Upgrade llmaz Controller Image

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/installation.md

Upgrades the llmaz controller by building and deploying a new image. Requires setting the 'IMG' environment variable to the desired image registry and tag before running the 'make helm-upgrade' command.

```cmd
IMG=<image-registry>:<tag> make helm-upgrade
```

--------------------------------

### Install llmaz Chatbot Components

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/open-webui.md

This command executes the `install-chatbot` target defined in the project's Makefile. Running this command initiates the deployment process for the chatbot components, including Open WebUI, based on the configuration specified in `values.global.yaml`.

```Shell
make install-chatbot
```

--------------------------------

### Example Response for Listing Models (JSON)

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/envoy-ai-gateway.md

Example JSON output showing the expected response format when querying the `/v1/models` endpoint of the Envoy AI Gateway. It lists the available models exposed through the gateway, including their IDs and creation timestamps.

```json
{
  "data": [
    {
      "id": "qwen2-0.5b",
      "created": 1745327294,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "qwen2.5-coder",
      "created": 1745327294,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    }
  ],
  "object": "list"
}
```

--------------------------------

### Uninstall llmaz Helm Release and Namespace

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/installation.md

Uninstalls the llmaz Helm release from the 'llmaz-system' namespace and then deletes the namespace itself. This removes the deployed resources.

```cmd
helm uninstall llmaz --namespace llmaz-system
kubectl delete ns llmaz-system
```

--------------------------------

### Example Response for Chat Completion (JSON)

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/envoy-ai-gateway.md

Example JSON output showing the expected response format from the `/v1/chat/completions` endpoint after sending a chat request. It includes the model's generated message, finish reason, usage statistics, and timing information.

```json
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?"
      }
    }
  ],
  "created": 1745327371,
  "model": "qwen2-0.5b",
  "system_fingerprint": "b5124-bc091a4d",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 10,
    "prompt_tokens": 10,
    "total_tokens": 20
  },
  "id": "chatcmpl-AODlT8xnf4OjJwpQH31XD4yehHLnurr0",
  "timings": {
    "prompt_n": 1,
    "prompt_ms": 319.876,
    "prompt_per_token_ms": 319.876,
    "prompt_per_second": 3.1262114069201816,
    "predicted_n": 10,
    "predicted_ms": 1309.393,
    "predicted_per_token_ms": 130.9393,
    "predicted_per_second": 7.63712651587415
  }
}
```

--------------------------------

### Enable Open WebUI in llmaz Configuration

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/open-webui.md

This YAML snippet shows how to enable the Open WebUI component within the llmaz global configuration file (`values.global.yaml`). Setting `enabled` to `true` activates the installation and deployment of Open WebUI. Optionally, `persistence: true` can be added for data persistence, recommended for production environments.

```YAML
open-webui:
  enabled: true
```

--------------------------------

### Querying Registered Models (Shell)

Source: https://github.com/inftyai/llmaz/blob/main/README.md

Uses `curl` to send an HTTP GET request to the local endpoint exposed via port-forwarding, querying the `/v1/models` path to list the models registered with the llmaz inference service.

```Shell
curl http://localhost:8080/v1/models
```

--------------------------------

### Delete llmaz Custom Resource Definitions (CRDs)

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/installation.md

Deletes the Custom Resource Definitions (CRDs) installed by llmaz. This is an optional step after uninstalling the Helm release to fully clean up.

```cmd
kubectl delete crd \
    openmodels.llmaz.io \
    backendruntimes.inference.llmaz.io \
    playgrounds.inference.llmaz.io \
    services.inference.llmaz.io
```

--------------------------------

### Loading Model from Object Store (YAML)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/llamacpp/README.md

YAML configuration snippet showing how to load a model from a private object store by providing the direct URI to the model file.

```yaml
source:
  uri: oss://llmaz.oss-ap-southeast-1-internal.aliyuncs.com/models/qwen2-0_5b-instruct-q5_k_m.gguf
```

--------------------------------

### Loading Model from Huggingface (YAML)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/llamacpp/README.md

YAML configuration snippet demonstrating how to specify a model to be loaded from Huggingface Model Hub by providing the model ID and the specific filename.

```yaml
source:
  modelHub:
    modelID: Qwen/Qwen2-0.5B-Instruct-GGUF
    filename: qwen2-0_5b-instruct-q5_k_m.gguf
```

--------------------------------

### List Kubernetes Services in llmaz-system Namespace

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/open-webui.md

This command-line snippet uses `kubectl` to list all services running in the `llmaz-system` Kubernetes namespace. The output helps identify the name and details of the `envoy-ai-gateway` service, which is required to configure the `openaiBaseApiUrl` for Open WebUI.

```CMD
kubectl get svc -n llmaz-system
```

--------------------------------

### Testing Deployed Model with Curl (CMD)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/llamacpp/README.md

Command line snippet using curl to send a POST request to the deployed service's completions endpoint, including necessary headers and a JSON payload with the prompt and prediction parameters.

```cmd
curl --request POST \
--url http://localhost:8080/v1/completions \
--header "Content-Type: application/json" \
--data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'
```

--------------------------------

### Build Model Loader Image Locally (Shell)

Source: https://github.com/inftyai/llmaz/blob/main/llmaz/README.md

This command uses the Makefile to build the Docker image for the model loader locally on your machine.

```Shell
make loader-image-load
```

--------------------------------

### Enabling Envoy Gateway and AI Gateway in YAML

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/envoy-ai-gateway.md

Configuration snippet to enable the deployment of Envoy Gateway and Envoy AI Gateway within the llmaz system via the `values.global.yaml` file. Setting these flags to `true` ensures the components are included in the deployment.

```yaml
envoy-gateway:
    enabled: true
envoy-ai-gateway:
    enabled: true
```

--------------------------------

### Requesting Text Completion using cURL

Source: https://github.com/inftyai/llmaz/blob/main/README.md

Demonstrates how to make a POST request to the local LLMaz completions endpoint using the cURL command-line tool. It sets the content type to JSON and includes a request body specifying the model, prompt, maximum tokens, and temperature for the text generation.

```cmd
curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "opt-125m",
    "prompt": "San Francisco is a",
    "max_tokens": 10,
    "temperature": 0
}'
```

--------------------------------

### Push Model Loader Image to Registry (Shell)

Source: https://github.com/inftyai/llmaz/blob/main/llmaz/README.md

This command uses the Makefile to push the locally built model loader Docker image to the default container registry, typically 'inftyai/model-loader:<version>'.

```Shell
make loader-image-push
```

--------------------------------

### Run Tests (Shell)

Source: https://github.com/inftyai/llmaz/blob/main/llmaz/README.md

This command executes the test suite for the model loader component, likely using pytest, to ensure functionality is correct.

```Shell
make pytest
```

--------------------------------

### Querying Chat Completions Endpoint (Bash)

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/envoy-ai-gateway.md

Bash command using `curl` to send a POST request with a JSON payload to the `/v1/chat/completions` endpoint. This demonstrates how to initiate a chat interaction with a specific model (`qwen2-0.5b`) by providing a system message.

```bash
curl -H "Content-Type: application/json"     -d '{
        "model": "qwen2-0.5b",
        "messages": [
            {
                "role": "system",
                "content": "Hi."
            }
        ]
    }'     http://localhost:8080/v1/chat/completions | jq .
```

--------------------------------

### Defining Playground Resource (YAML)

Source: https://github.com/inftyai/llmaz/blob/main/README.md

Defines a Playground custom resource for llmaz, specifying the number of replicas and claiming a previously defined OpenModel resource by name. This resource creates a runnable inference service for the specified model.

```YAML
apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: opt-125m
spec:
  replicas: 1
  modelClaim:
    modelName: opt-125m
```

--------------------------------

### Configure Basic Playground ElasticConfig (yaml)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/hpa/README.md

This YAML snippet shows the basic configuration for the 'elasticConfig' field within a Playground specification. It sets the minimum and maximum number of replicas for the Playground, assuming the scaling trigger is defined elsewhere (e.g., in the backendRuntime).

```yaml
spec:
  elasticConfig:
    minReplicas: 1
    maxReplicas: 3
```

--------------------------------

### Generate Table of Contents - Shell

Source: https://github.com/inftyai/llmaz/blob/main/docs/proposals/NNNN-template/README.md

This shell command is used to automatically generate the table of contents for the proposal document. It processes the markdown file and populates the section between the `<!-- toc -->` and `<!-- /toc -->` tags based on the document's headings.

```Shell
hack/update-toc.sh
```

--------------------------------

### Port Forward Open WebUI Service

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/open-webui.md

This command-line snippet uses `kubectl` to set up port forwarding from local port 8080 to port 80 of the `open-webui` service in the Kubernetes cluster. This allows accessing the Open WebUI interface running inside the cluster from the local machine via `http://localhost:8080`.

```Shell
kubectl port-forward svc/open-webui 8080:80
```

--------------------------------

### Exposing llmaz Service Locally (Shell)

Source: https://github.com/inftyai/llmaz/blob/main/README.md

Uses `kubectl port-forward` to forward a local port (8080) to the ClusterIP service created by llmaz for the deployed Playground, allowing local access to the inference endpoint.

```Shell
kubectl port-forward svc/opt-125m-lb 8080:8080
```

--------------------------------

### Configure OpenAI API Endpoint for Open WebUI

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/open-webui.md

This YAML snippet updates the `values.global.yaml` file to configure the `openaiBaseApiUrl` for Open WebUI. It sets the API endpoint to the internal cluster service name of the Envoy AI Gateway, allowing Open WebUI to communicate with the AI models through the gateway.

```YAML
open-webui:
  enabled: true
  openaiBaseApiUrl: http://envoy-default-default-envoy-ai-gateway-dbec795a.llmaz-system.svc.cluster.local/v1
```

--------------------------------

### Defining OpenModel Resource (YAML)

Source: https://github.com/inftyai/llmaz/blob/main/README.md

Defines an OpenModel custom resource for llmaz, specifying the model family, source (HuggingFace model ID), and inference configuration including GPU limits. This resource makes the model available for use within the llmaz platform.

```YAML
apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: opt-125m
spec:
  familyName: opt
  source:
    modelHub:
      modelID: facebook/opt-125m
  inferenceConfig:
    flavors:
      - name: default # Configure GPU type
        limits:
          nvidia.com/gpu: 1
```

--------------------------------

### Define ModelSource Schema

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines the structure for ModelSource, specifying where a model originates from. It supports loading from a ModelHub or a generic URI following a protocol.

```Definition
modelHub: ModelHub
uri: URIProtocol
```

--------------------------------

### Define ModelHub Schema

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines the structure for ModelHub, representing a model registry configuration for downloading models. It includes fields for the registry name, model identifier, specific filename, revision, and patterns for filtering files.

```Definition
name: string
modelID: string (Required)
filename: string (Required)
revision: string
allowPatterns: []string
ignorePatterns: []string
```

--------------------------------

### Port-forwarding Envoy AI Gateway Service (Bash)

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/integrations/envoy-ai-gateway.md

Bash command to port-forward the Envoy AI Gateway LoadBalancer service to localhost on port 8080, mapping to the service's port 80. This allows local access to the AI Gateway API endpoints for testing purposes.

```bash
kubectl port-forward svc/envoy-default-default-envoy-ai-gateway-dbec795a 8080:80
```

--------------------------------

### Configure Playground ElasticConfig with HPA CPU Trigger (yaml)

Source: https://github.com/inftyai/llmaz/blob/main/docs/examples/hpa/README.md

This YAML snippet demonstrates how to explicitly define an HPA scale trigger directly within the Playground's 'elasticConfig'. It configures HPA to scale the Playground based on the average CPU utilization across its pods, targeting 50% utilization.

```yaml
spec:
  elasticConfig:
    minReplicas: 1
    maxReplicas: 3
    scaleTrigger:
      hpa:
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 50
```

--------------------------------

### Define ModelSpec Schema

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines the desired state of a Model resource. It includes the model's family name, its source location, and required inference configurations.

```Definition
familyName: ModelName (Required)
source: ModelSource (Required)
inferenceConfig: InferenceConfig (Required)
```

--------------------------------

### Define ModelRef Schema

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines the structure for ModelRef, which links a model by name and assigns it a specific role within a configuration, useful when multiple models are involved (e.g., for speculative decoding).

```Definition
name: ModelName (Required)
role: ModelRole
```

--------------------------------

### Define ModelName Alias

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines ModelName as an alias for a string type, typically used to represent the name of a model.

```Definition
Alias of string
```

--------------------------------

### Define ModelRole Alias

Source: https://github.com/inftyai/llmaz/blob/main/site/content/en/docs/reference/core.v1alpha1.md

Defines ModelRole as an alias for a string type, used to specify the function or role of a model when multiple models are referenced.

```Definition
Alias of string
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.