### Initialize GradioServer and Start

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Initializes the GradioServer and starts it if no external Gradio server URL is provided. Exits if the server fails to start.

```python
gradio_server = None
GRADIO_HOST = os.getenv("GRADIO_HOST", "127.0.0.1")
GRADIO_PORT = int(os.getenv("GRADIO_PORT", "7860"))
GRADIO_URL = os.getenv("GRADIO_SERVER_URL", f"http://{GRADIO_HOST}:{GRADIO_PORT}")
LLAMA_ENDPOINT = os.getenv("LLAMA_ENDPOINT", "<YOUR_MODEL_API_ENDPOINT>")


class GradioServer:
    def __init__(self):
        self.host = GRADIO_HOST
        self.port = GRADIO_PORT
        self.process: Optional[multiprocessing.Process] = None
        self.url = GRADIO_URL

    async def chat_with_llama(self, message: str, history: List[List[str]]) -> str:
        """Make a request to the Llama endpoint"""
        # Convert history and new message into OpenAI chat format
        messages = []
        for h in history:
            messages.extend([
                {"role": "user", "content": h[0]},
                {"role": "assistant", "content": h[1]}
            ])
        messages.append({"role": "user", "content": message})

        async with httpx.AsyncClient() as client:
            try:
                response = await client.post(
                    f"{LLAMA_ENDPOINT}/v1/chat/completions",
                    json={
                        "messages": messages,
                        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
                        "stream": False,
                        "temperature": 0.7,
                        "top_p": 0.95
                    },
                    timeout=30.0
                )

                if response.status_code == 200:
                    response_data = response.json()
                    return response_data['choices'][0]['text']
                else:
                    return f"Error: Received status code {response.status_code} from Llama endpoint"
            except Exception as e:
                return f"Error communicating with Llama endpoint: {str(e)}"

    def run_server(self):
        interface = gr.ChatInterface(
            fn=self.chat_with_llama,
            type="messages",
            title="Chat with Llama",
            description="This is a chat interface powered by Llama 3.1 8B Instruct",
            examples=[
                ["What is the capital of France?"],
                ["Explain quantum computing in simple terms"],
                ["Write a short poem about technology"]
            ],
        )
        interface.launch(
            server_name=self.host,
            server_port=self.port,
            root_path=f"https://api.aws.us-east-1.cerebrium.ai/v4/{os.getenv('PROJECT_ID')}/{os.getenv('APP_NAME')}/",
            quiet=True
        )

    def start(self):
        print(f"Starting Gradio server at {self.url} port {self.port}")

        # Start Gradio in a separate process
        self.process = multiprocessing.Process(target=self.run_server)
        self.process.start()

        # Wait for Gradio to become ready
        max_retries = 30
        retry_delay = 1.0

        for _ in range(max_retries):
            try:
                response = requests.get(f"{self.url}/", timeout=1.0)
                if response.status_code == 200:
                    print(f"Gradio server is ready at {self.url}")
                    return True
            except requests.exceptions.ConnectionError:
                time.sleep(retry_delay)

        print("Failed to start Gradio server")
        self.stop()
        return False

    def stop(self):
        if self.process:
            self.process.terminate()
            self.process.join()
            self.process = None


@app.on_event("startup")
async def startup_event():
    global gradio_server
    if not os.getenv("GRADIO_SERVER_URL"):  # Only start local server if no external URL provided
        gradio_server = GradioServer()
        if not gradio_server.start():
            sys.exit(1)
```

--------------------------------

### Get Startup Time Metrics

Source: https://cerebrium.ai/docs/api-reference/metrics/get-startup-time-metrics

Retrieve cold start and container startup time metrics for a specific application within a project.

```APIDOC
## GET /v2/projects/{project_id}/apps/{app_id}/metrics/startup-time

### Description
Retrieve cold start and container startup time metrics for an app.

### Method
GET

### Endpoint
/v2/projects/{project_id}/apps/{app_id}/metrics/startup-time

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **app_id** (string) - Required - The ID of the application.

### Request Example
```json
{
  "example": "No request body for GET request"
}
```

### Response
#### Success Response (200)
- **metrics** (object) - Contains startup time metrics.
  - **cold_start_time** (number) - The cold start time in seconds.
  - **container_startup_time** (number) - The container startup time in seconds.

#### Response Example
```json
{
  "metrics": {
    "cold_start_time": 1.23,
    "container_startup_time": 0.45
  }
}
```

### Authentication
Service Account Token authentication. Include the token in the Authorization header:
`Authorization: Bearer <your-service-account-token>`
```

--------------------------------

### Download Example Audio File

Source: https://cerebrium.ai/docs/partner-services/deepgram

Use `wget` to download an example audio file for testing the Deepgram service.

```bash
wget https://dpgr.am/bueller.wav

```

--------------------------------

### Install LiveKit CLI

Source: https://cerebrium.ai/docs/v4/examples/livekit-outbound-agent

Install the LiveKit CLI package using Homebrew. Ensure Homebrew is installed first.

```bash
brew update && brew install livekit-cli
```

--------------------------------

### Gradio Application Setup

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Add the Gradio application setup code to `main.py`. This section initializes the Gradio server as a global variable.

```python
import multiprocessing
import os
import sys
import time

import gradio as gr

# Global variable for the Gradio server
gradio_server = None

```

--------------------------------

### Install and Initialize Cerebrium CLI

Source: https://cerebrium.ai/docs/v4/examples/livekit-outbound-agent

Install the Cerebrium CLI and initialize a new project for the LiveKit voice agent.

```bash
pip install cerebrium --upgrade
cerebrium login
cerebrium init livekit-voice-agent
```

--------------------------------

### Install and Login to Wandb

Source: https://cerebrium.ai/docs/v4/examples/wandb-sweep

Install the Wandb library and log in to your Wandb account to enable experiment tracking. You will need to copy an API key back into the terminal.

```bash
pip install wandb
wandb login
```

--------------------------------

### Install and Authenticate Twilio CLI

Source: https://cerebrium.ai/docs/v4/examples/livekit-outbound-agent

Installs the Twilio CLI using Homebrew and logs in to authenticate. Ensure you have Homebrew installed before running.

```bash
brew tap twilio/brew && brew install twilio
twilio login
```

--------------------------------

### FastAPI Application Setup

Source: https://cerebrium.ai/docs/v4/examples/high-throughput-embeddings

Sets up a FastAPI application with startup events for model initialization and health/readiness check endpoints.

```python
from fastapi import FastAPI, Body

app = FastAPI(title="High-Throughput Embedding Service")

@app.on_event("startup")
async def startup_event():
    """Initialize models on container startup"""
    await model.setup()


@app.get("/health")
async def health():
    return {"status": "healthy"}

@app.get("/ready")
async def ready():
    """Readiness endpoint to report model initialization state."""
    is_ready = model.engine_array is not None
    return {"ready": is_ready}
```

--------------------------------

### Install Cerebrium CLI and Initialize Project

Source: https://cerebrium.ai/docs/migrations/mystic

Install the Cerebrium command-line tool and initialize a new project for a stable diffusion model. Ensure you are logged in via the CLI.

```bash
pip install cerebrium --upgrade
cerebrium login  # You'll be redirected to the dashboard for login
cerebrium init stable-diffusion
cd stable-diffusion

```

--------------------------------

### Install and Initialize Cerebrium

Source: https://cerebrium.ai/docs/v4/examples/wandb-sweep

Install the Cerebrium library and initialize a new project for a Wandb sweep. This sets up the necessary project structure.

```bash
pip install cerebrium --upgrade
cerebrium login
cerebrium init wandb-sweep
```

--------------------------------

### Get GitHub Install URL

Source: https://cerebrium.ai/docs/api-reference/integrations/get-github-install-url

Generate a GitHub App installation URL for a project. This endpoint requires authentication using a Bearer token.

```APIDOC
## GET /v4/projects/{project_id}/integrations/github/install-url

### Description
Generate a GitHub App installation URL for a project.

### Method
GET

### Endpoint
/v4/projects/{project_id}/integrations/github/install-url

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project for which to generate the GitHub installation URL.

### Request Example
(No request body for GET request)

### Response
#### Success Response (200)
(Response body structure not detailed in the provided OpenAPI spec, but typically would contain the URL)

#### Response Example
(Example response body not provided in the source text)
```

--------------------------------

### FastAPI Startup Event for Gradio Server

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Initializes and starts the Gradio server when the FastAPI application starts up, but only if an external GRADIO_SERVER_URL is not provided.

```python
@app.on_event("startup")
async def startup_event():
    global gradio_server
    if not os.getenv("GRADIO_SERVER_URL"):  # Only start local server if no external URL provided
        gradio_server = GradioServer()
        if not gradio_server.start():
            sys.exit(1)
```

--------------------------------

### Install and Login Cerebrium CLI

Source: https://cerebrium.ai/docs/v4/examples/deploy-an-llm-with-tensorrtllm-tritonserver

Install the Cerebrium CLI and log in to your account. This is the first step to manage your Cerebrium projects.

```bash
pip install cerebrium
cerebrium login
```

--------------------------------

### Configure APT Dependencies

Source: https://cerebrium.ai/docs/toml-reference/toml-reference

List system packages required for the deployment. 'latest' installs the most recent version available.

```toml
[cerebrium.dependencies.apt]
ffmpeg = "latest"
libopenblas-base = "latest"
```

--------------------------------

### Enable UV for Faster Package Installation (TOML)

Source: https://cerebrium.ai/docs/toml-reference/toml-reference

Enable UV as the package installer in your `cerebrium.toml` file to significantly speed up Python dependency installation during deployment. This replaces pip with UV.

```toml
[cerebrium.deployment]
use_uv = true
```

--------------------------------

### Set Docker Entrypoint

Source: https://cerebrium.ai/docs/v4/examples/livekit-outbound-agent

Specifies the command to run when the Docker container starts. Ensure this matches your application's entry point.

```docker
CMD ["python3", "main.py", "start"]
```

--------------------------------

### Example SSE Response Snippet

Source: https://cerebrium.ai/docs/endpoints/streaming

This is an example of the server-sent events (SSE) response you can expect from a streaming endpoint. Data is sent progressively with the `data:` prefix.

```bash
HTTP/1.1 200 OK
cache-control: no-cache
content-encoding: gzip
content-type: text/event-stream; charset=utf-8
date: Tue, 28 May 2024 21:12:46 GMT
server: envoy
transfer-encoding: chunked
vary: Accept-Encoding
x-envoy-upstream-service-time: 198995
x-request-id: e6b55132-32af-96d7-a064-8915c4a42452

data: Number 0
```

--------------------------------

### Install Cerebrium CLI

Source: https://cerebrium.ai/docs/migrations/hugging-face

Install the Cerebrium CLI to manage your deployments. Ensure you have the latest version by using the --upgrade flag.

```bash
pip install cerebrium --upgrade
```

--------------------------------

### Create and Upload api.toml Configuration

Source: https://cerebrium.ai/docs/partner-services/deepgram

Example command to create an api.toml file and upload it to persistent storage for a Deepgram app.

```bash
cerebrium cp api.toml deepgram/api.toml
```

--------------------------------

### Define Pre-build Commands in TOML

Source: https://cerebrium.ai/docs/container-images/defining-container-images

Use pre_build_commands to execute scripts before dependency installation, useful for setting up specialized build tools.

```toml
[cerebrium.deployment]
pre_build_commands = [
    # Add specialized build tools
    "curl -o /usr/local/bin/pget -L 'https://github.com/replicate/pget/releases/download/v0.6.2/pget_linux_x86_64'",
    "chmod +x /usr/local/bin/pget"
]
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/v4/examples/transcribe-whisper

Initializes a new Cerebrium project for the Whisper transcription example. This command sets up the basic project structure.

```bash
cerebrium init 1-whisper-transcription
```

--------------------------------

### Initialize Infinity Project

Source: https://cerebrium.ai/docs/v4/examples/high-throughput-embeddings

Run this command to initialize a new project with the Infinity framework for high-throughput deployments. This sets up the necessary project structure.

```bash
cerebrium init infinity-throughput
```

--------------------------------

### GET Request with Custom Domain

Source: https://cerebrium.ai/docs/networking/custom-domains

Example of making a GET request to a Cerebrium app using a custom domain. Replace api.example.com with your configured custom domain.

```bash
curl https://api.example.com/v4/p-1234/my-app/run \
  -H "Authorization: Bearer {YOUR_API_KEY}"
```

--------------------------------

### Python Training Script for Llama 3.2 Fine-tuning

Source: https://cerebrium.ai/docs/v4/examples/wandb-sweep

This script configures and executes the fine-tuning process for a Llama 3.2 model. It includes dataset preparation, QLoRA implementation, training arguments setup, and model saving. Ensure necessary libraries like Hugging Face Transformers and PEFT are installed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoConfig
from peft import LoraConfig
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments
import wandb
import os

# Model and dataset names
model_name = "meta-llama/Llama-2-7b-hf"
dataset_name = "mosaicml/dolly-v2-3b"

# QLoRA configuration
lora_r = 64
lora_alpha = 16
lora_dropout = 0.1

# Training parameters
params = {
    "test_size": 0.1,
    "batch_size": 1,
    "gradient_accumulation_steps": 2,
    "epochs": 1,
    "eval_steps": 0.2,
    "logging_steps": 1,
    "warmup_steps": 10,
    "learning_rate": 2e-4,
    "fp16": False,
    "bf16": False,
    "group_by_length": True
}

# Output directory for the new model
new_model = "./cerebrium_llama_3_2_finetuned"

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=False,
    ),
    device_map="auto",
    trust_remote_code=True
)
model.config.use_cache = False
model.config.pretraining_tp = 1

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

# Load dataset
dataset = load_dataset(dataset_name, split="train")

# PEFT configuration
peft_config = LoraConfig(
    r=lora_r,
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    bias="none",
    task_type="CAUSAL_LM",
)

# Formatting function for chat template
def format_chat_template(row):
    row_json = [
        {"role": "user", "content": row["prompt"]},
        {"role": "assistant", "content": row["response"]}
    ]
    row["text"] = tokenizer.apply_chat_template(row_json, tokenize=False)
    return row

dataset = dataset.map(format_chat_template, num_proc=4)
dataset = dataset.train_test_split(test_size=params.get("test_size", 0.1))

# Training arguments
training_arguments = TrainingArguments(
    output_dir=new_model,
    per_device_train_batch_size=params.get("batch_size", 1),
    per_device_eval_batch_size=params.get("batch_size", 1),
    gradient_accumulation_steps=params.get("gradient_accumulation_steps", 2),
    optim="paged_adamw_32bit",
    num_train_epochs=params.get("epochs", 1),
    eval_strategy="steps",
    eval_steps=params.get("eval_steps", 0.2),
    logging_steps=params.get("logging_steps", 1),
    warmup_steps=params.get("warmup_steps", 10),
    learning_rate=params.get("learning_rate", 2e-4),
    fp16=params.get("fp16", False),
    bf16=params.get("bf16", False),
    group_by_length=params.get("group_by_length", True),
    report_to="wandb"
)

# Initialize trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    peft_config=peft_config,
    args=training_arguments,
)

# Train and save
model.config.use_cache = False
trainer.train()
model.config.use_cache = True

# Save model
trainer.model.save_pretrained(new_model)

wandb.finish()
return {"status": "success", "model_path": new_model}
```

--------------------------------

### Compare CUDA Base Images for Cold-Starts

Source: https://cerebrium.ai/docs/hardware/using-cuda

Choose between minimal runtime images for faster cold-starts or full development images with more tools, understanding the impact on initialization time.

```toml
# Minimal runtime image - faster cold-starts
docker_base_image_url = "debian:bookworm-slim"
```

```toml
# Full development image - slower cold-starts, more tools
docker_base_image_url = "nvidia/cuda:12.0.1-devel-ubuntu22.04"
```

--------------------------------

### Install Cerebrium CLI on Linux

Source: https://cerebrium.ai/docs/getting-started/introduction

Install the Cerebrium CLI on Linux systems. This includes options for Debian/Ubuntu using dpkg or a binary installation.

```bash
# Ubuntu/Debian
wget https://github.com/CerebriumAI/cerebrium/releases/latest/download/cerebrium_linux_amd64.deb
sudo dpkg -i cerebrium_linux_amd64.deb

# Or binary installation
curl -L https://github.com/CerebriumAI/cerebrium/releases/latest/download/cerebrium_cli_linux_amd64.tar.gz | tar xz
sudo mv cerebrium /usr/local/bin/
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/v4/examples/langchain-langsmith

Use this command to create a new Cerebrium project with a starter template. This command sets up the basic file structure, including the entrypoint and configuration files.

```bash
cerebrium init agent-tool-calling
```

--------------------------------

### Install Cerebrium CLI with Homebrew on macOS

Source: https://cerebrium.ai/docs/getting-started/introduction

Install the Cerebrium CLI on macOS using Homebrew. Ensure you have Homebrew installed before running this command.

```bash
brew tap cerebriumai/tap
brew install cerebrium
```

--------------------------------

### Install Dependencies

Source: https://cerebrium.ai/docs/v4/examples/wandb-sweep

Install required Python packages locally using pip. These are also needed on Cerebrium.

```bash
pip install -r requirements.txt
```

--------------------------------

### FastAPI Application Setup for Gradio Proxy

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Set up the main FastAPI application in `main.py`. This includes initializing the app, defining environment variables for the Gradio URL, setting up a health check endpoint, and creating a catchall route to proxy requests to the Gradio application.

```python
# at the top of your main.py file
import requests
from typing import Optional, List

import httpx
from fastapi import FastAPI, Request
from starlette.responses import Response as StarletteResponse

app = FastAPI()

# Get the Gradio app URL (when running on Cerebrium)
GRADIO_HOST = os.getenv("GRADIO_HOST", "127.0.0.1")
GRADIO_PORT = int(os.getenv("GRADIO_PORT", "7860"))
GRADIO_URL = os.getenv("GRADIO_SERVER_URL", f"http://{GRADIO_HOST}:{GRADIO_PORT}")

# Health check endpoint
@app.get("/health")
async def health_check():
    return {"status": "healthy"}


@app.route("/{path:path}", include_in_schema=False, methods=["GET", "POST"])
async def gradio(request: Request):
    print(f"Forwarding request path: {request.url.path}")

    headers = dict(request.headers)

    # Construct the full URL to Gradio, preserving the original path
    target_url = f"{GRADIO_URL}{request.url.path}"

    async with httpx.AsyncClient() as client:
        response = await client.request(
            request.method,
            target_url,
            headers=headers,
            data=await request.body(),
            params=request.query_params,
        )

        content = await response.aread()
        response_headers = dict(response.headers)
        return StarletteResponse(
            content=content,
            status_code=response.status_code,
            headers=response_headers,
        )
```

--------------------------------

### vLLM Model Setup and Inference

Source: https://cerebrium.ai/docs/v4/examples/mistral-vllm

Initialize the vLLM LLM with Mistral 7B and configure sampling parameters. The model loads once at startup. The `predict` function handles input and returns generated text.

```python
import os
from vllm import LLM, SamplingParams
from huggingface_hub import login

# Your huggingface token (HF_AUTH_TOKEN) should be stored in your project secrets on your dashboard
login(token=os.environ.get("HF_AUTH_TOKEN"))

# Initialize model with optimized settings
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1", dtype="bfloat16", max_model_len=20000, gpu_memory_utilization=0.9)


def predict(prompt, temperature=0.8, top_p=0.75, top_k=40, max_tokens=256, frequency_penalty=1):
    item = Item(
        prompt=prompt,
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        max_tokens=max_tokens,
        frequency_penalty=frequency_penalty
    )

    sampling_params = SamplingParams(
        temperature=item.temperature,
        top_p=item.top_p,
        top_k=item.top_k,
        max_tokens=item.max_tokens,
        frequency_penalty=item.frequency_penalty
    )

    outputs = llm.generate([item.prompt], sampling_params)

    generated_text = []
    for output in outputs:
        generated_text.append(output.outputs[0].text)

    return {"result": generated_text}

```

--------------------------------

### Create and Execute a LangChain Agent

Source: https://cerebrium.ai/docs/v4/examples/langchain-langsmith

Use 'create_tool_calling_agent' to combine the language model, tools, and prompt into an agent. Then, initialize an 'AgentExecutor' to run the agent with user input.

```python
from langchain.agents import AgentExecutor, create_tool_calling_agent

agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input": "what's 3 plus 5 raised to the 2.743. also what's 17.24 - 918.1241", })

```

--------------------------------

### Example Response from Cerebrium API

Source: https://cerebrium.ai/docs/v4/examples/deploy-a-vision-language-model-with-sglang

This is an example of the JSON response you can expect after successfully analyzing an ad using the Cerebrium vision-language model.

```json
{
  "success": true,
  "analysis": {
    "summary": "The company description is relevant to the image because it accurately reflects Nike's branding, which is showcased through the advertised sneaker and logo. The ad promotes Nike's core products—athletic footwear—and its values of performance, style, and inspiration, aligning with the brand's identity. The collaboration with a superhero theme further emphasizes innovation and empowerment, core ",
    "grade": "A"
  },
  "dimensions_evaluated": ["Effectiveness", "Clarity", "Appeal", "Credibility"]
}
```

--------------------------------

### Initialize a Cerebrium project

Source: https://cerebrium.ai/docs/getting-started/introduction

Initialize a new Cerebrium project with a specified name and navigate into the project directory. This sets up the basic project structure including `main.py` and `cerebrium.toml`.

```bash
cerebrium init my-first-app
cd my-first-app
```

--------------------------------

### Dockerfile for Runtime Stage

Source: https://cerebrium.ai/docs/container-images/custom-dockerfiles

This Dockerfile sets up the runtime environment by copying necessary files from a build stage. It exposes port 8192 and uses dumb-init to run the server.

```dockerfile
FROM gcr.io/distroless/base-debian12
WORKDIR /
COPY --from=build  /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib/x86_64-linux-gnu/libgcc_s.so.1
COPY --from=build /rs_server/target/release/rs_server /rs_server
COPY --from=build /usr/bin/dumb-init /usr/bin/dumb-init
EXPOSE 8192
CMD ["dumb-init", "--", "/rs_server"]
```

--------------------------------

### Install Cerebrium CLI on Windows

Source: https://cerebrium.ai/docs/getting-started/introduction

Install the Cerebrium CLI on Windows using PowerShell. This involves downloading a zip archive and extracting the executable.

```powershell
# PowerShell (Run as Administrator)
Invoke-WebRequest -Uri "https://github.com/CerebriumAI/cerebrium/releases/latest/download/cerebrium_cli_windows_amd64.zip" -OutFile "cerebrium.zip"
Expand-Archive -Path "cerebrium.zip" -DestinationPath "."
# Add cerebrium.exe to PATH
```

--------------------------------

### Initialize SGLang Runtime and FastAPI App

Source: https://cerebrium.ai/docs/v4/examples/deploy-a-vision-language-model-with-sglang

Sets up the FastAPI application and initializes the SGLang Runtime Engine with a specified multimodal model. This code should be placed in your `main.py` file and runs on container startup.

```python
import sglang as sgl
from sglang import function
from fastapi import FastAPI, HTTPException
from transformers import AutoProcessor

app = FastAPI(title="Vision Language SGLang API")
model_path = "Qwen/Qwen3-VL-30B-A3B-Instruct-FP8"
processor = AutoProcessor.from_pretrained(model_path)

@app.on_event("startup")
def _startup_warmup():
    # Initialize engine on main thread during app startup
    runtime = sgl.Runtime(
        model_path=model_path,
        enable_multimodal=True,
        mem_fraction_static=0.8,
        tp_size=1,
        attention_backend="flashinfer",
    )
    runtime.endpoint.chat_template = sgl.lang.chat_template.get_chat_template(
        "qwen2-vl"
    )
    sgl.set_default_backend(runtime)


@app.get("/health")
def health():
    return {
        "status": "healthy",
    }
```

--------------------------------

### Install Cerebrium CLI with pip

Source: https://cerebrium.ai/docs/getting-started/introduction

Install the Cerebrium CLI using pip for Python environments. This is the primary method for managing your Cerebrium projects and deployments.

```bash
pip install cerebrium
```

--------------------------------

### Python Project Dependencies

Source: https://cerebrium.ai/docs/v4/examples/livekit-outbound-agent

This requirements.txt file lists the necessary Python packages for the outbound AI agent. Install them using 'pip install -r requirements.txt'.

```python
livekit-agents>=0.11.1
livekit-plugins-openai>=0.10.5
livekit-plugins-deepgram
livekit-plugins-cartesia
livekit-plugins-silero>=0.7.3
livekit-plugins-rag>=0.2.2
python-dotenv~=1.0
aiofile~=3.8.8
fastapi
uvicorn
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/v4/examples/sdxl

Use this command to create a new project directory for the SDXL refiner model. Ensure you are using CLI v1.20 or later.

```bash
cerebrium init 2-sdxl-refiner
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/migrations/replicate

Use this command to create a new Cerebrium project directory.

```bash
cerebrium init cog-migration-sdxl
```

--------------------------------

### Start Gradio Server Process

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Initiates the Gradio server in a separate process to avoid blocking the main application. Includes a retry mechanism to confirm the server is ready.

```python
def start(self):
        print(f"Starting Gradio server at {self.url} port {self.port}")

        # Start Gradio in a separate process
        self.process = multiprocessing.Process(target=self.run_server)
        self.process.start()

        # Wait for Gradio to become ready
        max_retries = 30
        retry_delay = 1.0

        for _ in range(max_retries):
            try:
                response = requests.get(f"{self.url}/")
                if response.status_code == 200:
                    print(f"Gradio server is ready at {self.url}")
                    return True
            except requests.exceptions.ConnectionError:
                time.sleep(retry_delay)

        print("Failed to start Gradio server")
        self.stop()
        return False
```

--------------------------------

### Agent Setup with LangChain

Source: https://cerebrium.ai/docs/v4/examples/langchain-langsmith

This code sets up a LangChain agent executor. It defines a prompt template, specifies the LLM (GPT-3.5 Turbo), and integrates the previously defined tools for calendar management.

```python
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.tools import tool
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_messages([
    ("system", "you're a helpful assistant managing the calendar of Michael Louis. You need to book appointments for a user based on available capacity and their preference. You need to find out if the user is: From Michaels team, a customer of Cerebrium or a friend or entrepreneur. If the person is from his team, book a morning slot. If its a potential customer for Cerebrium, book an afternoon slot. If its a friend or entrepreneur needing help or advice, book a night time slot. If none of these are available, book the earliest slot. Do not book a slot without asking the user what their preferred time is. Find out from the user, their name and email address."),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

tools = [get_availability, book_slot]


llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0, api_key=os.environ.get("OPENAI_API_KEY"))
agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

```

--------------------------------

### Example Output of Streaming Chat Completion

Source: https://cerebrium.ai/docs/endpoints/openai-compatible-endpoints

This is an example of the output received when streaming chat completions from a Cerebrium OpenAI-compatible endpoint. Each chunk contains a part of the generated text.

```text
Starting to receive chunks...
ChatCompletionChunk(id='412f0e25-61c4-93b8-a00f-09a5076cd9fa', choices=[Choice(delta=None, finish_reason='stop', index=0, logprobs=None, text=' The')], created=1724166657, model='gpt-3.5-turbo', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='412f0e25-61c4-93b8-a00f-09a5076cd9fa', choices=[Choice(delta=None, finish_reason='stop', index=0, logprobs=None, text=' formation')], created=1724166657, model='gpt-3.5-turbo', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='412f0e25-61c4-93b8-a00f-09a5076cd9fa', choices=[Choice(delta=None, finish_reason='stop', index=0, logprobs=None, text=' of')], created=1724166657, model='gpt-3.5-turbo', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='412f0e25-61c4-93b8-a00f-09a5076cd9fa', choices=[Choice(delta=None, finish_reason='stop', index=0, logprobs=None, text=' the')], created=1724166657, model='gpt-3.5-turbo', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None)
ChatCompletionChunk(id='412f0e25-61c4-93b8-a00f-09a5076cd9fa', choices=[Choice(delta=None, finish_reason='stop', index=0, logprobs=None, text=' mist')], created=1724166657, model='gpt-3.5-turbo', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None)
...
```

--------------------------------

### Example API Response from Cerebrium Deployment

Source: https://cerebrium.ai/docs/v4/examples/langchain-langsmith

This is an example JSON response received after deploying to Cerebrium, showing the structure of a successful API call, including input, chat history, and output.

```json
{
    "run_id": "UHCJ_GkTKh451R_nKUd3bDxp8UJrcNoPWfEZ3AYiqdY85UQkZ6S1vg==",
    "status_code": 200,
    "result": {
        "result": {
            "input": "Hi! I would like to book a time with Michael the 18th of April 2024.",
            "chat_history": [],
            "output": "Michael is available on the 18th of April 2024 at the following times:\n1. 13:00 - 13:30\n2. 14:45 - 17:00\n3. 17:45 - 19:00\n\nPlease let me know your preferred time slot. Are you from Michael's team, a potential customer of Cerebrium, or a friend/entrepreneur seeking advice?"
        }
    },
    "run_time_ms": 6728.828907012939,
    "process_time_ms": 6730.178117752075
}
```

--------------------------------

### ComfyUI Server Startup Event

Source: https://cerebrium.ai/docs/v4/examples/comfyUI

Handles the application startup event by loading the workflow and starting the ComfyUI server process. It includes logic to wait for the server to become available via WebSocket.

```python
@app.on_event("startup")
async def startup_event():
    """Start ComfyUI server on application startup."""
    global json_workflow, side_process

    # Load workflow JSON
    json_workflow = load_workflow_file("workflow_api.json")
    logger.info("Loaded workflow from workflow_api.json")

    # Start ComfyUI process
    if side_process is None:
        side_process = Process(
            target=setup_comfyui,
            kwargs=dict(original_working_directory=original_working_directory, data_dir=""),
            daemon=True,
        )
        side_process.start()
        logger.info(f"Started ComfyUI process (PID: {side_process.pid})")

        for sig in [signal.SIGINT, signal.SIGTERM]:
            signal.signal(sig, lambda s, f: terminate_process())

        # Wait for ComfyUI to start
        max_attempts = 30
        for attempt in range(max_attempts):
            try:
                with websocket_connection() as (ws, _):
                    logger.info("Successfully connected to ComfyUI!")
                    break
            except Exception:
                logger.info(f"Waiting for ComfyUI to start... ({attempt + 1}/{max_attempts})")
                time.sleep(2)
        else:
            logger.warning("Could not confirm ComfyUI is running after multiple attempts")

```

--------------------------------

### FastAPI and Imports

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Basic setup for a FastAPI application, including necessary imports for multiprocessing, environment variables, time, typing, Gradio, httpx, requests, and FastAPI.

```python
import multiprocessing
import os
import sys
import time
from typing import Optional, List

import gradio as gr
import httpx
import requests
from fastapi import FastAPI, Request
from starlette.responses import Response as StarletteResponse

# Initialize FastAPI
app = FastAPI()
```

--------------------------------

### Get TOML Config

Source: https://cerebrium.ai/docs/api-reference/integrations/get-toml-config

Retrieve the cerebrium.toml configuration from a GitHub repository.

```APIDOC
## GET /v4/projects/{project_id}/integrations/github/repos/{owner}/{repo}/toml

### Description
Parse and retrieve the cerebrium.toml configuration from a GitHub repository.

### Method
GET

### Endpoint
/v4/projects/{project_id}/integrations/github/repos/{owner}/{repo}/toml

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **owner** (string) - Required - The owner of the GitHub repository.
- **repo** (string) - Required - The name of the GitHub repository.

### Responses
#### Success Response (200)
- **(object)** - OK

### Authentication
Service Account Token authentication. Include the service account token in the Authorization header of API requests:
`Authorization: Bearer <your-service-account-token>`
```

--------------------------------

### Get App Cost

Source: https://cerebrium.ai/docs/api-reference/apps/get-app-cost

Retrieve cost breakdown for a specific app.

```APIDOC
## GET /v2/projects/{project_id}/apps/{app_id}/cost

### Description
Retrieve cost breakdown for a specific app.

### Method
GET

### Endpoint
/v2/projects/{project_id}/apps/{app_id}/cost

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **app_id** (string) - Required - The ID of the app.

### Response
#### Success Response (200)
- **(No specific fields described in the provided OpenAPI spec)**

#### Response Example
(No example provided in the OpenAPI spec)
```

--------------------------------

### Configure HTTPS Server

Source: https://cerebrium.ai/docs/partner-services/deepgram

Enable HTTPS by providing paths to certificate and key files. This performs TLS termination only.

```toml
[server.https]
# cert_file = "/path/to/cert.pem"
# key_file = "/path/to/key.pem"
```

--------------------------------

### Get Subscription Information

Source: https://cerebrium.ai/docs/api-reference/subscriptions/get-subscription

Retrieve the current subscription plan and status for a specific project.

```APIDOC
## GET /v2/projects/{project_id}/subscription

### Description
Retrieve the current subscription plan and status for a project.

### Method
GET

### Endpoint
/v2/projects/{project_id}/subscription

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project to retrieve subscription information for.

### Request Example
(No request body for GET requests)

### Response
#### Success Response (200)
- **subscription_plan** (string) - The current subscription plan.
- **status** (string) - The current status of the subscription.

#### Response Example
(Example response body not provided in the source text)
```

--------------------------------

### Get Metrics Export Config

Source: https://cerebrium.ai/docs/api-reference/settings/get-metrics-export-config

Retrieve the OTLP metrics export configuration for a project.

```APIDOC
## GET /v2/metrics-export/{project_id}/config

### Description
Retrieve the OTLP metrics export configuration for a project.

### Method
GET

### Endpoint
/v2/metrics-export/{project_id}/config

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project for which to retrieve the metrics export configuration.

### Request Example
(No request body for GET request)

### Response
#### Success Response (200)
(Response structure not detailed in the provided text, but typically includes OTLP configuration details.)

#### Response Example
(Response example not provided in the source text.)
```

--------------------------------

### Initialize SGLang Project

Source: https://cerebrium.ai/docs/v4/examples/deploy-a-vision-language-model-with-sglang

Use the `cerebrium init` command to create a new project directory for your SGLang application. Navigate into the created directory to begin development.

```bash
cerebrium init 7-vision-language-sglang
cd 7-vision-language-sglang
```

--------------------------------

### Get Run Details

Source: https://cerebrium.ai/docs/api-reference/runs/get-run

Retrieve details for a specific run of an app within a project.

```APIDOC
## GET /v2/projects/{project_id}/apps/{app_id}/runs/{run_id}

### Description
Retrieve details for a specific run of an app.

### Method
GET

### Endpoint
/v2/projects/{project_id}/apps/{app_id}/runs/{run_id}

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **app_id** (string) - Required - The ID of the application.
- **run_id** (string) - Required - The ID of the run.

### Response
#### Success Response (200)
- **(No specific fields described in the provided OpenAPI spec)**

### Authentication
This endpoint uses Bearer Token authentication. Include your Service Account Token in the `Authorization` header:
`Authorization: Bearer <your-service-account-token>`
```

--------------------------------

### Get Project Cost

Source: https://cerebrium.ai/docs/api-reference/projects/get-project-cost

Retrieve current billing period cost breakdown for a project.

```APIDOC
## GET /v2/projects/{project_id}/cost

### Description
Retrieve current billing period cost breakdown for a project.

### Method
GET

### Endpoint
/v2/projects/{project_id}/cost

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project to retrieve cost information for.

### Request Example
(No request body for GET requests)

### Response
#### Success Response (200)
(Response schema not detailed in the provided text, but typically includes cost breakdown details)

#### Response Example
(Response example not detailed in the provided text)
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/v4/examples/asgi-gradio-interface

Use the Cerebrium CLI to initialize a new project directory for the Gradio interface.

```bash
cerebrium init 2-gradio-interface
```

--------------------------------

### Get Response Time Metrics

Source: https://cerebrium.ai/docs/api-reference/metrics/get-response-time-metrics

Retrieve end-to-end response time percentiles for an app.

```APIDOC
## GET /v2/projects/{project_id}/apps/{app_id}/metrics/response-time

### Description
Retrieve end-to-end response time percentiles for an app.

### Method
GET

### Endpoint
/v2/projects/{project_id}/apps/{app_id}/metrics/response-time

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **app_id** (string) - Required - The ID of the application.

### Response
#### Success Response (200)
- **response_time_metrics** (object) - Contains response time percentiles.

#### Response Example
{
  "response_time_metrics": {
    "p50": "123.45ms",
    "p90": "234.56ms",
    "p99": "345.67ms"
  }
}

### Security
- BearerAuth: Service Account Token authentication. Include the token in the Authorization header: `Authorization: Bearer <your-service-account-token>`
```

--------------------------------

### Initialize Cerebrium Project

Source: https://cerebrium.ai/docs/v4/examples/mistral-vllm

Use this command to initialize a new Cerebrium project for the vLLM example. Ensure you are using CLI v1.20 or later.

```bash
cerebrium init 1-faster-inference-with-vllm
```

--------------------------------

### Get Repo Metadata

Source: https://cerebrium.ai/docs/api-reference/integrations/get-repo-metadata

Retrieve metadata for a GitHub repository within a specific project.

```APIDOC
## GET /v4/projects/{project_id}/integrations/github/repos/{owner}/{repo}

### Description
Retrieve metadata for a GitHub repository.

### Method
GET

### Endpoint
/v4/projects/{project_id}/integrations/github/repos/{owner}/{repo}

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **owner** (string) - Required - The owner of the GitHub repository.
- **repo** (string) - Required - The name of the GitHub repository.

### Response
#### Success Response (200)
- **metadata** (object) - Description of the repository metadata (details not provided in source).

#### Response Example
```json
{
  "metadata": {}
}
```

### Security
- **BearerAuth**: Service Account Token authentication. Include the token in the `Authorization: Bearer <your-service-account-token>` header.
```

--------------------------------

### Initialize vLLM Engine and Login to Hugging Face

Source: https://cerebrium.ai/docs/v4/examples/openai-compatible-endpoint-vllm

Set up the vLLM asynchronous engine and log in to Hugging Face using a token stored in environment secrets. This code initializes the engine with a specific model and configures GPU memory utilization and model length.

```python
from vllm import SamplingParams, AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
from pydantic import BaseModel
from typing import Any
import time
import json
import os
from huggingface_hub import login

# Your huggingface token (HF_AUTH_TOKEN) should be stored in your project secrets on your dashboard
login(token=os.environ.get("HF_AUTH_TOKEN"))

engine_args = AsyncEngineArgs(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.9,  # Increase GPU memory utilization
    max_model_len=8192  # Decrease max model length
)
engine = AsyncLLMEngine.from_engine_args(engine_args)
```

--------------------------------

### Get Custom Domain

Source: https://cerebrium.ai/docs/api-reference/custom-domains/get-custom-domain

Retrieve details for a specific custom domain within a project.

```APIDOC
## GET /v2/projects/{project_id}/domains/{domain_id}

### Description
Retrieve details for a specific custom domain.

### Method
GET

### Endpoint
/v2/projects/{project_id}/domains/{domain_id}

### Parameters
#### Path Parameters
- **project_id** (string) - Required - The ID of the project.
- **domain_id** (string) - Required - The ID of the custom domain.

### Response
#### Success Response (200)
- **(object)** - Details of the custom domain.

#### Response Example
{
  "example": "{\"domain_id\": \"your_domain_id\", \"project_id\": \"your_project_id\", \"domain\": \"example.com\", \"status\": \"active\", \"created_at\": \"2023-01-01T12:00:00Z\", \"updated_at\": \"2023-01-01T12:00:00Z\"}"
}
```