### Python OpenAI SDK Quickstart for LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway

This Python snippet shows how to use the OpenAI SDK with the LLM Gateway. Configure the client with the gateway's base URL and your API key. Ensure you have the 'openai' library installed.

```python
from openai import OpenAI
import openai.types.chat.chat_completion as types

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key="<YOUR_API_KEY>",
)

import json

messages = [{"role": "user", "content": "What is the capital of france"}]
response = client.chat.completions.create(model="claude-sonnet-4.6", messages=messages)
print(response.choices[0].message.content)

```

--------------------------------

### Simple Prompt Example

Source: https://www.assemblyai.com/docs/llm-gateway/prompt-engineering

A basic prompt consisting of a single instruction to guide the LLM.

```text
Provide a summary of the meeting.
```

--------------------------------

### Python Quickstart for LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway

Use this Python snippet to send a chat completion request to the LLM Gateway. Ensure you have the 'requests' library installed and replace '<YOUR_API_KEY>' with your actual API key.

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json = {
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])

```

--------------------------------

### First Response: Tool Call Example

Source: https://www.assemblyai.com/docs/llm-gateway/agentic-workflows

This JSON shows the initial response from the LLM Gateway when it identifies a need to use a tool. It includes the tool's name and arguments.

```json
{
  "request_id": "abc123",
  "choices": [
    {
      "index": 0,
      "finish_reason": "tool_use",
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "search_employees",
              "arguments": "{\"first_name\": \"Sarah\"}"
            }
          }
        ]
      }
    }
  ],
  "request": { "model": "claude-sonnet-4-6", "max_tokens": 1000 },
  "usage": { "input_tokens": 150, "output_tokens": 30, "total_tokens": 180 }
}
```

--------------------------------

### JavaScript Quickstart for LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway

This JavaScript snippet demonstrates how to make a chat completion request using the Fetch API. Replace '<YOUR_API_KEY>' with your valid API key.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);

```

--------------------------------

### Install Requests Package for Python

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

Install the 'requests' package using pip. This is required for making API calls in Python.

```bash
pip install requests
```

--------------------------------

### Example Agentic Interaction

Source: https://www.assemblyai.com/docs/llm-gateway/agentic-workflows

Illustrates a multi-step agentic interaction where the model first searches for employee information and then uses that to find location details. This demonstrates autonomous tool selection and chaining.

```text
You: Where does Andrew live?
🔧 Calling: search_employees(first_name="Andrew")
✅ Result: Found 1 result
🔧 Calling: search_tavily(query="33186 zip code location")
✅ Result: The zip code 33186 is located in Miami, Florida...
Assistant: Andrew lives in Miami, Florida (zip code 33186).
```

--------------------------------

### Enable Global Routing with JavaScript

Source: https://www.assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency

This JavaScript example demonstrates how to make a POST request to the LLM Gateway with global routing. The `model_region` parameter must be set to 'global' for this feature to activate.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      model_region: "global",
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);

```

--------------------------------

### Chat Completions Request Example

Source: https://www.assemblyai.com/docs/llm-gateway/chat-completions

Use this cURL command to send a POST request to the LLM Gateway for chat completions. Ensure you replace 'YOUR_API_KEY' with your actual AssemblyAI API key.

```bash
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "max_tokens": 1000
  }'
```

--------------------------------

### Python: Structured Q&A with LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway/ask-questions

Use this Python script to transcribe audio, create a structured Q&A prompt, and send it to LLM Gateway for formatted responses. Ensure you have the 'requests' and 'time' libraries installed.

```python
import requests
import time

# Step 1: Transcribe the audio
base_url = "https://api.assemblyai.com"
headers = {"authorization": "<YOUR_API_KEY>"}

audio_url = "https://assembly.ai/meeting.mp4"
data = {"audio_url": audio_url}
response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)
transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()
    if transcript["status"] == "completed":
        break
    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")
    else:
        time.sleep(3)

# Step 2: Create structured Q&A prompt
questions = [
    "What are the top level KPIs for engineering? (KPI stands for key performance indicator)",
    "How many days has it been since the data team has gotten updated metrics? (Choose from: 1, 2, 3, 4, 5, 6, 7, more than 7)"
]

prompt = f"""Answer the following questions based on the meeting transcript. Format your response as:
Q1: [question]
A1: [answer]
Q2: [question]
A2: [answer]

Questions:
    Ꭵ. {{'. '.join([f'{i+1}. {q}' for i, q in enumerate(questions)])}}

Context: This is a GitLab meeting to discuss logistics."""

# Step 3: Send to LLM Gateway
llm_gateway_data = {
    "model": "claude-sonnet-4-5-20250929",
    "messages": [
        {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
    ],
    "transcript_id": transcript_id,
    "max_tokens": 1000
}

result = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json=llm_gateway_data
)
print(result.json()["choices"][0]["message"]["content"])

```

--------------------------------

### Example Structured JSON Response

Source: https://www.assemblyai.com/docs/llm-gateway/structured-outputs

This is an example of a JSON response when using structured outputs. The 'content' field contains a JSON string that conforms to a specified schema.

```json
{
  "request_id": "abc123",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "{\"steps\":[{\"explanation\":\"Start with the equation 8x + 7 = -23\",\"output\":\"8x + 7 = -23\"},{\"explanation\":\"Subtract 7 from both sides to isolate the term with x\",\"output\":\"8x = -30\"},{\"explanation\":\"Divide both sides by 8 to solve for x\",\"output\":\"x = -30/8 = -15/4 = -3.75\"}],\"final_answer\":\"x = -3.75\"}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input_tokens": 85,
    "output_tokens": 120,
    "total_tokens": 205
  }
}
```

--------------------------------

### Enable Global Routing with Python

Source: https://www.assemblyai.com/docs/llm-gateway/cloud-endpoints-and-data-residency

Use this snippet to send a request to the LLM Gateway with global routing enabled. Ensure you have the 'requests' library installed. Set `model_region` to 'global' to opt-in.

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "model_region": "global",
        "max_tokens": 1000
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])

```

--------------------------------

### Python: Transcribe Audio and Summarize with LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

This snippet shows how to transcribe an audio file using AssemblyAI and then send the transcript to LLM Gateway for summarization. It handles audio upload, polling for transcription completion, and making the LLM Gateway API call. Ensure you have the 'requests' and 'time' libraries installed.

```python
import requests
import time

# Step 1: Transcribe the audio
base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
# response = requests.post(base_url + "/v2/upload",
# headers=headers,
# data=f)
# upload_url = response.json()["upload_url"]
# Or use a publicly-accessible URL:

upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {
  "audio_url": upload_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Step 2: Send transcript to LLM Gateway
prompt = "Provide a brief summary of the transcript."

llm_gateway_data = {
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}"}
  ],
  "transcript_id": transcript_id,
  "max_tokens": 1000
}

response = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
print(response.json()["choices"][0]["message"]["content"])
```

--------------------------------

### LLM Gateway Chat Completion Request

Source: https://www.assemblyai.com/docs/llm-gateway/agentic-workflows

Example of a POST request to the chat completions endpoint. Use this to send messages and tool definitions to the LLM Gateway.

```bash
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "Where does Sarah work and what city is that in?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "search_employees",
          "description": "Search employee database",
          "parameters": {
            "type": "object",
            "properties": {
              "first_name": {
                "type": "string"
              }
            }
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "search_web",
          "description": "Search the web for information",
          "parameters": {
            "type": "object",
            "properties": {
              "query": {
                "type": "string"
              }
            }
          }
        }
      }
    ],
    "max_tokens": 1000
  }'
```

--------------------------------

### Specify Plain Text Answer Format

Source: https://www.assemblyai.com/docs/llm-gateway/prompt-engineering

Use 'Answer Format:' to guide the LLM to produce a concise, plain text summary.

```plaintext
Provide a summary of the podcast.

Context:
This is an episode of the Lex Fridman podcast with guest Sam Altman, the CEO of OpenAI.

Answer Format:
catchy title, no longer than 10 words
```

--------------------------------

### Python Example: Agentic Workflow Loop

Source: https://www.assemblyai.com/docs/llm-gateway/agentic-workflows

This Python snippet demonstrates how to implement an agentic workflow using a loop to handle sequential tool calls. It continues making calls until the model provides a final answer or reaches the maximum number of iterations. Ensure you have a `chat` function and `handle_tool_calls` defined.

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

conversation_history = [
    {"role": "user", "content": "Where does Sarah work and what city is that in?"}
]

max_iterations = 10
iteration = 0

while iteration < max_iterations:
    iteration += 1
    response = chat(conversation_history, model)
    choice = response["choices"][0]

    if choice.get("message", {}).get("tool_calls"):
        # Execute tools and add results to history
        handle_tool_calls(choice["message"]["tool_calls"], conversation_history)
        # Continue loop - model can make more tool calls
        continue
    else:
        # Model has final answer
        print(choice["message"]["content"])
        break
```

--------------------------------

### Log Request Details in JavaScript

Source: https://www.assemblyai.com/docs/llm-gateway/troubleshooting

Capture essential request and response details, including `request_id`, model, region, timestamp, status code, and errors, for debugging and support. This example uses the `fetch` API.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      max_tokens: 1000,
    }),
  }
);

const result = await response.json();
console.log({
  timestamp: Date.now(),
  region: "us",
  model: "claude-sonnet-4-6",
  status_code: response.status,
  request_id: result.request_id,
  error: result.error,
});
```

--------------------------------

### Log Request Details in Python

Source: https://www.assemblyai.com/docs/llm-gateway/troubleshooting

Capture essential request and response details, including `request_id`, model, region, timestamp, status code, and errors, for debugging and support. This example uses the `requests` library.

```python
import requests
import time

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers={\"authorization\": \"<YOUR_API_KEY>\"},
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": "What is the capital of France?"}],
        "max_tokens": 1000,
    },
)

result = response.json()
log_entry = {
    "timestamp": time.time(),
    "region": "us",
    "model": "claude-sonnet-4-6",
    "status_code": response.status_code,
    "request_id": result.get("request_id"),
    "error": result.get("error"),
}
print(log_entry)
```

--------------------------------

### Stream OpenAI Chat Completions in JavaScript

Source: https://www.assemblyai.com/docs/llm-gateway/chat-completions

Enable streaming by setting `stream: true` in the request body. This JavaScript example uses the Fetch API to read the response stream chunk by chunk.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-4.1",
      messages: [{ role: "user", content: "What is the capital of France?" }],
      stream: true,
      max_tokens: 1000,
    }),
  }
);

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  console.log(decoder.decode(value));
}
```

--------------------------------

### Send Chat Completion Request with cURL

Source: https://www.assemblyai.com/docs/llm-gateway/structured-outputs

Use this cURL command to send a chat completion request to the LLM Gateway. Ensure you replace '<YOUR_API_KEY>' with your actual API key. This example demonstrates sending a system message, a user message, and specifies a JSON schema for the response format.

```bash
curl -X POST "https://llm-gateway.assemblyai.com/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: <YOUR_API_KEY>" \
    -d '{
          "model": "gemini-2.5-flash-lite",
          "messages": [
            {
              "role": "system",
              "content": "You are a helpful math tutor. Guide the user through the solution step by step."
            },
            {
              "role": "user",
              "content": "how can I solve 8x + 7 = -23"
            }
          ],
          "response_format": {
            "type": "json_schema",
            "json_schema": {
              "name": "math_reasoning",
              "schema": {
                "type": "object",
                "properties": {
                  "steps": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "explanation": { "type": "string" },
                        "output": { "type": "string" }
                      },
                      "required": ["explanation", "output"],
                      "additionalProperties": false
                    }
                  },
                  "final_answer": { "type": "string" }
                },
                "required": ["steps", "final_answer"],
                "additionalProperties": false
              },
              "strict": true
            }
          }
        }'
```

--------------------------------

### JavaScript: Transcribe Audio and Summarize with LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

This snippet demonstrates transcribing an audio file using AssemblyAI and then using LLM Gateway for summarization in JavaScript. It includes file reading, API calls for upload and transcription, polling for completion, and sending data to the LLM Gateway. Requires 'fs-extra' and Node.js environment with fetch support.

```javascript
import fs from "fs-extra";

// Step 1: Transcribe the audio
const base_url = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${base_url}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();

const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL of an audio or video file on the web
};

res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

let transcript;
while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();

  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}

// Step 2: Send transcript to LLM Gateway
const prompt = "Provide a brief summary of the transcript.";

const llm_gateway_data = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcript_id,
  max_tokens: 1000,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
console.log(result.choices[0].message.content);
```

--------------------------------

### Get Existing Transcript (Python)

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

Retrieves a previously generated transcript using its ID. Requires the 'requests' library and your API key.

```python
transcript = requests.get("https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID", headers=headers).json()
```

--------------------------------

### Define and Use Tools with LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway/tool-calling

Define tools with their names, descriptions, and parameters. Then, use these tools in a chat completion request to allow the model to call them.

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_employees",
            "description": "Search employee database by name, department, or other criteria",
            "parameters": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "string",
                        "description": "Employee's first name"
                    },
                    "department": {
                        "type": "string",
                        "description": "Department name"
                    }
                },
                "required": []
            }
        }
    }
]

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers = headers,
    json={
        "model": "claude-sonnet-4-6",
        "messages": [{"role": "user", "content": "Find employees in Engineering"}],
        "tools": tools
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Get Existing Transcript (JavaScript)

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

Retrieves a previously generated transcript using its ID. Requires your API key and uses the fetch API.

```javascript
const res = await fetch("https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID", { headers });
if (!res.ok) throw new Error(`Error: ${res.status}`);
const transcript = await res.json();
```

--------------------------------

### Stream OpenAI Chat Completions in Python

Source: https://www.assemblyai.com/docs/llm-gateway/chat-completions

Use the `stream=True` parameter to receive responses as server-sent events. This Python example iterates over the response lines and decodes them.

```python
import requests

headers = {
  "authorization": "<YOUR_API_KEY>"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gpt-4.1",
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "stream": True,
        "max_tokens": 1000
    },
    stream=True
)

for line in response.iter_lines():
    if line:
        print(line.decode("utf-8"))
```

--------------------------------

### LLM Gateway Chat Completions API Request

Source: https://www.assemblyai.com/docs/llm-gateway/tool-calling

Example cURL request to the LLM Gateway's chat completions endpoint, demonstrating how to define a tool for searching employees.

```bash
curl -X POST \
  "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [
      {
        "role": "user",
        "content": "Find employees in Engineering"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "search_employees",
          "description": "Search employee database by name, department, or other criteria",
          "parameters": {
            "type": "object",
            "properties": {
              "department": {
                "type": "string",
                "description": "Department name"
              }
            }
          }
        }
      }
    ],
    "max_tokens": 1000
  }'
```

--------------------------------

### Transcribe Audio File (JavaScript)

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

Uploads an audio file from a local path and polls for transcription completion using fetch. Requires 'fs-extra' for file reading. Replace '<YOUR_API_KEY>' with your actual AssemblyAI API key.

```javascript
import fs from "fs-extra";

const base_url = "https://api.assemblyai.com";

const headers = {
  authorization: "<YOUR_API_KEY>",
};

const path = "./my-audio.mp3";
const audioData = await fs.readFile(path);
let res = await fetch(`${base_url}/v2/upload`, {
  method: "POST",
  headers,
  body: audioData,
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const uploadResponse = await res.json();
const uploadUrl = uploadResponse.upload_url;

const data = {
  audio_url: uploadUrl, // You can also use a URL of an audio or video file on the web
};

res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();

const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  const transcript = await res.json();

  if (transcript.status === "completed") {
    break;
  } else if (transcript.status === "error") {
    throw new Error(`Transcription failed: ${transcript.error}`);
  } else {
    await new Promise((resolve) => setTimeout(resolve, 3000));
  }
}
```

--------------------------------

### JavaScript: Structured Q&A with LLM Gateway

Source: https://www.assemblyai.com/docs/llm-gateway/ask-questions

This JavaScript code snippet demonstrates how to transcribe audio, construct a structured Q&A prompt, and submit it to LLM Gateway. It uses the Fetch API for network requests.

```javascript
// Step 1: Transcribe the audio
const base_url = "https://api.assemblyai.com";
const headers = { authorization: "<YOUR_API_KEY>" };

const audio_url = "https://assembly.ai/meeting.mp4";
const data = { audio_url };

let res = await fetch(base_url + "/v2/transcript", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const response = await res.json();
const transcript_id = response.id;
const polling_endpoint = base_url + `/v2/transcript/${transcript_id}`;

let transcript;
while (true) {
  res = await fetch(polling_endpoint, { headers });
  if (!res.ok) throw new Error(`Error: ${res.status}`);
  transcript = await res.json();
  if (transcript.status === "completed") break;
  if (transcript.status === "error")
    throw new Error(`Transcription failed: ${transcript.error}`);
  await new Promise((resolve) => setTimeout(resolve, 3000));
}

// Step 2: Create structured Q&A prompt
const questions = [
  "What are the top level KPIs for engineering? (KPI stands for key performance indicator)",
  "How many days has it been since the data team has gotten updated metrics? (Choose from: 1, 2, 3, 4, 5, 6, 7, more than 7)",
];

const prompt = `Answer the following questions based on the meeting transcript. Format your response as:
Q1: [question]
A1: [answer]
Q2: [question]
A2: [answer]

Questions:
    ${questions.map((q, i) => `${i + 1}. ${q}`).join("\n")} 

Context: This is a GitLab meeting to discuss logistics.`;

// Step 3: Send to LLM Gateway
const llm_gateway_data = {
  model: "claude-sonnet-4-6",
  messages: [
    { role: "user", content: `${prompt}\n\n{{ transcript }}` },
  ],
  transcript_id: transcript_id,
  max_tokens: 1000,
};

res = await fetch("https://llm-gateway.assemblyai.com/v1/chat/completions", {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify(llm_gateway_data),
});
if (!res.ok) throw new Error(`Error: ${res.status}`);
const result = await res.json();
console.log(result.choices[0].message.content);

```

--------------------------------

### Transcribe Audio and Ask Questions (Python)

Source: https://www.assemblyai.com/docs/llm-gateway/ask-questions

Use this Python script to transcribe an audio file and then query the transcript using the LLM Gateway. Ensure you have your API key and the transcript ID is valid.

```python
import requests
import time

# Step 1: Transcribe an audio file.
base_url = "https://api.assemblyai.com"

headers = {
  "authorization": "<YOUR_API_KEY>"
}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
# response = requests.post(base_url + "/v2/upload",
#                         headers=headers,
#                         data=f)
# upload_url = response.json()["upload_url"]

# Or use a publicly-accessible URL:
upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {
  "audio_url": upload_url
}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
  transcript = requests.get(polling_endpoint, headers=headers).json()

  if transcript["status"] == "completed":
    break

  elif transcript["status"] == "error":
    raise RuntimeError(f"Transcription failed: {transcript['error']}")

  else:
    time.sleep(3)

# Step 2: Define a prompt with your question(s).
prompt = "What is a runner's knee?"

# Step 3: Send to LLM Gateway.
llm_gateway_data = {
  "model": "claude-sonnet-4-5-20250929",
  "messages": [
    {"role": "user", "content": f"{prompt}\n\n{{{{ transcript }}}}}}"
  ],
  "transcript_id": transcript_id,
  "max_tokens": 1000
}

result = requests.post(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  headers=headers,
  json=llm_gateway_data
)
print(result.json()["choices"][0]["message"]["content"])
```

--------------------------------

### Example 404 Response Body

Source: https://www.assemblyai.com/docs/llm-gateway/troubleshooting

This JSON structure represents a typical 'transcript not found' error response from the API. It includes the HTTP status code, a message, and a request ID for debugging.

```json
{
  "code": 404,
  "message": "transcript not found",
  "request_id": "bf08febb-ee48-4ce1-b473-7c7b15561033"
}
```

--------------------------------

### Transcribe Audio File (Python)

Source: https://www.assemblyai.com/docs/llm-gateway/apply-llms-to-audio-files

Uploads an audio file from a URL and polls for transcription completion. Requires the 'requests' and 'time' libraries. Replace '<YOUR_API_KEY>' with your actual AssemblyAI API key.

```python
import requests
import time

base_url = "https://api.assemblyai.com"

headers = {"authorization": "<YOUR_API_KEY>"}

# You can use a local filepath:
# with open("./my-audio.mp3", "rb") as f:
#     response = requests.post(base_url + "/v2/upload", headers=headers, data=f)
#     upload_url = response.json()["upload_url"]

# Or use a publicly-accessible URL:
upload_url = "https://assembly.ai/sports_injuries.mp3"

data = {"audio_url": upload_url}

response = requests.post(base_url + "/v2/transcript", headers=headers, json=data)

transcript_id = response.json()["id"]
polling_endpoint = base_url + f"/v2/transcript/{transcript_id}"

while True:
    transcript = requests.get(polling_endpoint, headers=headers).json()

    if transcript["status"] == "completed":
        break

    elif transcript["status"] == "error":
        raise RuntimeError(f"Transcription failed: {transcript['error']}")

    else:
        time.sleep(3)
```

--------------------------------

### OpenAI Chat Completion Request (Python)

Source: https://www.assemblyai.com/docs/llm-gateway/prompt-caching

Demonstrates how to make a chat completion request to an OpenAI model through the LLM Gateway using Python. Caching is handled automatically by OpenAI, so no specific cache_control configuration is needed in the request.

```python
import os
import requests

headers = {
  "authorization": os.environ["ASSEMBLYAI_API_API_KEY"]
}

# OpenAI models cache automatically — no cache_control needed
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gpt-4.1",
        "messages": [
            {
                "role": "system",
                "content": "You are a customer support agent..."
            },
            {
                "role": "user",
                "content": "What is your return policy?"
            }
        ],
        "max_tokens": 1000
    }
)

result = response.json()

# Cached tokens still appear in usage
cache_details = result["usage"].get("prompt_tokens_details", {})
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")
```

--------------------------------

### JavaScript: Configure Chat Completion for JSON Schema Output

Source: https://www.assemblyai.com/docs/llm-gateway/structured-outputs

This JavaScript snippet shows how to make a request to the AssemblyAI Chat Completions API using `fetch` to obtain structured JSON output. It configures the `response_format` with a `json_schema` for a math tutoring scenario.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "gemini-2.5-flash-lite",
      messages: [
        {
          role: "system",
          content:
            "You are a helpful math tutor. Guide the user through the solution step by step.",
        },
        {
          role: "user",
          content: "how can I solve 8x + 7 = -23",
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "math_reasoning",
          schema: {
            type: "object",
            properties: {
              steps: {
                type: "array",
                items: {
                  type: "object",
                  properties: {
                    explanation: { type: "string" },
                    output: { type: "string" },
                  },
                  required: ["explanation", "output"],
                  additionalProperties: false,
                },
              },
              final_answer: { type: "string" },
            },
            required: ["steps", "final_answer"],
            additionalProperties: false,
          },
          strict: true,
        },
      },
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

--------------------------------

### Python Chat Completion with Gemini

Source: https://www.assemblyai.com/docs/llm-gateway/prompt-caching

This Python snippet demonstrates how to make a chat completion request to the LLM Gateway using a Gemini model. It shows how to set up headers, send the request, and access usage details, including cached tokens.

```python
import os
import requests

headers = {
  "authorization": os.environ["ASSEMBLYAI_API_KEY"]
}

# Gemini models cache automatically
response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash",
        "messages": [
            {
                "role": "system",
                "content": "You are a customer support agent..."
            },
            {
                "role": "user",
                "content": "What is your return policy?"
            }
        ],
        "max_tokens": 1000
    }
)

result = response.json()

# Cached tokens appear in usage
cache_details = result["usage"].get("prompt_tokens_details", {})
print(f"Cached tokens: {cache_details.get('cached_tokens', 0)}")
```

--------------------------------

### Python: Configure Chat Completion for JSON Schema Output

Source: https://www.assemblyai.com/docs/llm-gateway/structured-outputs

This Python snippet demonstrates how to set up a request to the AssemblyAI Chat Completions API to receive structured JSON output. It includes the necessary headers, message content, and the `response_format` parameter with a defined `json_schema` for a math reasoning task.

```python
import requests

headers = {
    "authorization": "<YOUR_API_KEY>",
    "content-type": "application/json"
}

response = requests.post(
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
    headers=headers,
    json={
        "model": "gemini-2.5-flash-lite",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful math tutor. Guide the user through the solution step by step."
            },
            {
                "role": "user",
                "content": "how can I solve 8x + 7 = -23"
            }
        ],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "math_reasoning",
                "schema": {
                    "type": "object",
                    "properties": {
                        "steps": {
                            "type": "array",
                            "items": {
                                "type": "object",
                                "properties": {
                                    "explanation": {"type": "string"},
                                    "output": {"type": "string"}
                                },
                                "required": ["explanation", "output"],
                                "additionalProperties": False
                            }
                        },
                        "final_answer": {"type": "string"}
                    },
                    "required": ["steps", "final_answer"],
                    "additionalProperties": False
                },
                "strict": True
            }
        }
    }
)

result = response.json()
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Inject Transcript into Prompt using transcript_id

Source: https://www.assemblyai.com/docs/llm-gateway/chat-completions

Example using curl to send a POST request to the chat completions endpoint, injecting a transcript by its ID. Ensure to replace YOUR_API_KEY and the transcript ID with your actual values.

```bash
curl -X POST "https://llm-gateway.assemblyai.com/v1/chat/completions" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-flash-lite",
    "messages": [
      {"role": "user", "content": "hi there"},
      {"role": "assistant", "content": "Hi! How can I help?"},
      {"role": "user", "content": "Here is a transcript: {{ transcript }}. Return the text verbatim."}
    ],
    "transcript_id": "065a71ac-dc3e-4e38-9374-e54c0bea564f"
  }'
```

--------------------------------

### Override Fallback Fields in JavaScript

Source: https://www.assemblyai.com/docs/llm-gateway/fallback

This JavaScript example demonstrates how to configure custom 'messages' and 'temperature' for a fallback model when calling the LLM Gateway. Unspecified fields in the fallback will use the primary request's values.

```javascript
const response = await fetch(
  "https://llm-gateway.assemblyai.com/v1/chat/completions",
  {
    method: "POST",
    headers: {
      authorization: "<YOUR_API_KEY>",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "kimi-k2.5",
      messages: [{ role: "user", content: "Tell me a fairy tale." }],
      temperature: 0.2,
      fallbacks: [
        {
          model: "claude-sonnet-4-6",
          messages: [
            { role: "user", content: "Tell me a fairy tale, but be very concise." },
          ],
          temperature: 0.4,
        },
      ],
    }),
  }
);

const result = await response.json();
console.log(result.choices[0].message.content);
```

--------------------------------

### Execute Tools and Continue Conversation

Source: https://www.assemblyai.com/docs/llm-gateway/tool-calling

This Python snippet demonstrates how to handle tool calls from the LLM Gateway. It includes appending tool calls to conversation history, executing the function, adding the tool result, and sending the updated history back to the API to continue the conversation.

```python
import requests
import json

headers = {
  "authorization": "<YOUR_API_KEY>"
}

choice = response.json()["choices"][0]

if choice.get("message", {}).get("tool_calls"):
    tool_call = choice["message"]["tool_calls"][0]

    # Add assistant message with tool calls to history
    conversation_history.append({
        "role": "assistant",
        "content": None,
        "tool_calls": [tool_call]
    })

    # Execute your function
    result = execute_function(
        tool_call["function"]["name"],
        json.loads(tool_call["function"]["arguments"])
    )

    # Add tool result to history
    conversation_history.append({
        "role": "tool",
        "tool_call_id": tool_call["id"],
        "content": json.dumps({"result": result})
    })

    # Continue conversation with the result
    response = requests.post(
        "https://llm-gateway.assemblyai.com/v1/chat/completions",
        headers = headers,
        json = {
            "model": "claude-sonnet-4-6",
            "messages": conversation_history,
            "tools": tools
        }
    )

result = response.json()
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Basic Chat Completions

Source: https://www.assemblyai.com/docs/llm-gateway/chat-completions

Send a message to the LLM Gateway and receive a text response. This is the most straightforward way to interact with the models.

```APIDOC
## POST /v1/chat/completions

### Description
Sends a message to the LLM Gateway and returns a response from the model.

### Method
POST

### Endpoint
/v1/chat/completions

### Request Body
- **model** (string) - Required - The model to use for the chat completion.
- **messages** (array) - Required - An array of message objects, where each object has a `role` (user, system, or assistant) and `content` (the message text).
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate in the response.

### Request Example
```json
{
  "model": "claude-sonnet-4-6",
  "messages": [
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "max_tokens": 1000
}
```

### Response
#### Success Response (200)
- **choices** (array) - An array containing the model's response(s).
  - **message** (object)
    - **content** (string) - The generated text response from the model.

#### Response Example
```json
{
  "choices": [
    {
      "message": {
        "content": "The capital of France is Paris."
      }
    }
  ]
}
```
```

--------------------------------

### Python OpenAI SDK Chat Completion

Source: https://www.assemblyai.com/docs/llm-gateway/quickstart

Utilize the OpenAI Python SDK to interact with the LLM Gateway. Configure the base URL and API key, then create a chat completion.

```python
from openai import OpenAI
import openai.types.chat.chat_completion as types

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key="<YOUR_API_KEY>",
)

import json

messages = [{"role": "user", "content": "What is the capital of france"}]
response = client.chat.completions.create(model="claude-sonnet-4.6", messages=messages)
print(response.choices[0].message.content)
```