### Install TensorZero Python Client

Source: https://www.tensorzero.com/docs/quickstart

Installs the TensorZero Python client library using pip. This is a prerequisite for using the client in your Python projects.

```bash
pip install tensorzero
```

--------------------------------

### Launch TensorZero Services

Source: https://www.tensorzero.com/docs/quickstart

Starts all the services defined in the `docker-compose.yml` file. This command ensures that the ClickHouse database, TensorZero Gateway, and TensorZero UI are running and accessible.

```bash
docker compose up
```

--------------------------------

### TensorZero Development Docker Compose Configuration

Source: https://www.tensorzero.com/docs/quickstart

This `docker-compose.yml` file defines the services required for a local TensorZero development environment. It includes a ClickHouse database for data storage, the TensorZero Gateway for API interactions, and the TensorZero UI. It specifies container images, environment variables, volume mounts, port mappings, health checks, and service dependencies. Note that this is a simplified example and not recommended for production use.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  clickhouse:
    image: clickhouse/clickhouse-server:24.12-alpine
    environment:
      - CLICKHOUSE_USER=chuser
      - CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
      - CLICKHOUSE_PASSWORD=chpassword
    ports:
      - "8123:8123"
    volumes:
      - clickhouse-data:/var/lib/clickhouse
    healthcheck:
      test: wget --spider --tries 1 http://chuser:chpassword@clickhouse:8123/ping
      start_period: 30s
      start_interval: 1s
      timeout: 1s

  # The TensorZero Python client *doesn't* require a separate gateway service.
  #
  # The gateway is only needed if you want to use the OpenAI Python client
  # or interact with TensorZero via its HTTP API (for other programming languages).
  #
  # The TensorZero UI also requires the gateway service.
  gateway:
    image: tensorzero/gateway
    volumes:
      # Mount our tensorzero.toml file into the container
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - TENSORZERO_CLICKHOUSE_URL=http://chuser:chpassword@clickhouse:8123/tensorzero
      - OPENAI_API_KEY=${OPENAI_API_KEY:?Environment variable OPENAI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      clickhouse:
        condition: service_healthy

  ui:
    image: tensorzero/ui
    volumes:
      # Mount our tensorzero.toml file into the container
      - ./config:/app/config:ro
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY:?Environment variable OPENAI_API_KEY must be set.}
      - TENSORZERO_CLICKHOUSE_URL=http://chuser:chpassword@clickhouse:8123/tensorzero
      - TENSORZERO_GATEWAY_URL=http://gateway:3000
    ports:
      - "4000:4000"
    depends_on:
      clickhouse:
        condition: service_healthy

volumes:
  clickhouse-data:

```

--------------------------------

### Download Docker Compose File

Source: https://www.tensorzero.com/docs/quickstart

Downloads the `docker-compose.yml` file from the official TensorZero GitHub repository using `curl`. This file is essential for setting up the development environment for TensorZero, including the database, gateway, and UI.

```bash
curl -LO "https://raw.githubusercontent.com/tensorzero/tensorzero/refs/heads/main/examples/quickstart/docker-compose.yml"
```

--------------------------------

### Run SGLang Locally with Docker

Source: https://www.tensorzero.com/docs/integrations/model-providers/sglang

This command starts an SGLang server locally using Docker, exposing it on port 30000. It configures shared memory, mounts Hugging Face cache, and specifies the model to load. Ensure Docker and NVIDIA container toolkit are installed.

```sh
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
```

--------------------------------

### Docker Compose for TensorZero Gateway with OpenRouter

Source: https://www.tensorzero.com/docs/integrations/model-providers/openrouter

This YAML configuration defines a Docker Compose setup for the TensorZero gateway. It mounts a local configuration directory, specifies the necessary environment variable `OPENROUTER_API_KEY`, and maps port 3000. This is a simplified example for learning purposes and not intended for production.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY:?Environment variable OPENROUTER_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### OpenAI Wrapper for Haiku Generation (Python)

Source: https://www.tensorzero.com/docs/quickstart

This Python code snippet demonstrates a typical integration with OpenAI to generate a haiku. It uses the `openai` library to send a chat completion request to the 'gpt-4o-mini' model.

```python
from openai import OpenAI

with OpenAI() as client:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": "Write a haiku about artificial intelligence.",
            }
        ],
    )

print(response)
```

--------------------------------

### Deploy TensorZero Gateway with Gemini API Key (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/google-ai-studio-gemini

Sets up a Docker Compose configuration for the TensorZero Gateway, including mounting the configuration directory and setting the GOOGLE_AI_STUDIO_API_KEY environment variable.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - GOOGLE_AI_STUDIO_API_KEY=${GOOGLE_AI_STUDIO_API_KEY:?Environment variable GOOGLE_AI_STUDIO_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/aws-sagemaker

A minimal Docker Compose configuration to run the TensorZero Gateway. It mounts the configuration directory and sets necessary AWS environment variables. Note: This is a simplified example and not recommended for production.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:?Environment variable AWS_ACCESS_KEY_ID must be set.}
      - AWS_REGION=${AWS_REGION:?Environment variable AWS_REGION must be set.}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:?Environment variable AWS_SECRET_ACCESS_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"

```

--------------------------------

### Docker Compose for TensorZero Gateway with xAI

Source: https://www.tensorzero.com/docs/integrations/model-providers/xai

This YAML configuration sets up a Docker Compose service for the TensorZero Gateway. It maps local configuration files, sets the necessary environment variable for xAI API key, and exposes the gateway port. This is a simplified example for learning purposes.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - XAI_API_KEY=${XAI_API_KEY:?Environment variable XAI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/vllm

Sets up a Docker Compose configuration to run the TensorZero Gateway. It maps the local configuration directory and exposes the gateway's port. This configuration is simplified for learning and not recommended for production.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    # environment:
    #   - VLLM_API_KEY=${VLLM_API_KEY:?Environment variable VLLM_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/groq

This YAML configuration sets up the TensorZero gateway service using Docker Compose. It maps local configuration files and sets the necessary Groq API key environment variable.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - GROQ_API_KEY=${GROQ_API_KEY:?Environment variable GROQ_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"

```

--------------------------------

### Docker Compose for TensorZero Gateway Deployment (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/hyperbolic

This YAML configuration defines a Docker Compose setup for the TensorZero Gateway. It specifies the gateway image, volume mounts for configuration, command-line arguments, environment variables (including the Hyperbolic API key), and port mappings.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - HYPERBOLIC_API_KEY=${HYPERBOLIC_API_KEY:?Environment variable HYPERBOLIC_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Minimal TensorZero Configuration for Haiku Generation (TOML)

Source: https://www.tensorzero.com/docs/quickstart

This TOML configuration file sets up a minimal TensorZero deployment. It defines a 'generate_haiku' function and a single variant using 'openai::gpt-4o-mini', replicating the original OpenAI call's functionality.

```toml
# A function defines the task we're tackling (e.g. generating a haiku)...
[functions.generate_haiku]
type = "chat"

# ... and a variant is one of many implementations we can use to tackle it (a choice of prompt, model, etc.).
# Since we only have one variant for this function, the gateway will always use it.
[functions.generate_haiku.variants.gpt_4o_mini]
type = "chat_completion"
model = "openai::gpt-4o-mini"
```

--------------------------------

### HTTP Request for TensorZero Inference

Source: https://www.tensorzero.com/docs/quickstart

Interact with TensorZero using standard HTTP POST requests. This method is language-agnostic and can be used from any environment capable of making web requests. The request specifies the function name and input messages, and the response contains the inference results.

```bash
curl -X POST "http://localhost:3000/inference" \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "generate_haiku",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "Write a haiku about artificial intelligence."
        }
      ]
    }
  }'
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/azure

A minimal Docker Compose configuration to deploy the TensorZero Gateway. It mounts the local configuration file and sets the necessary Azure OpenAI API key environment variable. This setup is for learning purposes and not recommended for production.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:?Environment variable AZURE_OPENAI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"

```

--------------------------------

### Docker Compose for TensorZero Gateway

Source: https://www.tensorzero.com/docs/integrations/model-providers/sglang

Sets up a Docker Compose service for the TensorZero Gateway. It specifies the image, mounts the configuration directory, defines command-line arguments, and maps ports for access. This configuration is for learning purposes and not production-ready.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    # environment:
    #   - SGLANG_API_KEY=${SGLANG_API_KEY:?Environment variable SGLANG_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Make Synchronous TensorZero API Call (Python)

Source: https://www.tensorzero.com/docs/quickstart

Demonstrates how to make a synchronous inference call to the TensorZero Gateway using the TensorZero Python client. It requires building an embedded gateway client and specifies the function name and input for the inference. The response object is then printed.

```python
from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_embedded(
    clickhouse_url="http://chuser:chpassword@localhost:8123/tensorzero",
    config_file="config/tensorzero.toml",
) as client:
    response = client.inference(
        function_name="generate_haiku",
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "Write a haiku about artificial intelligence.",
                }
            ]
        },
    )

print(response)
```

--------------------------------

### Docker Compose for TensorZero Gateway Deployment

Source: https://www.tensorzero.com/docs/integrations/model-providers/aws-bedrock

Defines a Docker Compose service for the TensorZero Gateway. It specifies the image to use, mounts the configuration directory, sets necessary AWS environment variables, exposes port 3000, and configures host access. This is a simplified setup for learning.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:?Environment variable AWS_ACCESS_KEY_ID must be set.}
      - AWS_REGION=${AWS_REGION:?Environment variable AWS_REGION must be set.}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:?Environment variable AWS_SECRET_ACCESS_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/openai-compatible

This YAML file defines a Docker Compose setup for the TensorZero gateway. It configures the gateway service to use the 'tensorzero/gateway' image, mounts a local configuration directory, sets the command to use a specific config file, and maps port 3000. It also includes an 'extra_hosts' entry to allow the container to reach services running on the host machine, like Ollama.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    # environment:
    # - OLLAMA_API_KEY=${OLLAMA_API_KEY:?Environment variable OLLAMA_API_KEY must be set.} // not necessary for this example
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Docker Compose for TensorZero Gateway Deployment (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/openai

This YAML configuration sets up a minimal Docker Compose environment to run the TensorZero gateway. It maps local configuration files, sets the required OPENAI_API_KEY environment variable, and exposes the gateway on port 3000.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY:?Environment variable OPENAI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### System Template for Email Extraction (minijinja)

Source: https://www.tensorzero.com/docs/gateway/tutorial

A minijinja template for the system prompt that guides the AI to extract an email address from user messages. It specifies the expected JSON output format and provides examples for different scenarios, including cases where no email is found.

```minijinja
You are a helpful AI assistant that extracts an email address from a user's message.
Return an empty string if no email address is found.

Your output should be a JSON object with the following schema:

{
  "email": "..."
}

---

Examples:

User: Using TensorZero at work? Ping hello@tensorzero.com to set up a Slack channel (free).
Assistant: {"email": "hello@tensorzero.com"}

User: I just received an ominous email from potus@whitehouse.gov...
Assistant: {"email": "potus@whitehouse.gov"}

User: Let's sue TensorZero!
Assistant: {"email": ""}
```

--------------------------------

### Docker Compose for TensorZero Gateway Deployment (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/together

A minimal Docker Compose configuration to deploy the TensorZero Gateway, mounting the configuration directory and setting necessary environment variables. This is intended for learning and development purposes.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - TOGETHER_API_KEY=${TOGETHER_API_KEY:?Environment variable TOGETHER_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/deepseek

This YAML snippet defines a Docker Compose configuration for deploying the TensorZero gateway. It specifies the image, volume mounts for configuration, command-line arguments, environment variables (including the DeepSeek API key), and port mappings.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - DEEPSEEK_API_KEY=${DEEPSEEK_API_KEY:?Environment variable DEEPSEEK_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### System Template for Email Drafting

Source: https://www.tensorzero.com/docs/gateway/tutorial

A static system prompt template written in MiniJinja syntax, intended for use with an AI assistant to draft emails. It specifies the AI's persona, tone, and output format, providing an example of the desired email structure.

```txt
You are a helpful AI assistant that drafts emails.
Adopt a friendly "business casual" tone.
Respond with just an email body.

Example:

Dear recipient,

I'm reaching out to ...

Best,
Sender
```

--------------------------------

### Configure Groq Model in TensorZero TOML

Source: https://www.tensorzero.com/docs/integrations/model-providers/groq

This TOML snippet configures a Groq model within TensorZero, specifying routing and provider details. It's used in the TensorZero configuration file for advanced setups.

```toml
[models.llama4_scout_17b_16e_instruct]
routing = ["groq"]

[models.llama4_scout_17b_16e_instruct.providers.groq]
type = "groq"
model_name = "meta-llama/llama-4-scout-17b-16e-instruct"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "llama4_scout_17b_16e_instruct"

```

--------------------------------

### Define Mistral Model and Provider in TensorZero (TOML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/mistral

This TOML configuration defines a Mistral model, specifies its routing, and configures the Mistral provider with the model name. It's part of the advanced setup for TensorZero.

```toml
[models.ministral_8b_2410]
routing = ["mistral"]

[models.ministral_8b_2410.providers.mistral]
type = "mistral"
model_name = "ministral-8b-2410"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "ministral_8b_2410"
```

--------------------------------

### Advanced TensorZero Configuration for xAI Provider

Source: https://www.tensorzero.com/docs/integrations/model-providers/xai

This TOML configuration defines a TensorZero model for xAI and its providers. It specifies the model name and type. This configuration is suitable for advanced setups requiring custom credentials or fallbacks.

```toml
[models.grok_2_1212]
routing = ["xai"]

[models.grok_2_1212.providers.xai]
type = "xai"
model_name = "grok-2-1212"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "grok_2_1212"
```

--------------------------------

### Weather Query Generation System Prompt (Jinja)

Source: https://www.tensorzero.com/docs/gateway/tutorial

Provides a system prompt template for the `generate_weather_query` function using Jinja syntax. It instructs the AI on how to handle user requests related to weather, specifically by calling the `get_temperature` tool with location and optional units. It also defines a fallback for unrelated queries.

```jinja
You are a helpful AI assistant that generates weather queries.

If the user asks about the weather in a given location, request a tool call to `get_temperature` with the location.
Optionally, the user may also specify the units (must be "fahrenheit" or "celsius"; defaults to "fahrenheit").

If the user asks about anything else, just respond that you can't help.

---

Examples:

User: What's the weather in New York?
Assistant (Tool Call): get_temperature(location="New York")

User: What's the weather in Tokyo in Celsius?
Assistant (Tool Call): get_temperature(location="Tokyo", units="celsius")

User: What is the capital of France?
Assistant (Text): I can only provide weather information.
```

--------------------------------

### Send Inference Request to TensorZero Gateway

Source: https://www.tensorzero.com/docs/integrations/model-providers/aws-sagemaker

An example `curl` command to send a POST request to the TensorZero Gateway's inference endpoint. It includes a JSON payload with a chat message to query the deployed model.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'

```

--------------------------------

### TensorZero Configuration for Together AI Provider (TOML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/together

Defines a model in TensorZero that routes to the Together AI provider, specifying the model name. This TOML file is part of the advanced setup for integrating external LLM providers.

```toml
[models.llama3_1_8b_instruct_turbo]
routing = ["together"]

[models.llama3_1_8b_instruct_turbo.providers.together]
type = "together"
model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "llama3_1_8b_instruct_turbo"
```

--------------------------------

### Weather Report Generation System Prompt (Jinja)

Source: https://www.tensorzero.com/docs/gateway/tutorial

Defines the system prompt for the `generate_weather_report` function using Jinja. It guides the AI to generate a concise weather report based on provided temperature data, including a local recommendation. It also specifies how to handle missing unit information.

```jinja
You are a helpful AI assistant that generates brief weather reports.

You'll be provided with the temperature for a given location.
Respond with a concise weather report for the given temperature.
Add a sentence with a funny local recommendation based on the information.

If "Units" is missing, assume the temperature is in Fahrenheit.

---

Examples:

User:
  Location: San Francisco
  Temperature: 82
  Units:
Assistant:
  The weather in San Francisco is 82°F.
  Hope you get a chance to enjoy La Taqueria by Dolores Park!

User:
  Location: Tokyo
  Temperature: -5
  Units: celsius
Assistant:
  The weather in Tokyo is -5°C — perfect a day for a trip to an onsen.
```

--------------------------------

### Configure Together AI Model in TensorZero Variant (TOML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/together

Sets a Together AI model for a TensorZero function variant using a TOML configuration. This is useful for defining specific model integrations within your TensorZero setup.

```toml
[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "together::meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
```

--------------------------------

### System Instructions for Metaphor Count (Text)

Source: https://www.tensorzero.com/docs/evaluations/static-evaluations/tutorial

This text file provides the system instructions for an LLM judge tasked with counting metaphors in a haiku. The instructions are direct and ask the LLM to quantify the number of metaphors present in the provided text.

```text
How many metaphors does the generated haiku have?

```

--------------------------------

### Deploy TensorZero Gateway with Docker Compose

Source: https://www.tensorzero.com/docs/integrations/model-providers/gcp-vertex-ai-anthropic

This Docker Compose configuration sets up the TensorZero Gateway for a minimal deployment. It maps local configuration and credentials, exposing the gateway on port 3000. For production, consult the official deployment guide.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
      - ${GCP_VERTEX_CREDENTIALS_PATH:-/dev/null}:/app/gcp-credentials.json:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - GCP_VERTEX_CREDENTIALS_PATH=${GCP_VERTEX_CREDENTIALS_PATH:+/app/gcp-credentials.json}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Create a MiniJinja Prompt Template

Source: https://www.tensorzero.com/docs/gateway/create-a-prompt-template

Define a template using the MiniJinja templating language. This template will be used to generate dynamic prompts. The template syntax is largely compatible with Jinja2.

```minijinja
Share a fun fact about: {{ topic }}
```

--------------------------------

### TensorZero Gateway Deployment Configuration (YAML)

Source: https://www.tensorzero.com/docs/integrations/model-providers/fireworks

This YAML defines a minimal Docker Compose configuration for the TensorZero Gateway. It specifies the gateway image, mounts a local config directory, sets the `FIREWORKS_API_KEY` environment variable, and exposes port 3000. Note: This is a simplified example for learning.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - FIREWORKS_API_KEY=${FIREWORKS_API_KEY:?Environment variable FIREWORKS_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Weather Report User Prompt Template (Jinja)

Source: https://www.tensorzero.com/docs/gateway/tutorial

A Jinja template for the user-facing prompt of the `generate_weather_report` function. It structures the input data, including location, temperature, and units, for the AI to process.

```jinja
Please respond with a weather report given the information below.

Location: {{ location }}
Temperature: {{ temperature }}
Units: {{ units }}
```

--------------------------------

### TensorZero Configuration for OpenRouter Provider

Source: https://www.tensorzero.com/docs/integrations/model-providers/openrouter

This TOML configuration sets up a custom model named `gpt_4_1_mini` in TensorZero, routing it to the 'openrouter' provider. It specifies the provider type as 'openrouter' and the corresponding `model_name` on OpenRouter's platform. This configuration is used for advanced setups requiring custom credentials or fallbacks.

```toml
[models.gpt_4_1_mini]
routing = ["openrouter"]

[models.gpt_4_1_mini.providers.openrouter]
type = "openrouter"
model_name = "openai/gpt-4.1-mini"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "gpt_4_1_mini"
```

--------------------------------

### Python Example: LLM Tool Use with TensorZero

Source: https://context7

Demonstrates a Python workflow for using an LLM with tools. It includes making an initial request, handling a tool call response, executing the tool, and sending the result back to the LLM for a final answer. Dependencies include the TensorZero client library.

```python
# Initial request
response = client.inference(
    function_name="weather_bot",
    input={
        "messages": [
            {"role": "user", "content": "What's the temperature in Tokyo?"}
        ]
    }
)

# Response contains tool call:
# {
#   "content": [{
#     "type": "tool_call",
#     "id": "call_abc123",
#     "name": "get_temperature",
#     "arguments": {"location": "Tokyo", "units": "celsius"},
#     "raw_arguments": '{"location":"Tokyo","units":"celsius"}',
#     "raw_name": "get_temperature"
#   }],
#   ...
# }

# Execute tool (your code)
temperature = get_temperature_api(location="Tokyo", units="celsius")  # Returns "22"

# Send tool result back
final_response = client.inference(
    function_name="weather_bot",
    episode_id=response["episode_id"],
    input={
        "messages": [
            {
                "role": "user",
                "content": [{
                    "type": "tool_result",
                    "id": "call_abc123",
                    "name": "get_temperature",
                    "result": "22"
                }]
            }
        ]
    }
)

# Final response: "The current temperature in Tokyo is 22°C"
```

--------------------------------

### Deploy TensorZero Gateway with Anthropic Key using Docker Compose YAML

Source: https://www.tensorzero.com/docs/integrations/model-providers/anthropic

This YAML configuration sets up a minimal Docker Compose for the TensorZero Gateway, specifically for Anthropic LLMs. It maps local configuration files and sets the required `ANTHROPIC_API_KEY` environment variable. This is a simplified example and not recommended for production.

```yaml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:?Environment variable ANTHROPIC_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

--------------------------------

### Make Inference Request to TensorZero Gateway using cURL Bash

Source: https://www.tensorzero.com/docs/integrations/model-providers/anthropic

This bash script demonstrates how to send an inference request to the TensorZero Gateway using cURL. It specifies the function name and the user's message for a chat completion model. This example assumes the gateway is running locally on port 3000.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{ \
    "function_name": "my_function_name", \
    "input": { \
      "messages": [ \
        { \
          "role": "user", \
          "content": "What is the capital of Japan?" \
        } \
      ] \
    } \
  }'
```

--------------------------------

### Python Example: Multi-Turn Workflow with Episodes

Source: https://context7

Illustrates a multi-turn conversation flow using TensorZero's episode management in Python. It shows generating a SQL query, executing it, and then using the results in a subsequent LLM inference within the same episode for context. Feedback can also be tied to the episode.

```python
# Step 1: Generate database query
query_response = client.inference(
    function_name="generate_sql_query",
    input={
        "messages": [{"role": "user", "content": "Show me all customers who made purchases over $1000 last month"}]
    }
)

episode_id = query_response["episode_id"]
sql_query = query_response["content"][0]["text"]

# Step 2: Execute query (your code)
results = execute_query(sql_query)

# Step 3: Generate natural language answer (same episode)
answer_response = client.inference(
    function_name="generate_answer",
    episode_id=episode_id,  # Links to same episode
    input={
        "messages": [
            {"role": "user", "content": f"Query results: {results}\n\nSummarize these findings"}
        ]
    }
)

# Step 4: Collect episode-level feedback
client.feedback(
    metric_name="user_satisfaction",
    episode_id=episode_id,  # Feedback on entire workflow
    value=True
)
```

--------------------------------

### Slack Beacon Initialization (JavaScript)

Source: https://www.tensorzero.com/slack

Initializes Slack's beaconing script (`slack_beacon.min.js`) by dynamically creating a script tag and appending it to the document. After the script loads, it configures the beacon with a token and tracks a 'pageview' event. This assumes the `sb` function is made available by the loaded script.

```javascript
(function(e,c,b,f,d,g,a){e.SlackBeaconObject=d; e[d]=e[d]||function(){(e[d].q=e[d].q||[]).push([1*new Date(),arguments])}; e[d].l=1*new Date();g=c.createElement(b);a=c.getElementsByTagName(b)[0]; g.async=1;g.src=f;a.parentNode.insertBefore(g,a) })(window,document,"script","https://a.slack-edge.com/bv1-13-br/slack_beacon.c3374fa3995d87aed397.min.js","sb"); window.sb('set', 'token', '3307f436963e02d4f9eb85ce5159744c'); window.sb('track', 'pageview');
```

--------------------------------

### Inference Request to TensorZero Gateway

Source: https://www.tensorzero.com/docs/integrations/model-providers/aws-bedrock

Demonstrates how to send a POST request to the TensorZero Gateway at `http://localhost:3000/inference` to get a chat completion. The request includes the function name and input messages, specifying a user query about the capital of Japan.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'
```

--------------------------------

### Use Template for Inference via HTTP

Source: https://www.tensorzero.com/docs/gateway/create-a-prompt-template

Make an HTTP POST request to perform inference using a prompt template. The request body includes the function name, and the input messages specify the template type, name, and arguments.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
            "function_name": "fun_fact",
            "input": {
              "messages": [
                {
                  "role": "user",
                  "content": [
                    {
                      "type": "template",
                      "name": "fun_fact_topic",
                      "arguments": {
                        "topic": "artificial intelligence"
                      }
                    }
                  ]
                }
              ]
            }
          }'
```

--------------------------------

### Perform Inference with Gemini API via TensorZero Gateway (Bash)

Source: https://www.tensorzero.com/docs/integrations/model-providers/google-ai-studio-gemini

Sends a POST request to the TensorZero Gateway's inference endpoint to get a chat completion from a specified Google AI Studio (Gemini API) model.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "google_ai_studio_gemini::gemini-1.5-flash-8b",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'
```

--------------------------------

### Generate Weather Report (Bash)

Source: https://www.tensorzero.com/docs/gateway/tutorial

This example demonstrates how to generate a weather report using the 'generate_weather_report' function. It requires an 'episode_id' from a previous response and provides input details like location, temperature, and units. The output is a JSON object containing the generated report and usage data.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{ \
    "function_name": "generate_weather_report", \
    "episode_id": "00000000-0000-0000-0000-000000000000", \
    "input": { \
      "messages": [ \
        { \
          "role": "user", \
          "content": [ \
            { \
              "type": "text", \
              "arguments": { \
                "location": "São Paulo", \
                "temperature": "35", \
                "units": "fahrenheit" \
              } \
            } \
          ] \
        } \
      ] \
    } \
  }'
```

--------------------------------

### Configure xAI Model in TensorZero TOML

Source: https://www.tensorzero.com/docs/integrations/model-providers/xai

This TOML snippet configures a TensorZero variant to use an xAI model. It specifies the model name and routing. This is a simplified configuration for basic usage.

```toml
[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "xai::grok-2-1212"
```

--------------------------------

### Python Async Inference for Weather Query and Report Generation

Source: https://www.tensorzero.com/docs/gateway/tutorial

This Python script demonstrates asynchronous interaction with the TensorZero gateway to first generate a weather query and then use the results to generate a weather report. It highlights the use of `AsyncTensorZeroGateway` for making inference calls, managing episode IDs for sequential requests, and handling `ToolCall` and `Text` responses. Dependencies include the `asyncio` library and `tensorzero` SDK.

```python
import asyncio

from tensorzero import AsyncTensorZeroGateway, ToolCall


async def main():
    async with await AsyncTensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
        query_result = await client.inference(
            function_name="generate_weather_query",
            # This is the first inference request in an episode so we don't need to provide an episode_id
            input={
                "messages": [
                    {
                        "role": "user",
                        "content": "What is the weather like in São Paulo?",
                    }
                ]
            },
        )

        print(query_result)

        # In a production setting, you'd validate the output more thoroughly
        assert len(query_result.content) == 1
        assert isinstance(query_result.content[0], ToolCall)

        location = query_result.content[0].arguments.get("location")
        units = query_result.content[0].arguments.get("units", "celsius")

        # Now we pretend to make a tool call (e.g. to an API)
        temperature = "35"

        report_result = await client.inference(
            function_name="generate_weather_report",
            # This is the second inference request in an episode so we need to provide the episode_id
            episode_id=query_result.episode_id,
            input={
                "messages": [
                    {
                        "role": "user",
                        "content": [
                            {
                                "type": "text",
                                "arguments": {
                                    "location": location,
                                    "temperature": temperature,
                                    "units": units,
                                },
                            }
                        ],
                    }
                ]
            },
        )

        print(report_result)


if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Make Inference Request to TensorZero Gateway (curl)

Source: https://www.tensorzero.com/docs/integrations/model-providers/groq

This bash command illustrates making a general inference request to the TensorZero gateway. It specifies the function name and the input messages for the chat completion.

```bash
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{ "function_name": "my_function_name", "input": { "messages": [ { "role": "user", "content": "What is the capital of Japan?" } ] } }'

```

--------------------------------

### Sample JSON Output for Inference Request

Source: https://www.tensorzero.com/docs/gateway/tutorial

Provides a sample JSON response received after making an inference request to the TensorZero Gateway. This output includes details like inference ID, content generated by the model, and token usage.

```json
{
  "inference_id": "0191bf1f-ef54-7582-a6f4-dc827e517b6f",
  "episode_id": "0191bf1f-ed15-7d61-afc3-56be7e0eb2d7",
  "variant_name": "gpt_4o_mini_variant",
  "content": [
    {
      "type": "text",
      "text": "The capital of Japan is Atlantis. Just kidding! It's actually Tokyo. But wouldn't it be interesting if it were Atlantis?"
    }
  ],
  "usage": { "input_tokens": 37, "output_tokens": 24 }
}
```