### Install Python SDK

Source: https://docs.letta.com/v1-sdk/client-sdks

Install the Python client library using pip.

```bash
pip install letta-client
```

--------------------------------

### Install Letta Evals

Source: https://docs.letta.com/v1-sdk/evals/getting-started

Install the evaluation package using pip or uv.

```bash
pip install letta-evals
```

```bash
uv pip install letta-evals
```

--------------------------------

### Install TypeScript SDK

Source: https://docs.letta.com/v1-sdk/client-sdks

Install the TypeScript client library using npm.

```bash
npm install @letta-ai/letta-client
```

--------------------------------

### Configure setup script

Source: https://docs.letta.com/v1-sdk/evals/configuration/suite-yaml

Specify a Python script and function for environment preparation.

```yaml
setup_script: setup.py:prepare_environment
```

--------------------------------

### Code Generation Dataset Examples

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example formats for code generation tasks.

```jsonl
{"input": "Write a function to reverse a string in Python"}
{"input": "Create a SQL query to find users older than 21"}
```

```csv
input
"Write a function to reverse a string in Python"
"Create a SQL query to find users older than 21"
```

--------------------------------

### Memory Testing Dataset Examples

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example formats for multi-turn memory testing.

```jsonl
{"input": ["Remember that my favorite color is blue", "What's my favorite color?"], "ground_truth": "blue"}
{"input": ["I live in Tokyo", "Where do I live?"], "ground_truth": "Tokyo"}
```

```csv
input,ground_truth
"[""Remember that my favorite color is blue"", ""What's my favorite color?""]","blue"
"[""I live in Tokyo"", ""Where do I live?""]","Tokyo"
```

--------------------------------

### Specify setup script

Source: https://docs.letta.com/v1-sdk/evals/concepts/suites

Points to a Python script and function to execute before the evaluation begins.

```yaml
setup_script: setup.py:prepare_environment # script.py:function_name
```

--------------------------------

### Start Letta server with Docker

Source: https://docs.letta.com/v1-sdk/evals/getting-started

Launch the Letta server using Docker to resolve connection refused errors.

```bash
docker run -p 8283:8283 -e OPENAI_API_KEY="your_api_key" letta/letta:latest
```

--------------------------------

### Tool Usage Testing Dataset Examples

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example formats for testing tool usage where ground truth is the expected tool name.

```jsonl
{"input": "Search for information about pandas", "ground_truth": "search"}
{"input": "Calculate 15 * 23", "ground_truth": "calculator"}
```

```csv
input,ground_truth
"Search for information about pandas","search"
"Calculate 15 * 23","calculator"
```

--------------------------------

### Verify and Reinstall Letta Evals

Source: https://docs.letta.com/v1-sdk/evals/troubleshooting

Use these commands to check if the package is installed or to perform a fresh installation.

```bash
# Verify installation
pip list | grep letta-evals


# Reinstall if needed
pip install --upgrade letta-evals
```

--------------------------------

### Define and example tool_call_message

Source: https://docs.letta.com/v1-sdk/messages/message-types

Structure and example for agent-initiated tool calls.

```typescript
{
  id: string;
  date: datetime;
  message_type: "tool_call_message";
  tool_call: {
    name: string;
    arguments: string; // JSON string
    tool_call_id: string;
  }
}
```

```json
{
  message_type: "tool_call_message",
  tool_call: {
    name: "archival_memory_search",
    arguments: '{"query": "user preferences", "page": 0}',
    tool_call_id: "call_abc123"
  }
}
```

--------------------------------

### Question Answering Dataset Examples

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example formats for question answering datasets.

```jsonl
{"input": "What is the capital of France?", "ground_truth": "Paris"}
{"input": "Who wrote Romeo and Juliet?", "ground_truth": "Shakespeare"}
```

```csv
input,ground_truth
"What is the capital of France?","Paris"
"Who wrote Romeo and Juliet?","Shakespeare"
```

--------------------------------

### Complete Dataset Example

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

A sample JSONL dataset showing various input types, ground truths, tags, and metadata.

```json
{"id": 1, "input": "What is the capital of France?", "ground_truth": "Paris", "tags": ["geography", "easy"], "metadata": {"region": "Europe"}}
{"id": 2, "input": "Calculate the square root of 144", "ground_truth": "12", "tags": ["math", "medium"]}
{"id": 3, "input": ["Hello", "What can you help me with?"], "tags": ["conversation"]}
```

--------------------------------

### Run Letta API proxy server with Docker

Source: https://docs.letta.com/v1-sdk/docker

Starts the Letta container with a volume mount for data persistence and an OpenAI API key provided as an environment variable.

```bash
# replace `~/.letta/.persist/pgdata` with wherever you want to store your agent data
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENAI_API_KEY="your_openai_api_key" \
  letta/letta:latest
```

--------------------------------

### GitHub Actions CI/CD configuration

Source: https://docs.letta.com/v1-sdk/evals/cli

Example workflow for running Letta evaluations in GitHub Actions.

```yaml
name: Run Evals
on: [push]


jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2


      - name: Install dependencies
        run: pip install letta-evals


      - name: Run evaluation
        env:
          LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          letta-evals run suite.yaml --quiet --output results/


      - name: Upload results
        uses: actions/upload-artifact@v2
        with:
          name: eval-results
          path: results/
```

--------------------------------

### Run Letta API proxy server with .env file

Source: https://docs.letta.com/v1-sdk/docker

Starts the Letta container using a .env file to manage environment variables instead of passing them directly in the command.

```bash
# using a .env file instead of passing environment variables
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  --env-file .env \
  letta/letta:latest
```

--------------------------------

### Define setup function signature

Source: https://docs.letta.com/v1-sdk/evals/concepts/suites

Required signature for the Python function invoked by the setup_script.

```python
def prepare_environment(suite: SuiteSpec) -> None:
    # Setup code here
    pass
```

--------------------------------

### GitLab CI configuration

Source: https://docs.letta.com/v1-sdk/evals/cli

Example configuration for running Letta evaluations in GitLab CI.

```yaml
evaluate:
  script:
    - pip install letta-evals
    - letta-evals run suite.yaml --quiet --output results/
  artifacts:
    paths:
      - results/
  variables:
    LETTA_API_KEY: $LETTA_API_KEY
    OPENAI_API_KEY: $OPENAI_API_KEY
```

--------------------------------

### Streaming Example Output

Source: https://docs.letta.com/v1-sdk/messages/streaming

Sample output format received from the streaming endpoint.

```text
data: {"id":"msg-123","message_type":"reasoning_message","reasoning":"User is asking a simple math question."}
data: {"id":"msg-456","message_type":"assistant_message","content":"2 + 2 equals 4!"}
data: {"message_type":"stop_reason","stop_reason":"end_turn"}
data: {"message_type":"usage_statistics","completion_tokens":50,"total_tokens":2821}
data: [DONE]
```

--------------------------------

### Agent Approval Request Examples

Source: https://docs.letta.com/v1-sdk/tools/human-in-the-loop

Examples of how an approval request appears in the API response across different interfaces.

```bash
curl --request POST \
  --url https://api.letta.com/v1/agents/$AGENT_ID/messages \
  --header 'Authorization: Bearer $LETTA_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [{
      "role": "user",
      "content": "Delete all test data from the database"
    }]
  }'


# Response includes approval request


{
"messages": [
{
"message_type": "reasoning_message",
"reasoning": "I need to delete test data from the database..."
},
{
"message_type": "approval_request_message",
"id": "message-abc123",
"tool_call": {
"name": "database_write",
"arguments": "{\"query\": \"DELETE FROM test_data\"}",
"tool_call_id": "tool-xyz789"
}
}
],
"stop_reason": "requires_approval"
}
```

```python
response = client.agents.messages.create(
    agent_id=agent.id,
    messages=[{
        "role": "user",
        "content": "Delete all test data from the database"
    }]
)


# Response includes approval request
{
  "messages": [
    {
      "message_type": "reasoning_message",
      "reasoning": "I need to delete test data from the database..."
    },
    {
      "message_type": "approval_request_message",
      "id": "message-abc123",
      "tool_call": {
        "name": "database_write",
        "arguments": "{\"query\": \"DELETE FROM test_data\"}",
        "tool_call_id": "tool-xyz789"
      }
    }
  ],
  "stop_reason": "requires_approval"
}
```

```typescript
const response = await client.agents.messages.create({
    agentId: agent.id,
    requestBody: {
        messages: [{
            role: "user",
            content: "Delete all test data from the database"
        }]
    }
});


// Response includes approval request
{
  "messages": [
    {
      "message_type": "reasoning_message",
      "reasoning": "I need to delete test data from the database..."
    },
    {
      "message_type": "approval_request_message",
      "id": "message-abc123",
      "tool_call": {
        "name": "database_write",
        "arguments": "{\"query\": \"DELETE FROM test_data\"}",
        "tool_call_id": "tool-xyz789"
      }
    }
  ],
  "stop_reason": "requires_approval"
}
```

--------------------------------

### Example Structured Response

Source: https://docs.letta.com/v1-sdk/messages/structured-outputs

A sample JSON output demonstrating the structure enforced by the schema.

```json
{
  "rank": 8,
  "reason": "Sushi is highly regarded for its fresh ingredients and artful presentation",
  "categories": [
    { "name": "taste", "score": 9 },
    { "name": "presentation", "score": 10 },
    { "name": "value", "score": 6 }
  ]
}
```

--------------------------------

### Dataset Tagging

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of using tags to categorize samples by functionality.

```json
{"tags": ["tool_usage", "search"]}
{"tags": ["memory", "recall"]}
{"tags": ["reasoning", "multi_step"]}
```

--------------------------------

### Multi-Run Evaluation Configuration

Source: https://docs.letta.com/v1-sdk/evals/results

Configuration examples for running evaluations multiple times using YAML or CLI arguments.

```yaml
name: my-eval-suite
dataset: dataset.jsonl
num_runs: 5 # Run 5 times
target:
  kind: agent
  agent_file: my_agent.af
graders:
  accuracy:
    kind: tool
    function: exact_match
gate:
  metric_key: accuracy
  op: gte
  value: 0.8
```

```bash
letta-evals run suite.yaml --num-runs 10 --output results/
```

--------------------------------

### Run Letta with MemFS enabled

Source: https://docs.letta.com/v1-sdk/docker

Use this docker run command to start the Letta container with local MemFS storage configured.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e LETTA_MEMFS_SERVICE_URL=local \
  -e OPENAI_API_KEY="your_key" \
  letta/letta:latest
```

--------------------------------

### View evaluation progress

Source: https://docs.letta.com/v1-sdk/evals/results

Example of the real-time console output generated during an evaluation run.

```text
Running evaluation: my-eval-suite
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3 100%


Results:
  Total samples: 3
  Attempted: 3
  Avg score: 0.83 (attempted: 0.83)
  Passed: 2 (66.7%)


Gate (quality >= 0.75): PASSED
```

--------------------------------

### Define input field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of providing single or multi-turn prompts for the input field.

```json
{ "input": "Hello, who are you?" }
```

```json
{ "input": ["Hello", "What's your name?", "Tell me about yourself"] }
```

--------------------------------

### Get Environment Information

Source: https://docs.letta.com/v1-sdk/evals/troubleshooting

Retrieve Python and package version details for troubleshooting reports.

```bash
# Get environment info
python --version
pip show letta-evals
```

--------------------------------

### Request Token Streaming via curl

Source: https://docs.letta.com/v1-sdk/messages/streaming

Example of initiating a streaming request using the curl command-line tool.

```bash
curl -N --request POST \
  --url https://api.letta.com/v1/agents/$AGENT_ID/messages/stream \
  --header "Authorization: Bearer $LETTA_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [{"role": "user", "content": "Tell me a joke"}],
    "stream_tokens": true
  }'
```

--------------------------------

### Initialize and Use Python SDK

Source: https://docs.letta.com/v1-sdk/client-sdks

Initialize the Letta client and perform basic agent operations like creation and messaging.

```python
from letta_client import Letta


client = Letta(api_key="your-api-key")


# Create an agent
agent = client.agents.create(
    model="openai/gpt-4.1",
    embedding="openai/text-embedding-3-small",
)


# Send a message
response = client.agents.messages.create(
    agent_id=agent.id,
    input="Hello!"
)
```

--------------------------------

### Create and Run Python Agent

Source: https://docs.letta.com/v1-sdk

Initialize a Letta client, create a stateful agent with memory, and send a message.

```python
from letta_client import Letta
import os


client = Letta(api_key=os.getenv("LETTA_API_KEY"))


# Create a stateful agent
agent = client.agents.create(
    model="openai/gpt-4.1",
    memory_blocks=[
        {
            "label": "human",
            "value": "Name: Timber. Status: dog. Occupation: building Letta, infrastructure to democratize self-improving superintelligence"
        },
        {
            "label": "persona",
            "value": "I am a self-improving superintelligence. Timber is my best friend and collaborator."
        }
    ]
)


print(f"Agent created with ID: {agent.id}")


# Send a message
response = client.agents.messages.create(
    agent_id=agent.id,
    input="What do you know about me?"
)


for message in response.messages:
    print(message)
```

```bash
python quickstart.py
```

```text
Agent created with ID: agent-abc123
ReasoningMessage(reasoning="Timber is asking what I know. I should reference my memory blocks.")
AssistantMessage(content="I know you're Timber, a dog who's building Letta!")
```

--------------------------------

### Create an agent with webpage fetching capabilities

Source: https://docs.letta.com/v1-sdk/tools/builtin-tools

Initializes an agent configured with the fetch_webpage tool to read and convert web content.

```python
agent = client.agents.create(
    tools=["fetch_webpage"],
    memory_blocks=[{
        "label": "persona",
        "value": "I fetch and read webpages to answer questions."
    }]
)
```

--------------------------------

### Define id field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example of overriding the default sample ID.

```json
{ "id": 42, "input": "Test case 42" }
```

--------------------------------

### Define tags field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of tagging samples and filtering by tags.

```json
{ "input": "Solve x^2 = 16", "ground_truth": "4", "tags": ["math", "algebra"] }
```

```yaml
sample_tags: [math] # Only samples tagged "math" will be evaluated
```

--------------------------------

### Show CLI help

Source: https://docs.letta.com/v1-sdk/evals/cli

Displays general help information or specific help for a command.

```bash
letta-evals --help
```

```bash
letta-evals run --help
letta-evals validate --help
```

--------------------------------

### Define metadata field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example of adding arbitrary metadata to a sample.

```json
{
  "input": "What is photosynthesis?",
  "ground_truth": "process where plants convert light into energy",
  "metadata": {
    "category": "biology",
    "difficulty": "medium"
  }
}
```

--------------------------------

### Create an agent with web search capabilities

Source: https://docs.letta.com/v1-sdk/tools/builtin-tools

Initializes an agent configured with the web_search tool and a persona defining its usage.

```python
agent = client.agents.create(
    tools=["web_search"],
    memory_blocks=[{
        "label": "persona",
        "value": "I use web_search for current events and external research."
    }]
)
```

--------------------------------

### Run Letta server with Docker

Source: https://docs.letta.com/v1-sdk/docker

Deploys the Letta server container with persistent storage and API keys for model providers.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENAI_API_KEY="your_openai_api_key" \
  -e ANTHROPIC_API_KEY="your_anthropic_api_key" \
  -e OLLAMA_BASE_URL="http://host.docker.internal:11434/v1" \
  letta/letta:latest
```

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  --network host \
  -e OPENAI_API_KEY="your_openai_api_key" \
  -e ANTHROPIC_API_KEY="your_anthropic_api_key" \
  -e OLLAMA_BASE_URL="http://localhost:11434/v1" \
  letta/letta:latest
```

--------------------------------

### Define ground_truth field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Example of providing an expected answer for tool graders.

```json
{ "input": "What is 2+2?", "ground_truth": "4" }
```

--------------------------------

### Importing AgentFiles

Source: https://docs.letta.com/v1-sdk/concepts/agent-file

Uploads a local .af file to the Letta platform to instantiate an agent.

```typescript
// Install SDK with `npm install @letta-ai/letta-client`
import { Letta, toFile } from "@letta-ai/letta-client";
import { readFileSync } from "fs";


// Create a client to connect to Letta
const client = new Letta({ apiKey: process.env.LETTA_API_KEY });


// Import your .af file from any location
const file = await toFile(readFileSync("/path/to/agent/file.af"), "agent.af");
const importResult = await client.agents.importFile({ file });


console.log(`Imported agent: ${importResult.agent_ids[0]}`);
```

```python
# Install SDK with `pip install letta-client`
from letta_client import Letta
import os


# Create a client to connect to Letta
client = Letta(api_key=os.getenv("LETTA_API_KEY"))


# Import your .af file from any location
imported_agent = client.agents.import_file(file=open("/path/to/agent/file.af", "rb"))


print(f"Imported agent: {imported_agent.id}")
```

```bash
curl -X POST "https://api.letta.com/v1/agents/import" \
     -H "Authorization: Bearer LETTA_API_KEY" \
     -F "file=@/path/to/agent/file.af"
```

--------------------------------

### Configure Google AI (Gemini) provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with a Gemini API key.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e GEMINI_API_KEY="your_gemini_api_key" \
  letta/letta:latest
```

--------------------------------

### GET /v1/runs/{run_id}/stream

Source: https://docs.letta.com/v1-sdk/messages/long-running

Resumes streaming data from a specific run.

```APIDOC
## GET /v1/runs/{run_id}/stream

### Description
Resumes streaming from the beginning or a specified sequence ID.

### Method
GET

### Endpoint
https://api.letta.com/v1/runs/{run_id}/stream

### Parameters
#### Path Parameters
- **run_id** (string) - Required - The ID of the run to stream

### Request Body
- **starting_after** (integer) - Optional - Sequence ID to start after
- **batch_size** (integer) - Optional - Number of historical chunks to fetch
```

--------------------------------

### GET /v1/runs/active

Source: https://docs.letta.com/v1-sdk/messages/long-running

Retrieves a list of active background streams for specified agents.

```APIDOC
## GET /v1/runs/active

### Description
Finds active background streams for your agents.

### Method
GET

### Endpoint
https://api.letta.com/v1/runs/active

### Request Body
- **agent_ids** (array) - Required - List of agent IDs to check
- **background** (boolean) - Required - Filter for background streams

### Response
- **run_id** (string) - The ID of the run
- **agent_id** (string) - The ID of the agent
- **status** (string) - The current status of the run
```

--------------------------------

### Initialize and Use TypeScript SDK

Source: https://docs.letta.com/v1-sdk/client-sdks

Initialize the Letta client and perform basic agent operations using asynchronous calls.

```typescript
import Letta from "@letta-ai/letta-client";


const client = new Letta({ apiKey: "your-api-key" });


// Create an agent
const agent = await client.agents.create({
  model: "openai/gpt-4.1",
  embedding: "openai/text-embedding-3-small",
});


// Send a message
const response = await client.agents.messages.create(agent.id, {
  input: "Hello!",
});
```

--------------------------------

### Configure model settings and context limits

Source: https://docs.letta.com/v1-sdk/models

Define provider-specific settings like temperature and output tokens alongside context window limits.

```python
agent = client.agents.create(
    model="openai/gpt-5.2",
    model_settings={
        "provider_type": "openai",
        "temperature": 0.7,
        "max_output_tokens": 4096,
    },
    context_window_limit=128000
)
```

```typescript
const agent = await client.agents.create({
  model: "openai/gpt-5.2",
  model_settings: {
    provider_type: "openai",
    temperature: 0.7,
    max_output_tokens: 4096,
  },
  context_window_limit: 128000,
});
```

--------------------------------

### Message Response JSON

Source: https://docs.letta.com/v1-sdk

Example response structure returned after sending a message to an agent.

```json
{
  "messages": [
    {
      "message_type": "reasoning_message",
      "reasoning": "Timber is asking what I know. I should reference my memory blocks."
    },
    {
      "message_name": "assistant_message",
      "content": "I know you're Timber, a dog who's building Letta - infrastructure to democratize self-improving superintelligence. We're best friends and collaborators!"
    }
  ]
}
```

--------------------------------

### Configure Google Vertex AI provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Authenticate with Google Cloud and run the Letta Docker container with project and location settings.

```bash
gcloud auth application-default login
```

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e GOOGLE_CLOUD_PROJECT="your-project-id" \
  -e GOOGLE_CLOUD_LOCATION="us-central1" \
  letta/letta:latest
```

--------------------------------

### Create and Run TypeScript Agent

Source: https://docs.letta.com/v1-sdk

Initialize a Letta client, create a stateful agent with memory, and send a message.

```typescript
import Letta from "@letta-ai/letta-client";


async function main() {
  const client = new Letta({ apiKey: process.env.LETTA_API_KEY });


  // Create a stateful agent
  const agent = await client.agents.create({
    model: "openai/gpt-4.1",
    memory_blocks: [
      {
        label: "human",
        value:
          "Name: Timber. Status: dog. Occupation: building Letta, infrastructure to democratize self-improving superintelligence",
      },
      {
        label: "persona",
        value:
          "I am a self-improving superintelligence. Timber is my best friend and collaborator.",
      },
    ],
  });


  console.log("Agent created with ID:", agent.id);


  // Send a message
  const response = await client.agents.messages.create(agent.id, {
    input: "What do you know about me?",
  });


  for (const message of response.messages) {
    console.log(message);
  }
}


main().catch(console.error);
```

```bash
npx tsx quickstart.ts
```

```text
Agent created with ID: agent-abc123
{ message_type: 'reasoning_message', reasoning: 'Timber is asking what I know...' }
{ message_type: 'assistant_message', content: "I know you're Timber, a dog who's building Letta!" }
```

--------------------------------

### Multi-Turn Conversations

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of multi-turn inputs for testing conversational context and memory updates.

```json
{"input": ["My name is Alice", "What's my name?"], "ground_truth": "Alice", "tags": ["memory", "recall"]}
{"input": ["Please remember that I like bananas.", "Actually, sorry, I meant I like apples."], "ground_truth": "apples", "tags": ["memory", "correction"]}
{"input": ["I work at Google", "Update my workplace to Microsoft", "Where do I work?"], "ground_truth": "Microsoft", "tags": ["memory", "multi_step"]}
```

--------------------------------

### Configure Together AI provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with a Together AI API key.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e TOGETHER_API_KEY="your_together_api_key" \
  letta/letta:latest
```

--------------------------------

### Define rubric_vars field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of using rubric variables for dynamic evaluation templates.

```json
{
  "input": "Write a function to calculate fibonacci numbers",
  "rubric_vars": {
    "reference_code": "def fib(n):\n    if n <= 1: return n\n    return fib(n-1) + fib(n-2)",
    "required_features": "recursion, base case"
  }
}
```

```csv
input,rubric_vars
"Write a function to calculate fibonacci numbers","{\"\"reference_code\"\": \"\"def fib(n):\n    if n <= 1: return n\n    return fib(n-1) + fib(n-2)\"\", \"\"required_features\"\": \"\"recursion, base case\"\"}"
```

```text
Evaluate the submitted code against this reference implementation:


{reference_code}


Required features: {required_features}


Score on correctness (0.6) and code quality (0.4).
```

```text
Evaluate the submitted code against this reference implementation:


def fib(n):
    if n <= 1: return n
    return fib(n-1) + fib(n-2)


Required features: recursion, base case


Score on correctness (0.6) and code quality (0.4).
```

--------------------------------

### Configure OpenAI provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with an OpenAI API key.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENAI_API_KEY="your_openai_api_key" \
  letta/letta:latest
```

--------------------------------

### Configure LM Studio local provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with LM Studio, using different network configurations for macOS/Windows and Linux.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e LMSTUDIO_BASE_URL="http://host.docker.internal:1234" \
  letta/letta:latest
```

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  --network host \
  -e LMSTUDIO_BASE_URL="http://localhost:1234" \
  letta/letta:latest
```

--------------------------------

### Configure multiple providers

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with multiple provider environment variables enabled simultaneously.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENAI_API_KEY="your_openai_api_key" \
  -e ANTHROPIC_API_KEY="your_anthropic_api_key" \
  -e OLLAMA_BASE_URL="http://host.docker.internal:11434/v1" \
  letta/letta:latest
```

--------------------------------

### View Streaming Output Format

Source: https://docs.letta.com/v1-sdk/messages/streaming

Example of the raw data format received during a streaming session.

```text
# Same ID across chunks of the same message
data: {"id":"msg-abc","message_type":"assistant_message","content":"Why"}
data: {"id":"msg-abc","message_type":"assistant_message","content":" did"}
data: {"id":"msg-abc","message_type":"assistant_message","content":" the"}
data: {"id":"msg-abc","message_type":"assistant_message","content":" scarecrow"}
data: {"id":"msg-abc","message_type":"assistant_message","content":" win"}
# ... more chunks with same ID
data: [DONE]
```

--------------------------------

### Configure OpenRouter provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with an OpenRouter API key.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OPENROUTER_API_KEY="your_openrouter_api_key" \
  letta/letta:latest
```

--------------------------------

### Stream Agent Messages

Source: https://docs.letta.com/v1-sdk/messages/streaming

Examples of how to initiate a streaming request to an agent and process the resulting chunks.

```typescript
import Letta from "@letta-ai/letta-client";
import type { LettaMessage } from "@letta-ai/letta-client/api/types";


const client = new Letta({ apiKey: process.env.LETTA_API_KEY });


const stream = await client.agents.messages.stream(agent.id, {
  messages: [{ role: "user", content: "What's 2+2?" }],
});


for await (const chunk of stream as AsyncIterable<LettaMessage>) {
  if (chunk.message_type === "reasoning_message") {
    console.log(`Thinking: ${(chunk as any).reasoning}`);
  } else if (chunk.message_type === "assistant_message") {
    console.log(`Response: ${(chunk as any).content}`);
  }
}
```

```python
stream = client.agents.messages.create(
    agent_id=agent.id,
    messages=[{"role": "user", "content": "What's 2+2?"}],
    streaming=True
)


for chunk in stream:
    if hasattr(chunk, 'message_type'):
        if chunk.message_type == 'reasoning_message':
            print(f"Thinking: {chunk.reasoning}")
        elif chunk.message_type == 'assistant_message':
            print(f"Response: {chunk.content}")
```

```bash
curl -N --request POST \
  --url https://api.letta.com/v1/agents/$AGENT_ID/messages/stream \
  --header "Authorization: Bearer $LETTA_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"messages": [{"role": "user", "content": "What is 2+2?"}]}'


# For self-hosted: Replace https://api.letta.com with http://localhost:8283
```

--------------------------------

### Error result structure

Source: https://docs.letta.com/v1-sdk/evals/results

Example of the JSON structure returned when an evaluation sample encounters an error.

```json
{
  "sample": {...},
  "submission": "",
  "grade": {
    "score": 0.0,
    "rationale": "Error during grading: Connection timeout",
    "metadata": {"error": "timeout", "error_type": "ConnectionError"}
  }
}
```

--------------------------------

### Diverse Test Cases

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of math-related test cases covering different complexity levels.

```json
{"input": "What is 2+2?", "ground_truth": "4", "tags": ["math", "easy"]}
{"input": "What is 0.1 + 0.2?", "ground_truth": "0.3", "tags": ["math", "floating_point"]}
{"input": "What is 999999999 + 1?", "ground_truth": "1000000000", "tags": ["math", "large_numbers"]}
```

--------------------------------

### Configure model handles

Source: https://docs.letta.com/v1-sdk/evals/configuration/suite-yaml

List model handles for cloud deployments.

```yaml
target:
  model_handles: ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"]
```

--------------------------------

### Ground Truth Formatting

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of appropriate versus overly strict ground truth definitions.

```json
{"input": "What's the largest planet?", "ground_truth": "Jupiter"}
```

```json
{"input": "What's the largest planet?", "ground_truth": "Jupiter is the largest planet in our solar system."}
```

--------------------------------

### Verify API key configuration

Source: https://docs.letta.com/v1-sdk/evals/cli

Use this command to ensure the LETTA_API_KEY environment variable is correctly set in your current session.

```bash
# Check environment variable is set
echo $LETTA_API_KEY
```

--------------------------------

### Define agent_args field

Source: https://docs.letta.com/v1-sdk/evals/concepts/datasets

Examples of passing custom agent arguments in JSONL and CSV formats.

```json
{
  "input": "What items do we have?",
  "agent_args": {
    "item": { "sku": "SKU-123", "name": "Widget A", "price": 19.99 }
  }
}
```

```csv
input,agent_args
"What items do we have?","{\"\"item\"\": {\"\"sku\"\": \"\"SKU-123\"\", \"\"name\"\": \"\"Widget A\"\", \"\"price\"\": 19.99}}"
```

--------------------------------

### List Graders and Check Configuration

Source: https://docs.letta.com/v1-sdk/evals/troubleshooting

Verify available graders and ensure the function name matches the suite configuration.

```bash
# List available graders
letta-evals list-graders


# Check spelling in suite.yaml
graders:
  my_metric:
    function: exact_match  # Correct
```

--------------------------------

### Define gate threshold values

Source: https://docs.letta.com/v1-sdk/evals/concepts/gates

Examples of setting threshold values for different aggregation types.

```yaml
gate:
  kind: simple
  aggregation: avg_score # Average score
  op: gte # >=
  value: 0.75 # 75% threshold
```

```yaml
gate:
  kind: simple
  aggregation: accuracy # Pass rate
  op: gte # >=
  value: 0.9 # 90% must pass
```

--------------------------------

### Configure AWS Bedrock provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with AWS credentials. Ensure the model is enabled in your AWS Model Catalog.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e AWS_ACCESS_KEY_ID="your_aws_access_key_id" \
  -e AWS_SECRET_ACCESS_KEY="your_aws_secret_access_key" \
  -e AWS_DEFAULT_REGION="your_aws_default_region" \
  letta/letta:latest
```

--------------------------------

### Resolve Import Errors

Source: https://docs.letta.com/v1-sdk/evals/troubleshooting

Ensure the correct Python environment is active before installing or running the package.

```bash
# Ensure you're in the right environment
which python


# Install in correct environment
source .venv/bin/activate
pip install letta-evals
```

--------------------------------

### GET /v1/agents/{AGENT_ID}/export

Source: https://docs.letta.com/v1-sdk/concepts/agent-file

Exports an existing agent configuration to a .af file format.

```APIDOC
## GET /v1/agents/{AGENT_ID}/export

### Description
Exports an agent into a serialized schema object.

### Method
GET

### Endpoint
https://api.letta.com/v1/agents/{AGENT_ID}/export

### Parameters
#### Path Parameters
- **AGENT_ID** (string) - Required - The unique identifier of the agent to export.
```

--------------------------------

### Configure Reasoning Messages

Source: https://docs.letta.com/v1-sdk/messages/streaming

Examples for enabling or disabling reasoning messages during agent creation in TypeScript and Python.

```typescript
// With reasoning (default) - includes reasoning_message events
const agent = await client.agents.create({
  model: "openai/gpt-4o-mini",
  // reasoning: true is the default
});


// Without reasoning - no reasoning_message events
const agentNoReasoning = await client.agents.create({
  model: "openai/gpt-4o-mini",
  reasoning: false, // Disable reasoning messages
});
```

```python
# With reasoning (default) - includes reasoning_message events
agent = client.agents.create(
    model="openai/gpt-4o-mini",
    # reasoning=True is the default
)


# Without reasoning - no reasoning_message events
agent = client.agents.create(
    model="openai/gpt-4o-mini",
    reasoning=False  # Disable reasoning messages
)
```

--------------------------------

### Create an agent with memory blocks

Source: https://docs.letta.com/v1-sdk/memory/memory-blocks

Initialize an agent with specific memory blocks using the Letta client. Ensure the API key is configured in your environment variables.

```typescript
// install letta-client with `npm install @letta-ai/letta-client`
import Letta from "@letta-ai/letta-client";


// create a client connected to the Letta API
const client = new Letta({
  apiKey: process.env.LETTA_API_KEY,
});


// create an agent with two basic self-editing memory blocks
const agentState = await client.agents.create({
  memory_blocks: [
    {
      label: "human",
      value: "The human's name is Bob the Builder.",
      limit: 5000,
    },
    {
      label: "persona",
      value: "My name is Sam, the all-knowing sentient AI.",
      limit: 5000,
    },
  ],
  model: "openai/gpt-4o-mini",
});
```

```python
# install letta_client with `pip install letta-client`
from letta_client import Letta
import os


# create a client connected to the Letta API
client = Letta(api_key=os.getenv("LETTA_API_KEY"))


# create an agent with two basic self-editing memory blocks
agent_state = client.agents.create(
    memory_blocks=[
        {
          "label": "human",
          "value": "The human's name is Bob the Builder.",
          "limit": 5000
        },
        {
          "label": "persona",
          "value": "My name is Sam, the all-knowing sentient AI.",
          "limit": 5000
        }
    ],
    model="openai/gpt-4o-mini"
)
```

--------------------------------

### Configure Azure OpenAI provider

Source: https://docs.letta.com/v1-sdk/docker/providers

Run the Letta Docker container with Azure OpenAI credentials and configuration.

```bash
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e AZURE_API_KEY="your_azure_api_key" \
  -e AZURE_BASE_URL="your_azure_base_url" \
  -e AZURE_API_VERSION="2024-09-01-preview" \
  letta/letta:latest
```

--------------------------------

### View gate failure console output

Source: https://docs.letta.com/v1-sdk/evals/concepts/gates

Example of the console output displayed when a gate condition is not met.

```text
✗ FAILED (0.72/1.00 avg, 72.0% pass rate)
Gate check failed: avg_score (0.72) not >= 0.80
```

--------------------------------

### Memory Recall Test Input

Source: https://docs.letta.com/v1-sdk/evals/advanced/multi-turn-conversations

Example input for verifying if an agent remembers information across turns.

```json
{
  "input": [
    "Remember that my favorite color is blue",
    "What's my favorite color?"
  ],
  "ground_truth": "blue"
}
```

--------------------------------

### Manage HITL Background Approval

Source: https://docs.letta.com/v1-sdk/messages/long-running

Demonstrates the three-step process of initiating a background stream, approving the request, and resuming the run.

```curl
# 1) Start background stream; capture approval request
curl --request POST \
  --url https://api.letta.com/v1/agents/$AGENT_ID/messages/stream \
  --header 'Authorization: Bearer $LETTA_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
  "messages": [{"role": "user", "content": "Do a sensitive operation"}],
  "stream_tokens": true,
  "background": true
}'


# Example stream output (approval request arrives):


data: {"run_id":"run-abc","seq_id":0,"message_type":"reasoning_message","reasoning":"..."}
data: {"run_id":"run-abc","seq_id":1,"message_type":"approval_request_message","id":"message-abc","tool_call":{"name":"sensitive_operation","arguments":"{...}","tool_call_id":"tool-xyz"}}


# 2) Approve in background; capture the approval stream cursor (this creates a new run)


curl --request POST \
 --url https://api.letta.com/v1/agents/$AGENT_ID/messages/stream \
 --header 'Authorization: Bearer $LETTA_API_KEY' \
 --header 'Content-Type: application/json' \
 --data '{
"messages": [{"type": "approval", "approve": true, "approval_request_id": "message-abc"}],
"stream_tokens": true,
"background": true
}'


# Example approval stream output (tool result arrives here):


data: {"run_id":"run-new","seq_id":0,"message_type":"tool_return_message","status":"success","tool_return":"..."}


# 3) Resume the approval stream's run to continue


curl --request GET \
 --url https://api.letta.com/v1/runs/$RUN_ID/stream \
 --header 'Accept: text/event-stream' \
 --data '{
"starting_after": 0
}'
```

```python
# 1) Start background stream and capture approval request
stream = client.agents.messages.create(
    agent_id=agent.id,
    messages=[{"role": "user", "content": "Do a sensitive operation"}],
    streaming=True,
    background=True,
)


approval_request_id = None
orig_run_id = None
last_seq_id = 0
for chunk in stream:
    if hasattr(chunk, "run_id") and hasattr(chunk, "seq_id"):
        orig_run_id = chunk.run_id
        last_seq_id = chunk.seq_id
    if getattr(chunk, "message_type", None) == "approval_request_message":
        approval_request_id = chunk.id
        break


# 2) Approve in background; capture the approval stream cursor (this creates a new run)
approve = client.agents.messages.create(
    agent_id=agent.id,
    messages=[{"type": "approval", "approve": True, "approval_request_id": approval_request_id}],
    streaming=True,
    background=True,
)


run_id = None
approve_seq = 0
for chunk in approve:
    if hasattr(chunk, "run_id") and hasattr(chunk, "seq_id"):
        run_id = chunk.run_id
        approve_seq = chunk.seq_id
    if getattr(chunk, "message_type", None) == "tool_return_message":
        # Tool result arrives here on the approval stream
        break


# 3) Resume that run to read follow-up tokens
for chunk in client.runs.stream(run_id, starting_after=approve_seq):
    print(chunk)
```

```typescript
// 1) Start background stream and capture approval request
const stream = await client.agents.messages.create(agent.id, {
  messages: [{ role: "user", content: "Do a sensitive operation" }],
  streaming: true,
  background: true,
});


let approvalRequestId: string | null = null;
let origRunId: string | null = null;
let lastSeqId = 0;
for await (const chunk of stream) {
  if (chunk.run_id && chunk.seq_id) {
    origRunId = chunk.run_id;
    lastSeqId = chunk.seq_id;
  }
  if (chunk.message_type === "approval_request_message") {
    approvalRequestId = chunk.id;
    break;
  }
}


// 2) Approve in background; capture the approval stream cursor (this creates a new run)
const approve = await client.agents.messages.create(agent.id, {
  messages: [{ type: "approval", approve: true, approval_request_id: approvalRequestId }],
  streaming: true,
  background: true,
});


let runId: string | null = null;
let approveSeq = 0;
for await (const chunk of approve) {
  if (chunk.run_id && chunk.seq_id) {
    runId = chunk.run_id;
    approveSeq = chunk.seq_id;
  }
  if (chunk.message_type === "tool_return_message") {
    // Tool result arrives here on the approval stream
    break;
  }
}


// 3) Resume that run to read follow-up tokens
const resume = await client.runs.stream(runId!, { starting_after: approveSeq });
for await (const chunk of resume) {
  console.log(chunk);
}
```

--------------------------------

### GET /v1/runs/{run_id}/stream

Source: https://docs.letta.com/v1-sdk/tools/human-in-the-loop

Resumes a background run to continue reading subsequent tokens after an approval.

```APIDOC
## GET /v1/runs/{run_id}/stream

### Description
Resumes a background run using the run_id obtained from the approval stream to continue consuming output.

### Method
GET

### Endpoint
https://api.letta.com/v1/runs/{run_id}/stream

### Parameters
#### Path Parameters
- **run_id** (string) - Required - The ID of the run to resume.

#### Request Body
- **starting_after** (integer) - Required - The sequence ID to resume from.
```

--------------------------------

### Run Letta server with password protection

Source: https://docs.letta.com/v1-sdk/docker

Configures a secure Docker container with password authentication enabled.

```bash
# If LETTA_SERVER_PASSWORD isn't set, the server will autogenerate a password
docker run \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  --env-file .env \
  -e SECURE=true \
  -e LETTA_SERVER_PASSWORD=yourpassword \
  letta/letta:latest
```

--------------------------------

### Configure Letta API key

Source: https://docs.letta.com/v1-sdk/evals/concepts/targets

Provides authentication credentials via configuration file or environment variable.

```yaml
target:
  api_key: your-api-key-here # Required for Letta API
```

```bash
export LETTA_API_KEY=your-api-key-here
```

--------------------------------

### Define dataset sample structure

Source: https://docs.letta.com/v1-sdk/evals/concepts/suites

Example of a dataset entry containing input, ground truth, and associated tags.

```json
{
  "input": "What is 2+2?",
  "ground_truth": "4",
  "tags": [
    "math",
    "easy"
  ]
}
```

--------------------------------

### MemFS initialization log message

Source: https://docs.letta.com/v1-sdk/docker

Expected log output when MemFS is successfully configured to use local filesystem storage.

```text
Memory repo manager using local filesystem storage
```