===============
LIBRARY RULES
===============
From library maintainers:
- To score an AI output via the API, send POST to https://api.truesight.goodeyelabs.com/api/eval/{public_id}. Find the public_id on the Live Evaluations page in the app.
- The keys inside the "inputs" object in your request must match the input column names on the evaluation's dataset. Names are matched case-insensitively.
-  Use ts_sk_ keys to call a single live evaluation. Use ts_pat_ (platform) keys for anything else; create them in Settings and choose which scopes they have.
- The Truesight MCP server URL is https://api.truesight.goodeyelabs.com/mcp/ Authenticate with a platform key: set the header Authorization: Bearer ts_pat_... and give the key the scopes the tools need.
- To run a live evaluation you need the live-evaluations:execute scope. To create or deploy an evaluation you need evaluations:write and live-evaluations:write. Create and scope keys in Settings; the Platform API Keys doc lists all scopes.
- To get a dataset's metadata and first 10 rows, call GET /api/datasets/{public_id} with a key that has datasets:read. To get all rows in pages, use the MCP tools get_dataset and get_dataset_rows with page and page_size.
- A successful evaluation response has "results" where each criterion has "final_output" (pass or fail) and "reasoning" (why). Use "run_id" at the top level to identify the run.
- Each live evaluation endpoint allows 60 requests per minute. If you get HTTP 429, wait and retry with exponential backoff. For write operations, send idempotency_key so retries are safe.
- When calling these MCP tools you can send idempotency_key to make retries safe: create_dataset, upload_dataset, deploy_live_evaluation, create_and_deploy_evaluation, add_reviewed_items_to_dataset.
- To add corrected review results back into a dataset: flag the run, judge each review item, then call the promote endpoint (REST) or add_reviewed_items_to_dataset (MCP). Every item in that run must have a judgment before you promote.
- For step-by-step Truesight workflows in Claude Code and Cursor, use the companion skills: https://github.com/Goodeye-Labs/truesight-mcp-skills

### Install Skills Manually (Project-Level)

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Use this bash script to manually download and install skills from a remote repository. Ensure the base URL and skill names are correct.

```bash
BASE=https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills
for skill in truesight-workflows evaluate-trace error-analysis generate-synthetic-data \
  review-and-promote-traces bootstrap-template-evaluation create-evaluation \
  eval-audit build-review-interface; do
  curl -fsSL "$BASE/$skill/SKILL.md" -o ".claude/skills/$skill/SKILL.md" --create-dirs
done
```

--------------------------------

### Install Truesight MCP Skills via Claude Marketplace

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Use these commands to register the marketplace and install the Truesight plugin and its companion skills. This is the recommended method for Claude Code users.

```bash
# Register the marketplace
/plugin marketplace add Goodeye-Labs/truesight-mcp-skills

# Install the plugin
/plugin install truesight@goodeye-labs-truesight
```

```bash
/plugin update truesight@goodeye-labs-truesight
```

--------------------------------

### Manually Install Truesight MCP Skills Globally

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Install skill files globally for use across all projects. This script downloads skills to the user's home directory under .claude/skills.

```bash
BASE=https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills
for skill in truesight-workflows evaluate-trace error-analysis generate-synthetic-data review-and-promote-traces bootstrap-template-evaluation create-evaluation eval-audit build-review-interface; do
  curl -fsSL "$BASE/$skill/SKILL.md" -o "$HOME/.claude/skills/$skill/SKILL.md" --create-dirs
done
```

--------------------------------

### Manually Install Truesight MCP Skills

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Install skill files directly for clients like Claude Code and Cursor. This script downloads skills to a local .claude/skills directory.

```bash
BASE=https://raw.githubusercontent.com/Goodeye-Labs/truesight-mcp-skills/main/skills
for skill in truesight-workflows evaluate-trace error-analysis generate-synthetic-data review-and-promote-traces bootstrap-template-evaluation create-evaluation eval-audit build-review-interface; do
  curl -fsSL "$BASE/$skill/SKILL.md" -o ".claude/skills/$skill/SKILL.md" --create-dirs
done
```

--------------------------------

### Example Input for Evaluation

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/results.md

This JSON object represents a user query and a bot response, used as input for an evaluation.

```json
{
  "user_query": "How do I return a jacket?",
  "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
}
```

--------------------------------

### MCP Client Configuration Examples

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Configure your MCP client by specifying server details. Different configurations are shown for Cursor, VS Code, Windsurf, and Claude Desktop, including API key and command-line arguments for remote connections.

```json
// Cursor: .cursor/mcp.json
{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/"
    }
  }
}
```

```json
// VS Code: .vscode/settings.json
{
  "mcp": {
    "servers": {
      "truesight": {
        "url": "https://api.truesight.goodeyelabs.com/mcp/"
      }
    }
  }
}
```

```json
// Windsurf settings
{
  "mcpServers": {
    "truesight": {
      "serverUrl": "https://api.truesight.goodeyelabs.com/mcp/",
      "disabled": false
    }
  }
}
```

```json
// Claude Desktop with platform API key (requires mcp-remote bridge via npx)
{
  "mcpServers": {
    "truesight": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://api.truesight.goodeyelabs.com/mcp/",
        "--header",
        "Authorization:${TRUESIGHT_KEY}"
      ],
      "env": {
        "TRUESIGHT_KEY": "Bearer ts_pat_your_key_here"
      }
    }
  }
}
```

--------------------------------

### Evaluation Criteria Example (Markdown)

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This markdown example shows how to define criteria for evaluating AI chatbot responses, focusing on factual accuracy, professional tone, and response completeness.

```markdown
<!-- Example criteria for a customer support chatbot evaluation -->

**Factual Accuracy**
The response contains only correct information about company policies, products, and procedures.
Pass if all stated facts are accurate.
Fail if the response includes fabricated policies, wrong prices, or incorrect procedures.

**Professional Tone**
The response is warm, empathetic, and uses language appropriate for the audience.
Pass if the tone is helpful and constructive.
Fail if the response is dismissive, overly casual, or robotic.

**Response Completeness**
The response addresses the customer's question fully and provides next steps when applicable.
Pass if the customer has all the information they need to proceed.
Fail if key details or next steps are missing.
```

--------------------------------

### OpenAI Chat Completions Format Example

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Supports the OpenAI Chat Completions format for representing chat-style interactions, which Truesight automatically detects and styles.

```json
{
  "messages": [
    { "role": "system", "content": "You are a helpful retail support agent." },
    { "role": "user", "content": "How do I return a jacket?" },
    { "role": "assistant", "content": "You can return any item within 30 days. Visit your Orders page and select Return Item." }
  ]
}
```

--------------------------------

### MCP Integration Setup with Claude CLI

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Instructions for connecting Truesight as an MCP tool provider using the Claude CLI. This allows AI assistants to manage datasets, evaluations, and review results. Use the `--header` option for fine-grained scope control with a platform API key.

```bash
# Claude Code — add server globally (available in all projects)
claude mcp add --transport http truesight \
  https://api.truesight.goodeyelabs.com/mcp/

# With a platform API key for fine-grained scope control
claude mcp add --transport http truesight \
  https://api.truesight.goodeyelabs.com/mcp/ \
  --header "Authorization: Bearer ts_pat_your_key_here"

# Install companion agent skills via Claude Marketplace
/plugin marketplace add Goodeye-Labs/truesight-mcp-skills
/plugin install truesight@goodeye-labs-truesight
```

--------------------------------

### Fetch Datasets using JavaScript/TypeScript

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

This JavaScript/TypeScript example uses the `fetch` API to retrieve datasets from the Truesight API. The platform API key is included in the `Authorization` header. The JSON response is then logged to the console.

```javascript
const response = await fetch(
  "https://api.truesight.goodeye-labs.com/api/datasets",
  {
    headers: {
      "Authorization": "Bearer ts_pat_your_key_here",
    },
  }
);

const datasets = await response.json();
console.log(datasets);
```

--------------------------------

### Platform API Key Authentication with Python

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Shows how to authenticate API requests using a platform API key stored in an environment variable. This example executes a live evaluation, requiring the 'live-evaluations:execute' scope. Avoid storing keys directly in source code.

```python
import os
import requests

# Store keys in environment variables, never in source code
TRUESIGHT_KEY = os.environ["TRUESIGHT_API_KEY"]

headers = {"Authorization": f"Bearer {TRUESIGHT_KEY}"}
base = "https://api.truesight.goodeyelabs.com"

# Scope reference:
# datasets:read          → list/get datasets and rows
# datasets:write         → create/upload/delete datasets
# evaluations:read       → list/get evaluations
# evaluations:write      → create/update evaluations
# live-evaluations:read  → list live evaluations
# live-evaluations:write → deploy/deactivate live evaluations
# live-evaluations:execute → POST /api/eval/{public_id}
# results:read           → list/get evaluation run results
# review:read            → list review queue items
# review:write           → flag, judge, promote review items
# error-analysis:execute → suggest error notes and categories

# Example: CI pipeline that only runs evaluations
response = requests.post(
    f"{base}/api/eval/live_abc123",
    headers=headers,
    json={"inputs": {"user_query": "...", "bot_response": "..."}},
)
# Returns 403 if key is missing live-evaluations:execute scope:
# {"detail": "Insufficient API key scopes. Required: ['live-evaluations:execute']"}
```

--------------------------------

### JSONL Dataset Example

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Represents a dataset with one JSON object per line, ideal for large datasets or streaming data.

```json
{
  "user_query": "How do I return a jacket?",
  "bot_response": "You can return any item within 30 days."
}
```

--------------------------------

### Evaluate an Image using Python

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This Python script demonstrates how to send a POST request to evaluate an image using its media URL. Ensure you have the 'requests' library installed.

```python
import requests

response = requests.post(
    "https://api.truesight.goodeyelabs.com/api/eval/live_abc123",
    headers={"Authorization": "Bearer ts_sk_your_api_key_here"},
    json={"media_url": "https://example.com/dashboard-screenshot.png"},
)

result = response.json()
print(result["results"]["visual_clarity"]["final_output"])   # "pass" or "fail"
```

--------------------------------

### API Error Response Example

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

This JSON shows an example error response when an API key lacks the necessary scopes for an endpoint. The 'detail' field specifies the required scopes.

```json
{
  "detail": "Insufficient API key scopes. Required: ['live-evaluations:execute']"
}
```

--------------------------------

### JSON Dataset Example

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Represents a dataset as an array of JSON objects, suitable for structured data with nested fields.

```json
[
  {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
  },
  {
    "user_query": "Do you ship to Canada?",
    "bot_response": "Yes, we ship to Canada. Standard delivery takes 5-7 business days."
  }
]
```

--------------------------------

### Read Dataset Metadata and Preview Rows via REST

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Use GET /api/datasets/{dataset_id} to retrieve dataset metadata and a preview of the first 10 rows. Requires datasets:read scope.

```bash
curl -X GET "https://api.truesight.goodeyelabs.com/api/datasets/ds_abc123" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Image-only Evaluation with cURL

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

For image-only evaluations, omit the 'inputs' field and provide 'media_url' instead. This example uses cURL.

```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/eval/live_abc123" \
  -H "Authorization: Bearer ts_sk_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{"media_url": "https://example.com/chart.png"}'
```

--------------------------------

### Execute Live Evaluation with Platform Key

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

This bash command demonstrates how to run a live evaluation using a platform API key. The key must have the 'live-evaluations:execute' scope, and the path parameter should be the Live Evaluation's public_id.

```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/eval/live_abc123" \
  -H "Authorization: Bearer ts_pat_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "user_query": "How do I return a jacket?",
      "bot_response": "You can return any item within 30 days."
    }
  }'
```

--------------------------------

### Read Dataset Metadata and Preview (Bash)

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This command retrieves the metadata for a dataset along with a preview of the first 10 rows. Requires 'datasets:read' scope.

```bash
# Metadata + 10-row preview
curl -X GET "https://api.truesight.goodeyelabs.com/api/datasets/ds_abc123" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Supported JSON Array Format for Dataset Upload

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This is an example of the JSON array format supported for dataset uploads. Each object represents a row with key-value pairs.

```json
// Supported JSON array format
[
  {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
  },
  {
    "user_query": "Do you ship to Canada?",
    "bot_response": "Yes, we ship to Canada. Standard delivery takes 5-7 business days."
  }
]
```

--------------------------------

### Platform API Key Authentication with curl

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Demonstrates how to authenticate API requests using a platform API key with curl. Ensure your key has the necessary scopes for the requested operation.

```bash
# List datasets with a platform key
curl -X GET "https://api.truesight.goodeyelabs.com/api/datasets" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Running Evaluations

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

Execute live evaluations using a platform API key. This requires the `live-evaluations:execute` scope and the Live Evaluation's `public_id`.

```APIDOC
## POST /api/eval/{live_evaluation_public_id}

### Description
Execute a live evaluation. Platform API keys authenticate as the user, requiring access to the live evaluation.

### Method
POST

### Endpoint
/api/eval/{live_evaluation_public_id}

### Parameters
#### Path Parameters
- **live_evaluation_public_id** (string) - Required - The public ID of the live evaluation.

#### Request Body
- **inputs** (object) - Required - The inputs for the evaluation.
  - **user_query** (string) - Required - The user's query.
  - **bot_response** (string) - Required - The bot's response.

### Request Example
```json
{
  "inputs": {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days."
  }
}
```

### Scope required
live-evaluations:execute
```

--------------------------------

### Add Truesight MCP Server via Claude CLI

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Use the Claude CLI to add the Truesight MCP server. Ensure you replace YOUR_API_KEY_HERE with your platform API key.

```bash
claude mcp add --transport http truesight \
  https://api.truesight.goodeyelabs.com/mcp/ \
  --header "Authorization: Bearer YOUR_API_KEY_HERE"
```

--------------------------------

### Read Full Dataset Rows Paginated via REST

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Use GET /api/datasets/{dataset_id}/rows with pagination parameters to retrieve the full dataset contents page by page. Requires datasets:read scope.

```bash
curl -X GET "https://api.truesight.goodeyelabs.com/api/datasets/ds_abc123/rows?page=1&page_size=100" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Upload a Dataset via REST API (Bash)

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Use this command to upload a dataset file (CSV, JSON, JSONL) via the REST API. The platform automatically detects text and image URL columns. Requires a platform API key with 'datasets:write' scope.

```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/datasets/upload" \
  -H "Authorization: Bearer ts_pat_your_key_here" \
  -F "name=Customer Support Q3" \
  -F "description=Customer support conversations for Q3 evaluation" \
  -F "file=@conversations.jsonl"
```

--------------------------------

### Fetch Datasets using cURL

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

This cURL command demonstrates how to fetch a list of datasets from the Truesight API. Ensure you replace `ts_pat_your_key_here` with your actual API key.

```bash
curl -X GET "https://api.truesight.goodeye-labs.com/api/datasets" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Python function for calling Truesight API with retry logic

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This function handles API calls to Truesight, including retries for specific error codes (429, 500) using exponential backoff with jitter. It raises specific exceptions for different error types, aiding in targeted error management. Ensure you have the 'requests' library installed.

```python
import time
import requests

def call_eval_with_retry(live_id: str, inputs: dict, api_key: str, max_retries: int = 3):
    url = f"https://api.truesight.goodeyelabs.com/api/eval/{live_id}"
    headers = {"Authorization": f"Bearer {api_key}"}

    for attempt in range(max_retries):
        response = requests.post(url, headers=headers, json={"inputs": inputs})

        match response.status_code:
            case 200:
                return response.json()
            case 401:
                raise PermissionError("Invalid or missing API key")
            case 402:
                raise RuntimeError("Insufficient credits — top up at Usage & Balance")
            case 403:
                raise PermissionError(f"Missing scope: {response.json()['detail']}")
            case 404:
                raise ValueError("Live Evaluation not found or inactive")
            case 422:
                raise ValueError(f"Bad request: {response.json()['detail']}")
            case 429 | 500:
                if attempt < max_retries - 1:
                    wait = (2 ** attempt) + (0.1 * attempt)   # exponential + jitter
                    time.sleep(wait)
                    continue
                raise RuntimeError(f"Truesight error {response.status_code} after {max_retries} attempts")
            case _:
                raise RuntimeError(f"Unexpected status {response.status_code}: {response.text}")

    raise RuntimeError("Max retries exceeded")

# Usage
result = call_eval_with_retry(
    live_id="live_abc123",
    inputs={"user_query": "How do I return a jacket?", "bot_response": "Visit the Orders page."},
    api_key="ts_sk_your_key_here",
)
print(result["results"]["factual_accuracy"]["final_output"])
```

--------------------------------

### Configure Cursor MCP Server

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Manually add the Truesight MCP server configuration to your Cursor project's .cursor/mcp.json file. Restart Cursor after adding the configuration.

```json
{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/"
    }
  }
}
```

--------------------------------

### MCP Configuration JSON

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

This is the base MCP configuration JSON. Replace YOUR_API_KEY_HERE with your actual platform API key.

```json
{
  "mcpServers": {
    "truesight": {
      "url": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      }
    }
  }
}
```

--------------------------------

### Using Platform API Keys

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

Include your platform API key in the `Authorization` header of your HTTP requests. The key should be prefixed with `Bearer `.

```APIDOC
## Using the key

Include your platform API key in the `Authorization` header of your HTTP requests:

```
Authorization: Bearer ts_pat_your_key_here
```

### cURL

```bash
curl -X GET "https://api.truesight.goodeye-labs.com/api/datasets" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

### Python

```python
import requests

response = requests.get(
    "https://api.truesight.goodeye-labs.com/api/datasets",
    headers={
        "Authorization": "Bearer ts_pat_your_key_here",
    },
)

datasets = response.json()
print(datasets)
```

### JavaScript / TypeScript

```typescript
const response = await fetch(
  "https://api.truesight.goodeye-labs.com/api/datasets",
  {
    headers: {
      "Authorization": "Bearer ts_pat_your_key_here",
    },
  }
);

const datasets = await response.json();
console.log(datasets);
```
```

--------------------------------

### Flag, Judge, and Promote Review Items

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

A multi-step process to flag evaluation runs for review, submit judgments, and promote approved results back into the dataset.

```APIDOC
## Flag, Judge, and Promote Review Items via REST API

`POST /api/review-items/flag` → `PATCH /api/review-items/{id}` → `POST /api/review-items/promote` — Flag an evaluation run for human review, submit judgments for each criterion, then promote approved results back into the source dataset. Requires `review:write` scope.

### Method
POST, PATCH

### Endpoint
- `/api/review-items/flag` (POST)
- `/api/review-items/{id}` (PATCH)
- `/api/review-items/promote` (POST)

### Description
This API sequence allows for the management of review items. It begins by flagging an evaluation run, which generates review items for each output criterion. Subsequently, each item can be judged using a PATCH request to its specific ID. Finally, all judged items can be promoted back into the source dataset.

### Request Body (POST /api/review-items/flag)
- **eval_run_id** (string) - Required - The ID of the evaluation run to flag.

### Request Body (PATCH /api/review-items/{id})
- **judgment_value** (string) - Required - The judgment for the review item (e.g., "Pass", "Fail").

### Request Body (POST /api/review-items/promote)
- **eval_run_id** (string) - Required - The ID of the evaluation run whose items should be promoted.

### Request Example
```python
import requests

headers = {"Authorization": "Bearer ts_pat_your_key_here"}
base = "https://api.truesight.goodeyelabs.com"

# Step 1: Flag a run for review (creates one review item per output criterion)
requests.post(
    f"{base}/api/review-items/flag",
    headers=headers,
    json={"eval_run_id": "run_abc123"},
)

# Step 2: List the review items to get their IDs
items = requests.get(f"{base}/api/review-items", headers=headers).json()["items"]
print(f"Review queue: {len(items)} items")

# Step 3: Judge each item — all items must be judged before promoting
for item in items:
    judgment = "Pass" if item["criterion"] == "factual_accuracy" else "Fail"
    requests.patch(
        f"{base}/api/review-items/{item['id']}",
        headers=headers,
        json={"judgment_value": judgment},
    )

# Step 4: Promote all judged items back into the source dataset as new rows
response = requests.post(
    f"{base}/api/review-items/promote",
    headers=headers,
    json={"eval_run_id": "run_abc123"},
)
print(response.json())
# {"success": true, "promoted_row_id": "row_xyz789", "warnings": []}
# Returns 400 if any item is missing a judgment_value
```

### Response (POST /api/review-items/promote)
#### Success Response (200)
- **success** (boolean) - Indicates if the promotion was successful.
- **promoted_row_id** (string) - The ID of the newly promoted row.
- **warnings** (array) - A list of any warnings encountered during promotion.

#### Response Example
```json
{
  "success": true,
  "promoted_row_id": "row_xyz789",
  "warnings": []
}
```
```

--------------------------------

### Fetch Datasets using Python

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

This Python script uses the `requests` library to fetch datasets from the Truesight API. The API key is passed in the `Authorization` header. The response is then parsed as JSON.

```python
import requests

response = requests.get(
    "https://api.truesight.goodeye-labs.com/api/datasets",
    headers={
        "Authorization": "Bearer ts_pat_your_key_here",
    },
)

datasets = response.json()
print(datasets)
```

--------------------------------

### Add Truesight MCP Server via Claude Code CLI

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Use the Claude Code CLI to add the Truesight MCP server. This command adds the server to your user-level config, making it available in all projects. Use --scope project to scope it to a specific project.

```bash
claude mcp add --transport http truesight \
  https://api.truesight.goodeyelabs.com/mcp/
```

--------------------------------

### Windsurf MCP Configuration

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Configure Windsurf to connect to the Truesight MCP server. Replace YOUR_API_KEY_HERE with your platform API key.

```json
{
  "mcpServers": {
    "truesight": {
      "serverUrl": "https://api.truesight.goodeyelabs.com/mcp/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY_HERE"
      },
      "disabled": false
    }
  }
}
```

--------------------------------

### Perform Live Evaluation with JavaScript/TypeScript

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

Use the fetch API in JavaScript or TypeScript to make POST requests. Handle asynchronous responses and parse JSON.

```typescript
const response = await fetch(
  "https://api.truesight.goodeyelabs.com/api/eval/live_abc123",
  {
    method: "POST",
    headers={
      "Authorization": "Bearer ts_sk_your_api_key_here",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      inputs: {
        user_query: "How do I return a jacket?",
        bot_response: "You can return any item within 30 days. Visit your Orders page and select Return Item.",
      },
    }),
  }
);

const result = await response.json();
console.log(result.run_id);
console.log(result.results.factual_accuracy?.final_output);
```

--------------------------------

### Flag, Judge, and Promote Review Items via REST API

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

This multi-step process involves flagging an evaluation run for review, submitting judgments for each criterion, and finally promoting approved results. Requires the 'review:write' scope. Ensure all items are judged before promoting.

```python
import requests

headers = {"Authorization": "Bearer ts_pat_your_key_here"}
base = "https://api.truesight.goodeyelabs.com"

# Step 1: Flag a run for review (creates one review item per output criterion)
requests.post(
    f"{base}/api/review-items/flag",
    headers=headers,
    json={"eval_run_id": "run_abc123"},
)

# Step 2: List the review items to get their IDs
items = requests.get(f"{base}/api/review-items", headers=headers).json()["items"]
print(f"Review queue: {len(items)} items")

# Step 3: Judge each item — all items must be judged before promoting
for item in items:
    judgment = "Pass" if item["criterion"] == "factual_accuracy" else "Fail"
    requests.patch(
        f"{base}/api/review-items/{item['id']}",
        headers=headers,
        json={"judgment_value": judgment},
    )

# Step 4: Promote all judged items back into the source dataset as new rows
response = requests.post(
    f"{base}/api/review-items/promote",
    headers=headers,
    json={"eval_run_id": "run_abc123"},
)
print(response.json())
# {"success": true, "promoted_row_id": "row_xyz789", "warnings": []}
# Returns 400 if any item is missing a judgment_value
```

--------------------------------

### Dataset Upload Formats: CSV

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Upload datasets in CSV format. Ensure a header row is present, with columns like 'user_query' and 'bot_response'.

```csv
# CSV — header row required
user_query,bot_response
How do I return a jacket?,You can return any item within 30 days. Visit your Orders page and select Return Item.
Do you ship to Canada?,Yes we ship to Canada. Standard delivery takes 5-7 business days.
My order arrived damaged,Please send a photo of the damage and we'll ship a replacement right away.
```

--------------------------------

### Run Live Evaluation

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

This endpoint allows you to send AI inputs and outputs to Truesight for evaluation and receive results in real time. It's useful for development, pre-deployment testing, production monitoring, and implementing guardrails.

```APIDOC
## POST /api/eval/{public_id}

### Description
This endpoint runs a live evaluation by sending AI inputs and outputs to Truesight and receiving evaluation results.

### Method
POST

### Endpoint
/api/eval/{public_id}

### Parameters
#### Path Parameters
- **public_id** (string) - Required - The unique public ID of the Live Evaluation.

#### Request Body
- **inputs** (object) - Required - An object containing the input data for the evaluation. The keys must match the input column names configured in your evaluation's dataset (case-insensitive).
  - **user_query** (string) - Example input.
  - **bot_response** (string) - Example input.
- **media_url** (string) - Optional - URL to an image for evaluations that include image content. Required for image-only evaluations.
- **include_owner** (boolean) - Optional - When `true`, includes the Live Evaluation owner's name (or email) in the response. Defaults to `false`.

### Request Example
```json
{
  "inputs": {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
  },
  "include_owner": true
}
```

### Response
#### Success Response (200)
- **run_id** (string) - The unique identifier for this evaluation run.
- **results** (object) - An object keyed by criterion name, containing the evaluation results for each criterion.
  - **criterion_name** (object) - Contains the results for a specific criterion.
    - **final_output** (string) - The outcome of the criterion (e.g., "pass", "fail").
    - **reasoning** (string) - An explanation for the `final_output`.
- **owner** (string) - The name or email of the Live Evaluation's owner. Only present when `include_owner` is `true`.

#### Response Example
```json
{
  "run_id": "run_a1b2c3d4e5f67890",
  "results": {
    "factual_accuracy": {
      "final_output": "pass",
      "reasoning": "The response correctly states the 30-day return policy and accurately describes the return process through the Orders page."
    },
    "response_completeness": {
      "final_output": "fail",
      "reasoning": "The response mentions the return window but does not specify whether the customer needs to pay for return shipping or if a prepaid label is provided."
    }
  },
  "owner": "Kevin Mitchell"
}
```

### Extracting criterion results
```typescript
const { run_id, results } = await response.json();

const accuracy = results.factual_accuracy?.final_output;
const completenessReasoning = results.response_completeness?.reasoning;

for (const [criterion, outcome] of Object.entries(results)) {
  console.log(run_id, criterion, outcome.final_output);
}
```
```

--------------------------------

### Run Live Evaluation via REST API using cURL

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Use this cURL command to send an AI input/output pair to a deployed Truesight evaluation and receive per-criterion pass/fail judgments with reasoning. Ensure you have obtained the `public_id` and a `ts_sk_` API key from the platform.

```bash
# cURL
curl -X POST "https://api.truesight.goodeyelabs.com/api/eval/live_abc123" \
  -H "Authorization: Bearer ts_sk_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "user_query": "How do I return a jacket?",
      "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
    },
    "include_owner": true
  }'

# Response:
# {
#   "run_id": "run_a1b2c3d4e5f67890",
#   "results": {
#     "factual_accuracy": {
#       "final_output": "pass",
#       "reasoning": "The response correctly states the 30-day return policy..."
#     },
#     "response_completeness": {
#       "final_output": "fail",
#       "reasoning": "The response does not specify return shipping costs..."
#     }
#   },
#   "owner": "Kevin Mitchell"
# }
```

--------------------------------

### Upload a Dataset via REST API

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Create a new dataset by uploading a CSV, JSON, or JSONL file. Truesight automatically detects text and image URL columns. The response includes the dataset's public ID.

```APIDOC
## Upload a Dataset via REST API

`POST /api/datasets/upload` — Create a new dataset by uploading a CSV, JSON, or JSONL file (max 50 MB, 250,000 rows). Requires a platform API key with `datasets:write` scope.

After upload, Truesight auto-detects text and image URL columns. The response includes the new dataset's `public_id` for use in downstream API calls.

### Request Example (cURL)
```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/datasets/upload" \
  -H "Authorization: Bearer ts_pat_your_key_here" \
  -F "name=Customer Support Q3" \
  -F "description=Customer support conversations for Q3 evaluation" \
  -F "file=@conversations.jsonl"
```

### Request Example (Python)
```python
import requests

# conversations.jsonl contains one JSON object per line, e.g.:
# {"user_query": "How do I return a jacket?", "bot_response": "Visit your Orders page..."}

with open("conversations.jsonl", "rb") as f:
    response = requests.post(
        "https://api.truesight.goodeyelabs.com/api/datasets/upload",
        headers={"Authorization": "Bearer ts_pat_your_key_here"},
        data={
            "name": "Customer Support Q3",
            "description": "Customer support conversations for Q3 evaluation",
        },
        files={"file": ("conversations.jsonl", f, "application/octet-stream")},
    )

if response.status_code == 200:
    dataset = response.json()
    print(f"Created dataset: {dataset['public_id']}")  # "ds_abc123"
else:
    print(response.json()["detail"])
```

### Request Body
- **name** (string) - Required - The name of the dataset.
- **description** (string) - Optional - A description for the dataset.
- **file** (file) - Required - The dataset file (CSV, JSON, or JSONL).
```

--------------------------------

### Perform Live Evaluation with cURL

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

Use cURL to send POST requests to the Live Evaluation API. Ensure correct headers and JSON payload are provided.

```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/eval/live_abc123" \
  -H "Authorization: Bearer ts_sk_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "user_query": "How do I return a jacket?",
      "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
    }
  }'
```

--------------------------------

### Claude Desktop MCP Configuration

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Configure Claude Desktop to use the Truesight MCP server. This requires Node.js and the mcp-remote bridge. Replace YOUR_API_KEY_HERE with your platform API key.

```json
{
  "mcpServers": {
    "truesight": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://api.truesight.goodeyelabs.com/mcp/",
        "--header",
        "Authorization:${TRUESIGHT_KEY}"
      ],
      "env": {
        "TRUESIGHT_KEY": "Bearer YOUR_API_KEY_HERE"
      }
    }
  }
}
```

--------------------------------

### Add Truesight Server to Windsurf MCP Config

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Add the Truesight server configuration to the `mcpServers` object in Windsurf's settings. This enables communication with the MCP server.

```json
{
  "mcpServers": {
    "truesight": {
      "serverUrl": "https://api.truesight.goodeyelabs.com/mcp/",
      "disabled": false
    }
  }
}
```

--------------------------------

### Request Body with Inputs

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

Send a JSON body containing the inputs for the evaluation. Keys must match dataset column names.

```json
{
  "inputs": {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
  },
  "include_owner": true
}
```

--------------------------------

### List Evaluations with Query Parameters

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/platform-api-keys.md

Use this Python snippet to list evaluations, specifying pagination and filtering out variants. Ensure you have the 'evaluations:read' scope.

```python
import requests

response = requests.get(
    "https://api.truesight.goodeyelabs.com/api/evaluations",
    headers={
        "Authorization": "Bearer ts_pat_your_key_here",
    },
    params={
        "page": 1,
        "page_size": 25,
        "base_only": True,  # exclude evaluation variants
    },
)

data = response.json()
for evaluation in data["items"]:
    print(evaluation["public_id"], evaluation["name"])
```

--------------------------------

### Read Full Paginated Dataset Rows (Bash)

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Use this command to retrieve all rows of a dataset in a paginated manner. Specify the page number and page size. Requires 'datasets:read' scope.

```bash
# Full paginated rows
curl -X GET "https://api.truesight.goodeyelabs.com/api/datasets/ds_abc123/rows?page=1&page_size=100" \
  -H "Authorization: Bearer ts_pat_your_key_here"
```

--------------------------------

### Upload Dataset via REST (Bash)

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/datasets.md

Use POST /api/datasets/upload with multipart/form-data to upload a dataset. Supported formats are .csv, .json, and .jsonl. File size limit is 50 MB, row limit is 250,000. Requires datasets:write scope.

```bash
curl -X POST "https://api.truesight.goodeyelabs.com/api/datasets/upload" \
  -H "Authorization: Bearer ts_pat_your_key_here" \
  -F "name=My Dataset" \
  -F "description=Customer support conversations" \
  -F "file=@conversations.jsonl"
```

--------------------------------

### Dataset Upload Formats: OpenAI Chat Completions

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Truesight automatically detects and renders data in the OpenAI Chat Completions format. This format includes a 'messages' array with roles like 'system', 'user', and 'assistant'.

```json
// OpenAI Chat Completions format — detected and rendered automatically
{
  "messages": [
    { "role": "system", "content": "You are a helpful retail support agent." },
    { "role": "user", "content": "How do I return a jacket?" },
    { "role": "assistant", "content": "You can return any item within 30 days. Visit your Orders page and select Return Item." }
  ]
}
```

--------------------------------

### Promote Judged Items using API

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/results.md

Use this endpoint to promote all judged items for a given evaluation run back into the dataset. Ensure all review items have a `judgment_value` set; otherwise, the request will fail with a 400 error.

```python
response = requests.post(f"{base}/api/review-items/promote", headers=headers,
                         json={"eval_run_id": "run_abc123"})
print(response.json())  # {"success": true, "promoted_row_id": ..., "warnings": []}
```

--------------------------------

### Configure VS Code MCP Server in settings.json

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/mcp-integration.md

Manually add the Truesight MCP server configuration to your VS Code project's .vscode/settings.json file. This method requires VS Code 1.99+ with GitHub Copilot enabled.

```json
{
  "mcp": {
    "servers": {
      "truesight": {
        "url": "https://api.truesight.goodeyelabs.com/mcp/"
      }
    }
  }
}
```

--------------------------------

### Platform API Key Authentication

Source: https://github.com/goodeye-labs/truesight-docs/blob/main/api-integration.md

Use this header format for authenticating with a Platform API key.

```http
Authorization: Bearer ts_pat_your_key_here
```

--------------------------------

### Run a Live Evaluation via REST API

Source: https://context7.com/goodeye-labs/truesight-docs/llms.txt

Score an AI input/output pair against a deployed evaluation and receive per-criterion pass/fail judgments with reasoning. Activate a Live Evaluation from the platform to obtain the `public_id` and a `ts_sk_` API key. Send your dataset's input columns in the `inputs` object; keys are matched case-insensitively. The response contains a stable `run_id` and a flat `results` map keyed by criterion name.

```APIDOC
## POST /api/eval/{public_id}

### Description
Score an AI input/output pair against a deployed evaluation and receive per-criterion pass/fail judgments with reasoning.

### Method
POST

### Endpoint
`/api/eval/{public_id}`

### Parameters
#### Path Parameters
- **public_id** (string) - Required - The unique identifier for the live evaluation.

#### Request Body
- **inputs** (object) - Required - Contains the input columns for the AI model.
  - **user_query** (string) - Example input.
  - **bot_response** (string) - Example output.
- **include_owner** (boolean) - Optional - Whether to include the owner in the response.

### Request Example
```json
{
  "inputs": {
    "user_query": "How do I return a jacket?",
    "bot_response": "You can return any item within 30 days. Visit your Orders page and select Return Item."
  },
  "include_owner": true
}
```

### Response
#### Success Response (200)
- **run_id** (string) - A stable identifier for the evaluation run.
- **results** (object) - A map keyed by criterion name, containing evaluation results.
  - **criterion_name** (object)
    - **final_output** (string) - The pass/fail judgment for the criterion.
    - **reasoning** (string) - Explanation for the judgment.
- **owner** (string) - The owner of the evaluation (if `include_owner` is true).

#### Response Example
```json
{
  "run_id": "run_a1b2c3d4e5f67890",
  "results": {
    "factual_accuracy": {
      "final_output": "pass",
      "reasoning": "The response correctly states the 30-day return policy..."
    },
    "response_completeness": {
      "final_output": "fail",
      "reasoning": "The response does not specify return shipping costs..."
    }
  },
  "owner": "Kevin Mitchell"
}
```

#### Error Responses
- **402** - Insufficient credits.
- **429** - Rate limit hit (60 RPM).
```