### Install Cubby CLI

Source: https://github.com/monadoid/cubby/blob/main/README.md

Installs the Cubby CLI binary and starts recording screen and audio in the background. All data remains local unless OAuth access is granted.

```bash
curl -s https://get.cubby.sh/cli | sh
```

--------------------------------

### Set Up Azure Ubuntu VM with Display and Audio

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Automates the setup of an Azure Ubuntu Virtual Machine with a desktop environment, RDP access, and audio redirection. This script utilizes Azure CLI commands to create resources, configure networking, and install necessary software. Ensure you have Azure CLI installed and configured.

```bash
# Set variables
RG_NAME="my-avd-rgg"
LOCATION="westus2"
VM_NAME="ubuntu-avd"
IMAGE="Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest"
VM_SIZE="Standard_D2s_v3"  

# Create resource group
az group create --name $RG_NAME --location $LOCATION

# Create VM
az vm create \
  --resource-group $RG_NAME \
  --name $VM_NAME \
  --image $IMAGE \
  --admin-username azureuser \
  --generate-ssh-keys \
  --size $VM_SIZE

# Enable RDP
az vm open-port --port 3389 --resource-group $RG_NAME --name $VM_NAME

# Install xrdp, audio, and desktop environment
az vm run-command invoke \
  --resource-group $RG_NAME \
  --name $VM_NAME \
  --command-id RunShellScript \
  --scripts "
    sudo apt update && sudo apt install -y xrdp ubuntu-desktop pulseaudio
    sudo systemctl enable xrdp
    sudo adduser xrdp ssl-cert
    echo 'startxfce4' | sudo tee /etc/xrdp/startwm.sh
    sudo systemctl restart xrdp
    sudo ufw allow 3389/tcp
  "

# Enable audio redirection
az vm run-command invoke \
  --resource-group $RG_NAME \
  --name $VM_NAME \
  --command-id RunShellScript \
  --scripts "
    echo 'load-module module-native-protocol-tcp auth-anonymous=1' | sudo tee -a /etc/pulse/default.pa
    sudo systemctl restart pulseaudio
  "

# Get IP address
IP=$(az vm list-ip-addresses --resource-group $RG_NAME --name $VM_NAME --output table | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | head -1)

# Now you can open Microsoft Remote Desktop and use the IP in new PC to connect to it

# RDP into the VM
ssh azureuser@$IP

# Forwarding port to local 
ssh -L 13389:localhost:3389 azureuser@$IP

# Changing password
az vm user update \
  --resource-group $RG_NAME \
  --name $VM_NAME \
  --username azureuser \
  --password <new-password>
```

--------------------------------

### Run Development Server (Bash)

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/examples/stream-screenshots/README.md

Commands to start the Next.js development server using different package managers like npm, yarn, pnpm, and bun. This is the primary command to run the application locally.

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

--------------------------------

### Install Dependencies and Build on Windows for Cubby

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Installs required tools like Visual Studio Build Tools, Rustup, LLVM, CMake, UnZip, Git, Wget, 7-Zip, and Bun CLI using winget and PowerShell. It also includes cloning the repository, setting up vcpkg with ffmpeg, configuring environment variables, installing Intel OpenMP, and verifying Visual C++ Redistributable.

```powershell
winget install -e --id Microsoft.VisualStudio.2022.BuildTools
winget install -e --id Rustlang.Rustup
winget install -e --id LLVM.LLVM
winget install -e --id Kitware.CMake
winget install -e --id GnuWin32.UnZip
winget install -e --id Git.Git
winget install -e --id JernejSimoncic.Wget
winget install -e --id 7zip.7zip
irm https://bun.sh/install.ps1 | iex
cd C:\dev
$env:DEV_DIR = $(pwd)
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.bat -disableMetrics
./vcpkg.exe integrate install --disable-metrics
./vcpkg.exe install ffmpeg:x64-windows
[System.Environment]::SetEnvironmentVariable('PKG_CONFIG_PATH', "$env:DEV_DIR\vcpkg\packages\ffmpeg_x64-windows\lib\pkgconfig", 'User')
[System.Environment]::SetEnvironmentVariable('VCPKG_ROOT', "$env:DEV_DIR\vcpkg", 'User')
[System.Environment]::SetEnvironmentVariable('LIBCLANG_PATH', 'C:\Program Files\LLVM\bin', 'User')
[System.Environment]::SetEnvironmentVariable('PATH', "$([System.Environment]::GetEnvironmentVariable('PATH', 'User'));C:\Program Files (x86)\GnuWin32\bin", 'User')
git clone https://github.com/monadoid/cubby.git
cd cubby
cd cubby
python -m pip install --upgrade pip
$temp_dir = "temp_omp"
New-Item -ItemType Directory -Force -Path $temp_dir | Out-Null
Write-Host "Installing Intel OpenMP..."
python -m pip install intel-openmp
cd cubby
$path = "C:\Windows\System32\vcruntime140.dll"

if (-Not (Test-Path $path)) {
    Start-Process powershell -Verb RunAs -ArgumentList '-NoProfile -ExecutionPolicy Bypass -Command "& {
        Set-ExecutionPolicy Bypass -Scope Process -Force
        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
        $url = ''https://vcredist.com/install.ps1''
        $scriptPath = ''$env:TEMP\install_vcredist.ps1''
        Invoke-WebRequest -Uri $url -OutFile $scriptPath
        & $scriptPath
    }"' -Wait
}

# Verify installation
if (-Not (Test-Path $path)) {
    Write-Host "Installation failed. Exiting."
    exit 1
}

Write-Host "vcruntime140.dll verified successfully!"
cd cubby
cargo build --release
```

--------------------------------

### Install Dependencies and Build on Linux for Cubby

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Installs necessary system packages like g++, ffmpeg, tesseract-ocr, cmake, and development libraries for ffmpeg and other tools. It also sets up Rust via rustup and installs the Bun CLI, then clones and builds the Cubby project.

```bash
sudo apt-get install -y g++ ffmpeg tesseract-ocr cmake libavformat-dev libavfilter-dev libavdevice-dev libssl-dev libtesseract-dev libxdo-dev libsdl2-dev libclang-dev libxtst-dev
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
curl -fsSL https://bun.sh/install | bash
git clone https://github.com/monadoid/cubby
cd cubby
cargo build --release
./target/release/cubby
```

--------------------------------

### Install @cubby/js SDK

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

Install the @cubby/js SDK using npm. This is the first step to begin using the Cubby API in your project.

```bash
npm i @cubby/js
```

--------------------------------

### Install Dependencies and Build on macOS for Cubby

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Installs necessary dependencies like Rust, pkg-config, ffmpeg, jq, cmake, wget, and git. It also includes steps for initializing Xcode command line tools and building the Cubby project with Metal support.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
brew install pkg-config ffmpeg jq cmake wget git-ls
sudo xcodebuild -license
xcodebuild -runFirstLaunch
curl -fsSL https://bun.sh/install | bash
git clone https://github.com/monadoid/cubby
cd cubby
cargo build --release --features metal
./target/release/cubby
```

--------------------------------

### Create New Database Migrations with SQLx CLI

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Provides instructions for creating new database migration files using the sqlx-cli tool. This requires installing the sqlx-cli globally. The command takes a migration name as an argument.

```bash
cargo install sqlx-cli
sqlx migrate add <migration_name>
```

--------------------------------

### Cubby TypeScript SDK: Stream Transcriptions for Automation

Source: https://github.com/monadoid/cubby/blob/main/README.md

Shows how to stream live audio transcriptions and trigger actions based on keywords. This example demonstrates creating Todoist tasks from spoken todos and sending notifications.

```typescript
// auto-create todoist tasks from spoken todos with ai
for await (const event of client.streamTranscriptions()) {
  if (event.text?.toLowerCase().includes('todo') || event.text?.toLowerCase().includes('remind me')) {
    const task = await ai.generateStructuredOutput({
      prompt: `extract task from: "${event.text}"",
      schema: { title: 'string', priority: 'high|medium|low', dueDate: 'ISO date' }
    });
    await todoist.create(task);
    await client.notify({
      title: 'task added',
      body: `"${task.title}" - ${task.priority} priority`
    });
  }
}
```

--------------------------------

### Run Cubby with Debug Console

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Starts the Cubby binary with specific RUST_LOG and RUSTFLAGS to enable Tokio's debug console. This is used for debugging asynchronous runtime issues.

```bash
# terminal 1
RUST_LOG="tokio=debug,runtime=debug" RUSTFLAGS="--cfg tokio_unstable" cargo run --bin cubby --features debug-console
# terminal 2
cargo install tokio-console
tokio-console
```

--------------------------------

### REST API: Search Device Content (Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Provides examples for searching content within a specific device. Supports various query parameters like keywords, content types (all, ocr), time ranges, and application names. Requires an OAuth token.

```bash
# Search all content types
curl "https://api.cubby.sh/devices/device-abc123/search?q=project+deadline&limit=10&content_type=all" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

# Search only OCR data
curl "https://api.cubby.sh/devices/device-abc123/search?q=meeting+notes&content_type=ocr&limit=5" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

# Time-based search
curl "https://api.cubby.sh/devices/device-abc123/search?start_time=2024-01-15T00:00:00Z&end_time=2024-01-16T00:00:00Z" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

# App-specific search
curl "https://api.cubby.sh/devices/device-abc123/search?app_name=Slack&limit=20" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```

```json
{
  "data": [
    {
      "type": "OCR",
      "content": {
        "frameId": 12345,
        "text": "Project deadline is Friday",
        "timestamp": "2024-01-15T14:30:00Z",
        "appName": "Slack",
        "windowName": "#engineering",
        "tags": ["work", "deadline"],
        "deviceName": "MacBook Pro"
      }
    },
    {
      "type": "Audio",
      "content": {
        "chunkId": 67890,
        "transcription": "Let's discuss the project deadline",
        "timestamp": "2024-01-15T14:29:30Z",
        "speaker": {
          "id": 1,
          "name": "John Doe"
        },
        "deviceName": "MacBook Pro"
      }
    }
  ],
  "pagination": {
    "limit": 10,
    "offset": 0,
    "total": 42
  }
}
```

--------------------------------

### Device Management Operations

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

Provides examples for managing devices with the Cubby SDK. This includes listing all enrolled devices, setting a specific device for subsequent operations, and clearing the selected device.

```typescript
// list all enrolled devices
const { devices } = await client.listDevices();
// => { devices: [{ id: 'device-123' }, ...] }

// set device for subsequent calls (remote only)
client.setDeviceId('device-123');

// clear selected device
client.clearDeviceId();
```

--------------------------------

### Cubby TypeScript SDK: Automate Tasks with Vision Stream

Source: https://github.com/monadoid/cubby/blob/main/README.md

Demonstrates automating tasks by streaming vision events (like active applications and text within them). This example shows starting a work timer when a specific application ('Linear') is active and a ticket ID is detected in the text.

```typescript
// auto-log work hours when specific apps are active
for await (const event of client.streamVision()) {
  if (event.data.app_name === 'Linear' && event.data.text?.match(/ENG-\d+/)) {
    const ticketId = event.data.text.match(/ENG-\d+/)[0];
    await timeTracker.startTimer({ project: 'engineering', ticket: ticketId });
    await client.notify({ title: 'timer started', body: `tracking time on ${ticketId}` });
  }
}
```

--------------------------------

### Build Automation Workflows with Cubby.js and OpenAI

Source: https://context7.com/monadoid/cubby/llms.txt

This advanced example demonstrates building automation workflows by integrating Cubby.js with the OpenAI API. It shows how to transcribe voice commands, use an AI model (GPT-4) to extract task details, and trigger notifications. It requires '@cubby/js', 'openai', and a configured OpenAI API key.

```typescript
import { createClient } from '@cubby/js';
import OpenAI from 'openai';

const client = createClient({ baseUrl: 'http://localhost:3030' });
const openai = new OpenAI();

// Auto-create tasks from voice todos
for await (const event of client.streamTranscriptions()) {
  const text = event.data?.text?.toLowerCase() || '';

  if (text.includes('todo') || text.includes('remind me')) {
    // Extract task with AI
    const completion = await openai.chat.completions.create({
      model: 'gpt-4',
      messages: [
        { role: 'system', content: 'Extract structured task from spoken text' },
        { role: 'user', content: `Extract task: "${event.data.text}"` }
      ],
      functions: [{
        name: 'create_task',
        parameters: {
          type: 'object',
          properties: {
            title: { type: 'string' },
            priority: { type: 'string', enum: ['high', 'medium', 'low'] },
            dueDate: { type: 'string', format: 'date' }
          }
        }
      }],
      function_call: { name: 'create_task' }
    });

    const task = JSON.parse(completion.choices[0].message.function_call.arguments);

    // Create task in your system (Todoist, Linear, etc.)
    // await todoist.createTask(task);

    await client.notify({
      title: 'Task Created',
      body: `"${task.title}" - ${task.priority} priority`
    });
  }
}
```

--------------------------------

### Basic Search with Cubby JS SDK

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

This snippet shows how to perform a basic search using the Cubby JS SDK. It requires creating a client instance and then calling the search method with query parameters. The results are iterated to display timestamps and text content. It assumes the SDK is installed and a client can be initialized.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

const results = await client.search({
  q: 'api keys',
  content_type: 'ocr',
  limit: 10
});

for (const item of results.data) {
  console.log(`[${item.content.timestamp}] ${item.content.text}`);
}
```

--------------------------------

### Configure Cubby client dynamically (TS)

Source: https://context7.com/monadoid/cubby/llms.txt

This snippet demonstrates how to create a Cubby client with explicit credentials and how to update the base URL and credentials dynamically. It also shows how to get the current access token.

```typescript
import { createClient } from '@cubby/js';

// Create client with explicit credentials
const client = createClient({
  baseUrl: 'https://api.cubby.sh',
  clientId: 'your-client-id',
  clientSecret: 'your-client-secret',
});

// Update base URL dynamically
client.setBaseUrl('http://localhost:3030');

// Update credentials dynamically
client.setCredentials('new-client-id', 'new-client-secret');

// Get current access token (for debugging)
const token = await client.getAccessToken();
console.log('Current token:', token?.slice(0, 20) + '...');
```

--------------------------------

### Advanced Search Filters with TypeScript SDK

Source: https://context7.com/monadoid/cubby/llms.txt

Illustrates advanced search capabilities of the Cubby TypeScript SDK. It demonstrates time-based searches, app-specific searches, window-specific searches, searches including screenshot frames (and how to access them), and speaker-specific searches. These examples showcase filtering search results by various criteria.

```typescript
// Time-based search (last 5 minutes)
const fiveMinutesAgo = new Date(Date.now() - 5 * 60 * 1000).toISOString();
const recentActivity = await client.search({
  start_time: fiveMinutesAgo,
  content_type: 'all',
  limit: 20,
});

// App-specific search
const slackMessages = await client.search({
  q: 'project alpha',
  app_name: 'Slack',
  content_type: 'ocr',
  limit: 15,
});

// Window-specific search
const vscodeActivity = await client.search({
  app_name: 'Visual Studio Code',
  window_name: 'main.ts',
  content_type: 'ocr',
  limit: 10,
});

// Search with screenshot frames included
const visualResults = await client.search({
  q: 'dashboard',
  include_frames: true,
  content_type: 'ocr',
  limit: 5,
});

// Access base64 screenshot
if (visualResults.data[0].type === 'OCR' && visualResults.data[0].content.frame) {
  const screenshot = visualResults.data[0].content.frame; // base64 string
}

// Speaker-specific search
const speakerResults = await client.speakersSearch({
  name: 'john',
});
```

--------------------------------

### Trigger Notifications with Cubby.js

Source: https://context7.com/monadoid/cubby/llms.txt

This example demonstrates how to trigger various types of notifications using the Cubby API, including simple, timed, persistent, and remote device notifications. It utilizes the '@cubby/js' library and requires a Cubby instance.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

// Simple notification
await client.notify({
  title: 'Task Complete',
  body: 'Your build finished successfully',
});

// Notification with timeout
await client.notify({
  title: 'Reminder',
  body: 'Meeting starts in 5 minutes',
  timeout: 5000, // milliseconds
});

// Persistent notification
await client.notify({
  title: 'Critical Alert',
  body: 'Server response time exceeded threshold',
  persistent: true,
});

// Remote device notification
const { devices } = await client.listDevices();
client.setDeviceId(devices[0].id);
await client.notify({
  title: 'Remote Alert',
  body: 'Task completed on remote device',
});
```

--------------------------------

### Search Historical Data with TypeScript SDK

Source: https://context7.com/monadoid/cubby/llms.txt

Shows how to search historical data using the Cubby TypeScript SDK. It covers searching across all content types with query parameters like 'q', 'content_type', 'limit', and 'offset'. The example also details how to process and interpret the search results based on the 'type' of data (OCR, Audio, UI).

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

// Search across all content types
const results = await client.search({
  q: 'project deadline meeting notes',
  content_type: 'all', // 'ocr' | 'audio' | 'ui' | 'all'
  limit: 50,
  offset: 0,
});

console.log(`Found ${results.pagination.total} results`);

// Process search results
for (const item of results.data) {
  console.log(`[${item.type}] ${item.content.timestamp}`);

  if (item.type === "OCR") {
    console.log(`App: ${item.content.appName}, Text: ${item.content.text}`);
    console.log(`Window: ${item.content.windowName}`);
    if (item.content.browserUrl) {
      console.log(`URL: ${item.content.browserUrl}`);
    }
  } else if (item.type === "Audio") {
    console.log(`Transcription: ${item.content.transcription}`);
    if (item.content.speaker) {
      console.log(`Speaker: ${item.content.speaker.name || 'Unknown'} (ID: ${item.content.speaker.id})`);
    }
  } else if (item.type === "UI") {
    console.log(`UI Element: ${item.content.text}`);
    console.log(`App: ${item.content.appName}, Window: ${item.content.windowName}`);
  }
}
```

--------------------------------

### Real-time Transcription with Cubby JS SDK

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

This example demonstrates how to use the Cubby JS SDK to stream real-time transcriptions. It involves creating a client and then using the streamTranscriptions method, which returns an async iterator. The code continuously logs incoming transcription text. This functionality is useful for live audio processing.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

console.log('listening for transcriptions...');

for await (const event of client.streamTranscriptions()) {
  console.log(event.text);
}
```

--------------------------------

### Run Development Instance

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Launches a development instance of the Cubby application on a specified port and data directory, avoiding conflicts with existing processes. Useful for running alongside a production instance or after database migrations.

```bash
./target/release/cubby --port 3035 --data-dir /tmp/sp
```

--------------------------------

### Client Creation Options

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

Demonstrates the parameters available when creating a Cubby client, including base URL and optional credentials. Credentials can be automatically inferred from environment variables.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({
  baseUrl: 'http://localhost:3030', // or 'https://api.cubby.sh'
  clientId: 'your-client-id', // optional, auto-detected from CUBBY_CLIENT_ID env var
  clientSecret: 'your-client-secret', // optional, auto-detected from CUBBY_CLIENT_SECRET env var
});
```

--------------------------------

### MCP Tools: Device and Gateway Operations (TypeScript)

Source: https://context7.com/monadoid/cubby/llms.txt

Demonstrates how to use MCP tools to interact with devices. Includes gateway tools that do not require device selection and device-specific tools that require a device to be set first. These tools allow for listing devices, setting the active device, searching content, and performing actions like opening applications or sending notifications.

```typescript
// Gateway tools (no device selection required)
await mcp.tools.call("devices/list", {});
await mcp.tools.call("devices/set", { device_id: "device-123" });

// Device tools (require device selection via devices/set)
await mcp.tools.call("device/search", {
  q: "project deadline",
  content_type: "all",
  limit: 10
});

await mcp.tools.call("device/search-keyword", {
  keyword: "meeting",
  limit: 5
});

await mcp.tools.call("device/speakers/search", {
  name: "john"
});

await mcp.tools.call("device/speakers/similar", {
  speaker_id: 1
});

await mcp.tools.call("device/speakers/unnamed", {});

await mcp.tools.call("device/audio/list", {});

await mcp.tools.call("device/vision/list", {});

await mcp.tools.call("device/frames/get", {
  frame_id: 12345
});

await mcp.tools.call("device/tags/get", {
  content_type: "ocr"
});

await mcp.tools.call("device/embeddings", {
  text: "sample text"
});

await mcp.tools.call("device/add", {
  content: "custom data",
  timestamp: new Date().toISOString()
});

await mcp.tools.call("device/open-application", {
  app_name: "Slack"
});

await mcp.tools.call("device/open-url", {
  url: "https://github.com"
});

await mcp.tools.call("device/notify", {
  title: "Alert",
  body: "Message from MCP"
});
```

--------------------------------

### MCP Server - Open Application

Source: https://github.com/monadoid/cubby/blob/main/README.md

Launches a specified application on the active Cubby device.

```APIDOC
## POST /mcp/device/open-application

### Description
Opens a specified application on the target Cubby device.

### Method
POST

### Endpoint
`/mcp/device/open-application`

### Parameters
None

### Request Body
- **applicationName** (string) - Required - The name or path of the application to launch.

### Request Example
```json
{
  "applicationName": "/Applications/Calculator.app"
}
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation that the application launch command was sent.

#### Response Example
```json
{
  "message": "Application launch command sent for /Applications/Calculator.app."
}
```
```

--------------------------------

### Build Release with Features

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Builds the project in release mode, enabling specific features like 'metal' or 'cuda' for hardware acceleration. This command is used for creating production-ready binaries.

```bash
cargo build --release --features metal # or cuda, depending on your computer's NPU
```

--------------------------------

### Device and Media Endpoints

Source: https://github.com/monadoid/cubby/blob/main/README.md

Endpoints for listing available audio devices and monitors.

```APIDOC
## GET /audio/list

### Description
Lists all available audio input and output devices.

### Method
GET

### Endpoint
/audio/list

### Parameters
None

### Request Example
```json
{}
```

### Response
#### Success Response (200)
- **devices** (array) - A list of audio devices.

#### Response Example
```json
{
  "devices": [
    {
      "id": "audio-1",
      "name": "Microphone (Built-in)",
      "type": "input"
    },
    {
      "id": "audio-2",
      "name": "Speakers (Realtek)",
      "type": "output"
    }
  ]
}
```

## GET /vision/list

### Description
Lists all connected monitors or display devices.

### Method
GET

### Endpoint
/vision/list

### Parameters
None

### Request Example
```json
{}
```

### Response
#### Success Response (200)
- **monitors** (array) - A list of connected monitors.

#### Response Example
```json
{
  "monitors": [
    {
      "id": "monitor-1",
      "name": "Dell U2719D",
      "resolution": "2560x1440"
    }
  ]
}
```
```

--------------------------------

### Generate OpenAPI Specification

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Instructions for generating the OpenAPI (Swagger) specification file for the project. This involves running the application first to expose the API, then accessing the `/openapi.yaml` endpoint. A command to open the URL in the default browser is also provided. This file is used for documentation generation.

```bash
open http://localhost:3030/openapi.yaml
```

--------------------------------

### Create Cubby Client with TypeScript SDK

Source: https://context7.com/monadoid/cubby/llms.txt

Demonstrates creating a Cubby client instance using the TypeScript SDK. It shows how to configure the client for local usage (defaulting to localhost:3030) and remote usage with OAuth credentials, including device selection for remote operations.

```typescript
import { createClient } from '@cubby/js';

// Local usage (no auth required)
const localClient = createClient({ baseUrl: 'http://localhost:3030' });

// Remote usage (requires OAuth credentials)
const remoteClient = createClient({
  baseUrl: 'https://api.cubby.sh',
  clientId: process.env.CUBBY_CLIENT_ID,
  clientSecret: process.env.CUBBY_CLIENT_SECRET,
});

// List devices and select one for remote operations
const { devices } = await remoteClient.listDevices();
console.log(`Found ${devices.length} devices`);
remoteClient.setDeviceId(devices[0].id);
```

--------------------------------

### Cubby TypeScript SDK - Stream Transcriptions

Source: https://github.com/monadoid/cubby/blob/main/README.md

Stream live audio transcriptions as they are generated, allowing for real-time processing and actions.

```APIDOC
## GET /stream/transcriptions

### Description
Establishes a stream to receive real-time audio transcriptions as they are generated by Cubby.

### Method
GET

### Endpoint
`/stream/transcriptions`

### Parameters
None

### Request Example
(This is a streaming endpoint, no request body is sent)

### Response
#### Success Response (200)
- **event** (object) - Represents a transcription event.
  - **timestamp** (string) - The timestamp of the transcription.
  - **text** (string) - The transcribed text.
  - **speaker** (string) - The identified speaker (if available).

#### Response Example
(This is a streaming response, events are sent continuously)
```json
{
  "timestamp": "2023-10-27T11:00:00Z",
  "text": "Remember to pick up milk on the way home.",
  "speaker": "User"
}
```
```

--------------------------------

### Device Automation Commands

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

Shows how to automate device actions using the Cubby SDK. This includes opening applications, opening URLs in a browser (optionally specifying the browser), and sending desktop notifications.

```typescript
// open applications
await client.device.openApplication('Slack');
await client.device.openApplication('Visual Studio Code');

// open urls in browser
await client.device.openUrl('https://github.com');
await client.device.openUrl('https://google.com', 'Safari');

// send desktop notifications
await client.notify({
  title: 'reminder',
  body: 'meeting in 5 minutes'
});
```

--------------------------------

### Cubby TypeScript SDK: Search Functionality

Source: https://github.com/monadoid/cubby/blob/main/README.md

Demonstrates how to initialize the Cubby TypeScript client, set a device ID, and perform a search query for specific content. It requires Cubby client ID and secret for authentication, which can be auto-detected from environment variables.

```typescript
import { createClient } from '@cubby/js';

// credentials auto-detected from env, or pass explicitly:
const client = createClient({
  baseUrl: 'https://api.cubby.sh',
  clientId: process.env.CUBBY_CLIENT_ID,
  clientSecret: process.env.CUBBY_CLIENT_SECRET,
});

// list devices and select one (for remote usage)
const { devices } = await client.listDevices();
client.setDeviceId(devices[0].id);

// find that article about dolphins you read last week
const results = await client.search({
  q: 'find me that website about dolphins',
  content_type: 'ocr',
  limit: 5
});
```

--------------------------------

### REST API: List Devices (Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Shows how to list all available devices associated with the authenticated user. This endpoint requires a valid OAuth token in the Authorization header.

```bash
curl https://api.cubby.sh/devices \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```

```json
{
  "devices": [
    {
      "id": "device-abc123",
      "userId": "user-xyz789",
      "createdAt": "2024-01-15T10:30:00Z",
      "updatedAt": "2024-01-15T10:30:00Z"
    }
  ]
}
```

--------------------------------

### REST API: Open Application on Device (Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Shows how to remotely open an application on a specified device. This requires the device ID, an OAuth token, and a JSON payload with the `app_name` parameter.

```bash
curl -X POST https://api.cubby.sh/devices/device-abc123/open-application \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "app_name": "Slack" }'
```

--------------------------------

### Create Cubby Client (Local Usage)

Source: https://github.com/monadoid/cubby/blob/main/cubby-js/README.md

Create a Cubby client instance for local development, specifying the base URL. This client can then be used to search your local data.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

// search your screen and audio history
const results = await client.search({
  q: 'project deadline',
  limit: 10
});

console.log(`found ${results.pagination.total} results`);
```

--------------------------------

### Cubby TypeScript SDK - Stream Vision

Source: https://github.com/monadoid/cubby/blob/main/README.md

Stream live screen capture data (OCR and screenshots) as it is processed, enabling real-time analysis of visual information.

```APIDOC
## GET /stream/vision

### Description
Establishes a stream to receive real-time screen capture data, including OCR results and screenshot information.

### Method
GET

### Endpoint
`/stream/vision`

### Parameters
None

### Request Example
(This is a streaming endpoint, no request body is sent)

### Response
#### Success Response (200)
- **event** (object) - Represents a vision event.
  - **timestamp** (string) - The timestamp of the screen capture.
  - **data** (object) - Contains details about the screen capture.
    - **app_name** (string) - The name of the active application.
    - **window_title** (string) - The title of the active window.
    - **text** (string) - Extracted OCR text from the screen.
    - **image_url** (string) - URL to a screenshot of the current screen (if available).

#### Response Example
(This is a streaming response, events are sent continuously)
```json
{
  "timestamp": "2023-10-27T11:05:00Z",
  "data": {
    "app_name": "Slack",
    "window_title": "#general - my-team",
    "text": "Project Alpha is on track.",
    "image_url": "https://cubby.sh/screenshots/abc123xyz.png"
  }
}
```
```

--------------------------------

### Control Device Applications with Cubby.js

Source: https://context7.com/monadoid/cubby/llms.txt

This snippet illustrates how to control applications on a device using the Cubby API. It covers opening applications by name, opening multiple applications, opening URLs in the default browser, and opening URLs in a specific browser. The '@cubby/js' library is required.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({ baseUrl: 'http://localhost:3030' });

// Open application by name
const slackOpened = await client.device.openApplication('Slack');
if (slackOpened) {
  console.log('Slack launched successfully');
}

// Open multiple applications
await client.device.openApplication('Visual Studio Code');
await client.device.openApplication('Google Chrome');

// Open URL in default browser
await client.device.openUrl('https://github.com');

// Open URL in specific browser
await client.device.openUrl('https://google.com', 'Safari');
await client.device.openUrl('https://stackoverflow.com', 'Firefox');
```

--------------------------------

### Run Cargo Benchmarks

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Executes the benchmark suite for the project using Cargo. This is useful for performance testing and identifying regressions. No external dependencies are required beyond the Rust toolchain.

```bash
cargo bench
```

--------------------------------

### Configure remote MCP Server with OAuth (JSON, Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Configuration for connecting to a remote Model Context Protocol (MCP) server using OAuth authentication. Includes a bash script to obtain the access token.

```bash
# Get OAuth credentials at https://cubby.sh/dashboard
# Exchange for access token
curl -X POST https://api.cubby.sh/oauth/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=YOUR_ID&client_secret=YOUR_SECRET&scope=read:cubby"
```

```json
{
  "mcpServers": {
    "cubby-remote": {
      "url": "https://api.cubby.sh/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_ACCESS_TOKEN"
      }
    }
  }
}
```

--------------------------------

### Open Application on Device

Source: https://context7.com/monadoid/cubby/llms.txt

Request to open a specified application on a target device.

```APIDOC
## POST /devices/{deviceId}/open-application

### Description
Request to open a specified application on a target device.

### Method
POST

### Endpoint
/devices/{deviceId}/open-application

### Parameters
#### Path Parameters
- **deviceId** (string) - Required - The ID of the device to open the application on

#### Headers
- **Authorization** (string) - Required - Bearer token for authentication (e.g., 'Bearer YOUR_ACCESS_TOKEN')
- **Content-Type** (string) - Required - Set to 'application/json'

#### Request Body
- **app_name** (string) - Required - The name of the application to open

### Request Example
```bash
curl -X POST https://api.cubby.sh/devices/device-abc123/open-application \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "app_name": "Slack" }'
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the request to open the application was successful

#### Response Example
```json
{
  "success": true
}
```
```

--------------------------------

### Validate and Fix OpenAPI File with Mintlify

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Provides commands and prompts for using Mintlify's scraping tool to validate and potentially fix an OpenAPI YAML file. This is crucial for ensuring the documentation generated by Mintlify is accurate. The process involves running a scraping command and iterating based on feedback or agent-assisted prompts.

```bash
npx @mintlify/scraping@latest openapi-file content/docs-mintlify-mig-tmp/openapi.yaml -o /tmp
```

--------------------------------

### REST API: Open URL on Device (Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Demonstrates how to open a URL in a web browser on a specific device. This POST request requires the device ID, an OAuth token, and a JSON payload containing the URL and an optional browser name.

```bash
curl -X POST https://api.cubby.sh/devices/device-abc123/open-url \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://github.com", "browser": "Chrome" }'
```

--------------------------------

### Run Tests

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Executes all tests for the project to ensure code integrity before submitting a pull request. This is a standard command for verifying code functionality.

```bash
cargo test
```

--------------------------------

### MCP Server - List Devices

Source: https://github.com/monadoid/cubby/blob/main/README.md

Retrieves a list of all devices enrolled with Cubby and accessible via the MCP server.

```APIDOC
## GET /mcp/devices/list

### Description
Lists all Cubby-enrolled devices available through the MCP server.

### Method
GET

### Endpoint
`/mcp/devices/list`

### Parameters
None

### Request Example
```bash
curl -X GET "http://localhost:3030/mcp/devices/list"
```

### Response
#### Success Response (200)
- **devices** (array) - A list of device objects.
  - **id** (string) - The unique identifier for the device.
  - **name** (string) - The human-readable name of the device.

#### Response Example
```json
{
  "devices": [
    {
      "id": "device-123abc",
      "name": "My MacBook Pro"
    }
  ]
}
```
```

--------------------------------

### List Devices

Source: https://context7.com/monadoid/cubby/llms.txt

Retrieve a list of all devices associated with your account.

```APIDOC
## GET /devices

### Description
Retrieve a list of all devices associated with your account.

### Method
GET

### Endpoint
/devices

### Parameters
#### Headers
- **Authorization** (string) - Required - Bearer token for authentication (e.g., 'Bearer YOUR_ACCESS_TOKEN')

### Request Example
```bash
curl https://api.cubby.sh/devices \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```

### Response
#### Success Response (200)
- **devices** (array) - An array of device objects
  - **id** (string) - Unique identifier for the device
  - **userId** (string) - Identifier for the user
  - **createdAt** (string) - Timestamp of device creation
  - **updatedAt** (string) - Timestamp of last device update

#### Response Example
```json
{
  "devices": [
    {
      "id": "device-abc123",
      "userId": "user-xyz789",
      "createdAt": "2024-01-15T10:30:00Z",
      "updatedAt": "2024-01-15T10:30:00Z"
    }
  ]
}
```
```

--------------------------------

### Debug GitHub Actions with tmate

Source: https://github.com/monadoid/cubby/blob/main/CONTRIBUTING.md

Sets up a tmate session within a GitHub Actions workflow for debugging, specifically for Windows-latest platform matrix. This allows interactive SSH access to the runner.

```yaml
- name: Setup tmate session # HACK
  if: matrix.platform == 'windows-latest'
  uses: mxschmitt/action-tmate@v3
```

--------------------------------

### Cubby TypeScript SDK - Search

Source: https://github.com/monadoid/cubby/blob/main/README.md

Search through your recorded content (OCR and audio transcriptions) using natural language queries.

```APIDOC
## POST /api/search

### Description
Searches through recorded screen content (OCR) and audio transcriptions stored locally by Cubby.

### Method
POST

### Endpoint
`/api/search`

### Parameters
#### Query Parameters
- **q** (string) - Required - The search query string.
- **content_type** (string) - Optional - Specifies the type of content to search ('ocr', 'transcription'). Defaults to both.
- **limit** (integer) - Optional - The maximum number of results to return.

### Request Body
```json
{
  "q": "find me that website about dolphins",
  "content_type": "ocr",
  "limit": 5
}
```

### Response
#### Success Response (200)
- **results** (array) - An array of search result objects.
  - **timestamp** (string) - The timestamp of the recorded event.
  - **content_type** (string) - The type of content ('ocr' or 'transcription').
  - **content** (string) - The extracted text content.
  - **metadata** (object) - Additional metadata about the event (e.g., application name, file path).

#### Response Example
```json
{
  "results": [
    {
      "timestamp": "2023-10-27T10:00:00Z",
      "content_type": "ocr",
      "content": "The website discusses dolphin behavior.",
      "metadata": {
        "app_name": "Chrome",
        "window_title": "Dolphins - Wikipedia"
      }
    }
  ]
}
```
```

--------------------------------

### REST API: Stream Events via WebSocket (Bash)

Source: https://context7.com/monadoid/cubby/llms.txt

Illustrates how to establish a WebSocket connection to receive real-time events from a specific device. The connection URL includes the device ID and can optionally include parameters like `include_images`. Requires an OAuth token.

```bash
# WebSocket connection for real-time events
wscat -c "wss://api.cubby.sh/devices/device-abc123/ws/events?include_images=false" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```

```json
{"name":"transcription","data":{"text":"hello world","timestamp":"2024-01-15T14:30:00Z","is_final":true}}
{"name":"ocr_result","data":{"app_name":"Slack","text":"new message","timestamp":"2024-01-15T14:30:01Z"}}
{"name":"ui_frame","data":{"app_name":"Chrome","text":"button clicked","timestamp":"2024-01-15T14:30:02Z"}}
```

--------------------------------

### MCP Server - Search Content

Source: https://github.com/monadoid/cubby/blob/main/README.md

Searches for content across screen OCR and audio transcriptions on the currently selected device.

```APIDOC
## POST /mcp/device/search

### Description
Performs a search query against the content (OCR and transcriptions) of the active Cubby device.

### Method
POST

### Endpoint
`/mcp/device/search`

### Parameters
None

### Request Body
- **q** (string) - Required - The search query.
- **content_type** (string) - Optional - Filter search by type ('ocr', 'transcription').
- **limit** (integer) - Optional - Maximum number of results.

### Request Example
```json
{
  "q": "meeting notes project alpha",
  "content_type": "ocr",
  "limit": 10
}
```

### Response
#### Success Response (200)
- **results** (array) - A list of matching content entries.
  - **timestamp** (string) - Timestamp of the content.
  - **content_type** (string) - Type of content ('ocr' or 'transcription').
  - **content** (string) - The actual content found.

#### Response Example
```json
{
  "results": [
    {
      "timestamp": "2023-10-27T12:00:00Z",
      "content_type": "ocr",
      "content": "Project Alpha meeting: Discussed Q4 roadmap."
    }
  ]
}
```
```

--------------------------------

### Manage devices using Cubby API (TS)

Source: https://context7.com/monadoid/cubby/llms.txt

This snippet demonstrates how to list, select, and clear devices using the Cubby API. It also shows how to override the device for a single operation.

```typescript
import { createClient } from '@cubby/js';

const client = createClient({
  baseUrl: 'https://api.cubby.sh',
  clientId: process.env.CUBBY_CLIENT_ID,
  clientSecret: process.env.CUBBY_CLIENT_SECRET,
});

// List all enrolled devices
const { devices } = await client.listDevices();
console.log(`Found ${devices.length} devices:`);
devices.forEach(device => {
  console.log(`- ${device.id} (created: ${device.createdAt})`);
});

// Select device for operations
client.setDeviceId(devices[0].id);

// All subsequent operations use the selected device
const results = await client.search({ q: 'hello', limit: 5 });
await client.notify({ title: 'test', body: 'works!' });

// Switch to another device
if (devices.length > 1) {
  client.setDeviceId(devices[1].id);
}

// Clear device selection
client.clearDeviceId();

// Override device for single operation
await client.search({ q: 'hello', deviceId: devices[0].id });
```