### Project Setup and Development Commands

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

This section provides essential bash commands for setting up and running the Next.js project. It covers dependency installation, database migrations, asset import, development server startup, local workflow inspection, and Remotion application development and deployment.

```bash
# Install dependencies
npm install

# Run database migrations (requires DATABASE_URL)
npm run db:migrate

# Import Mux assets and generate embeddings
npm run import-mux-assets
npm run import-mux-assets -- --language en  # Specific caption language

# Start development server
npm run dev

# Inspect workflow runs locally
npx workflow web

# Remotion development
npm run remotion:studio                              # Open Remotion Studio
npm run remotion:render:local default-composition    # Test local render
npm run remotion:render:local default-composition out/video.mp4

# Production: Deploy Remotion to AWS Lambda
npm run remotion:deploy

# Database utilities
npm run db:generate   # Generate migrations from schema changes
npm run db:studio     # Open Drizzle Studio for inspection

```

--------------------------------

### Install and Run Development Server

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Installs project dependencies and starts the development server for local testing and development.

```bash
npm install
npm run dev
```

--------------------------------

### Mux Client Module Setup (TypeScript)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Sets up a centralized Mux client module (`app/lib/mux.ts`). This module provides essential read-only helper functions for interacting with Mux assets, including retrieving assets, extracting playback IDs, and managing audio/text tracks.

```typescript
```typescript
// Add `app/lib/mux.ts` wrapper that exports the minimal read helpers we need (assets list/retrieve, playback ID extraction, audio track helpers)
// Add text track helpers (`getReadyTextTracks`, `findTextTrack`, `getTranscript`, `getTrackVtt`)
```
```

--------------------------------

### Start Vercel Workflow from Route Handler (TypeScript)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Illustrates how to initiate a Vercel Workflow from a Next.js API route handler. The `start()` function from `workflow/api` is used to trigger the workflow asynchronously, allowing the route handler to return immediately. This pattern is suitable for background processing.

```typescript
import { NextResponse } from "next/server";
import { start } from "workflow/api";

// app/api/workflows/translate-captions/route.ts
import { translateCaptionsWorkflow } from "@/workflows/translate-captions";

export async function POST(request: Request) {
  const { assetId, targetLang } = await request.json();

  // Executes asynchronously — returns immediately
  await start(translateCaptionsWorkflow, [assetId, targetLang]);

  return NextResponse.json({ message: "Workflow started" });
}
```

--------------------------------

### Start Development Server (Bash)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

Command to run the Next.js development server locally. This command compiles and serves the application, allowing for real-time development and testing of features, including local workflow execution.

```bash
npm run dev
```

--------------------------------

### POST /api/lambda/render

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Starts a video rendering job using Remotion for social clip creation. This multi-step workflow composes AI-generated content with Remotion, supporting various aspect ratios and burnt-in captions.

```APIDOC
## POST /api/lambda/render

### Description
Initiates a video rendering process using Remotion for creating social media clips. This workflow supports multiple aspect ratios and can include burnt-in captions.

### Method
POST

### Endpoint
/api/lambda/render

### Parameters
#### Request Body
- **id** (string) - Required - Identifier for the rendering configuration (e.g., 'social-clip-portrait', 'social-clip-square', 'social-clip-landscape').
- **fileName** (string) - Required - The desired filename for the output video.
- **inputProps** (object) - Required - Properties for the video composition.
  - **audioUrl** (string) - Required - URL of the audio source.
  - **startTime** (number) - Required - Start time in seconds for the clip.
  - **endTime** (number) - Required - End time in seconds for the clip.
  - **title** (string) - Optional - Title to be displayed in the video.
  - **captions** (array) - Optional - An array of caption objects to be burnt into the video.
    - **id** (string) - Unique identifier for the caption.
    - **startTime** (number) - Start time in seconds for the caption.
    - **endTime** (number) - End time in seconds for the caption.
    - **text** (string) - The caption text.

### Request Example
```json
{
  "id": "social-clip-portrait",
  "fileName": "my-clip.mp4",
  "inputProps": {
    "audioUrl": "https://stream.mux.com/playback_id/audio.m4a",
    "startTime": 30,
    "endTime": 45,
    "title": "Key Insights from the Talk",
    "captions": [
      { "id": "1", "startTime": 30, "endTime": 33, "text": "Welcome to the presentation" },
      { "id": "2", "startTime": 33, "endTime": 37, "text": "Today we'll discuss..." },
      { "id": "3", "startTime": 37, "endTime": 45, "text": "Building scalable video pipelines" }
    ]
  }
}
```

### Response
#### Success Response (200)
- **type** (string) - Indicates the status of the operation (e.g., 'success').
- **data** (object) - Contains the rendering details.
  - **renderId** (string) - The ID of the initiated render job.
  - **bucketName** (string) - The name of the bucket where the render will be stored.

#### Response Example
```json
{
  "type": "success",
  "data": {
    "renderId": "render_123",
    "bucketName": "remotion-bucket"
  }
}
```

### Polling Render Progress
To monitor the progress of a render job, send a POST request to the `/api/lambda/progress` endpoint.

#### Method
POST

#### Endpoint
/api/lambda/progress

#### Parameters
#### Request Body
- **id** (string) - Required - The render ID obtained from the `/api/lambda/render` response.
- **bucketName** (string) - Required - The bucket name obtained from the `/api/lambda/render` response.

#### Response Example (Progress)
```json
{
  "type": "success",
  "data": {
    "type": "progress",
    "progress": 0.65
  }
}
```

#### Response Example (Complete)
```json
{
  "type": "success",
  "data": {
    "type": "done",
    "url": "https://...",
    "size": 5242880
  }
}
```

#### Response Example (Error)
```json
{
  "type": "success",
  "data": {
    "type": "error",
    "message": "Render failed"
  }
}
```

### Aspect Ratio Configurations
The `id` parameter in the `/api/lambda/render` request body corresponds to predefined aspect ratios:
- **portrait**: { id: "social-clip-portrait", width: 1080, height: 1920 } (9:16)
- **square**: { id: "social-clip-square", width: 1080, height: 1080 } (1:1)
- **landscape**: { id: "social-clip-landscape", width: 1920, height: 1080 } (16:9)
```

--------------------------------

### Start Audio Dubbing Workflow with Mux AI and ElevenLabs

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Initiates a durable workflow for audio dubbing using ElevenLabs. This workflow requires an ELEVENLABS_API_KEY to be configured. It takes an asset ID and target language, returning a run ID to poll for completion. Upon success, it provides an audio track ID.

```typescript
// POST /api/workflows/translate-audio
// Requires ELEVENLABS_API_KEY
const response = await fetch("/api/workflows/translate-audio", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    assetId: "asset_abc123",
    targetLang: "fr"  // French
  })
});

const { runId, status, message } = await response.json();
// { runId: "run_audio_456", status: "running", message: "Audio translation workflow started" }

// Poll for completion
const statusResponse = await fetch(`/api/workflows/translate-audio?runId=${runId}`);
const statusResult = await statusResponse.json();
// { 
//   runId: "run_audio_456",
//   status: "completed",
//   success: true,
//   completedSteps: ["prepare", "dub", "finalize"],
//   result: { audioTrackId: "track_french_audio_789" }
// }

// Error response when ElevenLabs not configured:
// { error: "ElevenLabs env key required", status: 501 }
```

--------------------------------

### ESLint Import Ordering Example

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

Demonstrates the required import ordering for the project, categorized into side-effect styles, built-in modules, external packages, internal modules, and parent/sibling/index imports. Blank lines are mandatory between these groups.

```typescript
// 1. Side-effect styles
import "./styles.css";

// 2. Built-in modules
import { Buffer } from "node:buffer";

// 5. Parent/sibling/index
import { env } from "@/lib/env";
// 3. External packages
import { z } from "zod";

// 4. Internal (@mux/ai is treated as internal)
import { getSummaryAndTags } from "@mux/ai/workflows";

```

--------------------------------

### API Route for Creating Multi-Step Clips (Next.js Server Action)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Initiates a multi-step clip creation workflow. This server action accepts parameters such as asset ID, start and end times, source language, target languages, and a preset. It starts the workflow and returns the workflow run ID.

```javascript
```javascript
// POST /api/clips/create
// body: { assetId, startTime, endTime, sourceLang, targetLangs, preset }
// starts multi-step clip creation workflow
// returns: { workflowRunId }
```
```

--------------------------------

### Client-Side Workflow State Management with localStorage

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Manages the state of in-flight video AI workflows using localStorage for a resumable user experience. Provides functions to start, track progress, update status (running, completed, failed), retrieve all in-flight workflows for an asset, and clear completed workflow states.

```typescript
// app/lib/workflow-state.ts
import {
  startWorkflow,
  getWorkflowProgress,
  markWorkflowRunning,
  markWorkflowCompleted,
  markWorkflowFailed,
  getAllInFlightWorkflows,
  clearWorkflowProgress
} from "@/app/lib/workflow-state";

// Start tracking a workflow
startWorkflow(
  "asset_abc123",
  "translateCaptions",  // WorkflowType
  "es",                 // targetLang (optional)
  "run_xyz789"          // workflowRunId from server
);
// Stores: localStorage["workflow:asset_abc123:translateCaptions:es"]

// Get current progress
const progress = getWorkflowProgress("asset_abc123", "translateCaptions", "es");
// progress: {
//   workflowRunId: "run_xyz789",
//   status: "running",  // "queued" | "running" | "completed" | "failed"
//   startedAt: "2024-01-15T10:30:00.000Z",
//   completedAt?: "2024-01-15T10:35:00.000Z",
//   error?: "Translation failed"
// }

// Update status
markWorkflowRunning("asset_abc123", "translateCaptions", "es");
markWorkflowCompleted("asset_abc123", "translateCaptions", "es");
markWorkflowFailed("asset_abc123", "translateCaptions", "es", "Network error");

// Get all in-flight workflows for an asset
const inFlight = getAllInFlightWorkflows("asset_abc123");
// [{ workflowType: "translateCaptions", targetLang: "es", progress: {...} }]

// Clear completed workflow
clearWorkflowProgress("asset_abc123", "translateCaptions", "es");

// WorkflowTypes available:
// "summarizeAndTag" | "translateCaptions" | "translateAudio" | "createClip" | "renderVideo" | "socialClip"

```

--------------------------------

### Start Workflow from Next.js Route Handler

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

This snippet shows how to initiate a workflow asynchronously from a Next.js API route handler. It uses the `start()` function from `workflow/api` to trigger a specified workflow with provided arguments. The workflow runs in the background, and the route handler returns immediately.

```typescript
import { NextResponse } from "next/server";
import { start } from "workflow/api";

// app/api/workflows/translate-captions/route.ts
import { translateCaptionsWorkflow } from "@/workflows/translate-captions";

export async function POST(request: Request) {
  const { assetId, targetLang } = await request.json();

  // Executes asynchronously and doesn't block your app
  await start(translateCaptionsWorkflow, [assetId, targetLang]);

  return NextResponse.json({ message: "Workflow started" });
}
```

--------------------------------

### Start Caption Translation Workflow with Mux AI

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Initiates a durable workflow to translate video captions. It takes an asset ID, source language, and target language as input. The workflow can be polled for completion status and returns a track ID upon success. Rate limiting headers are provided in the response.

```typescript
// POST /api/workflows/translate-captions
// Start caption translation workflow
const response = await fetch("/api/workflows/translate-captions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    assetId: "asset_abc123",
    sourceLang: "en",
    targetLang: "es"  // Spanish
  })
});

const { runId, status, message } = await response.json();
// { runId: "run_xyz789", status: "running", message: "Caption translation workflow started" }

// Poll for completion
const statusResponse = await fetch(`/api/workflows/translate-captions?runId=${runId}`);
const statusResult = await statusResponse.json();
// { 
//   runId: "run_xyz789",
//   status: "completed",       // "running" | "completed" | "failed" 
//   success: true,
//   completedSteps: ["prepare", "translate", "finalize"],
//   result: { trackId: "track_spanish_123" }
// }

// Rate limit headers included:
// X-RateLimit-Limit: 10
// X-RateLimit-Remaining: 9
// X-RateLimit-Reset: 2024-01-15T00:00:00.000Z
```

--------------------------------

### Database Schema Management with Drizzle ORM and pgvector

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Defines PostgreSQL tables for storing video metadata and embeddings using Drizzle ORM. Includes examples for inserting and querying video data, as well as updating video summaries and tags. Supports storing embeddings with a specified dimension count.

```typescript
// db/schema.ts
import { videos, videoChunks, rateLimits, featureMetrics } from "@/db/schema";
import { db } from "@/db";
import { eq } from "drizzle-orm";

// Videos table - stores Mux asset metadata
await db.insert(videos).values({
  muxAssetId: "asset_abc123",
  muxPlaybackId: "playback_xyz",
  title: "Building Video AI Pipelines",
  summary: "A comprehensive guide...",
  tags: ["video", "ai", "mux"],
  duration: 1234.5,
  aspectRatio: "16:9",
  transcriptVtt: "WEBVTT\n\n00:00:00.000 --> 00:00:05.000\nHello...",
  meta: { /* full Mux asset JSON */ }
});

// Video chunks with embeddings (1536 dimensions for OpenAI text-embedding-3-small)
await db.insert(videoChunks).values({
  videoId: "video_uuid",
  chunkIndex: 0,
  startTime: 0,
  endTime: 30,
  embedding: [0.123, -0.456, ...] // 1536 float values
});

// Query videos
const video = await db.select().from(videos).where(eq(videos.muxAssetId, "asset_abc123"));

// Update summary and tags
await db.update(videos)
  .set({ summary: "Updated summary", tags: ["new", "tags"], updatedAt: new Date() })
  .where(eq(videos.muxAssetId, "asset_abc123"));

```

--------------------------------

### Render Social Clips with Mux AI and Remotion

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Starts a video rendering workflow using Remotion for social clip creation. This workflow supports portrait, square, and landscape aspect ratios and can burn in captions. It requires an asset ID, desired aspect ratio, and input properties like audio URL and captions.

```typescript
// POST /api/lambda/render
// Start Remotion Lambda render
const response = await fetch("/api/lambda/render", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    id: "social-clip-portrait",  // "social-clip-portrait" | "social-clip-square" | "social-clip-landscape"
    fileName: "my-clip.mp4",
    inputProps: {
      audioUrl: "https://stream.mux.com/playback_id/audio.m4a",
      startTime: 30,
      endTime: 45,
      title: "Key Insights from the Talk",
      captions: [
        { id: "1", startTime: 30, endTime: 33, text: "Welcome to the presentation" },
        { id: "2", startTime: 33, endTime: 37, text: "Today we'll discuss..." },
        { id: "3", startTime: 37, endTime: 45, text: "Building scalable video pipelines" }
      ]
    }
  })
});

const { type, data } = await response.json();
// { type: "success", data: { renderId: "render_123", bucketName: "remotion-bucket" } }

// Poll render progress
const progressResponse = await fetch("/api/lambda/progress", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    id: "render_123",
    bucketName: "remotion-bucket"
  })
});

const progressData = await progressResponse.json();
// Progress: { type: "success", data: { type: "progress", progress: 0.65 } }
// Complete: { type: "success", data: { type: "done", url: "https://...", size: 5242880 } }
// Error: { type: "success", data: { type: "error", message: "Render failed" } }

// Aspect ratio configurations
const ASPECT_RATIOS = {
  portrait: { id: "social-clip-portrait", width: 1080, height: 1920 },  // 9:16
  square: { id: "social-clip-square", width: 1080, height: 1080 },      // 1:1
  landscape: { id: "social-clip-landscape", width: 1920, height: 1080 } // 16:9
};
```

--------------------------------

### Define and Start Durable Workflows in Next.js

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

This TypeScript code illustrates how to define and execute durable workflows using the 'workflow' library in a Next.js application. Workflows are defined with 'use workflow' and steps with 'use step' directives, enabling step-based retry and resume semantics. Workflows can be initiated from API routes and their status polled.

```typescript
// workflows/translate-captions.ts
import { translateCaptions } from "@mux/ai/workflows";
import { start } from "workflow/api";

// Define workflow with "use workflow" directive inside function body
export async function translateCaptionsWorkflow(
  assetId: string,
  sourceLang: string,
  targetLang: string
) {
  "use workflow";

  const result = await doTranslation(assetId, sourceLang, targetLang);
  await persistTrackId(assetId, targetLang, result.trackId);
  return result;
}

// Define steps with "use step" directive inside function body
async function doTranslation(assetId: string, sourceLang: string, targetLang: string) {
  "use step";
  return await translateCaptions(assetId, sourceLang, targetLang, {
    uploadToMux: true  // Attaches track directly to Mux asset
  });
}

async function persistTrackId(assetId: string, targetLang: string, trackId: string) {
  "use step";
  // Save to database...
}

// Start workflow from route handler (non-blocking)
import { start, getRun } from "workflow/api";

export async function POST(request: Request) {
  const { assetId, sourceLang, targetLang } = await request.json();

  const run = await start(translateCaptionsWorkflow, [assetId, sourceLang, targetLang]);
  // Returns immediately - workflow runs in background

  return NextResponse.json({ runId: run.runId, status: "running" });
}

// Poll for status
const run = getRun(runId);
const status = await run.status;  // "pending" | "running" | "completed" | "failed"
const result = await run.returnValue;  // Final result when completed

// Inspect workflows locally
// $ npx workflow web

```

--------------------------------

### API Route for Translating Audio (Next.js Server Action)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Starts an audio dubbing workflow. This server action requires the asset ID and the target language in the request body. It initiates the dubbing process and returns the corresponding workflow run ID.

```javascript
```javascript
// POST /api/workflows/translate-audio
// body: { assetId, targetLang }
// starts audio dubbing workflow
// returns: { workflowRunId }
```
```

--------------------------------

### Remotion Local Development Commands

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Commands for local development with Remotion, including launching the Studio for live preview and rendering videos locally for testing.

```bash
# Open the Remotion Studio for live preview and iteration
npm run remotion:studio

# Render a video locally (for testing)
# Pass the composition name as an argument
npm run remotion:render:local default-composition

# Optionally specify an output path
npm run remotion:render:local default-composition out/foo.mp4
```

--------------------------------

### Run Database Migrations

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Applies database migrations defined in `db/migrations/`. This process creates necessary tables, indexes, and enables the pgvector extension for semantic search capabilities.

```bash
npm run db:migrate
```

--------------------------------

### Media Detail Page Structure (Directory Structure)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Illustrates the co-located feature folder structure for the media detail page (`/media/[slug]`). Each folder contains relevant actions, UI components, and helper files for specific functionalities.

```treeview
app/media/[slug]/
├── media-content.tsx
├── page.tsx
├── localization/
│   ├── actions.ts      (captions & audio translation)
│   ├── constants.ts
│   └── ui.tsx
├── player/
│   ├── context.ts
│   ├── provider.tsx
│   ├── ui.tsx
│   └── use-player.ts
├── social-clips/
│   ├── actions.ts      (clip creation & Remotion Lambda rendering)
│   ├── constants.ts
│   ├── preview.tsx     (client-side Remotion Player preview)
│   └── ui.tsx
├── summarize-and-tag/
│   ├── actions.ts      (start/poll summary generation workflow)
│   └── ui.tsx
├── transcript/
│   ├── actions.ts      (semantic search within video transcript)
│   ├── helpers.ts
│   └── ui.tsx
└── workflows-panel/
    ├── helpers.ts
    └── ui.tsx          (includes StatusBadge, StepProgress, etc.)

```

--------------------------------

### Workflow Definitions

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

Examples of defining durable and composable workflows using the 'use workflow' and 'use step' directives.

```APIDOC
## Workflow Definitions

### Description
Defines reusable workflow logic. Workflows can be composed of multiple steps, and individual steps can be wrapped for retries and progress tracking.

### Method
N/A (Workflow definition)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A

#### translateCaptionsWorkflow Example
```typescript
export async function translateCaptionsWorkflow(assetId: string, targetLang: string) {
  "use workflow";
  const result = await doTranslation(assetId, targetLang);
  await persistTrackId(assetId, targetLang, result.trackId);
  return result;
}

async function doTranslation(assetId: string, targetLang: string) {
  "use step";
  return await translateCaptions(assetId, "en", targetLang, { uploadToMux: true });
}
```

#### createClipWorkflow Example
```typescript
export async function createClipWorkflow(input: ClipInput) {
  "use workflow";
  const captions = await translateAllCaptions(input.assetId, input.targetLangs);
  const audio = await dubAllAudio(input.assetId, input.targetLangs);
  const { videoBuffer, posterBuffer } = await renderClipStep(input, captions, audio);
  const { videoUrl, posterUrl } = await uploadArtifacts(videoBuffer, posterBuffer);
  await finalizeClip(input.clipId, videoUrl, posterUrl);
  return { videoUrl, posterUrl };
}
```
```

--------------------------------

### POST /api/workflows/translate-captions

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

This endpoint starts the 'translateCaptions' workflow asynchronously. It accepts an asset ID and a target language in the request body and returns a confirmation message.

```APIDOC
## POST /api/workflows/translate-captions

### Description
Starts the 'translateCaptions' workflow asynchronously. The workflow runs in the background and does not block the application.

### Method
POST

### Endpoint
/api/workflows/translate-captions

### Parameters
#### Query Parameters
None

#### Request Body
- **assetId** (string) - Required - The ID of the asset to translate captions for.
- **targetLang** (string) - Required - The target language for the captions.

### Request Example
```json
{
  "assetId": "your_asset_id",
  "targetLang": "es"
}
```

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating the workflow has started.

#### Response Example
```json
{
  "message": "Workflow started"
}
```
```

--------------------------------

### Database Migration Scripts (Bash)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Provides commands for managing database migrations using Drizzle ORM. Includes generating new migrations, applying them, and opening a local studio for inspection.

```bash
npm run db:generate

```

```bash
npm run db:migrate

```

```bash
npm run db:studio

```

--------------------------------

### Remotion Production Deployment

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Deploys the Remotion site to AWS Lambda for serverless rendering, suitable for production environments. This command is not intended for development.

```bash
# Deploy Remotion site to AWS Lambda for serverless rendering
npm run remotion:deploy
```

--------------------------------

### Mux Client Helpers for Asset Management

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Provides a singleton Mux client with typed helper functions for common Mux operations. These include retrieving and listing assets, extracting playback IDs, managing text tracks, fetching transcripts and VTT content, generating audio and instant clip URLs, and creating signed playback tokens. It simplifies interaction with the Mux API for video asset management.

```typescript
// app/lib/mux.ts
import {
  mux,
  getAsset,
  getPlaybackIdForAsset,
  listAssets,
  getReadyTextTracks,
  findTextTrack,
  getTranscript,
  getTrackVtt,
  getMuxAudioUrl,
  getMuxInstantClipUrl,
  generatePlaybackToken
} from "@/lib/mux";

// List all Mux assets with pagination
const assets = await listAssets({ limit: 10, page: 1 });

// Get a single asset by ID
const asset = await getAsset("asset_abc123");

// Get asset with playback ID in one call
const { asset, playbackId, policy } = await getPlaybackIdForAsset("asset_abc123");
// policy: "public" | "signed"

// Find text tracks for an asset
const textTracks = getReadyTextTracks(asset);
const englishTrack = findTextTrack(asset, "en");

// Fetch transcript content
const transcript = await getTranscript(playbackId, englishTrack.id);
const vttContent = await getTrackVtt(playbackId, englishTrack.id);

// Generate audio URL (supports signed playback)
const audioUrl = await getMuxAudioUrl(playbackId, policy, "audio.m4a");

// Generate instant clip URL for streaming segments
const clipUrl = await getMuxInstantClipUrl(playbackId, policy, 30, 45, "audio");

// Generate signed playback token (for signed assets)
const token = await generatePlaybackToken(playbackId, "video"); // "video" | "thumbnail" | "gif" | "storyboard"

```

--------------------------------

### Run Remotion Studio for Local Preview

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/AGENTS.md

This command initiates the Remotion Studio development server, allowing for live preview and iteration on video compositions directly in the browser. It's a free and unlimited phase for styling and tweaking.

```bash
npm run remotion:studio
```

--------------------------------

### Environment Variables for Mux, OpenAI, and Database

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Essential environment variables required for Mux authentication, OpenAI API access for embeddings, and database connection details for storing Mux catalog metadata.

```bash
# Mux credentials
MUX_TOKEN_ID=
MUX_TOKEN_SECRET=

# OpenAI (required for embeddings)
OPENAI_API_KEY=

# Database (PostgreSQL with pgvector) — required to store/search the Mux catalog metadata
DATABASE_URL=
```

--------------------------------

### Database Connection Configuration

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Configures the database connection string in a `.env.local` file, which is used by both Drizzle ORM and the data import script. Ensure your Postgres database supports pgvector.

```bash
# Database (PostgreSQL + pgvector)
DATABASE_URL="postgresql://USER:PASSWORD@HOST:5432/DB_NAME"

# Mux (used by the import script)
MUX_TOKEN_ID="..."
MUX_TOKEN_SECRET="..."

# Embeddings (used by the import script)
OPENAI_API_KEY="..."
```

--------------------------------

### API Route for Translating Captions (Next.js Server Action)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Initiates a caption translation workflow. This server action accepts the asset ID, source language, and target language as input in the request body. It starts the translation process and returns the ID of the workflow run.

```javascript
```javascript
// POST /api/workflows/translate-captions
// body: { assetId, sourceLang, targetLang }
// starts caption translation workflow
// returns: { workflowRunId }
```
```

--------------------------------

### Inspect Workflow Runs Locally

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Launches a local interface to inspect and manage Vercel workflow runs, useful for debugging and monitoring asynchronous processes.

```bash
npx workflow web
```

--------------------------------

### Import Mux Assets and Generate Embeddings (Bash)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/README.md

Imports Mux assets, generates embeddings, and upserts data into the database. Supports specifying a language for subtitle embedding, defaulting to 'en'.

```bash
npm run import-mux-assets

```

```bash
npm run import-mux-assets -- --language en

```

--------------------------------

### AI Video Summarization and Tagging Workflow

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Implements an AI-powered video summarization workflow using `@mux/ai/workflows`. This function directly generates a title, description, and tags from an asset by analyzing storyboard frames and its transcript. It supports configurable options for tone, transcript inclusion, and AI provider (Anthropic, OpenAI, Google), with API keys passed securely. The workflow progress can be tracked client-side.

```typescript
// workflows/get-summary-and-tags.ts
import { getSummaryAndTags } from "@mux/ai/workflows";
import { start } from "workflow/api";

// Start summary workflow (wrapped with progress tracking)
const run = await start(getSummaryAndTagsWorkflow, [
  assetId,
  {
    tone: "professional",        // "neutral" | "professional" | "playful"
    includeTranscript: true,
    cleanTranscript: true,
    provider: "anthropic",       // "openai" | "anthropic" | "google"
    anthropicApiKey: env.ANTHROPIC_API_KEY,
  }
]);

// Poll for results
const result = await run.returnValue;
// result: {
//   success: true,
//   currentStep: "finalize",
//   completedSteps: ["prepare", "generate", "finalize"],
//   result: {
//     title: "Building Scalable Video AI Pipelines",
//     description: "A deep dive into...",
//     tags: ["video", "ai", "workflows", "mux"],
//     storyboardUrl: "https://...",
//   }
// }

// Server action usage (app/media/[slug]/summarize-and-tag/actions.ts)
import { startSummaryWorkflowAction, pollSummaryWorkflowAction } from "./actions";

const { runId, status, error } = await startSummaryWorkflowAction(assetId, "professional");
const pollResult = await pollSummaryWorkflowAction(runId, 0);
// pollResult.status: "running" | "completed" | "failed"
// pollResult.result: SummaryResult when completed

```

--------------------------------

### Implement Durable AI Workflows with Vercel (TypeScript)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Shows how to implement durable AI workflows using Vercel Workflows for tasks like caption translation. Workflows are non-blocking, resumable, and observable, suitable for long-running operations. The example defines a workflow and its steps using the "use workflow" and "use step" directives.

```typescript
// workflows/translate-captions.ts
import { translateCaptions } from "@mux/ai/workflows";

export async function translateCaptionsWorkflow(assetId: string, targetLang: string) {
  "use workflow";

  const result = await doTranslation(assetId, targetLang);
  await persistTrackId(assetId, targetLang, result.trackId);

  return result;
}

async function doTranslation(assetId: string, targetLang: string) {
  "use step";
  return await translateCaptions(assetId, "en", targetLang, {
    uploadToMux: true,
  });
}

async function persistTrackId(assetId: string, targetLang: string, trackId: string) {
  "use step";
  await db.media.update({
    where: { muxAssetId: assetId },
    data: { [`captionTrack_${targetLang}`]: trackId },
  });
}
```

--------------------------------

### Manual Remotion Deployment Script

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/DOCS/AUTOMATED-REMOTION-DEPLOYMENTS.md

This command initiates a manual deployment of the Remotion site and Lambda function. It requires AWS credentials to be set as environment variables. Ensure these secrets are configured correctly before running.

```bash
npm run remotion:deploy
```

--------------------------------

### Create Video Clip Workflow with Mux AI and Remotion (TypeScript)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

This TypeScript workflow orchestrates the creation of a video clip by translating captions, dubbing audio, rendering the clip with Remotion, uploading artifacts, and finalizing the clip record. It demonstrates multi-step execution and parallel processing.

```typescript
// workflows/create-clip.ts
import { Buffer } from "node:buffer";

import { renderClip, uploadToStorage } from "@/lib/remotion";

import { translateAudio, translateCaptions } from "@mux/ai/workflows";

// The main workflow function orchestrates the steps
export async function createClipWorkflow(input: ClipInput) {
  "use workflow";

  const { assetId, startTime, endTime, targetLangs, preset } = input;

  // Orchestrate the steps
  const captionResults = await translateAllCaptions(assetId, targetLangs);
  const audioResults = await dubAllAudio(assetId, targetLangs);
  const { videoBuffer, posterBuffer } = await renderClipStep(assetId, startTime, endTime, preset, captionResults, audioResults);
  const { videoUrl, posterUrl } = await uploadArtifacts(videoBuffer, posterBuffer);
  await finalizeClip(input.clipId, videoUrl, posterUrl);

  return { videoUrl, posterUrl };
}

// Step 1: Translate captions for each target language
async function translateAllCaptions(assetId: string, targetLangs: string[]) {
  "use step";
  return await Promise.all(
    targetLangs.map(lang => translateCaptions(assetId, "en", lang, { uploadToMux: true }))
  );
}

// Step 2: Dub audio for each target language
async function dubAllAudio(assetId: string, targetLangs: string[]) {
  "use step";
  return await Promise.all(
    targetLangs.map(lang => translateAudio(assetId, lang, { uploadToMux: true }))
  );
}

// Step 3: Render the clip with Remotion
async function renderClipStep(
  assetId: string,
  startTime: number,
  endTime: number,
  preset: string,
  captionTracks: CaptionResult[],
  audioTracks: AudioResult[]
) {
  "use step";
  return await renderClip({ assetId, startTime, endTime, preset, captionTracks, audioTracks });
}

// Step 4: Upload artifacts to storage
async function uploadArtifacts(videoBuffer: Buffer, posterBuffer: Buffer) {
  "use step";
  return await uploadToStorage(videoBuffer, posterBuffer);
}

// Step 5: Finalize the clip record
async function finalizeClip(clipId: string, videoUrl: string, posterUrl: string) {
  "use step";
  await db.clip.update({
    where: { id: clipId },
    data: { status: "ready", renderedUrl: videoUrl, posterUrl },
  });
}

```

--------------------------------

### Initialize Mux Client in TypeScript

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Initializes a singleton Mux client using credentials from validated environment variables. It exports typed helpers for common Mux operations and defines necessary types for assets and playback. This centralizes credential management and error handling, promoting reuse across the application.

```typescript
import { Mux } from "@mux/mux-node";
import { env } from "@/lib/env";

const mux = new Mux(env.MUX_TOKEN_ID, env.MUX_TOKEN_SECRET);

export const listAssets = mux.video.assets.list;
export const getAsset = mux.video.assets.retrieve;
export const getPlaybackIdForAsset = (assetId: string) => mux.video.assets.retrieve(assetId).then(asset => asset.playback_ids?.[0]?.id);
export const getReadyAudioTracks = (assetId: string) => mux.video.assets.retrieve(assetId).then(asset => asset.audio_tracks?.find(track => track.status === "ready"));
export const findAudioTrack = (assetId: string, trackId: string) => mux.video.assets.retrieve(assetId).then(asset => asset.audio_tracks?.find(track => track.id === trackId));

export type MuxAsset = Awaited<ReturnType<typeof mux.video.assets.retrieve>>;
export type AssetTrack = MuxAsset["tracks"][0];
export type PlaybackPolicy = "public" | "private" | "signed";
export type PlaybackAsset = {
  playbackId: string;
  policy: PlaybackPolicy;
};

export default mux;

```

--------------------------------

### Implement IP-Based Rate Limiting in Next.js

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

This TypeScript code demonstrates how to set up IP-based rate limiting for API endpoints in a Next.js application. It utilizes helper functions to check limits, retrieve status, create error responses, and add rate limit headers. Rate limiting is bypassed in development environments.

```typescript
// app/lib/rate-limit.ts
import {
  checkRateLimit,
  getRateLimitStatus,
  createRateLimitError,
  addRateLimitHeaders,
  RATE_LIMITS
} from "@/app/lib/rate-limit";

// Rate limit configuration
const limits = RATE_LIMITS;
// {
//   "translate-audio": { maxRequests: 3, windowHours: 24 },
//   "translate-captions": { maxRequests: 10, windowHours: 24 },
//   "render": { maxRequests: 6, windowHours: 24 },
//   "summary": { maxRequests: 10, windowHours: 24 },
//   "search": { maxRequests: 50, windowHours: 1 }
// }

// Check rate limit (increments counter if allowed)
const clientIp = getClientIpFromRequest(request);
const result = await checkRateLimit(clientIp, "translate-captions");
// result: {
//   allowed: true,
//   remaining: 9,
//   resetAt: Date,
//   limit: 10
// }

if (!result.allowed) {
  const error = createRateLimitError(result);
  // { error: "Rate limit exceeded. Try again tomorrow morning.", remaining: 0, ... }
  return NextResponse.json(error, { status: 429 });
}

// Get status without incrementing
const status = await getRateLimitStatus(clientIp, "render");

// Add headers to response
addRateLimitHeaders(response.headers, result);
// X-RateLimit-Limit: 10
// X-RateLimit-Remaining: 9
// X-RateLimit-Reset: 2024-01-16T00:00:00.000Z

// Note: Rate limiting bypassed in development (NODE_ENV === "development")

```

--------------------------------

### Fetch Storyboard Metadata and VTT with Mux Node.js Client

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Retrieves storyboard metadata as JSON or VTT format using a playback ID. These functions are useful for generating preview images or for debugging the storyboard generation process. The returned data can be cached for performance.

```typescript
import mux from "@/lib/mux";

async function getStoryboardJson(playbackId: string) {
  try {
    const storyboardMeta = await mux.video.playback.storyboardMeta(playbackId, {
      // Optional parameters like 'quality', 'width', etc.
    });
    return storyboardMeta; // Returns storyboard JSON as a string
  } catch (error) {
    console.error(`Error retrieving storyboard metadata for playback ID ${playbackId}:`, error);
    throw error;
  }
}

async function getStoryboardVtt(playbackId: string) {
  try {
    const storyboardVtt = await mux.video.playback.storyboardVtt(playbackId, {
      // Optional parameters like 'quality', 'width', etc.
    });
    return storyboardVtt; // Returns storyboard VTT as a string
  } catch (error) {
    console.error(`Error retrieving storyboard VTT for playback ID ${playbackId}:`, error);
    throw error;
  }
}

```

--------------------------------

### Summarize Asset with getSummaryAndTags

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/application-explained.md

Generates a title, description, and tags for a Mux asset using its storyboard frames and optionally a transcript. Outputs include the generated metadata and a storyboard URL. This primitive is useful for quick content analysis and metadata enrichment.

```javascript
import { getSummaryAndTags } from "@mux/ai/workflows";

async function summarizeAsset(assetId, options) {
  const result = await getSummaryAndTags(assetId, options);
  console.log("Summary:", result.title, result.description);
  console.log("Tags:", result.tags);
  console.log("Storyboard URL:", result.storyboardUrl);
  return result;
}
```

--------------------------------

### Database Configuration with Postgres and pgvector (Drizzle ORM)

Source: https://github.com/muxinc/nextjs-video-ai-workflows/blob/main/context/implementation-explained.md

Configures the application to use a Postgres database with the pgvector extension for storing embeddings. It involves setting the `DATABASE_URL` environment variable and running Drizzle migrations to create `videos` and `video_chunks` tables for persisting asset metadata and embeddings.

```javascript
```javascript
// Configure `DATABASE_URL` (Postgres + pgvector)
// Run Drizzle migrations to create `videos` and `video_chunks`
// Store asset metadata + embeddings to enable fast search and reduce repeated Mux API calls
```
```

--------------------------------

### POST /api/workflows/translate-captions

Source: https://context7.com/muxinc/nextjs-video-ai-workflows/llms.txt

Initiates a durable workflow to translate video captions. This endpoint attaches translated tracks directly to the Mux asset and supports retry/resume semantics.

```APIDOC
## POST /api/workflows/translate-captions

### Description
Starts a durable workflow for translating video captions. The translated tracks are attached directly to the Mux asset, with retry and resume capabilities.

### Method
POST

### Endpoint
/api/workflows/translate-captions

### Parameters
#### Request Body
- **assetId** (string) - Required - The ID of the Mux asset to translate captions for.
- **sourceLang** (string) - Required - The source language code of the captions (e.g., 'en' for English).
- **targetLang** (string) - Required - The target language code for translation (e.g., 'es' for Spanish).

### Request Example
```json
{
  "assetId": "asset_abc123",
  "sourceLang": "en",
  "targetLang": "es"
}
```

### Response
#### Success Response (200)
- **runId** (string) - The ID of the workflow run.
- **status** (string) - The current status of the workflow (e.g., 'running').
- **message** (string) - A message indicating the status of the workflow initiation.

#### Response Example
```json
{
  "runId": "run_xyz789",
  "status": "running",
  "message": "Caption translation workflow started"
}
```

### Polling for Completion
To check the status of the translation workflow, poll the same endpoint with the `runId` as a query parameter:

#### Method
GET

#### Endpoint
/api/workflows/translate-captions?runId={runId}

#### Response Example (Completed)
```json
{
  "runId": "run_xyz789",
  "status": "completed",
  "success": true,
  "completedSteps": ["prepare", "translate", "finalize"],
  "result": { "trackId": "track_spanish_123" }
}
```

#### Response Example (Running)
```json
{
  "runId": "run_xyz789",
  "status": "running"
}
```

#### Rate Limit Headers
Responses include rate limit information:
- X-RateLimit-Limit
- X-RateLimit-Remaining
- X-RateLimit-Reset
```