### Start Interactive Setup Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md Run this command inside Claude Code to configure the plugin through an interactive wizard. ```bash /setup-video-vision ``` -------------------------------- ### Install Plugin Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md After adding the plugin, run this command to install it. ```bash /plugin install claude-video-vision ``` -------------------------------- ### Clone and Install Dependencies Source: https://github.com/jordanrendric/claude-video-vision/blob/main/CONTRIBUTING.md Clone the repository, navigate to the server directory, and install Node.js dependencies. ```bash git clone https://github.com/jordanrendric/claude-video-vision.git cd claude-video-vision/mcp-server npm install ``` -------------------------------- ### Local Development Setup Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md Clone the repository and link it to your Claude instance for local development. ```bash git clone https://github.com/jordanrendric/claude-video-vision.git claude --plugin-dir /path/to/claude-video-vision ``` -------------------------------- ### Install Whisper-cpp Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md On macOS, use Homebrew to install whisper-cpp for local audio processing. ```bash brew install whisper-cpp ``` -------------------------------- ### Install yt-dlp Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md On macOS, use Homebrew to install yt-dlp for downloading YouTube videos. ```bash brew install yt-dlp ``` -------------------------------- ### Install Plugin via Marketplace Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md Use this command within Claude Code to add the plugin from its GitHub repository. ```bash /plugin marketplace add https://github.com/jordanrendric/claude-video-vision ``` -------------------------------- ### Run in Watch Mode Source: https://github.com/jordanrendric/claude-video-vision/blob/main/CONTRIBUTING.md Start the project in watch mode for continuous development. ```bash npm run dev ``` -------------------------------- ### video_detail Example Workflow - Call 1 Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md First call in a video_detail workflow: extracts frames from a specified segment and returns a sample of 3 frames (start, middle, end). ```typescript video_detail({ segments: [{ start: "00:23:10", end: "00:23:18", fps: 3 }], view_sample: 3 }) ``` -------------------------------- ### Create Session Manifest Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Initializes a new session manifest object with basic video information and an empty resolutions map. Use this to start tracking frames for a video. ```typescript // mcp-server/tests/session/manifest.test.ts import { describe, it, expect } from "vitest"; import { createManifest, mergeFrames, getUncachedTimestamps, sampleFrameIndices, } from "../../src/session/manifest.js"; describe("manifest", () => { describe("createManifest", () => { it("creates a manifest with empty resolutions", () => { const m = createManifest("abc123", "/test.mp4"); expect(m.video_hash).toBe("abc123"); expect(m.resolutions).toEqual({}); expect(m.created_at).toBeDefined(); }); }); // ... other tests ... }); ``` ```typescript // mcp-server/src/session/manifest.ts import type { SessionManifest } from "../types.js"; type ManifestFrame = { timestamp: string; file: string }; export function createManifest(videoHash: string, videoPath: string): SessionManifest { return { video_hash: videoHash, video_path: videoPath, created_at: new Date().toISOString(), resolutions: {}, }; } // ... other functions ... ``` -------------------------------- ### video_detail Example Workflow - Call 2 Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Second call in a video_detail workflow: views specific cached frames without re-extraction. ```typescript video_detail({ view: ["00:23:15", "00:23:16"] }) ``` -------------------------------- ### video_detail Example Workflow - Call 3 Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Third call in a video_detail workflow: re-extracts frames with a new resolution and returns a single high-resolution frame. ```typescript video_detail({ view: ["00:23:15"], segments: [{ start: "00:23:14", end: "00:23:16", fps: 3, resolution: 1024 }] }) ``` -------------------------------- ### Segment Interface Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Defines a video segment with start and end times, frames per second (FPS), and an optional resolution. ```typescript interface Segment { start: string; end: string; fps: number; resolution?: number; } ``` -------------------------------- ### Get Session Directory Function Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Determines the session directory path based on the sessions root directory and the video path. It utilizes the computed video hash to create a unique directory for each session. ```typescript export function getSessionDir(sessionsRoot: string, videoPath: string): string { const hash = computeVideoHash(videoPath); return join(sessionsRoot, hash); } ``` -------------------------------- ### video_analyze Tool Filters Schema (TypeScript) Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Defines the schema for the video_analyze tool, specifying available ffmpeg filters and their boolean enable/disable options. This schema guides Claude in selecting appropriate filters based on user intent. ```typescript { path: z.string().describe("Absolute or relative path to the video file"), filters: z.object({ scene_changes: z.boolean().default(false), // scdet — where cuts happen black_intervals: z.boolean().default(false), // blackdetect — transitions, fades silence: z.boolean().default(false), // silencedetect — pauses, chapter breaks freeze: z.boolean().default(false), // freezedetect — still images, slides, paused content motion: z.boolean().default(false), // siti — spatial info (complexity) + temporal info (motion) blur: z.boolean().default(false), // blurdetect — sharpness score per frame exposure: z.boolean().default(false), // signalstats — brightness, contrast, saturation per frame loudness: z.boolean().default(false), // ebur128 — momentary loudness, speech vs music transcription: z.boolean().default(false), // audio transcription via configured backend }), } ``` -------------------------------- ### Parse Black Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Parses the stderr output from FFmpeg's `blackdetect` filter to extract the start time, end time, and duration of black intervals. Expects lines with `black_start`, `black_end`, and `black_duration`. ```typescript const stderr = "[blackdetect @ 0x1] black_start:23.5 black_end:25.0 black_duration:1.5\n"; const intervals = parseBlackdetectOutput(stderr); expect(intervals).toHaveLength(1); expect(intervals[0]).toEqual({ start: "00:00:23", end: "00:00:25", duration: 1.5 }); ``` -------------------------------- ### Segment-Based Frame Extraction Tests Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Unit tests for segment-based frame extraction, covering timestamp generation and frame extraction for specified segments. Includes setup for temporary output directories and cleanup. ```typescript import { describe, it, expect, afterEach } from "vitest"; import { extractFramesBySegments, generateTimestampsForSegment } from "../../src/extractors/frames.js"; import { join } from "path"; import { rmSync, existsSync } from "fs"; import { tmpdir } from "os"; import type { Segment } from "../../src/types.js"; const FIXTURE = join(import.meta.dirname, "../fixtures/test-3s.mp4"); const OUT_DIR = join(tmpdir(), "cvv-segments-test-" + Date.now()); describe("segment-based frame extraction", () => { afterEach(() => { if (existsSync(OUT_DIR)) rmSync(OUT_DIR, { recursive: true, force: true }); }); describe("generateTimestampsForSegment", () => { it("generates timestamps at the given fps within the range", () => { const timestamps = generateTimestampsForSegment( { start: "00:00:00", end: "00:00:04", fps: 1 }, ); expect(timestamps).toEqual(["00:00:00", "00:00:01", "00:00:02", "00:00:03"]); }); it("handles fractional fps", () => { const timestamps = generateTimestampsForSegment( { start: "00:00:00", end: "00:00:10", fps: 0.5 }, ); expect(timestamps).toEqual(["00:00:00", "00:00:02", "00:00:04", "00:00:06", "00:00:08"]); }); }); describe("extractFramesBySegments", () => { it("extracts frames for a single segment", async () => { const segments: Segment[] = [ { start: "00:00:00", end: "00:00:03", fps: 1, resolution: 256 }, ]; const result = await extractFramesBySegments(FIXTURE, segments, OUT_DIR); expect(result.length).toBeGreaterThanOrEqual(2); expect(result[0].timestamp).toBeDefined(); expect(result[0].image).toBeDefined(); expect(result[0].resolution).toBe(256); }); }); }); ``` -------------------------------- ### Build Project Source: https://github.com/jordanrendric/claude-video-vision/blob/main/CONTRIBUTING.md Compile the project using npm. ```bash npm run build ``` -------------------------------- ### Run Tests Source: https://github.com/jordanrendric/claude-video-vision/blob/main/CONTRIBUTING.md Execute the project's test suite. ```bash npm test ``` -------------------------------- ### Interval Interface Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Defines a time interval with start and end timestamps, and its calculated duration. ```typescript interface Interval { start: string; end: string; duration: number; } ``` -------------------------------- ### Get Uncached Timestamps Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Filters a list of wanted timestamps to return only those not present in the cached manifest for a given resolution. ```typescript export function getUncachedTimestamps( manifest: SessionManifest, resolution: string, wanted: string[], ): string[] { const cached = new Set( (manifest.resolutions[resolution]?.frames ?? []).map((f) => f.timestamp), ); return wanted.filter((ts) => !cached.has(ts)); } ``` -------------------------------- ### Build mcp-server Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Run this command to build the mcp-server project. Ensure there are no errors during the build process. ```bash cd mcp-server && npm run build ``` -------------------------------- ### Register video_detail Tool Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Import and register the `video_detail` tool with the server. This is required after implementing the tool's logic. ```typescript import { registerVideoDetail } from "./tools/video-detail.js"; // ... registerVideoDetail(server); ``` -------------------------------- ### Run Vitest Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Executes the session manifest tests using Vitest. ```bash cd mcp-server && npx vitest run tests/session/manifest.test.ts ``` -------------------------------- ### Get Uncached Timestamps Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Identifies which timestamps from a given list are not yet present in the manifest for a specific resolution. Returns an empty array if the resolution does not exist in the manifest. ```typescript // mcp-server/tests/session/manifest.test.ts describe("getUncachedTimestamps", () => { it("returns all timestamps when nothing is cached", () => { const m = createManifest("abc", "/test.mp4"); const wanted = ["00:00:02", "00:00:04", "00:00:06"]; expect(getUncachedTimestamps(m, "512", wanted)).toEqual(wanted); }); it("excludes already-cached timestamps", () => { const m = createManifest("abc", "/test.mp4"); m.resolutions["512"] = { frames: [{ timestamp: "00:00:02", file: "512/frame_00_00_02.jpg" }], }; const wanted = ["00:00:02", "00:00:04", "00:00:06"]; expect(getUncachedTimestamps(m, "512", wanted)).toEqual(["00:00:04", "00:00:06"]); }); it("returns all when resolution bucket does not exist", () => { const m = createManifest("abc", "/test.mp4"); m.resolutions["512"] = { frames: [{ timestamp: "00:00:02", file: "512/frame_00_00_02.jpg" }], }; expect(getUncachedTimestamps(m, "1024", ["00:00:02"])).toEqual(["00:00:02"]); }); }); ``` ```typescript // mcp-server/src/session/manifest.ts export function getUncachedTimestamps(manifest: SessionManifest, resolution: string, wantedTimestamps: string[]): string[] { const cachedTimestamps = new Set(manifest.resolutions[resolution]?.frames.map((f) => f.timestamp) ?? []); return wantedTimestamps.filter((ts) => !cachedTimestamps.has(ts)); } ``` -------------------------------- ### Run mcp-server Test Suite Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Execute the full test suite for the mcp-server project. All tests are expected to pass upon successful integration. ```bash cd mcp-server && npm test ``` -------------------------------- ### Parse FFmpeg Silence Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Extracts silence intervals from FFmpeg's stderr output. Each interval includes start time, end time, and duration. ```typescript export function parseSilenceOutput(stderr: string): Interval[] { const results: Interval[] = []; const startRegex = /silence_start:\s*([\d.]+)/g; const endRegex = /silence_end:\s*([\d.]+)\s*\|\s*silence_duration:\s*([\d.]+)/g; const starts: number[] = []; let match: RegExpExecArray | null; while ((match = startRegex.exec(stderr)) !== null) { starts.push(parseFloat(match[1])); } let i = 0; while ((match = endRegex.exec(stderr)) !== null) { const startSec = starts[i] ?? 0; results.push({ start: formatHMS(startSec), end: formatHMS(parseFloat(match[1])), duration: parseFloat(match[2]), }); i++; } return results; } ``` -------------------------------- ### Parse FFmpeg Black Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Extracts black interval data from FFmpeg's stderr output. Each interval includes start time, end time, and duration. ```typescript export function parseBlackdetectOutput(stderr: string): Interval[] { const results: Interval[] = []; const regex = /black_start:([\d.]+)\s+black_end:([\d.]+)\s+black_duration:([\d.]+)/g; let match: RegExpExecArray | null; while ((match = regex.exec(stderr)) !== null) { results.push({ start: formatHMS(parseFloat(match[1])), end: formatHMS(parseFloat(match[2])), duration: parseFloat(match[3]), }); } return results; } ``` -------------------------------- ### Watch Video via Slash Command Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md Use the /watch-video command with a local file path or YouTube URL. Optionally, provide a query for Claude to answer about the video. ```bash /watch-video path/to/video.mp4 /watch-video tutorial.mp4 "what language is used in this tutorial?" /watch-video https://www.youtube.com/watch?v=... "summarize this video" ``` -------------------------------- ### Combine Video and Audio Filters in FFmpeg Command Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Shows how `buildAnalysisCommand` can generate a single FFmpeg command that includes both video (`-vf`) and audio (`-af`) filters when multiple analysis types are requested. ```typescript const filters: AnalysisFilters = { scene_changes: true, black_intervals: false, silence: true, freeze: false, motion: false, blur: false, exposure: false, loudness: false, transcription: false, }; const cmd = buildAnalysisCommand("/test.mp4", filters, "/tmp/work"); expect(cmd!.args).toContain("-vf"); expect(cmd!.args).toContain("-af"); ``` -------------------------------- ### Parse FFmpeg Freeze Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Extracts freeze frame intervals from FFmpeg's stderr output. Each interval includes start time, end time, and duration. ```typescript export function parseFreezeOutput(stderr: string): Interval[] { const results: Interval[] = []; const startRegex = /freeze_start:\s*([\d.]+)/g; const endRegex = /freeze_end:\s*([\d.]+)/g; const durationRegex = /freeze_duration:\s*([\d.]+)/g; const starts: number[] = []; const ends: number[] = []; const durations: number[] = []; let match: RegExpExecArray | null; while ((match = startRegex.exec(stderr)) !== null) starts.push(parseFloat(match[1])); while ((match = endRegex.exec(stderr)) !== null) ends.push(parseFloat(match[1])); while ((match = durationRegex.exec(stderr)) !== null) durations.push(parseFloat(match[1])); for (let i = 0; i < starts.length; i++) { results.push({ start: formatHMS(starts[i]), end: formatHMS(ends[i] ?? starts[i] + (durations[i] ?? 0)), duration: durations[i] ?? 0, }); } return results; } ``` -------------------------------- ### Test Plugin Locally Source: https://github.com/jordanrendric/claude-video-vision/blob/main/CONTRIBUTING.md Run the claude CLI tool with a specified plugin directory to test the plugin locally. ```bash claude --plugin-dir /path/to/claude-video-vision ``` -------------------------------- ### Register video_analyze Tool Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Import and register the `video_analyze` tool with the server. This is a necessary step after creating the tool's implementation file. ```typescript import { registerVideoAnalyze } from "./tools/video-analyze.js"; // ... registerVideoAnalyze(server); ``` -------------------------------- ### Add New Schema Parameters for Video Configure Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Update the schema for video_configure to include new configuration options: enable_index, session_max_age_days, and clear_sessions. ```typescript enable_index: z.boolean().optional(), session_max_age_days: z.number().min(1).optional(), clear_sessions: z.boolean().optional(), ``` -------------------------------- ### Parse Silence Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Parses the stderr output from FFmpeg's `silencedetect` filter to extract silence intervals, including start time, end time, and duration. Expects lines indicating `silence_start` and `silence_end`. ```typescript const stderr = " [silencedetect @ 0x1] silence_start: 5.234 [silencedetect @ 0x1] silence_end: 8.567 | silence_duration: 3.333 "; const intervals = parseSilenceOutput(stderr); expect(intervals).toHaveLength(1); expect(intervals[0]).toEqual({ start: "00:00:05", end: "00:00:08", duration: 3.333 }); ``` -------------------------------- ### Commit video_detail Tool Changes Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Stage and commit the newly created `video_detail.ts` tool file and its registration in `index.ts`. ```bash git add mcp-server/src/tools/video-detail.ts mcp-server/src/index.ts git commit -m "feat(tools): add video_detail tool with drill-down and session caching" ``` -------------------------------- ### Verify TypeScript Files Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md List all TypeScript files within the mcp-server/src directory and sort them alphabetically. This helps verify that all new files have been correctly added. ```bash find mcp-server/src -name "*.ts" | sort ``` -------------------------------- ### Parse Freeze Frame Detection Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Parses the stderr output from FFmpeg's `freezedetect` filter to extract freeze frame information, including start time, end time, and duration. Expects lines with `freeze_start`, `freeze_end`, and `freeze_duration`. ```typescript const stderr = " [freezedetect @ 0x1] lavfi.freezedetect.freeze_start: 10.000 [freezedetect @ 0x1] lavfi.freezedetect.freeze_duration: 5.500 [freezedetect @ 0x1] lavfi.freezedetect.freeze_end: 15.500 "; const intervals = parseFreezeOutput(stderr); expect(intervals).toHaveLength(1); expect(intervals[0]).toEqual({ start: "00:00:10", end: "00:00:15", duration: 5.5 }); ``` -------------------------------- ### Add Schema Parameters for Segments and View Sample Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Extend the schema to include parameters for variable FPS/resolution segments and for selecting a number of evenly spaced sample frames. ```typescript segments: z.array(z.object({ start: z.string().regex(HMS_REGEX, "Must be HH:MM:SS format"), end: z.string().regex(HMS_REGEX, "Must be HH:MM:SS format"), fps: z.number().positive(), resolution: z.number().min(128).max(2048).optional(), })).optional().describe("Variable FPS/resolution segments — overrides global fps/start_time/end_time"), view_sample: z.number().min(1).optional().describe("Return only N evenly spaced frames"), ``` -------------------------------- ### Commit Changes for Startup Session Cleanup Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Commit the index.ts file with a message indicating the addition of expired session cleanup during server startup. ```bash git add mcp-server/src/index.ts git commit -m "feat(startup): add expired session cleanup on server boot" ``` -------------------------------- ### Create and Write Session Manifest Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Ensures the session directory exists and writes the session manifest to manifest.json. This is part of the session creation process. ```typescript if (!existsSync(sessionDir)) mkdirSync(sessionDir, { recursive: true }); writeFileSync(join(sessionDir, "manifest.json"), JSON.stringify(manifest, null, 2)); ``` -------------------------------- ### Commit Changes Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Add all changes to the Git repository and commit them with a descriptive message indicating the completion of the smart video analysis integration. ```bash git add -A git commit -m "chore: verify integration — smart video analysis v2 complete" ``` -------------------------------- ### Add Session Cleanup on Server Startup Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Implement logic in the server's index.ts to clean up expired sessions upon startup, conditional on the enable_index configuration. ```typescript const CONFIG_PATH = join(homedir(), ".claude-video-vision", "config.json"); const config = loadConfig(CONFIG_PATH); if (config.enable_index) { const sessionsDir = join(homedir(), ".claude-video-vision", "sessions"); cleanExpiredSessions(sessionsDir, config.session_max_age_days); } ``` -------------------------------- ### Handle No Filters Selected in FFmpeg Command Build Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Demonstrates that `buildAnalysisCommand` returns `null` when no analysis filters are selected. This prevents unnecessary FFmpeg process execution. ```typescript const filters: AnalysisFilters = { scene_changes: false, black_intervals: false, silence: false, freeze: false, motion: false, blur: false, exposure: false, loudness: false, transcription: false, }; const cmd = buildAnalysisCommand("/test.mp4", filters, "/tmp/work"); expect(cmd).toBeNull(); ``` -------------------------------- ### Build FFmpeg Command with Audio Filters (Silence and Loudness) Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Constructs an FFmpeg command to analyze audio for silence using `silencedetect` and loudness using `ebur128`. Ensure `silence` and `loudness` are enabled in `AnalysisFilters`. ```typescript const filters: AnalysisFilters = { scene_changes: false, black_intervals: false, silence: true, freeze: false, motion: false, blur: false, exposure: false, loudness: true, transcription: false, }; const cmd = buildAnalysisCommand("/test.mp4", filters, "/tmp/work"); const afIndex = cmd!.args.indexOf("-af"); expect(afIndex).toBeGreaterThan(-1); expect(cmd!.args[afIndex + 1]).toContain("silencedetect"); expect(cmd!.args[afIndex + 1]).toContain("ebur128"); ``` -------------------------------- ### Commit Changes for Video Perception Skill Rewrite Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Commit the updated SKILL.md file for the video-perception skill, reflecting the new analyze-first workflow and parameter changes. ```bash git add skills/video-perception/SKILL.md git commit -m "feat(skill): rewrite video-perception with analyze-first workflow" ``` -------------------------------- ### Load Manifest Function Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Loads a session manifest from a given session directory. Returns null if the manifest file does not exist. ```typescript export function loadManifest(sessionDir: string): SessionManifest | null { const manifestPath = join(sessionDir, "manifest.json"); if (!existsSync(manifestPath)) return null; return JSON.parse(readFileSync(manifestPath, "utf-8")); } ``` -------------------------------- ### Manifest JSON Structure for Video Analysis Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md The manifest file organizes frames by resolution, using timestamps as deduplication keys. New resolutions create new assets, but changing FPS for existing timestamps does not. ```json { "video_hash": "a1b2c3d4e5f6", "video_path": "/path/to/video.mp4", "created_at": "2026-04-25T16:00:00Z", "resolutions": { "512": { "frames": [ { "timestamp": "00:00:02", "file": "512/frame_00_00_02.jpg" }, { "timestamp": "00:00:04", "file": "512/frame_00_00_04.jpg" } ] }, "1024": { "frames": [ { "timestamp": "00:00:04", "file": "1024/frame_00_00_04.jpg" } ] } }, "analysis": null } ``` -------------------------------- ### FFmpeg Command for Video Analysis Filters Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Illustrates a single ffmpeg command that chains multiple analytical filters for video and audio processing. The server dynamically constructs this command based on selected filters. ```bash ffmpeg -i input.mp4 \ -vf "scdet=threshold=8,signalstats,siti,blurdetect,metadata=mode=print:file=video_meta.txt" \ -af "ebur128=metadata=1,silencedetect=noise=-30dB:d=0.5,ametadata=mode=print:file=audio_meta.txt" \ -f null - ``` -------------------------------- ### Commit video_analyze Tool Changes Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Stage and commit the newly created `video_analyze.ts` tool file and its registration in `index.ts`. ```bash git add mcp-server/src/tools/video-analyze.ts mcp-server/src/index.ts git commit -m "feat(tools): add video_analyze tool with ffmpeg filter orchestration" ``` -------------------------------- ### Video Session Persistence Directory Structure Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md When index is enabled, extracted data persists between tool calls in a structured directory. This structure organizes frames by resolution and includes a manifest file. ```text ~/.claude-video-vision/sessions/ {video-hash-12chars}/ manifest.json analysis.json 512/ frame_00_00_02.jpg frame_00_00_04.jpg 1024/ frame_00_00_04.jpg ``` -------------------------------- ### Build FFmpeg Analysis Command Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Constructs the FFmpeg command arguments and identifies metadata output files based on the provided analysis filters. Returns null if no filters are specified. ```typescript export function buildAnalysisCommand( videoPath: string, filters: AnalysisFilters, workDir: string, ): AnalysisCommand | null { const videoFilters: string[] = []; const audioFilters: string[] = []; let videoMetaFile: string | null = null; let audioMetaFile: string | null = null; if (filters.scene_changes) videoFilters.push("scdet=threshold=8"); if (filters.black_intervals) videoFilters.push("blackdetect=d=0.5:pix_th=0.10:pic_th=0.90"); if (filters.freeze) videoFilters.push("freezedetect=noise=0.003:duration=1"); if (filters.motion) videoFilters.push("siti=print_summary=1"); if (filters.blur) videoFilters.push("blurdetect"); if (filters.exposure) videoFilters.push("signalstats"); if (filters.silence) audioFilters.push("silencedetect=noise=-30dB:duration=0.5"); if (filters.loudness) audioFilters.push("ebur128=metadata=1:peak=true"); if (videoFilters.length === 0 && audioFilters.length === 0) return null; const args: string[] = ["-i", videoPath]; if (videoFilters.length > 0) { videoMetaFile = join(workDir, "video_meta.txt"); videoFilters.push(`metadata=mode=print:file=${videoMetaFile}`); args.push("-vf", videoFilters.join(",")); } if (audioFilters.length > 0) { audioMetaFile = join(workDir, "audio_meta.txt"); audioFilters.push(`ametadata=mode=print:file=${audioMetaFile}`); args.push("-af", audioFilters.join(",")); } args.push("-f", "null", "-"); return { args, videoMetaFile, audioMetaFile }; } ``` -------------------------------- ### Build FFmpeg Command with Scene Change Detection Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Constructs an FFmpeg command to analyze video for scene changes using the `scdet` filter. Ensure `scene_changes` is enabled in the `AnalysisFilters`. ```typescript const filters: AnalysisFilters = { scene_changes: true, black_intervals: false, silence: false, freeze: false, motion: false, blur: false, exposure: false, loudness: false, transcription: false, }; const cmd = buildAnalysisCommand("/test.mp4", filters, "/tmp/work"); expect(cmd).not.toBeNull(); expect(cmd!.args).toContain("-i"); expect(cmd!.args).toContain("/test.mp4"); const vfIndex = cmd!.args.indexOf("-vf"); expect(vfIndex).toBeGreaterThan(-1); expect(cmd!.args[vfIndex + 1]).toContain("scdet"); ``` -------------------------------- ### Add Tests for New Defaults Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md These tests verify the behavior of new configuration defaults for 'enable_index' and 'session_max_age_days'. They ensure the values are correctly loaded and preserved. ```typescript it("returns new defaults for enable_index and session_max_age_days", () => { const config = loadConfig(join(TEST_DIR, "config.json")); expect(config.enable_index).toBe(false); expect(config.session_max_age_days).toBe(7); }); it("preserves enable_index when set", () => { const configPath = join(TEST_DIR, "config.json"); writeFileSync(configPath, JSON.stringify({ enable_index: true })); const loaded = loadConfig(configPath); expect(loaded.enable_index).toBe(true); expect(loaded.session_max_age_days).toBe(7); }); ``` -------------------------------- ### video_watch New Parameters Schema Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md TypeScript schema for the video_watch tool, including new parameters for 'segments' and 'view_sample' to enable variable FPS extraction and frame sampling. ```typescript { // ... all existing params unchanged ... segments: z.array(z.object({ start: z.string().regex(/^\d{2}:\d{2}:\d{2}$/), end: z.string().regex(/^\d{2}:\d{2}:\d{2}$/), fps: z.number().positive(), resolution: z.number().min(128).max(2048).optional(), })).optional(), view_sample: z.number().optional(), } ``` -------------------------------- ### SessionManifest Interface Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Represents a manifest for a video session, storing video details, creation timestamp, extracted frame information across resolutions, and optional analysis results. ```typescript interface SessionManifest { video_hash: string; video_path: string; created_at: string; resolutions: Record; }>; analysis?: VideoAnalysis; } ``` -------------------------------- ### Add New Imports for Video Watch Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Import necessary functions and types for enhanced video processing, including segment extraction, session management, and manifest handling. ```typescript import { extractFramesBySegments } from "../extractors/frames.js"; import { getSessionDir, loadManifest, saveManifest, computeVideoHash } from "../session/manager.js"; import { createManifest, mergeFrames, sampleFrameIndices } from "../session/manifest.js"; import type { Segment } from "../types.js"; ``` -------------------------------- ### Config Interface with New Fields Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md Defines new fields for the configuration interface, including enabling indexing and setting the maximum session age in days. ```typescript interface Config { // ... existing fields ... enable_index: boolean; // default: false session_max_age_days: number; // default: 7 } ``` -------------------------------- ### Commit Changes Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Stages and commits the updated session manifest files. ```bash git add mcp-server/src/session/manifest.ts mcp-server/tests/session/manifest.test.ts git commit -m "feat(session): add manifest merge, dedup, and sampling logic" ``` -------------------------------- ### Session Manager Tests Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Comprehensive tests for the session manager module, including hashing, session directory creation, and cleanup logic. Ensure the test directory and sessions directory are created before tests and cleaned up afterward. ```typescript // mcp-server/tests/session/manager.test.ts import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { mkdirSync, rmSync, writeFileSync, existsSync } from "fs"; import { join } from "path"; import { tmpdir } from "os"; import { computeVideoHash, getSessionDir, cleanExpiredSessions, } from "../../src/session/manager.js"; const TEST_DIR = join(tmpdir(), "cvv-session-test-" + Date.now()); const SESSIONS_DIR = join(TEST_DIR, "sessions"); describe("session manager", () => { beforeEach(() => { mkdirSync(SESSIONS_DIR, { recursive: true }); }); afterEach(() => { rmSync(TEST_DIR, { recursive: true, force: true }); }); describe("computeVideoHash", () => { it("returns a 12-char hex string", () => { const testFile = join(TEST_DIR, "test.mp4"); writeFileSync(testFile, Buffer.alloc(128 * 1024, "x")); const hash = computeVideoHash(testFile); expect(hash).toMatch(/^[a-f0-9]{12}$/); }); it("returns same hash for same content", () => { const file1 = join(TEST_DIR, "a.mp4"); const file2 = join(TEST_DIR, "b.mp4"); const content = Buffer.alloc(128 * 1024, "hello"); writeFileSync(file1, content); writeFileSync(file2, content); expect(computeVideoHash(file1)).toBe(computeVideoHash(file2)); }); it("returns different hash for different content", () => { const file1 = join(TEST_DIR, "a.mp4"); const file2 = join(TEST_DIR, "b.mp4"); writeFileSync(file1, Buffer.alloc(128 * 1024, "aaa")); writeFileSync(file2, Buffer.alloc(128 * 1024, "bbb")); expect(computeVideoHash(file1)).not.toBe(computeVideoHash(file2)); }); }); describe("getSessionDir", () => { it("returns path under sessions dir using video hash", () => { const testFile = join(TEST_DIR, "test.mp4"); writeFileSync(testFile, Buffer.alloc(128 * 1024, "x")); const dir = getSessionDir(SESSIONS_DIR, testFile); expect(dir).toContain(SESSIONS_DIR); expect(dir).toMatch(/[a-f0-9]{12}$/); }); }); describe("cleanExpiredSessions", () => { it("removes sessions older than maxAgeDays", () => { const oldSession = join(SESSIONS_DIR, "old123456ab"); mkdirSync(oldSession, { recursive: true }); const manifest = { video_hash: "old123456ab", video_path: "/old.mp4", created_at: new Date(Date.now() - 10 * 86400_000).toISOString(), resolutions: {}, }; writeFileSync(join(oldSession, "manifest.json"), JSON.stringify(manifest)); cleanExpiredSessions(SESSIONS_DIR, 7); expect(existsSync(oldSession)).toBe(false); }); it("keeps sessions newer than maxAgeDays", () => { const newSession = join(SESSIONS_DIR, "new123456ab"); mkdirSync(newSession, { recursive: true }); const manifest = { video_hash: "new123456ab", video_path: "/new.mp4", created_at: new Date().toISOString(), resolutions: {}, }; writeFileSync(join(newSession, "manifest.json"), JSON.stringify(manifest)); cleanExpiredSessions(SESSIONS_DIR, 7); expect(existsSync(newSession)).toBe(true); }); it("does nothing if sessions dir does not exist", () => { expect(() => cleanExpiredSessions("/nonexistent/path", 7)).not.toThrow(); }); }); }); ``` -------------------------------- ### Commit Changes for Video Configure Enhancement Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Commit the modified video_configure.ts file with a message detailing the addition of new configuration parameters. ```bash git add mcp-server/src/tools/video-configure.ts git commit -m "feat(configure): add enable_index, session_max_age_days, clear_sessions" ``` -------------------------------- ### Commit Configuration Changes Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md This command stages and commits the changes made to `config.ts` and `config.test.ts`. It includes a conventional commit message indicating a new feature related to configuration. ```bash git add mcp-server/src/config.ts mcp-server/tests/config.test.ts git commit -m "feat(config): add enable_index and session_max_age_days" ``` -------------------------------- ### video_detail Tool Schema Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md TypeScript schema for the video_detail tool, defining parameters for extracting and viewing video segments. Use 'segments' for extraction and 'view'/'view_sample' for viewing. ```typescript { path: z.string().describe("Absolute or relative path to the video file"), // What to EXTRACT to disk segments: z.array(z.object({ start: z.string().regex(/^\d{2}:\d{2}:\d{2}$/, "Must be HH:MM:SS format"), end: z.string().regex(/^\d{2}:\d{2}:\d{2}$/, "Must be HH:MM:SS format"), fps: z.number().positive(), resolution: z.number().min(128).max(2048).optional(), })).optional(), // What to VIEW (return as images) view: z.array(z.string()).optional(), // specific timestamps: ["00:23:14", "00:23:18"] view_sample: z.number().optional(), // OR: N evenly spaced frames from what's extracted skip_cached: z.boolean().default(true), // skip re-extraction of cached frames } ``` -------------------------------- ### Default Configuration for Claude Video Vision Source: https://github.com/jordanrendric/claude-video-vision/blob/main/README.md This JSON object represents the default configuration settings for the claude-video-vision tool. It specifies backend, whisper engine and model, frame processing options, and session management. ```json { "backend": "local", "whisper_engine": "cpp", "whisper_model": "auto", "whisper_at": false, "frame_mode": "images", "frame_format": "jpeg", "frame_resolution": 512, "default_fps": "auto", "max_frames": 100, "frame_describer_model": "sonnet", "enable_index": false, "session_max_age_days": 7, "downloads_max_age_days": 7 } ``` -------------------------------- ### Frame Description Format Source: https://github.com/jordanrendric/claude-video-vision/blob/main/agents/frame-describer.md Use this format to describe video frames, ensuring key visual information is captured concisely. ```text Frame at [timestamp] — [1-3 sentence description covering the above] ``` -------------------------------- ### Derive Content Profile from Motion and Complexity Analysis Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Determines a content profile description based on Signal-to-Noise (SI) average and Temporal Information (TI) average values. Provides a human-readable description for various combinations of SI and TI, or a fallback string if data is missing or unrecognized. ```typescript export function deriveContentProfile(siAvg: number | undefined, tiAvg: number | undefined): string { if (siAvg === undefined || tiAvg === undefined) return "unknown (no motion analysis data)"; const siLabel = siAvg > 50 ? "high" : siAvg > 25 ? "moderate" : "low"; const tiLabel = tiAvg > 30 ? "high" : tiAvg > 10 ? "moderate" : "low"; const descriptions: Record = { "high-high": "high visual complexity, high motion (action, sports, fast-cutting)", "high-moderate": "high visual complexity, moderate motion (documentary, detailed scenes)", "high-low": "high visual complexity, low motion (detailed static shots, landscapes)", "moderate-high": "moderate complexity, high motion (animation, fast graphics)", "moderate-moderate": "moderate complexity, moderate motion (dialogue scenes, tutorials)", "moderate-low": "moderate complexity, low motion (presentations, talking head)", "low-high": "low complexity, high motion (simple fast-moving graphics)", "low-moderate": "low complexity, moderate motion (screencast with scrolling)", "low-low": "low complexity, low motion (static slides, title cards)", }; return descriptions[`${siLabel}-${tiLabel}`] ?? `SI=${siAvg.toFixed(1)}, TI=${tiAvg.toFixed(1)}`; } ``` -------------------------------- ### Sample Frame Indices Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Calculates a list of indices to sample from a total number of frames, aiming for even distribution. Useful for selecting a representative subset of frames. ```typescript // mcp-server/tests/session/manifest.test.ts describe("sampleFrameIndices", () => { it("returns all indices when count >= total", () => { expect(sampleFrameIndices(5, 10)).toEqual([0, 1, 2, 3, 4]); }); it("returns evenly spaced indices", () => { const indices = sampleFrameIndices(10, 3); expect(indices).toEqual([0, 5, 9]); }); it("returns single index for count=1", () => { expect(sampleFrameIndices(10, 1)).toEqual([0]); }); it("returns empty for totalFrames=0", () => { expect(sampleFrameIndices(0, 3)).toEqual([]); }); }); ``` ```typescript // mcp-server/src/session/manifest.ts export function sampleFrameIndices(totalFrames: number, count: number): number[] { if (totalFrames === 0) { return []; } if (count >= totalFrames) { return Array.from({ length: totalFrames }, (_, i) => i); } if (count === 1) { return [0]; } const indices: number[] = []; const step = (totalFrames - 1) / (count - 1); for (let i = 0; i < count; i++) { indices.push(Math.round(i * step)); } return indices; } ``` -------------------------------- ### Parse FFmpeg Signalstats Output Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Parses metadata for signal statistics, extracting brightness (YAVG) and saturation (SATAVG) at specific timestamps. It handles cases where saturation might be missing, defaulting to 0. ```typescript export function parseSignalstatsOutput(metaFileContent: string): Array<{ timestamp: number; brightness: number; saturation: number }> { const results: Array<{ timestamp: number; brightness: number; saturation: number }> = []; let currentPtsTime: number | null = null; let currentYavg: number | null = null; let currentSat: number | null = null; for (const line of metaFileContent.split("\n")) { const ptsMatch = line.match(/pts_time:([\d.]+)/); if (ptsMatch) { if (currentPtsTime !== null && currentYavg !== null) { results.push({ timestamp: currentPtsTime, brightness: currentYavg, saturation: currentSat ?? 0 }); } currentPtsTime = parseFloat(ptsMatch[1]); currentYavg = null; currentSat = null; } const yavgMatch = line.match(/lavfi.signalstats.YAVG=([\d.]+)/); if (yavgMatch) currentYavg = parseFloat(yavgMatch[1]); const satMatch = line.match(/lavfi.signalstats.SATAVG=([\d.]+)/); if (satMatch) currentSat = parseFloat(satMatch[1]); } if (currentPtsTime !== null && currentYavg !== null) { results.push({ timestamp: currentPtsTime, brightness: currentYavg, saturation: currentSat ?? 0 }); } return results; } ``` -------------------------------- ### Smart Video Analysis Response Format Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/specs/2026-04-25-smart-video-analysis-design.md This JSON structure represents the output of a smart video analysis, including scene information, black intervals, silence intervals, frame statistics, loudness summary, and transcription. ```json { "scenes": [ { "time": "00:01:23", "score": 64.3 }, { "time": "00:03:45", "score": 43.1 } ], "black_intervals": [ { "start": "00:01:22", "end": "00:01:24", "duration": 1.5 } ], "silence_intervals": [ { "start": "00:05:00", "end": "00:05:03", "duration": 3.0 } ], "freeze_intervals": [], "frame_stats": [ { "timestamp": "00:00:00", "si": 65.2, "ti": 12.3, "blur": 0.23, "brightness": 128, "saturation": 45 } ], "loudness_summary": { "mean_lufs": -23.5, "range_lu": 12.3 }, "transcription": [ { "start": "00:00:01", "end": "00:00:05", "text": "Welcome to today's lecture" } ], "content_profile": "high visual complexity, low motion (detailed static shots)" } ``` -------------------------------- ### Compute Video Hash Function Source: https://github.com/jordanrendric/claude-video-vision/blob/main/docs/superpowers/plans/2026-04-25-smart-video-analysis.md Computes a 12-character hexadecimal hash for a given video file. It reads a chunk of the file and its size to generate the hash. ```typescript // mcp-server/src/session/manager.ts import { createHash } from "crypto"; import { readFileSync, existsSync, readdirSync, rmSync, statSync, openSync, readSync, closeSync, writeFileSync, mkdirSync } from "fs"; import { join } from "path"; import type { SessionManifest } from "../types.js"; export function computeVideoHash(videoPath: string): string { const fd = openSync(videoPath, "r"); const chunkSize = 64 * 1024; const buffer = Buffer.alloc(chunkSize); const bytesRead = readSync(fd, buffer, 0, chunkSize, 0); closeSync(fd); const fileSize = statSync(videoPath).size; const hash = createHash("sha256"); hash.update(buffer.subarray(0, bytesRead)); hash.update(String(fileSize)); return hash.digest("hex").slice(0, 12); } ```