VLM Run

https://docs.vlm.run
Admin
A platform providing tools and APIs for Vision Language Models, including capabilities for agents,...
Tokens:270,228
Snippets:2,512
Trust Score:7.6
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Basic VLM Run Python SDK

Source: https://docs.vlm.run/sdk-reference/getting-started

Install the core Python SDK using pip. This is the first step to start interacting with the VLM Run API.

```bash
pip install vlmrun --upgrade
```

--------------------------------

### List All Models

Source: https://docs.vlm.run/cli/models

Use this command to see all available models on the VLM Run platform. No setup is required beyond having the CLI installed.

```bash
# List all models
vlmrun models list
```

--------------------------------

### List All Skills (Node.js SDK)

Source: https://docs.vlm.run/api-reference/v1/skills/get-skills-list

This Node.js example demonstrates how to list skills using the VLMRun SDK. Ensure the SDK is installed and provide your API key and base URL for the client initialization.

```typescript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  baseURL: "https://api.vlm.run/v1",
  apiKey: "<VLMRUN_API_KEY>"
});

const skills = await client.agent.skills.list();
console.log(skills);
```

--------------------------------

### Install VLM Run Node.js SDK

Source: https://docs.vlm.run/sdk-reference/node/getting-started

Install the SDK using npm, yarn, or pnpm.

```bash
# Using npm
npm install vlmrun

# Using yarn
yarn add vlmrun

# Using pnpm
pnpm add vlmrun
```

--------------------------------

### Install VLM Run SDK with Optional Dependencies

Source: https://docs.vlm.run/sdk-reference/getting-started

Install the VLM Run Python SDK with specific optional dependency packs for features like video or document processing, or install all features.

```bash
# For video processing
pip install "vlmrun[video]"

# For document processing
pip install "vlmrun[doc]"

# For all features
pip install "vlmrun[all]"
```

--------------------------------

### AgentSkill Reference Example

Source: https://docs.vlm.run/api-reference/v1/post-audio-generate

Example of how to reference a skill stored in the database. Ensure the skill_id and skill_version are correctly specified.

```json
{
  "type": "skill_reference",
  "skill_id": "pillow",
  "skill_version": "latest"
}
```

--------------------------------

### Node.js SDK to Get Skill Download URL

Source: https://docs.vlm.run/api-reference/v1/skills/get-skill-download

This Node.js example demonstrates how to get a presigned skill download URL using the 'vlmrun' SDK. It logs the download URL to the console. Make sure to set up the client with your API key and the correct base URL.

```TypeScript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  baseURL: "https://api.vlm.run/v1",
  apiKey: "<VLMRUN_API_KEY>"
});

const download = await client.agent.skills.download({ skillId: "<skill-id>" });
console.log(download.downloadUrl);
```

--------------------------------

### Referenced Agent Skill Example

Source: https://docs.vlm.run/api-reference/v1/post-document-generate

Example of configuring a referenced agent skill using its ID and version.

```json
{
    "type": "skill_reference",
    "skill_id": "pillow",
    "skill_version": "latest"
}
```

--------------------------------

### Legacy Inline Agent Skill Example

Source: https://docs.vlm.run/api-reference/v1/post-document-generate

Example of configuring an inline agent skill using the legacy 'bundle' field.

```json
{
    "type": "inline",
    "name": "csv-insights",
    "description": "Summarize CSV files.",
    "bundle": "<base64-zip>"
}
```

--------------------------------

### Example: Parse PDF Form

Source: https://docs.vlm.run/cli/generate

This example shows how to parse a PDF document as a form using the `document.form` domain.

```bash
# Parse a PDF form
vlmrun generate document form.pdf --domain document.form
```

--------------------------------

### Install VLMRun Node.js Package

Source: https://docs.vlm.run/skills/quickstart

Install the VLMRun Node.js client library using npm.

```bash
npm install vlmrun
```

--------------------------------

### Install VLM Run Plugin

Source: https://docs.vlm.run/integrations/integrations-voxel51

Install the VLM Run plugin for FiftyOne directly from GitHub. This command downloads the plugin from the specified repository.

```bash
fiftyone plugins download \
    https://github.com/vlm-run/vlmrun-voxel51-plugin
```

--------------------------------

### Referenced Agent Skill Example

Source: https://docs.vlm.run/api-reference/v1/post-video-generate

Example of how to reference a skill stored in the database by its ID or name, optionally specifying a version.

```json
{
  "type": "skill_reference",
  "skill_id": "uuid-or-name-of-skill",
  "skill_version": 2
}
```

--------------------------------

### Example: Extract Invoice Data from Image

Source: https://docs.vlm.run/cli/generate

This example demonstrates extracting invoice data from an image file using the `document.invoice` domain.

```bash
# Extract invoice data from an image
vlmrun generate image invoice.jpg --domain document.invoice
```

--------------------------------

### Complete Workflow Example

Source: https://docs.vlm.run/agents/agent-execution

Demonstrates the full workflow from uploading a file to executing an agent and retrieving results. Use this for end-to-end agent execution.

```python
from pathlib import Path
from vlmrun.client import VLMRun
import time

# Initialize client
client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Step 1: Upload file
print("Uploading file...")
file = client.files.upload(file=Path("invoice.pdf"))
print(f"✓ File uploaded: {file.file_id}")

# Step 2: Execute agent
print("Executing agent...")
execution = client.agent.execute(
    name="invoice-extractor:v1",
    inputs={"file": file.public_url}
)
print(f"✓ Execution started: {execution.execution_id}")

# Step 3: Poll for completion
print("Waiting for results...")
while True:
    result = client.agent.executions.get(execution_id=execution.execution_id)

    if result.status == "completed":
        print("✓ Processing complete!")
        print(f"\nExtracted Data:")
        for key, value in result.response.items():
            print(f"  {key}: {value}")
        break
    elif result.status == "failed":
        print(f"✗ Processing failed: {result.error}")
        break

    print("  Processing...", end="\r")
    time.sleep(2)
```

--------------------------------

### Complete Workflow Example

Source: https://docs.vlm.run/agents/agent-execution

Demonstrates the full workflow from uploading a file to executing an agent and retrieving results using the Node.js SDK. Use this for end-to-end agent execution.

```typescript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  baseUrl: "https://api.vlm.run/v1",
  apiKey: "<VLMRUN_API_KEY>"
});

async function processInvoice() {
  // Step 1: Upload file
  console.log("Uploading file...");
  const file = await client.files.upload({ filePath: "invoice.pdf" });
  console.log(`✓ File uploaded: ${file.file_id}`);

  // Step 2: Execute agent
  console.log("Executing agent...");
  const execution = await client.agent.execute({
    name: "invoice-extractor:v1",
    inputs: { file: file.public_url }
  });
  console.log(`✓ Execution started: ${execution.execution_id}`);

  // Step 3: Poll for completion
  console.log("Waiting for results...");
  while (true) {
    const result = await client.agent.executions.get({
      executionId: execution.execution_id
    });

    if (result.status === "completed") {
      console.log("✓ Processing complete!");
      console.log("\nExtracted Data:");
      console.log(result.response);
      break;
    } else if (result.status === "failed") {
      console.log(`✗ Processing failed: ${result.error}`);
      break;
    }

    process.stdout.write("  Processing...\r");
    await new Promise(resolve => setTimeout(resolve, 2000));
  }
}

processInvoice();
```

--------------------------------

### Lookup Skill by ID (Node.js)

Source: https://docs.vlm.run/skills/manage/list-lookup

Find a skill by its unique ID using the Node.js client. This example demonstrates how to pass the skill ID to the `get` method.

```typescript
const skill = await client.skills.get({ id: "<skill-id>" });
```

--------------------------------

### Quick CLI Configuration Setup

Source: https://docs.vlm.run/cli/getting-started

Initialize the VLM Run CLI configuration file and set your API key using the config commands.

```bash
vlmrun config init
vlmrun config set --api-key "your-api-key"
```

--------------------------------

### List Models with Python SDK

Source: https://docs.vlm.run/api-reference/v1/get-models

Use the Python SDK to get a list of supported models. Ensure you have installed the 'vlmrun' package and have your API key ready.

```python
!pip install vlmrun

from vlmrun.client import VLMRun

client = VLMRun(api_key="<VLMRUN_API_KEY>")
response = client.models.list()
```

--------------------------------

### Initialize Default Configuration

Source: https://docs.vlm.run/cli/config

Create a default configuration file. Use the --force flag to overwrite an existing config file.

```bash
# Create config file
vlmrun config init

# Overwrite existing config
vlmrun config init --force
```

--------------------------------

### Get Agent Execution by ID in Node.js

Source: https://docs.vlm.run/api-reference/v1/get-agent-executions-by-id

Utilize the VLMRun Node.js SDK to retrieve a specific agent execution by its ID. This example demonstrates asynchronous fetching.

```typescript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  baseURL: "https://api.vlm.run/v1",
  apiKey: "<VLMRUN_API_KEY>"
});

const response = await client.executions.get("<execution_id>");
console.log(response);
```

--------------------------------

### Create and Use VLM Run Skills with Python SDK

Source: https://docs.vlm.run/skills/reference

Demonstrates how to use the Python SDK to create skills, either by uploading them to the server or by creating an inline skill bundle. Both methods require a local directory containing a SKILL.md file.

```python
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentSkill

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Server-side: upload and create a reusable skill
skill_ref = client.skills.create_from_directory(Path("./my-skill"))

# Inline: build a self-contained skill bundle (no upload)
skill_inline = AgentSkill.from_directory(Path("./my-skill"))
```

--------------------------------

### Get Agent Execution by ID in Python

Source: https://docs.vlm.run/api-reference/v1/get-agent-executions-by-id

Use the VLMRun Python client to fetch a specific agent execution by its ID. Ensure you have installed the 'vlmrun' package.

```python
!pip install vlmrun

from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionResponse

client = VLMRun(api_key="<VLMRUN_API_KEY>")
response: AgentExecutionResponse = client.executions.get("<execution_id>")
```

--------------------------------

### Get Prediction by ID in Python

Source: https://docs.vlm.run/api-reference/v1/predictions/get-predictions-by-id

Fetches prediction results for a given prediction ID using the Python client. Ensure you have installed the vlmrun library and have your API key ready.

```python
!pip install vlmrun

from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse

client = VLMRun(api_key="<VLMRUN_API_KEY>")
response: PredictionResponse = client.predictions.get("<prediction_id>")
```

--------------------------------

### Get File by ID using Node.js SDK

Source: https://docs.vlm.run/api-reference/v1/files/get-files-by-id

Utilize the VLMRun Node.js SDK to fetch file information by its ID. This example demonstrates asynchronous retrieval and logging of the response.

```TypeScript
import { VlmRun } from "vlmrun";

const client = new VlmRun({ apiKey: "<VLMRUN_API_KEY>" });

// Get the file by ID
const response = await client.files.get("file_123");
console.log(response);
```

--------------------------------

### Initialize VLM Run Client

Source: https://docs.vlm.run/sdk-reference/components/client

Demonstrates how to initialize the VLMRun client using environment variables, an explicit API key, or custom configuration including base URL and retry settings.

```python
from vlmrun.client import VLMRun

# Using environment variable (recommended)
client = VLMRun()

# Or with explicit API key
client = VLMRun(api_key="your-api-key")

# With custom configuration
client = VLMRun(
    api_key="your-api-key",
    base_url="https://custom-endpoint.com/v1",
    timeout=60.0,
    max_retries=3
)
```

--------------------------------

### Get File by ID using Python SDK

Source: https://docs.vlm.run/api-reference/v1/files/get-files-by-id

Use the VLMRun Python SDK to retrieve file details by providing the file ID. Ensure you have installed the 'vlmrun' package and have your API key configured.

```Python
!pip install vlmrun

from vlmrun.client import VLMRun

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Get the file by ID
response = client.files.get(file_id="file_123")
```

--------------------------------

### Python: Execute Agent with Image and Text Input

Source: https://docs.vlm.run/api-reference/v1/post-agent-execute

This example shows how to execute an agent with an image and a text instruction. The image is provided via a URL, and the instruction specifies the task for the agent.

```python
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse
from vlmrun.types import MessageContent, ImageUrl

class ExecutionInputs(BaseModel):
  image: MessageContent = Field(..., description="Image to analyze")
  instruction: MessageContent = Field(..., description="What to extract or do")

client = VLMRun(api_key="<VLMRUN_API_KEY>")

response: AgentExecutionResponse = client.agent.execute(
  name="<agent-name>:<agent-version>",
  inputs=ExecutionInputs(
    image=MessageContent(
      type="image_url",
      image_url=ImageUrl(url="https://example.com/photo.jpg", detail="high"),
    ),
    instruction=MessageContent(
      type="text",

```

--------------------------------

### Example Audio Transcription Output

Source: https://docs.vlm.run/guides/audio-ai/guide-audio-transcription

This JSON object represents the transcribed output of a 5-hour podcast. It includes metadata about the audio content and a list of segments, each with start time, end time, and the transcribed text.

```json
{
  "metadata": {
    "language": null,
    "content": null,
    "topics": null,
    "duration": 18375.22
  },
  "segments": [
    {
      "start_time": 0,
      "end_time": 20.83,
      "content": " The following is a conversation with Dylan Patel and Nathan Lampert. Dylan runs semi-analysis, a well-respected research and analysis company that specializes in semiconductors, GPUs, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI,"
    },
    {
      "start_time": 20.83,
      "end_time": 47.04,
      "content": " and is the author of the amazing blog on AI called Interconnects. They are both highly respected, read, and listened to by the experts, researchers, and engineers in the field of AI. And personally, I'm just a fan of the two of them. So I use the deep seek moment that shook the AI world a bit as an opportunity to sit down with them and lay it all"
    },
    {
      "start_time": 47.04,
      "end_time": 70.17,
      "content": " out from DeepSeek, Open AI, Google XAI, Metanthropic to Nvidia and DSMC, and to US-Ch., China, Taiwan relations, and everything else that is happening at the cutting edge of AI. This conversation is a deep dive into many critical aspects of the AI industry. While it does get super technical,"
    },
    {
      "start_time": 70.75,
      "end_time": 91.67,
      "content": " we try to make sure that it's still accessible to folks outside of the AI field by defining terms, stating important concepts explicitly, spelling out acronyms, and in general, always moving across the several layers of abstraction and levels of detail. There is a lot of hype in the media about what AI is and isn't."
    },
    {
      "start_time": 92.53,
      "end_time": 113.23,
      "content": " The purpose of this podcast, in part, is to cut through the hype, through the bullshit, and the low-resolution analysis, and to discuss in detail how stuff works and what the implications are. Let me also, if I may, comment on the new OpenA.O3 mini reasoning model, the release of which we were"
    },
    {
      "start_time": 113.23,
      "end_time": 134.71,
      "content": " anticipating during the conversation, and it did indeed come out right after. Its capabilities and costs are on par with our expectations, as we stated. OpenAI 03 Mini is indeed a great model, but it should be stated that Deep Seekar 1 has similar performance on benchmarks. It's still cheaper and it"
    },
    {
      "start_time": 134.71,
      "end_time": 161.17,
      "content": " reveals its chain of thought reasoning, which O3Mini does not. It only shows a summary of the reasoning. Plus R1 is open weight. And O3 Mini is not. By the way, I got a chance to play with 03 Mini. And anecdotal vibe check-wise, I felt that 03 Mini, specifically O3 Mini High, is better than R1."
    },
    {
      "start_time": 161.96,
      "end_time": 182.44,
      "content": " Still, for me personally, I find that Claude Sonna 3-5 is the best model for programming, except for tricky cases where I will use O1 Pro to brainstorm. Either way, many more better AI models will come, including reasoning models, both from American and Chinese companies. They will continue to shift the cost curve."
    },
    {
      "start_time": 183.42,
      "end_time": 205.54,
      "content": " But the quote, deep seek moment is indeed real. I think it will still be remembered five years from now as a pivotal event in tech history, due in part to the geopolitical implications, but for other reasons too, as we discuss in detail from many perspectives in this conversation. This is the Lex Friedman podcast. To support it, please check out"
    },
    {
      "start_time": 205.54,
      "end_time": 227.12,
      "content": " our sponsors in the description. And now, dear friends, here's Dylan Patel and Nathan Lambert. A lot of people are curious to understand China's deep seek AI models. So let's lay it out. Nathan, can you describe what deep seek v3 and deep seek r1r how they work how they're trained let's look at the big picture"
    },
    {
      "start_time": 227.12,
      "end_time": 249.79,
      "content": " and then we'll zoom in on the details. Yeah, so DeepSeek v3 is a new mixture of experts transformer language model from Deepseek, who is based in China. They have some new specifics in the model that we'll get into. Largely, this is a open weight model, and it's a instruction model like what you would use in chat GPT."
    },
    {
      "start_time": 250.83,
      "end_time": 270.39,
      "content": ""
    }
  ]
}
```

--------------------------------

### Example Temporal Grounding Output Structure

Source: https://docs.vlm.run/capabilities/temporal-grounding

This JSON structure shows the temporal information returned for video transcription, including segment start and end times, and the transcribed audio and video content for each segment.

```json
{
  "metadata": {
    "duration": 248.67   // total duration of the video in seconds
  },
  "segments": [
    {
      "start_time": 0,   // start time of the first segment in seconds
      "end_time": 21.33, // end time of the first segment in seconds
      "audio": {
        "content": " The Keys Rocks is a rough and tumble area just outside Pittsburgh. My name is Scott Baker. My family has been connected with this community for generations. In 1941, my grandfather opened his bakery here, and he called it Jenny Lee. We used to always go to Jenny Lee's after church. If you were good in church,"
      },
      "video": {
        "content": "In this image we can see many buildings and trees. There is a bridge in the image. There are many towers in the background of the image and the sky is in white color."
      }
    },
    {
      "start_time": 21.33, // start time of the second segment in seconds
      "end_time": 42.5,    // end time of the second segment in seconds
      "audio": {
        "content": " oh, egg custard pot. Gooden Church. Oh, I ate custard pies, homemade bread that was still warm. Years later, I worked in the store. I did wedding cakes. My father, Bernie, took over after my grandfather retired. I am a baker. took over after my grandfather retired. I am a baker by name, baker by trade. I started coming in to the bakery with my dad when I was seven or eight years old."
      },
      "video": {
        "content": "In this image we can see a man standing on the floor. We can also see a group of people standing beside a table containing some food items in a cupboard. On the backside we can a wall with some photo frames and a roof with some ceiling lights."
      }
    },
    ...
    {
      "start_time": 230.67, // start time of the last segment in seconds
      "end_time": 248.67,   // end time of the last segment in seconds
      "audio": {
        "content": " I made that. I'm proud to say that. You know, You know, You know, Thank you."
      },
      "video": {
        "content": "In this image we can see a person holding a food item in his hand. In front of him there is a table. In the background of the image there are some objects."
      }
    }
  ]
}
```

--------------------------------

### Execute Agent with All Input Types

Source: https://docs.vlm.run/api-reference/v1/post-agent-execute

This example demonstrates how to combine various input modalities, including uploaded files, file URLs, image URLs, video URLs, audio URLs, text instructions, and plain JSON primitives, in a single agent execution request. It showcases the flexibility of the 'inputs' field for complex multi-modal interactions.

```json
{
  "name": "<agent-name>:<agent-version>",
  "inputs": {
    "file": {
      "type": "input_file",
      "file_id": "dbb28d43-d741-4e0c-b25b-04ddc69b3197"
    },
    "supporting_document": {
      "type": "file_url",
      "file_url": { "url": "https://example.com/referral.pdf" }
    },
    "reference_image": {
      "type": "image_url",
      "image_url": { "url": "https://example.com/layout.png", "detail": "high" }
    },
    "demo_video": {
      "type": "video_url",
      "video_url": { "url": "https://example.com/clip.mp4" }
    },
    "voicemail": {
      "type": "audio_url",
      "audio_url": { "url": "https://example.com/voicemail.mp3" }
    },
    "instruction": {
      "type": "text",
      "text": "Schedule the patient and confirm insurance eligibility."
    },
    "email_details": "<div dir=\"ltr\">Hi,<br />Please see the attached order form for Oscar Bhujel. Kindly let us know once the appointment is scheduled.<br />Thank you,<br />Camielle Jane Lim</div>",
    "metadata": {
      "received_at": "2026-04-20T16:30:00Z",
      "priority": "normal",
      "source": "gmail"
    }
  },
  "batch": true
}
```

--------------------------------

### Python SDK to Get Skill Download URL

Source: https://docs.vlm.run/api-reference/v1/skills/get-skill-download

Use the Python SDK to obtain a presigned download URL for a skill. Ensure you have installed the 'vlmrun' package. The output includes the download URL and its expiration time in seconds.

```Python
!pip install vlmrun

from vlmrun.client import VLMRun

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Get a presigned download URL for a skill zip
download = client.agent.skills.download(skill_id="<skill-id>")
print(f"Download URL: {download.download_url}")
print(f"Expires in: {download.expires_in} seconds")
```

--------------------------------

### Node.js: Basic Image Pointing

Source: https://docs.vlm.run/agents/capabilities/image/pointing

Use this snippet to get a text response indicating the locations of objects in an image using Node.js. Ensure you have the `vlmrun` package installed and initialize the client with your API key and base URL.

```typescript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  apiKey: "<VLMRUN_API_KEY>",
  baseURL: "https://api.vlm.run/v1"
});

const response = await client.agent.completions.create({
  model: "vlmrun-orion-1:auto",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Point to all the cars and doors in this image" },
        { type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
      ]
    }
  ]
});

console.log(response.choices[0].message.content);
```

--------------------------------

### Client Initialization

Source: https://docs.vlm.run/sdk-reference/components/client

Demonstrates how to initialize the VLMRun client, with options for using environment variables, explicit API keys, or custom configurations including base URL, timeout, and retry settings.

```APIDOC
## Client Initialization

### Description
Initialize the VLMRun client. It's recommended to use the `VLMRUN_API_KEY` environment variable for API key authentication. Custom configurations for `base_url`, `timeout`, and `max_retries` are also supported.

### Method
```python
from vlmrun.client import VLMRun

# Using environment variable (recommended)
client = VLMRun()

# Or with explicit API key
client = VLMRun(api_key="your-api-key")

# With custom configuration
client = VLMRun(
    api_key="your-api-key",
    base_url="https://custom-endpoint.com/v1",
    timeout=60.0,
    max_retries=3
)
```

### Parameters
#### Initialization Parameters
- **api_key** (str) - Optional - Your API key (falls back to `VLMRUN_API_KEY` env var).
- **base_url** (str) - Optional - API endpoint URL. Defaults to `"https://api.vlm.run/v1"`.
- **timeout** (float) - Optional - Request timeout in seconds. Defaults to `120.0`.
- **max_retries** (int) - Optional - Maximum retry attempts for failed requests. Defaults to `5`.
```

--------------------------------

### Using Local and Server-Side Skills

Source: https://docs.vlm.run/cli/execute

Shows how to attach skills to an agent execution, either from a local directory or by referencing a server-side skill ID.

```bash
vlmrun execute -n my-agent:v1 -i img.jpg --skill ./my-skill
```

```bash
vlmrun execute -n my-agent:v1 -i img.jpg --skill-id my-skill:latest
```

--------------------------------

### Get Skill by ID using Python SDK

Source: https://docs.vlm.run/api-reference/v1/skills/get-skill-by-id

Use the Python SDK to retrieve a skill by its ID. Ensure you have installed the 'vlmrun' package and have your API key ready. This method returns skill details including name, version, and description.

```Python
!pip install vlmrun

from vlmrun.client import VLMRun

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Get skill by ID
skill = client.agent.skills.get(id="<skill-id>")
print(f"{skill.name} (v{skill.version}): {skill.description}")
```

--------------------------------

### Inline Skill Example

Source: https://docs.vlm.run/api-reference/v1/post-agent-create

Provide an inline skill by including its name, description, and a base64-encoded zip bundle of its source files. The zip must contain a single SKILL.md file.

```json
{
    "type": "inline",
    "name": "csv-insights",
    "description": "Summarize CSV files.",
    "source": {
        "type": "base64",
        "media_type": "application/zip",
        "data": "<base64-zip>"
    }
}
```

--------------------------------

### Get Skill by ID using Node.js SDK

Source: https://docs.vlm.run/api-reference/v1/skills/get-skill-by-id

This Node.js example demonstrates fetching skill information via the VLM Run SDK. It requires initializing the client with your API key and the base URL. The retrieved skill object is then logged to the console.

```typescript
import { VlmRun } from "vlmrun";

const client = new VlmRun({
  baseURL: "https://api.vlm.run/v1",
  apiKey: "<VLMRUN_API_KEY>"
});

const skill = await client.agent.skills.get({ id: "<skill-id>" });
console.log(skill);
```

--------------------------------

### Install VLM Run CLI

Source: https://docs.vlm.run/cli/getting-started

Install the VLM Run CLI using pip. The recommended installation includes the 'cli' extra for a richer terminal experience.

```bash
pip install vlmrun
```

```bash
pip install vlmrun[cli]
```

--------------------------------

### Launch FiftyOne App with Dataset

Source: https://docs.vlm.run/integrations/integrations-voxel51

Launch the FiftyOne App and load a sample dataset. This is the initial step to start using the VLM Run operators within FiftyOne.

```python
import fiftyone as fo
import fiftyone.zoo as foz

# Load a sample dataset
dataset = foz.load_zoo_dataset("quickstart", max_samples=10)
session = fo.launch_app(dataset)
```

--------------------------------

### Configure Instructor Client with VLM Run

Source: https://docs.vlm.run/agents/integrations/integrations-instructor

Set up the OpenAI client to point to the VLM Run API endpoint and then create an Instructor client instance. This allows using Instructor's features with VLM Run.

```python
import instructor
import openai

# Configure the OpenAI client
client = openai.OpenAI(
    base_url="https://api.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>"
)

# Configure the Instructor client
inst_client = instructor.from_openai(
    client, mode=instructor.Mode.MD_JSON
)
```

--------------------------------

### Install VLMRun Python Package

Source: https://docs.vlm.run/skills/quickstart

Install the VLMRun Python client library using pip.

```bash
pip install vlmrun
```

--------------------------------

### Python: Execute Agent with File URL Input

Source: https://docs.vlm.run/api-reference/v1/post-agent-execute

This example demonstrates executing an agent using a file URL. It includes uploading a file and then executing the agent with the file's public URL. The agent can be executed by name or by an inline prompt, with an optional response model for structured output.

```python
!pip install vlmrun

from pathlib import Path
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionResponse, AgentExecutionConfig
from vlmrun.types import MessageContent, FileUrl

# Define a Pydantic model for the execution inputs
class ExecutionInputs(BaseModel):
  file: MessageContent = Field(..., description="The file to extract data from")

# Define a Pydantic model for the response
class Invoice(BaseModel):
  invoice_id: str = Field(..., description="The ID of the invoice")
  total_amount: float = Field(..., description="The total amount of the invoice")

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Upload the file to the object store
file = client.files.upload(file=Path("test.pdf"))

# Execute the agent (by name and version)
response: AgentExecutionResponse = client.agent.execute(
  name="<agent-name>:<agent-version>",
  inputs=ExecutionInputs(
    file=MessageContent(type="file_url", file_url=FileUrl(url=file.public_url))
  ),
  batch=True,
)

# Execute the agent (by inline prompt)
response: AgentExecutionResponse = client.agent.execute(
  inputs=ExecutionInputs(
    file=MessageContent(type="file_url", file_url=FileUrl(url=file.public_url))
  ),
  config=AgentExecutionConfig(
    prompt="Extract the invoice_id and total amount from the invoice.",
    response_model=Invoice,
  ),
  batch=True,
)
```

--------------------------------

### Create Skill from File (Python)

Source: https://docs.vlm.run/api-reference/v1/skills/post-skill-create

Use this snippet to create a skill from a pre-uploaded skill zip file. The zip must contain a SKILL.md file. Requires the vlmrun library.

```Python
skill = client.agent.skills.create(
    name="invoice-extraction",
    description="Extract structured data from invoices",
    file_id="<file-id>"  # Uploaded skill zip containing SKILL.md
)
```

--------------------------------

### MarkdownPage Schema Example

Source: https://docs.vlm.run/guides/doc-ai/guide-parsing-documents

This snippet shows an example of the MarkdownPage schema, which can be used to represent structured document content.

```json
{
  "content": "# Overview\n\nFine-tuning involves adjusting the parameters of pre-trained models on a specific dataset or task. This process enhances the model's ability to generate more accurate and relevant responses for the given context by adapting it to the nuances and specific requirements of the task at hand.\n\n**Example use cases**\n-   Generate output in a consistent format\n-   Process input by following specific instructions\n\n## What we'll cover\n\n*   When to fine-tune\n*   Preparing the dataset\n*   Best practices\n*   Hyperparameters\n*   Fine-tuning advances\n*   Resources\n\n---\n3",
  "markdown_content": "# Overview\n\nFine-tuning involves adjusting the parameters of pre-trained models on a specific dataset or task. This process enhances the model's ability to generate more accurate and relevant responses for the given context by adapting it to the nuances and specific requirements of the task at hand.\n\n**Example use cases**\n-   Generate output in a consistent format\n-   Process input by following specific instructions\n\n## What we'll cover\n\n*   When to fine-tune\n*   Preparing the dataset\n*   Best practices\n*   Hyperparameters\n*   Fine-tuning advances\n*   Resources\n\n---\n3",
  "tables": null,
  "figures": null
}
```

--------------------------------

### Quick Example: Transcribe Audio from URL

Source: https://docs.vlm.run/sdk-reference/getting-started

Transcribe an audio file by providing its URL and specifying the transcription domain.

```python
# Transcribe an audio file
response = client.audio.generate(
    url="https://storage.googleapis.com/vlm-data-public-prod/examples/audio/sample.mp3",
    domain="audio.transcription"
)
```

--------------------------------

### Basic CLI Chat Usage Examples

Source: https://docs.vlm.run/cli/chat

Demonstrates how to use the `vlmrun chat` command for common tasks like describing images, analyzing documents, processing videos, and comparing files. Use the `-i` flag to specify input files.

```bash
# Describe an image
vlmrun chat "Describe this image" -i photo.jpg

# Analyze a document
vlmrun chat "Extract the key information" -i document.pdf

# Process a video
vlmrun chat "Summarize this video" -i video.mp4

# Compare multiple files
vlmrun chat "Compare these images" -i image1.jpg -i image2.jpg
```

--------------------------------

### JSON Theme Example

Source: https://docs.vlm.run/agents/structured-responses

Example of a JSON object defining a theme with light and dark modes for syntax highlighting.

```json
JSON theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "invoice_number": "INV-2024-001",
  "invoice_date": "2024-09-15T00:00:00",
  "total_amount": 1250.00,
  "vendor_name": "Acme Corporation"
}
```

--------------------------------

### List Files with Node.js SDK

Source: https://docs.vlm.run/api-reference/v1/files/get-files-list

This Node.js example demonstrates how to fetch a list of files using the vlmrun SDK. Initialize the client with your API key and specify the desired limit for the file list.

```TypeScript
import { VlmRun } from "vlmrun";

const client = new VlmRun({ apiKey: "<VLMRUN_API_KEY>" });

// Get the list of files
const response = await client.files.list({ limit: 10 });
console.log(response);
```

--------------------------------

### Verify VLM Run CLI Installation

Source: https://docs.vlm.run/cli/getting-started

Check if the VLM Run CLI has been installed correctly by running the version command.

```bash
vlmrun --version
```

--------------------------------

### List All Skills (CLI)

Source: https://docs.vlm.run/skills/manage/list-lookup

Use the VLMRun CLI to list skills. Options include listing the latest 25, grouping by name to show the latest version, or specifying a custom limit and sort order.

```bash
# List all skills (latest 25)
vlmrun skills list

# List with grouping (latest version per name)
vlmrun skills list --grouped

# List with custom limit and sort
vlmrun skills list --limit 50 --order-by name --asc
```
Tokens:
Raw