Moondream

https://docs.moondream.ai
Admin
Moondream is an open-source family of Vision Language Models designed for powerful, efficient visual...
Tokens:40,389
Snippets:343
Trust Score:8.2
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Moondream SDK

Source: https://docs.moondream.ai/quickstart

Install the required package for your environment.

```bash
pip install moondream
```

```bash
npm install moondream
```

--------------------------------

### Segmentation with Spatial Hints

Source: https://docs.moondream.ai/skills/segment

Examples of using point and bounding box hints to guide the segmentation process.

```python
# Point hint - segment object near center
result = model.segment(image, "person", spatial_refs=[[0.5, 0.5]])

# Bounding box hint - segment object within region
result = model.segment(image, "person", spatial_refs=[[0.1, 0.2, 0.6, 0.8]])

# Multiple hints
result = model.segment(image, "person", spatial_refs=[[0.3, 0.4], [0.7, 0.6]])
```

--------------------------------

### Install Moondream Python Client

Source: https://docs.moondream.ai/running-locally

Installs the Moondream library with built-in support for the Photon inference engine.

```bash
pip install moondream
```

--------------------------------

### SFT Training Group Example

Source: https://docs.moondream.ai/finetuning/http-api-reference

Example of a training group configured for SFT mode.

```json
{
  "mode": "sft",
  "request": {
    "skill": "query",
    "question": "What country is this?",
    "image_url": "data:image/jpeg;base64,..."
  },
  "target": {
    "answer": "United States"
  }
}
```

--------------------------------

### Install Dependencies

Source: https://docs.moondream.ai/finetuning/quickstart

Install the required Python packages for Moondream.

```bash
pip install moondream datasets pillow  
```

--------------------------------

### Install Moondream on JetPack 7

Source: https://docs.moondream.ai/running-locally

Standard installation for JetPack 7 using the PyPI PyTorch wheel.

```bash
pip install moondream  
```

--------------------------------

### RL Training Group Example

Source: https://docs.moondream.ai/finetuning/http-api-reference

Example of a training group configured for RL mode.

```json
{
  "mode": "rl",
  "request": {
    "finetune_id": "01HXYZ...",
    "num_rollouts": 2,
    "request": {
      "skill": "query",
      "question": "What color is the car?",
      "image_url": "data:image/jpeg;base64,...",
      "reasoning": false
    }
  },
  "rollouts": [
    {
      "skill": "query",
      "output": {
        "answer": "red"
      }
    },
    {
      "skill": "query",
      "output": {
        "answer": "dark red"
      }
    }
  ],
  "rewards": [0.8, 0.6]
}
```

--------------------------------

### Batch Status Response Examples

Source: https://docs.moondream.ai/batch

Example JSON responses for processing and completed batch job states.

```json
{
  "id": "01JQXYZ9ABCDEF123456",
  "status": "processing",
  "model": "moondream3.1-9B-A2B",
  "progress": { "total": 1000, "completed": 450 },
  "created_at": "2025-01-10T12:00:00Z"
}
```

```json
{
  "id": "01JQXYZ9ABCDEF123456",
  "status": "completed",
  "model": "moondream3.1-9B-A2B",
  "progress": { "total": 1000, "completed": 998, "failed": 2 },
  "usage": { "input_tokens": 1500000, "output_tokens": 50000 },
  "outputs": [
    { "index": 0, "url": "https://..." },
    { "index": 1, "url": "https://..." }
  ],
  "created_at": "2025-01-10T12:00:00Z",
  "completed_at": "2025-01-10T12:45:00Z",
  "expires_at": "2025-01-17T12:45:00Z"
}
```

--------------------------------

### Install PyTorch for JetPack 6

Source: https://docs.moondream.ai/running-locally

Installs the specific NVIDIA PyTorch wheel required for JetPack 6.x environments.

```bash
pip install https://developer.download.nvidia.com/compute/redist/jp/v61/pytorch/torch-2.5.0a0+872d972e41.nv24.08.17622132-cp310-cp310-linux_aarch64.whl  
```

```bash
pip install https://developer.download.nvidia.com/compute/redist/jp/v60/pytorch/torch-2.4.0a0+07cecf4168.nv24.05.14710581-cp310-cp310-linux_aarch64.whl  
```

--------------------------------

### Verify Moondream Installation

Source: https://docs.moondream.ai/running-locally

Checks the PyTorch version and GPU device name to confirm a successful installation.

```bash
python3 -c "import torch, moondream; print(torch.__version__, torch.cuda.get_device_name(0))"  
```

--------------------------------

### Configure API Key

Source: https://docs.moondream.ai/sample-projects/automatic-detection-labeling

Set the required environment variable for authentication before starting the application.

```bash
export MOONDREAM_API_KEY="your_key_here"  
npm start  
```

--------------------------------

### GET /v1/batch

Source: https://docs.moondream.ai/batch

Lists existing batches.

```APIDOC
## GET /v1/batch

### Description
Retrieves a paginated list of batches.

### Method
GET

### Endpoint
/v1/batch?limit={n}&cursor={cursor}

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Results per page (default 50, max 100)
- **cursor** (string) - Optional - Pagination cursor

### Response
#### Success Response (200)
- **batches** (array) - List of batch objects
- **next_cursor** (string) - Cursor for the next page
- **has_more** (boolean) - Indicates if more results are available
```

--------------------------------

### Query with a finetuned model

Source: https://docs.moondream.ai/finetuning/http-api-reference

Example request using a specific finetuned model checkpoint for inference.

```bash
curl -X POST https://api.moondream.ai/v1/query \
  -H "X-Moondream-Auth: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moondream3-preview/01HXYZ...@500",
    "image_url": "data:image/jpeg;base64,...",
    "question": "What do you see?"
  }'
```

--------------------------------

### Captioning Example Response

Source: https://docs.moondream.ai/transformers

The expected JSON output format for a captioning request.

```json
{
    "caption": "Four brown horses with red and black harnesses plow a field, guided by two men in western attire, with a majestic mountain range in the background."
}
```

--------------------------------

### Configure SFT training groups

Source: https://docs.moondream.ai/finetuning/python-sdk

Provides target answers for SFT training. Includes examples for query, reasoning, point, and detect skills.

```python
ft.train_step([{
    "mode": "sft",
    "request": {
        "skill": "query",
        "image": pil_image,
        "question": "What country is this?",
    },
    "target": {"answer": "United States"},
}])
```

```python
ft.train_step([{
    "mode": "sft",
    "request": {
        "skill": "query",
        "image": pil_image,
        "question": "What country is this?",
        "reasoning": True,
    },
    "target": {
        "answer": "United States",
        "reasoning": {"text": "The road markings and signs match the US."},
    },
}])
```

```python
ft.train_step([{
    "mode": "sft",
    "request": {
        "skill": "point",
        "image": pil_image,
        "object": "the red button",
    },
    "target": {"points": [{"x": 0.52, "y": 0.31}]},
}])
```

```python
"target": {"boxes": [{"x_min": 0.45, "y_min": 0.22, "x_max": 0.58, "y_max": 0.39}]}
```

```python
ft.train_step([{
    "mode": "sft",
    "request": {
        "skill": "detect",
        "image": pil_image,
        "object": "vehicles",
    },
    "target": {"boxes": [
        {"x_min": 0.10, "y_min": 0.20, "x_max": 0.40, "y_max": 0.60},
    ]},
}])
```

--------------------------------

### Install Missing CUDA Libraries on JetPack 6

Source: https://docs.moondream.ai/running-locally

Installs additional system packages if library errors persist after setting the path.

```bash
sudo apt install cuda-cupti-12-6 libnvtoolsext1  
```

--------------------------------

### Launch Triton Inference Server

Source: https://docs.moondream.ai/running-locally

Commands to start the Triton server container with the necessary environment variables and volume mounts.

```bash
docker run --gpus all --rm -it \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v ./triton_server/model_repository:/models \
  -e MOONDREAM_API_KEY=your-api-key \
  -e KESTREL_MODEL=moondream3-preview \
  nvcr.io/nvidia/tritonserver:24.08-py3 \
  bash -c "pip install kestrel && tritonserver --model-repository=/models"  
```

```bash
docker run --gpus all --rm -it \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v ./triton_server/model_repository:/models \
  -v /path/to/model/weights:/model_weights \
  -e MOONDREAM_API_KEY=your-api-key \
  -e KESTREL_MODEL_PATH=/model_weights \
  nvcr.io/nvidia/tritonserver:24.08-py3 \
  bash -c "pip install kestrel && tritonserver --model-repository=/models"  
```

--------------------------------

### Query Target Examples

Source: https://docs.moondream.ai/finetuning/http-api-reference

Target structures for the query skill, including optional reasoning fields.

```json
{
  "answer": "United States"
}
```

```json
{
  "answer": "United States",
  "reasoning": {
    "text": "The road markings and signs match the US."
  }
}
```

--------------------------------

### Detect Target Example

Source: https://docs.moondream.ai/finetuning/http-api-reference

Target structure for the detect skill using boxes.

```json
{
  "boxes": [
    { "x_min": 0.12, "y_min": 0.22, "x_max": 0.39, "y_max": 0.58 },
    { "x_min": 0.51, "y_min": 0.18, "x_max": 0.74, "y_max": 0.49 }
  ]
}
```

--------------------------------

### Query Example Response

Source: https://docs.moondream.ai/transformers

The expected JSON output format for query requests, including reasoning and grounding data.

```json
{
    "answer": "The prominent colors in this image are brown, black, white, and blue."
}

# Answer with reasoning
{
    "reasoning": {
        "text": "I see 4 horses.",
        "grounding": [
            {
                "start_idx": 6,
                "end_idx": 14,
                "points": [
                    (0.359375, 0.4423828125),
                    (0.5791015625, 0.5),
                    (0.810546875, 0.4423828125),
                    (0.9345703125, 0.5)
                ]
            }
        ]
    },
    "answer": "4"
}
```

--------------------------------

### Point Target Examples

Source: https://docs.moondream.ai/finetuning/http-api-reference

Target structures for the point skill, supporting both points and boxes.

```json
{
  "points": [
    { "x": 0.52, "y": 0.31 }
  ]
}
```

```json
{
  "boxes": [
    { "x_min": 0.45, "y_min": 0.22, "x_max": 0.58, "y_max": 0.39 }
  ]
}
```

--------------------------------

### Example API Response

Source: https://docs.moondream.ai/skills/query

The standard JSON structure returned by the /query endpoint.

```json
{
  "request_id": "2025-03-25_query_2025-03-25-21:00:39-715d03",
  "answer": "Detailed text answer to your question..."
}
```

--------------------------------

### Object Detection with Moondream

Source: https://docs.moondream.ai/skills/detect

Examples of how to use the /detect endpoint to identify objects in an image across different environments.

```python
import moondream as md  
from PIL import Image  
import matplotlib.pyplot as plt  
import matplotlib.patches as patches  
  
# Initialize with API key  
model = md.vl(api_key="your-api-key")  
  
# Load an image  
image = Image.open("path/to/image.jpg")  
  
# Detect objects (e.g., "person", "car", "face", etc.)  
result = model.detect(image, "person")  
detections = result["objects"]  
request_id = result["request_id"]  
print(f"Found {len(detections)} people")  
print(f"Request ID: {request_id}")  
  
# Visualize the detections  
plt.figure(figsize=(10, 10))  
plt.imshow(image)  
ax = plt.gca()  
  
for obj in detections:  
    # Convert normalized coordinates to pixel values  
    x_min = obj["x_min"] * image.width  
    y_min = obj["y_min"] * image.height  
    x_max = obj["x_max"] * image.width  
    y_max = obj["y_max"] * image.height  
      
    # Calculate width and height for the rectangle  
    width = x_max - x_min  
    height = y_max - y_min  
      
    # Create a rectangle patch  
    rect = patches.Rectangle(  
        (x_min, y_min), width, height,   
        linewidth=2, edgecolor='r', facecolor='none'  
    )  
    ax.add_patch(rect)  
    plt.text(  
        x_min, y_min, "Person",   
        color='white', fontsize=12,  
        bbox=dict(facecolor='red', alpha=0.5)  
    )  
  
plt.axis('off')  
plt.savefig("output_with_detections.jpg")  
plt.show()  
```

```javascript
import { vl } from 'moondream';  
import fs from 'fs';  
  
// Initialize with API key  
const model = new vl({ apiKey: "your-api-key" });  
  
// Load an image  
const image = fs.readFileSync("path/to/image.jpg");  
  
// Detect objects  
const result = await model.detect({  
  image: image,  
  object: "person"  
});  
console.log(`Found ${result.objects.length} people`);  
console.log(`Request ID: ${result.request_id}`);  
  
// Process the detections  
result.objects.forEach((obj, index) => {  
  console.log(`Person ${index + 1}: x_min=${obj.x_min}, y_min=${obj.y_min}, x_max=${obj.x_max}, y_max=${obj.y_max}`);  
});  
```

```bash
curl -X POST https://api.moondream.ai/v1/detect \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{
    "model": "moondream3.1-9B-A2B",
    "image_url": "data:image/jpeg;base64,...",
    "object": "person"
  }'  
```

--------------------------------

### Query Skill Input

Source: https://docs.moondream.ai/batch

Example input for the query skill. Requires a question and either an image or an array of images.

```json
{"skill": "query", "image": "<base64>", "question": "What is this?", "reasoning": true}
```

--------------------------------

### Detect Skill Input

Source: https://docs.moondream.ai/batch

Example input for the detect skill. The object field specifies the item to detect.

```json
{"skill": "detect", "image": "<base64>", "object": "car"}
```

--------------------------------

### Perform Visual Question Answering with Moondream

Source: https://docs.moondream.ai/skills/query

Examples for querying images using the Moondream SDKs and direct API requests.

```python
import moondream as md  
from PIL import Image  
  
# Initialize with API key  
model = md.vl(api_key="your-api-key")  
  
# Load an image  
image = Image.open("path/to/image.jpg")  
  
# Ask a question  
result = model.query(image, "What's in this image?")  
answer = result["answer"]  
request_id = result["request_id"]  
print(f"Answer: {answer}")  
print(f"Request ID: {request_id}")  
  
# Stream the response  
stream_result = model.query(image, "What's in this image?", stream=True)  
for chunk in stream_result["chunk"]:
    print(chunk, end="", flush=True)
```

```javascript
import { vl } from 'moondream';  
import fs from 'fs';  
  
// Initialize with API key  
const model = new vl({ apiKey: "your-api-key" });  
  
// Load an image  
const image = fs.readFileSync("path/to/image.jpg");  
  
// Ask a question  
const result = await model.query({  
  image: image,  
  question: "What's in this image?"  
});  
console.log(`Answer: ${result.answer}`);  
console.log(`Request ID: ${result.request_id}`);  
  
// Stream the response  
const stream = await model.query({  
  image: image,  
  question: "What's in this image?",  
  stream: true  
});  
for await (const chunk of stream.answer) {  
  process.stdout.write(chunk);  
}
```

```bash
curl -X POST https://api.moondream.ai/v1/query \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{
    "model": "moondream3.1-9B-A2B",
    "image_url": "data:image/jpeg;base64,...",
    "question": "What is in this image?"
  }'
```

--------------------------------

### Locate objects with the Point endpoint

Source: https://docs.moondream.ai/skills/point

Examples for requesting object center points using Python, Node.js, and cURL.

```python
import moondream as md  
from PIL import Image  
import matplotlib.pyplot as plt  
  
# Initialize with API key  
model = md.vl(api_key="your-api-key")  
  
# Load an image  
image = Image.open("path/to/image.jpg")  
  
# Locate objects (e.g., "person", "car", "face", etc.)  
result = model.point(image, "face")  
points = result["points"]  
request_id = result["request_id"]  
print(f"Found {len(points)} faces")  
print(f"Request ID: {request_id}")  
  
# Visualize the points  
plt.figure(figsize=(10, 10))  
plt.imshow(image)  
  
for point in points:  
    # Convert normalized coordinates to pixel values  
    x = point["x"] * image.width  
    y = point["y"] * image.height  
      
    # Plot the point  
    plt.plot(x, y, 'ro', markersize=15, alpha=0.7)  
    plt.text(  
        x + 10, y, "Face",   
        color='white', fontsize=12,  
        bbox=dict(facecolor='red', alpha=0.5)  
    )  
  
plt.axis('off')  
plt.savefig("output_with_points.jpg")  
plt.show()  
```

```javascript
import { vl } from 'moondream';  
import fs from 'fs';  
  
// Initialize with API key  
const model = new vl({ apiKey: "your-api-key" });  
  
// Load an image  
const image = fs.readFileSync("path/to/image.jpg");  
  
// Locate objects  
const result = await model.point({  
  image: image,  
  object: "face"  
});  
console.log(`Found ${result.points.length} faces`);  
console.log(`Request ID: ${result.request_id}`);  
  
// Process the points  
result.points.forEach((point, index) => {  
  console.log(`Face ${index + 1}: x=${point.x}, y=${point.y}`);  
});  
```

```bash
curl -X POST https://api.moondream.ai/v1/point \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{  
    "model": "moondream3.1-9B-A2B",  
    "image_url": "data:image/jpeg;base64,...",  
    "object": "face"  
  }'  
```

--------------------------------

### Point Skill Input

Source: https://docs.moondream.ai/batch

Example input for the point skill. The object field specifies the item to locate.

```json
{"skill": "point", "image": "<base64>", "object": "door handle"}
```

--------------------------------

### Generate Image Captions

Source: https://docs.moondream.ai/skills/caption

Examples of generating standard, short, and streaming captions using different programming languages.

```python
import moondream as md  
from PIL import Image  
  
# Initialize with API key  
model = md.vl(api_key="your-api-key")  
  
# Load an image  
image = Image.open("path/to/image.jpg")  
  
# Generate a caption  
result = model.caption(image)  
caption = result["caption"]  
request_id = result["request_id"]  
print(f"Caption: {caption}")  
print(f"Request ID: {request_id}")  
  
# Generate a short caption  
short_result = model.caption(image, length="short")  
short_caption = short_result["caption"]  
print(f"Short Caption: {short_caption}")  
  
# Stream the response  
stream_result = model.caption(image, stream=True)  
for chunk in stream_result["chunk"]:
    print(chunk, end="", flush=True)
```

```javascript
import { vl } from 'moondream';  
import fs from 'fs';  
  
// Initialize with API key  
const model = new vl({ apiKey: "your-api-key" });  
  
// Load an image  
const image = fs.readFileSync("path/to/image.jpg");  
  
// Generate a caption  
const result = await model.caption({ image: image });  
console.log(`Caption: ${result.caption}`);  
console.log(`Request ID: ${result.request_id}`);  
  
// Generate a short caption  
const shortResult = await model.caption({  
  image: image,  
  length: "short"  
});  
console.log(`Short Caption: ${shortResult.caption}`);  
  
// Stream the response  
const stream = await model.caption({  
  image: image,  
  stream: true  
});  
for await (const chunk of stream.caption) {  
  process.stdout.write(chunk);  
}
```

```bash
# Normal caption  
curl -X POST https://api.moondream.ai/v1/caption \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{  
    "model": "moondream3.1-9B-A2B",  
    "image_url": "data:image/jpeg;base64,...",  
    "length": "normal",  
    "stream": false  
  }'  
  
# Short caption  
curl -X POST https://api.moondream.ai/v1/caption \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{  
    "model": "moondream3.1-9B-A2B",  
    "image_url": "data:image/jpeg;base64,...",  
    "length": "short",  
    "stream": false  
  }'  
  
# Streaming caption  
curl -X POST https://api.moondream.ai/v1/caption \
  -H 'Content-Type: application/json' \
  -H 'X-Moondream-Auth: YOUR_API_KEY' \
  -d '{  
    "model": "moondream3.1-9B-A2B",  
    "image_url": "data:image/jpeg;base64,...",  
    "length": "normal",  
    "stream": true  
  }'
```

--------------------------------

### Chat Completions with OpenAI SDKs and curl

Source: https://docs.moondream.ai/openai

Examples of how to interact with the Moondream chat endpoint using the official OpenAI Python and Node.js SDKs, as well as a direct curl request.

```python
from openai import OpenAI  
  
client = OpenAI(  
    api_key="YOUR_API_KEY",  
    base_url="https://api.moondream.ai/v1",  
)  
  
response = client.chat.completions.create(  
    model="moondream3.1-9B-A2B",  
    messages=[{"role": "user", "content": "What is 2 + 2?"}],  
)  
print(response.choices[0].message.content)  
```

```javascript
import OpenAI from "openai";  
  
const client = new OpenAI({  
  apiKey: "YOUR_API_KEY",  
  baseURL: "https://api.moondream.ai/v1",  
});  
  
const response = await client.chat.completions.create({  
  model: "moondream3.1-9B-A2B",  
  messages: [{ role: "user", content: "What is 2 + 2?" }],  
});  
console.log(response.choices[0].message.content);  
```

```bash
curl https://api.moondream.ai/v1/chat/completions \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{  
    "model": "moondream3.1-9B-A2B",  
    "messages": [{"role": "user", "content": "What is 2 + 2?"}]  
  }'  
```

--------------------------------

### Initialize Environment

Source: https://docs.moondream.ai/finetuning/quickstart

Import necessary libraries and define the classification question.

```python
import os  
import time  
from itertools import cycle  
  
from datasets import load_dataset  
import moondream as md  
  
QUESTION = "Is this rock, paper, or scissors? Respond with rock, paper, or scissors only."
```

--------------------------------

### Create Finetune Instance

Source: https://docs.moondream.ai/finetuning/quickstart

Initialize a new finetuning session using an API key.

```python
ft = md.ft(  
    api_key=os.environ["MOONDREAM_API_KEY"],  
    name=f"rps-query-{int(time.time())}",  
    rank=8,  
)  
print(f"Created finetune: {ft.finetune_id}")  
```

--------------------------------

### POST /v1/batch?action=mpu-create

Source: https://docs.moondream.ai/batch

Initializes a new batch upload session.

```APIDOC
## POST /v1/batch?action=mpu-create

### Description
Initializes a new batch upload session and returns identifiers required for subsequent part uploads.

### Method
POST

### Endpoint
/v1/batch?action=mpu-create

### Response
#### Success Response (200)
- **fileId** (string) - Unique identifier for the file
- **uploadId** (string) - Unique identifier for the upload session
- **key** (string) - Storage key for the batch file
```

--------------------------------

### Create or Load a Finetune

Source: https://docs.moondream.ai/finetuning/python-sdk

Initialize a new finetune or load an existing one using its ID.

```python
import moondream as md  
  
ft = md.ft(  
    api_key="your-api-key",  
    name="my-finetune",  
    rank=8,  
)  
```

```python
ft = md.ft(  
    api_key="your-api-key",  
    finetune_id="01HXYZ...",  
)  
```

--------------------------------

### Initialize and Run Local Inference

Source: https://docs.moondream.ai/running-locally

Demonstrates how to initialize the model for local execution and perform various vision tasks. Requires a valid API key and a compatible hardware environment.

```python
import moondream as md
from PIL import Image

# Initialize with local inference (NVIDIA GPU or Apple Silicon)
model = md.vl(api_key="YOUR_API_KEY", local=True)

# Load an image
image = Image.open("path/to/image.jpg")

# Generate a caption
caption = model.caption(image)["caption"]
print("Caption:", caption)

# Ask a question
answer = model.query(image, "What's in this image?")["answer"]
print("Answer:", answer)

# Detect objects
objects = model.detect(image, "person")["objects"]
for obj in objects:
    print(f"Bounds: ({obj['x_min']}, {obj['y_min']}) to ({obj['x_max']}, {obj['y_max']})")

# Locate objects
points = model.point(image, "person")["points"]
for point in points:
    print(f"Center: ({point['x']}, {point['y']})")
```

--------------------------------

### md.ft()

Source: https://docs.moondream.ai/finetuning/python-sdk

Initializes a finetune instance. Can be used to create a new finetune or load an existing one by ID.

```APIDOC
## md.ft()

### Description
Initializes a finetune instance. If creating a new finetune, provide name and rank. If loading an existing one, provide the finetune_id.

### Parameters
- **api_key** (str) - Required - Moondream API key
- **name** (str) - Required* - Unique name for new finetune
- **rank** (int) - Required* - LoRA rank (8, 16, 24, or 32)
- **finetune_id** (str) - Optional - ID of existing finetune
- **endpoint** (str) - Optional - API endpoint URL

*Required when creating a new finetune.
```

--------------------------------

### Install Moondream on JetPack 6

Source: https://docs.moondream.ai/running-locally

Installs Moondream with a NumPy version constraint to ensure compatibility with the JetPack 6 PyTorch wheel.

```bash
pip install "numpy<2" moondream  
```

--------------------------------

### GET /v1/batch/:batchId

Source: https://docs.moondream.ai/batch

Retrieves the status of a specific batch.

```APIDOC
## GET /v1/batch/:batchId

### Description
Returns the current status of a batch job.

### Method
GET

### Endpoint
/v1/batch/:batchId

### Parameters
#### Path Parameters
- **batchId** (string) - Required - The batch identifier

### Response
#### Success Response (200)
- **status** (string) - Current status (chunking, processing, completed, failed)
```

--------------------------------

### Prepare Batch Input File

Source: https://docs.moondream.ai/batch

Create a JSONL file where each line contains a JSON object specifying the image, skill, and required parameters.

```json
{"id": "img_001", "skill": "caption", "image": "<base64>", "length": "normal"}  
{"id": "img_002", "skill": "query", "image": "<base64>", "question": "What color is the car?"}  
{"id": "img_003", "skill": "detect", "image": "<base64>", "object": "person"}  
```

--------------------------------

### POST /v1/point

Source: https://docs.moondream.ai/api

Get precise center coordinates for objects.

```APIDOC
## POST /v1/point

### Description
Get precise center coordinates for objects.

### Method
POST

### Endpoint
https://api.moondream.ai/v1/point

### Request Body
- **model** (string) - Required - The model identifier
- **image_url** (string) - Required - Base64 encoded image data
- **object** (string) - Required - The object to locate

### Response
#### Success Response (200)
- **request_id** (string) - Unique identifier for the request
- **points** (array) - List of objects with x and y center coordinates
```

--------------------------------

### GET /finetunes

Source: https://docs.moondream.ai/finetuning/http-api-reference

Retrieves a list of all fine-tuning jobs associated with the organization.

```APIDOC
## GET /finetunes

### Description
List all finetunes for your organization. Results are ordered by creation time (oldest first).

### Method
GET

### Endpoint
/finetunes

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Maximum results per page (1–100), default 20
- **cursor** (string) - Optional - Pagination cursor from previous response

### Response
#### Success Response (200)
- **finetunes** (array) - List of finetune objects
- **next_cursor** (string) - Pagination cursor for the next page
- **has_more** (boolean) - Indicates if more results are available

#### Response Example
{
  "finetunes": [
    {
      "finetune_id": "01HXYZ...",
      "name": "my-finetune",
      "rank": 32,
      "created_at_ms": 1736937000000,
      "updated_at_ms": 1736937000000
    }
  ],
  "next_cursor": "eyJjIjoiMjAyNS0wMS0xNVQxMDozMDowMFoifQ==",
  "has_more": true
}
```

--------------------------------

### ft.rollouts()

Source: https://docs.moondream.ai/finetuning/python-sdk

Generates rollouts for a single request to the finetune model.

```APIDOC
## ft.rollouts()

### Description
Generates rollouts for a single request, supporting query, point, or detect skills.

### Parameters
- **skill** (str) - Required - "query", "point", or "detect"
- **image** (PIL.Image/EncodedImage) - Depends - Required for point/detect
- **question** (str) - Depends - Required for query
- **object** (str) - Depends - Required for point/detect
- **num_rollouts** (int) - Optional - Number of attempts (1-16)
- **settings** (dict) - Optional - Sampling settings
- **reasoning** (bool) - Optional - Enable reasoning (query only)
- **spatial_refs** (list) - Optional - Spatial references (query only)
- **ground_truth** (dict) - Optional - For automatic reward computation

### Response
Returns a dictionary containing the request metadata, rollouts, and rewards (if ground_truth provided).
```

--------------------------------

### GET /v1/models

Source: https://docs.moondream.ai/openai

Lists the available models for the Moondream AI service.

```APIDOC
## GET /v1/models

### Description
List the available models with the standard OpenAI models endpoint.

### Method
GET

### Endpoint
https://api.moondream.ai/v1/models

### Request Example
curl https://api.moondream.ai/v1/models \
  -H 'Authorization: Bearer YOUR_API_KEY'
```

--------------------------------

### Clone Kestrel Repository

Source: https://docs.moondream.ai/running-locally

Initial step to obtain the Triton model repository required for deployment.

```bash
git clone https://github.com/m87-labs/kestrel.git  
cd kestrel  
```

--------------------------------

### Get Batch Status

Source: https://docs.moondream.ai/batch

Retrieves the current processing status of a specific batch.

```http
GET /v1/batch/:batchId
```

--------------------------------

### GET /finetunes/:finetuneId/checkpoints

Source: https://docs.moondream.ai/finetuning/http-api-reference

Retrieve a list of saved checkpoints for a specific finetune.

```APIDOC
## GET /finetunes/:finetuneId/checkpoints

### Description
List saved checkpoints for a finetune.

### Method
GET

### Endpoint
/finetunes/:finetuneId/checkpoints

### Parameters
#### Path Parameters
- **finetuneId** (string) - Required - The identifier of the finetune.

#### Query Parameters
- **limit** (integer) - Optional - Maximum results per page (1–100), default 20
- **cursor** (string) - Optional - Pagination cursor from previous response

### Response
#### Success Response (200)
- **checkpoints** (array) - List of checkpoint objects
- **next_cursor** (string) - Pagination cursor for the next page
- **has_more** (boolean) - Whether there are more results

#### Response Example
{
  "checkpoints": [
    {
      "checkpoint_id": "01JXYZ...",
      "finetune_id": "01HXYZ...",
      "step": 100,
      "created_at_ms": 1736937000000,
      "updated_at_ms": 1736937600000
    }
  ],
  "next_cursor": "eyJzIjo1MDB9",
  "has_more": true
}
```

--------------------------------

### GET /finetunes/:finetuneId

Source: https://docs.moondream.ai/finetuning/http-api-reference

Retrieves the details of a specific finetune job by its ID.

```APIDOC
## GET /finetunes/:finetuneId

### Description
Get details for a specific finetune.

### Method
GET

### Endpoint
/finetunes/:finetuneId

### Parameters
#### Path Parameters
- **finetuneId** (string) - Required - The unique identifier of the finetune.

### Response
#### Success Response (200)
- **finetune** (object) - The finetune object containing id, name, rank, and timestamps.

#### Response Example
{
  "finetune": {
    "finetune_id": "01HXYZ...",
    "name": "my-finetune",
    "rank": 32,
    "created_at_ms": 1736937000000,
    "updated_at_ms": 1736937000000
  }
}
```

--------------------------------

### Save a finetune checkpoint

Source: https://docs.moondream.ai/finetuning/http-api-reference

Returns the checkpoint details upon successful save. Fails with 404 if the finetune does not exist.

```json
{
  "ok": true,
  "checkpoint": {
    "checkpoint_id": "01JXYZ...",
    "finetune_id": "01HXYZ...",
    "step": 100,
    "created_at_ms": 1736937000000,
    "updated_at_ms": 1736937600000
  }
}
```

--------------------------------

### Configure Moondream Model Selection

Source: https://docs.moondream.ai/running-locally

Initialize the model with local execution enabled. Specify the model parameter to switch between Moondream 3 Preview and Moondream 2.

```python
# Moondream 3 Preview (default)  
model = md.vl(api_key="YOUR_API_KEY", local=True)  
  
# Moondream 2  
model = md.vl(api_key="YOUR_API_KEY", local=True, model="moondream2")
```

--------------------------------

### md.vl()

Source: https://docs.moondream.ai/running-locally

Initializes the Moondream vision-language model for local execution.

```APIDOC
## md.vl(api_key, local, model)

### Description
Initializes the Moondream model instance. Setting `local=True` enables local execution.

### Parameters
- **api_key** (string) - Required - Your Moondream API key.
- **local** (boolean) - Required - Set to True to run the model locally.
- **model** (string) - Optional - The model identifier (e.g., "moondream2") or a finetuned model string in the format "{base_model}/{finetune_id}@{step}".
```

--------------------------------

### Caption API Response Formats

Source: https://docs.moondream.ai/skills/caption

Examples of non-streaming and streaming response structures for the caption endpoint.

```json
{
  "request_id": "2025-03-25_caption_2025-03-25-21:00:39-715d03",
  "caption": "A detailed caption describing the image..."
}
```

```json
{data: {"chunk": "A scene ", "completed": false, "request_id": "2025-03-25_caption_123456"}}
{data: {"chunk": "showing a mountain", "completed": false, "request_id": "2025-03-25_caption_123456"}}
{data: {"chunk": " landscape.", "completed": false, "request_id": "2025-03-25_caption_123456"}}
{data: {"completed": true, "chunk": "", "org_id": "a349504c-8006-54f3-8862-ba0c41d2b4d7", "request_id": "2025-03-25_caption_123456"}}
```

--------------------------------

### Query with Reasoning via Python SDK

Source: https://docs.moondream.ai/reasoning

Use the Python SDK to perform a query with reasoning enabled.

```python
import moondream as md
from PIL import Image

# Initialize with your API key
model = md.vl(api_key="YOUR_API_KEY")

# Load an image
image = Image.open("path/to/image.jpg")

# Query with reasoning enabled
result = model.query(image, "What is in this image?", reasoning=True)
print(result["answer"])
```

--------------------------------

### Execute Training Loop

Source: https://docs.moondream.ai/finetuning/quickstart

Perform RL training steps by generating rollouts and updating the model.

```python
train_iter = cycle(train_data)  
  
for i in range(20):  
    example = next(train_iter)  
  
    # Generate 4 rollouts  
    response = ft.rollouts(  
        "query",  
        image=example["image"],  
        question=QUESTION,  
        num_rollouts=4,  
        settings={"temperature": 1.0, "max_tokens": 4},  
    )  
  
    # Score: 1.0 if correct, 0.0 if not  
    rewards = [  
        float(r["output"]["answer"].strip().lower() == example["class"])  
        for r in response["rollouts"]  
    ]  
  
    # Train  
    step = ft.train_step([{
        "mode": "rl",  
        "request": response["request"],  
        "rollouts": response["rollouts"],  
        "rewards": rewards,  
    }], lr=2e-4)  
  
    print(f"step={step['step']} reward={sum(rewards)/len(rewards):.2f}")  
```

--------------------------------

### GET /v1/batch/{batch_id}

Source: https://docs.moondream.ai/batch

Poll the status of a batch job to track progress and determine when it has completed or failed.

```APIDOC
## GET /v1/batch/{batch_id}

### Description
Polls the current status and progress of a batch processing job.

### Method
GET

### Endpoint
https://api.moondream.ai/v1/batch/{batch_id}

### Parameters
#### Path Parameters
- **batch_id** (string) - Required - The unique identifier of the batch job.

### Response
#### Success Response (200)
- **id** (string) - The batch ID.
- **status** (string) - Current status (e.g., "processing", "completed", "failed").
- **model** (string) - The model version used.
- **progress** (object) - Progress metrics including total and completed counts.
- **usage** (object) - Token usage statistics (present in completed status).
- **outputs** (array) - List of result objects containing URLs (present in completed status).

#### Response Example
{
  "id": "01JQXYZ9ABCDEF123456",
  "status": "completed",
  "model": "moondream3.1-9B-A2B",
  "progress": { "total": 1000, "completed": 998, "failed": 2 },
  "usage": { "input_tokens": 1500000, "output_tokens": 50000 },
  "outputs": [
    { "index": 0, "url": "https://..." },
    { "index": 1, "url": "https://..." }
  ]
}
```

--------------------------------

### Get model ID for inference

Source: https://docs.moondream.ai/finetuning/python-sdk

Retrieves the model ID for a specific checkpoint to be used with inference endpoints.

```python
model_id = ft.model(step=100)  
# "moondream3-preview/01HXYZ...@100"  
```

--------------------------------

### POST /train_step

Source: https://docs.moondream.ai/finetuning/http-api-reference

Apply one training step using RL or SFT.

```APIDOC
## POST /train_step

### Description
Apply one training step using RL or SFT. You can mix skills across groups in the same train step.

### Method
POST

### Endpoint
/train_step

### Parameters
#### Request Body
- **finetune_id** (string) - Required - The finetune ID to train
- **groups** (array) - Required - List of training groups
- **lr** (number) - Optional - Learning rate (default: 2e-4)

### Request Example
{
  "finetune_id": "01HXYZ...",
  "groups": [
    {
      "mode": "rl",
      "request": { ... },
      "rollouts": [ ... ],
      "rewards": [0.8, 0.3, 0.6, 0.5]
    }
  ],
  "lr": 2e-4
}

### Response
#### Success Response (200)
- **step** (integer) - Training step after this request
- **applied** (boolean) - Whether the step was applied
- **kl** (number) - RL KL divergence
- **router_kl** (number) - Router KL divergence
- **grad_norm** (number) - Gradient norm
- **reward_mean** (number) - Mean reward across RL groups
- **reward_std** (number) - Reward standard deviation across RL rollouts
- **sft_loss** (number) - SFT loss when SFT groups are included

#### Response Example
{
  "step": 101,
  "applied": true,
  "kl": 0.12,
  "router_kl": 0.004,
  "grad_norm": 1.42,
  "reward_mean": 0.58,
  "reward_std": 0.19,
  "sft_loss": 1.23
}
```
Tokens:
Raw