Luxonis Software v3

https://docs.luxonis.com/software-v3/
Admin
Luxonis Software v3 is a comprehensive SDK and documentation for developing AI vision applications...
Tokens:3,633,020
Snippets:29,528
Trust Score:9.2
Update:5 days ago
Show doc for...
Tokens:
Raw
### Build and install UVC C++ example

Source: https://docs.luxonis.com/software-v3/oak-apps/self-hosted-oak-agent.md?install-toolkit=1

Steps to list devices, connect to the agent, clone the examples repository, build the app, and install it on an OAK4 device.

```bash
oakctl device list
+---+------------------+--------------------------------+----------------------------------------+------------------------+---------------+-------------------+
| # | Serial Number    | Device                         | Connection                             | OS                     | Agent Version | Setup             |
+=============================================================================================================================================================+
| 1 | 0e40028535f13370 | Raspberry Pi 5 Model B Rev 1.1 | 10.12.11.231, 10.12.102.35, 172.17.0.1 | Ubuntu 24.04.4 LTS     | 0.21.0        | Setup (unadopted) |
+---+------------------+--------------------------------+----------------------------------------+------------------------+---------------+-------------------+
| 2 | 2795412959       | Luxonis, Inc. OAK4-D R9        | 10.12.102.126, 172.23.136.130          | Luxonis OS RVC4 1.32.1 | 0.20.0        | Setup (unadopted) |
+---+------------------+--------------------------------+----------------------------------------+------------------------+---------------+-------------------+
oakctl device connection set 1 # Connect to the first device
```

```bash
git clone --depth 1 --branch main https://github.com/luxonis/oak-examples
cd oak-examples
git submodule update --init --recursive
```

```bash
cd cpp/uvc
oakctl app build .
```

```bash
oakctl app install *.oakapp -d 2
```

```bash
oakctl app list -d 2
+---+--------+---------------------------------+---------+--------------+
| E | App Id | Identifier                      | Status  | Frontend URL |
+=======================================================================+
| * | d4ed4  | com.example.cpp.uvc:3.0.0       | running |              |
+---+--------+---------------------------------+---------+--------------+
```

--------------------------------

### Run an example App in Peripheral mode on Windows

Source: https://docs.luxonis.com/software-v3?install-toolkit=1

Commands to clone the examples repository, install requirements, and run the application locally on Windows.

```bash
# Clone the repo
git clone --depth 1 --branch main https://github.com/luxonis/oak-examples.git
# Change directory to the generic-example
cd oak-examples\neural-networks\generic-example\
# Install requirements
pip install -r requirements.txt
# Run the app and follow instructions to open the web viewer
python main.py
```

--------------------------------

### Setup workspace and build from source

Source: https://docs.luxonis.com/software-v3/depthai/ros/build-from-source.md?install-toolkit=1

Creates a workspace, clones the required repositories, installs dependencies, and builds the project.

```bash
mkdir -p dai_ws/src
cd dai_ws/src
git clone https://github.com/luxonis/depthai-core.git
cd depthai-core && git submodule update --init --recursive && cd ..
git clone https://github.com/luxonis/depthai-ros.git

# checkout the release branch matching your target ROS distro
# Humble: depthai-core -> v3-humble, depthai-ros -> v3-humble
# Jazzy:  depthai-core -> v3-jazzy,  depthai-ros -> v3-jazzy
# Kilted: depthai-core -> main,      depthai-ros -> kilted

cd ..
rosdep install --from-paths src --ignore-src -r -y
source /opt/ros/$ROS_DISTRO/setup.bash
MAKEFLAGS="-j1 -l1" colcon build
source install/setup.bash
```

--------------------------------

### GPUStereo Python Pipeline Setup

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/gpu_stereo.md

Example of initializing a pipeline with GPUStereo in Python, including a check for device support.

```python
import depthai as dai

device = dai.Device()
if not device.isGpuStereoSupported():
    raise RuntimeError("GPUStereo is not supported on this device.")

with dai.Pipeline(device) as pipeline:
    cam_left = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
    cam_right = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)

    gpu = pipeline.create(dai.node.GPUStereo)
    cam_left.requestFullResolutionOutput().link(gpu.left)
    cam_right.requestFullResolutionOutput().link(gpu.right)
```

--------------------------------

### GPUStereo C++ Pipeline Setup

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/gpu_stereo.md

Example of initializing a pipeline with GPUStereo in C++, including a check for device support.

```cpp
#include <depthai/depthai.hpp>

auto device = std::make_shared<dai::Device>();
if(!device->isGpuStereoSupported()) {
    throw std::runtime_error("GPUStereo is not supported on this device.");
}

dai::Pipeline pipeline(device);
auto camLeft = pipeline.create<dai::node::Camera>();
auto camRight = pipeline.create<dai::node::Camera>();
camLeft->build(dai::CameraBoardSocket::CAM_B);
camRight->build(dai::CameraBoardSocket::CAM_C);

auto gpu = pipeline.create<dai::node::GPUStereo>();
camLeft->requestFullResolutionOutput()->link(gpu->left);
camRight->requestFullResolutionOutput()->link(gpu->right);
```

--------------------------------

### Dynamic Calibration C++ Example

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/dynamic_calibration.md

This example demonstrates setting up a pipeline with the DynamicCalibration node, starting the calibration process, and handling calibration results and coverage data.

```cpp
// examples/cpp/DynamicCalibration/calibrate.cpp
#include <chrono>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <opencv2/opencv.hpp>
#include <thread>

#include "depthai/depthai.hpp"

int main() {
    auto device = std::make_shared<dai::Device>();

    // ---------- Pipeline definition ----------
    dai::Pipeline pipeline(device);

    auto monoLeft = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_B);
    auto monoRight = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_C);

    auto* leftOut = monoLeft->requestFullResolutionOutput();
    auto* rightOut = monoRight->requestFullResolutionOutput();

    // Dynamic-calibration node
    auto dynCalib = pipeline.create<dai::node::DynamicCalibration>();
    leftOut->link(dynCalib->left);
    rightOut->link(dynCalib->right);

    auto stereo = pipeline.create<dai::node::StereoDepth>();
    leftOut->link(stereo->left);
    rightOut->link(stereo->right);

    // In-pipeline host queues
    auto leftSyncedQueue = stereo->syncedLeft.createOutputQueue();
    auto rightSyncedQueue = stereo->syncedRight.createOutputQueue();
    auto depthQueue = stereo->depth.createOutputQueue();

    auto dynCalibOutQ = dynCalib->calibrationOutput.createOutputQueue();
    auto dynCoverageOutQ = dynCalib->coverageOutput.createOutputQueue();

    auto dynCalibInputControl = dynCalib->inputControl.createInputQueue();

    device->setCalibration(device->getCalibration());

    pipeline.start();
    std::this_thread::sleep_for(std::chrono::seconds(1));  // wait for autoexposure to settle

    using DCC = dai::DynamicCalibrationControl;
    // Optionally set performance mode:
    dynCalibInputControl->send(DCC::setPerformanceMode(DCC::PerformanceMode::OPTIMIZE_PERFORMANCE));

    // Start calibration (optimize performance)
    dynCalibInputControl->send(DCC::startCalibration());

    while(pipeline.isRunning()) {
        auto leftSynced = leftSyncedQueue->get<dai::ImgFrame>();
        auto rightSynced = rightSyncedQueue->get<dai::ImgFrame>();
        auto depth = depthQueue->get<dai::ImgFrame>();

        cv::imshow("left", leftSynced->getCvFrame());
        cv::imshow("right", rightSynced->getCvFrame());

        cv::imshow("depth", dai::utility::colorizeDepthFrame(*depth).getCvFrame());

        // Coverage (non-blocking)
        if(auto coverageMsg = dynCoverageOutQ->tryGet<dai::CoverageData>()) {
            std::cout << "2D Spatial Coverage = " << coverageMsg->meanCoverage << "  / 100 [%]\n";
            std::cout << "Data Acquired       = " << coverageMsg->dataAcquired << "  / 100 [%]\n";
        }

        // Calibration result (non-blocking)
        if(auto dynCalibrationResult = dynCalibOutQ->tryGet<dai::DynamicCalibrationResult>()) {
            std::cout << "Dynamic calibration status: " << dynCalibrationResult->info << std::endl;

            if(dynCalibrationResult->calibrationData) {
                std::cout << "Successfully calibrated." << std::endl;

                // Apply the produced calibration
                const auto& newCalib = dynCalibrationResult->calibrationData->newCalibration;
                dynCalibInputControl->send(DCC::applyCalibration(newCalib));

                // Print quality deltas
                const auto& q = dynCalibrationResult->calibrationData->calibrationDifference;

                if(!q.pairwiseRotationDifference.empty()) {
                    const auto& pairwiseRotation = q.pairwiseRotationDifference.begin()->second;
                    float rotDiff = 0.0f;
                    for(const auto axis : pairwiseRotation) rotDiff += axis * axis;
                    rotDiff = std::sqrt(rotDiff);
                    std::cout << "Rotation difference: " << rotDiff << " deg\n";
                }
                std::cout << "Mean Sampson error achievable = " << q.sampsonErrorNew << " px\n";
                std::cout << "Mean Sampson error current    = " << q.sampsonErrorCurrent << " px\n";
            }

            const bool invalidTranslation =
                dynCalibrationResult->info == "A multisensor pairwise recalibration changed the translation direction by 15 degrees or more";
            if(dynCalibrationResult->calibrationData || invalidTranslation) {
                dynCalibInputControl->send(DCC::resetData());
                dynCalibInputControl->send(DCC::startCalibration());
            }
        }

        int key = cv::waitKey(1);
        if(key == 'q') break;
    }

    return 0;
}
```

--------------------------------

### Install and Setup Template App

Source: https://docs.luxonis.com/software-v3

Commands to install the DepthAI core package, clone the template repository, and install necessary dependencies.

```bash
# Install core packages
pip install depthai --force-reinstall
# Clone the template app
git clone https://github.com/luxonis/oak-template.git
# Change directory to the template app
cd oak-template
# Install requirements
pip install -r requirements.txt
```

--------------------------------

### Python: Initialize BenchmarkOut

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/benchmark_out.md

Basic setup for the BenchmarkOut node in Python, including setting the FPS and sending an initial frame to start the stream.

```python
import depthai as dai

pipeline = dai.Pipeline()
benchmarkOut = pipeline.create(dai.node.BenchmarkOut)

# Choose whether the node runs on the host or on the device
benchmarkOut.setRunOnHost(False)
# Set the desired output frame rate (FPS)
benchmarkOut.setFps(30)

# To initialize, send a single input message (e.g., an ImgFrame)
inputQueue = benchmarkOut.input.createInputQueue()
# Create and configure your initial frame as needed
initialFrame = dai.ImgFrame()
inputQueue.send(initialFrame)
```

--------------------------------

### Porting a simple RGB stream from v2 to v3

Source: https://docs.luxonis.com/software-v3/depthai/tutorials/v2-vs-v3.md?install-toolkit=1

This example demonstrates the transition from v2 to v3 by showing the removal of explicit XLink nodes and the device context, replaced by direct output queue creation and the pipeline start method.

```python
#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)

# ORIG – explicit XLink removed in v3
# xoutVideo = pipeline.create(dai.node.XLinkOut)
# xoutVideo.setStreamName("video")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setVideoSize(1920, 1080)

# Linking
# ORIG
# camRgb.video.link(xoutVideo.input)
# NEW – output queue straight from the node
videoQueue = camRgb.video.createOutputQueue()

# ORIG – entire `with dai.Device` block removed
# with dai.Device(pipeline) as device:
#   video = device.getOutputQueue(name="video", maxSize=1, blocking=False)
#   while True:
# NEW – start the pipeline
pipeline.start()
while pipeline.isRunning():
    videoIn = videoQueue.get()  # blocking
    cv2.imshow("video", videoIn.getCvFrame())
    if cv2.waitKey(1) == ord('q'):
        break
```

--------------------------------

### Install and clone the template application

Source: https://docs.luxonis.com/software-v3?install-toolkit=1

Commands to install core packages, clone the template repository, and install necessary requirements.

```bash
1# Install core packages
2pip install depthai --force-reinstall
3# Clone the template app
4git clone https://github.com/luxonis/oak-template.git
5# Change directory to the template app
6cd oak-template
7# Install requirements
8pip install -r requirements.txt
```

--------------------------------

### GracefulInterruptCallback.setup

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/graceful_interrupt.md?install-toolkit=1

Installs the signal handler when a fit starts. Lightning calls this hook at the start of every stage. For 'fit', it saves the current handlers of SIGINT and SIGTERM and installs its own handler.

```APIDOC
## GracefulInterruptCallback.setup

### Description
Installs the signal handler when a fit starts. Lightning calls this hook at the start of every stage. The hook stores `trainer` for every stage, so that the handler can save a checkpoint and stop the trainer. For a stage other than `"fit"`, the hook does nothing else. For `"fit"`, in the process that created the callback, the hook saves the current handlers of `SIGINT` and `SIGTERM` and installs its own handler. In any process, the hook then logs `Added GracefulInterrupt callback` at the `INFO` level.

### Method
setup

### Signature
```python
def setup(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, stage: str | None = None):
```

### Parameters
- **trainer** (pl.Trainer) - Required - The trainer to save and stop on an interrupt.
- **pl_module** (lxt.LuxonisLightningModule) - Required - The model. Unused.
- **stage** (str | None) - Optional - The stage that starts, for example `"fit"`.
```

--------------------------------

### Counter example: starting at 5

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/utils/general.md?install-toolkit=1

Shows how Counter returns sequential integers starting from a given value. The example creates a counter with start=5 and calls it three times, producing (5, 6, 7).

```pycon
>>> counter = Counter(start=5)
>>> counter(), counter(), counter()
(5, 6, 7)
```

--------------------------------

### Install the Luxonis Toolkit

Source: https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/offline?install-toolkit=1

Installs the Luxonis toolkit from the marketplace. Run after adding the marketplace.

```text
/plugin install luxonis@luxonis
```

--------------------------------

### LuxonisFileSystem.__init__

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/utils/filesystem.md

Initializes a LuxonisFileSystem instance with the given path and configuration options.

```APIDOC
## LuxonisFileSystem.__init__

### Description
Initializes a LuxonisFileSystem instance. The backend is chosen by the protocol of the URL (file://, s3://, gcs://, mlflow://). A path with no protocol is treated as local. For MLflow, the artifact path is optional and defaults to the run's artifact root. A bare mlflow:// binds to the active run if allowed.

### Method
Constructor

### Signature
```python
def __init__(path: str, allow_active_mlflow_run: bool | None = False, allow_local: bool | None = True, cache_storage: str | None = None, put_file_plugin: str | None = None, tracking_uri: str | None = None):
```

### Parameters
- **path** (str) - Required - Input path consisting of a protocol and path, or only a path for local files. MLflow paths are of the form mlflow://<experiment_id>/<run_id>/<artifact_path>, where the artifact path is optional, or a bare mlflow:// to use the active run.
- **allow_active_mlflow_run** (bool | None) - Optional - Whether operations are allowed on the active MLflow run. Defaults to False.
- **allow_local** (bool | None) - Optional - Whether operations are allowed on the local file system. Defaults to True.
- **cache_storage** (str | None) - Optional - Path to an fsspec filecache directory, used only by the s3:// and gcs:// backends. No cache is used if not set, and it is ignored, with a warning, for file:// and mlflow://.
- **put_file_plugin** (str | None) - Optional - Name of a registered function in PUT_FILE_REGISTRY to use instead of LuxonisFileSystem.put_file. The registered function must conform to the PutFile protocol.
- **tracking_uri** (str | None) - Optional - MLflow tracking URI to use. If omitted, MLFLOW_TRACKING_URI from the environment is used.

### Raises
- **ValueError** - If the protocol is not supported; if the protocol is "file" but local files are not allowed; if the protocol is "mlflow" but no MLflow path is specified and using the active MLflow run is not allowed; if the protocol is "mlflow" but there is no "MLFLOW_TRACKING_URI" in environment variables.

### Example
```pycon
fs = LuxonisFileSystem(
    "mlflow://12/abc123/checkpoints",
    tracking_uri="http://localhost:5000",
)
```
```

--------------------------------

### FOMOHead Usage Example

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/fomo_head.md

Example demonstrating how to initialize the FOMOHead and perform a forward pass to get heatmap output.

```pycon
>>> import torch
>>> from torch import Size
>>> from luxonis_train.nodes import FOMOHead
>>> sizes = [Size([1, 8, 64, 64]), Size([1, 16, 32, 32])]
>>> head = FOMOHead(
...     n_classes=2,
...     input_shapes=[{"features": sizes}],
...     original_in_shape=Size([3, 128, 128]),
... )
>>> head.in_channels
16
>>> head(torch.zeros(1, 16, 32, 32))["heatmap"].shape
torch.Size([1, 2, 32, 32])
```

--------------------------------

### Run an example App in Peripheral mode on Linux/MacOS

Source: https://docs.luxonis.com/software-v3?install-toolkit=1

Commands to clone the examples repository, set up a Python virtual environment, and run the application locally.

```bash
# Clone the repo
git clone --depth 1 --branch main https://github.com/luxonis/oak-examples.git
# Change directory to the generic-example
cd oak-examples/neural-networks/generic-example/
# Create and enter virtual python environment
python3 -m venv venv
source venv/bin/activate
# Install requirements into the environment
pip install -r requirements.txt
# Run the app and follow instructions to open the web viewer
python3 main.py
```

--------------------------------

### Initialize and Run LuxonisModel

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/core/core.md

Demonstrates initializing a LuxonisModel with a configuration file and overrides, followed by training, testing, and exporting the model.

```python
from luxonis_train import LuxonisModel

model = LuxonisModel("config.yaml", opts={"trainer.epochs": 10})
model.train()
model.test(view="test")
model.export()
```

--------------------------------

### console

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/luxonis_progress_bar.md

The rich console that the bar prints to. The Lightning RichProgressBar base class creates the console when a stage starts and the bar is enabled. A bar that is disabled when the stage starts gets no console.

```APIDOC
## console

### Description
The rich console that the bar prints to. The Lightning RichProgressBar base class creates the console when a stage starts and the bar is enabled. A bar that is disabled when the stage starts gets no console.

### Attribute
console

### Raises
- **RuntimeError** - When the console does not exist yet. The message asks the user to set rich_logging to False in the config.
```

--------------------------------

### Deploy an example App in Standalone mode

Source: https://docs.luxonis.com/software-v3?install-toolkit=1

Commands to connect to an OAK4 device, clone the examples repository, and run an application using oakctl.

```bash
# Connect to the device
oakctl connect
# Clone the repo
git clone --depth 1 --branch main https://github.com/luxonis/oak-examples.git
# Change directory to the generic-example
cd oak-examples/neural-networks/generic-example/
# Run the app and follow instructions to open the web viewer
oakctl app run .
```

--------------------------------

### ConfigItem example

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/typing.md

Shows how to instantiate a ConfigItem and access its name and params attributes.

```pycon
>>> ConfigItem(name="Resize", params={"height": 256}).name
'Resize'
>>> ConfigItem(name="Normalize").params
{}
```

--------------------------------

### setup

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/gpu_stats_monitor.md

Checks the trainer and finds the GPUs to query. Called by Lightning at the start of every stage.

```APIDOC
## setup

### Description
Check the trainer and find the GPUs to query. Lightning calls this hook at the start of every stage. The hook stores the sorted, unique device indices of trainer. It maps each index to a physical GPU ID through the CUDA_VISIBLE_DEVICES environment variable. When the variable is not set, each GPU ID is equal to its index.

### Method
setup

### Signature
```python
def setup(trainer: pl.Trainer, pl_module: pl.LightningModule, stage: str | None = None):
```

### Parameters
- **trainer** (pl.Trainer): The trainer. It must have a logger.
- **pl_module** (pl.LightningModule): The model. Unused.
- **stage** (str | None): The stage that starts. Unused.

### Raises
- **MisconfigurationException**: When trainer has no logger, or when CUDA is not available.
```

--------------------------------

### GracefulInterruptCallback.setup

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/callbacks/graceful_interrupt.md

Installs the signal handler when a fit starts. Lightning calls this hook at the start of every stage. The hook stores `trainer` for every stage, so that the handler can save a checkpoint and stop the trainer.

```APIDOC
## GracefulInterruptCallback.setup

### Description
Installs the signal handler when a fit starts. Lightning calls this hook at the start of every stage. The hook stores `trainer` for every stage, so that the handler can save a checkpoint and stop the trainer. For a stage other than `"fit"`, the hook does nothing else.

### Method
Hook

### Signature
```python
def setup(trainer: pl.Trainer, pl_module: lxt.LuxonisLightningModule, stage: str | None = None):
```

### Parameters
- **trainer** (pl.Trainer) - Required - The trainer to save and stop on an interrupt.
- **pl_module** (lxt.LuxonisLightningModule) - Required - The model. Unused.
- **stage** (str | None) - Optional - The stage that starts, for example `"fit"`.
```

--------------------------------

### Example configuration file

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/training.md

A sample YAML configuration file defining model, loader, trainer, and tracker settings.

```yaml
model:
  name: my_detector
  predefined_model:
    name: DetectionModel
    variant: light

loader:
  params:
    dataset_name: my_dataset

trainer:
  batch_size: 8
  epochs: 100
  n_workers: 4
  validation_interval: 5
  preprocessing:
    train_image_size: [384, 512]
    keep_aspect_ratio: true
    normalize:
      active: true
      params:
        mean: [0.0, 0.0, 0.0]
        std: [1.0, 1.0, 1.0]
    augmentations:
      - name: HorizontalFlip
        params:
          p: 0.5
  optimizer:
    name: Adam
    params:
      lr: 0.001
  scheduler:
    name: ConstantLR

tracker:
  project_name: detection
  save_directory: output
  is_tensorboard: true
```

--------------------------------

### Initialize LuxonisLoader with filtering and update options

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-loader.md

Example of initializing the LuxonisLoader with specific task filtering, categorical metadata preservation, and remote media update settings.

```python
loader = LuxonisLoader(
    dataset,
    view="train",
    filter_task_names=["detection"],
    exclude_empty_annotations=True,
    keep_categorical_as_strings=True,
    update_mode="missing",
)
```

--------------------------------

### Accessing the LuxonisFileSystem abstraction

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/utils.md

Demonstrates how to import and initialize the LuxonisFileSystem to perform directory operations.

```python
from luxonis_ml.utils import LuxonisFileSystem

fs = LuxonisFileSystem("file:///tmp/dataset")
files = fs.walk_dir(".")
```

--------------------------------

### Python DynamicCalibration pipeline example

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/dynamic_calibration.md

Run this example by following the README on Github. It sets up a DepthAI pipeline with two cameras, links them to the DynamicCalibration node, and processes calibration and coverage outputs. The example sends control commands to set performance mode, start calibration, and apply calibration results when available.

```python
import depthai as dai
import numpy as np
import time
import cv2

# ---------- Pipeline definition ----------
with dai.Pipeline() as pipeline:
    # Cameras
    monoLeft  = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
    monoRight = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)

    # Full-res NV12 outputs
    monoLeftOut  = monoLeft.requestFullResolutionOutput()
    monoRightOut = monoRight.requestFullResolutionOutput()

    # Initialize the DynamicCalibration node
    dynCalib = pipeline.create(dai.node.DynamicCalibration)

    # Link the cameras to the DynamicCalibration
    monoLeftOut.link(dynCalib.left)
    monoRightOut.link(dynCalib.right)

    # Stereo (for depth + synced previews)
    stereo = pipeline.create(dai.node.StereoDepth)
    monoLeftOut.link(stereo.left)
    monoRightOut.link(stereo.right)

    # Output queues
    syncedLeftQueue  = stereo.syncedLeft.createOutputQueue()
    syncedRightQueue = stereo.syncedRight.createOutputQueue()
    depthQueue       = stereo.depth.createOutputQueue()

    # Initialize the command output queues for calibration and coverage
    dynCalibCalibrationQueue = dynCalib.calibrationOutput.createOutputQueue()
    dynCalibCoverageQueue    = dynCalib.coverageOutput.createOutputQueue()

    # Initialize the command input queue
    dynCalibInputControl = dynCalib.inputControl.createInputQueue()

    device = pipeline.getDefaultDevice()
    device.setCalibration(device.getCalibration())

    pipeline.start()
    time.sleep(1) # wait for auto exposure to settle

    # Set performance mode
    dynCalibInputControl.send(
        dai.DynamicCalibrationControl.setPerformanceMode(
            dai.DynamicCalibrationControl.PerformanceMode.OPTIMIZE_PERFORMANCE
        )
    )

    # Start periodic calibration
    dynCalibInputControl.send(
        dai.DynamicCalibrationControl.startCalibration()
    )

    while pipeline.isRunning():
        leftSynced  = syncedLeftQueue.get()
        rightSynced = syncedRightQueue.get()
        depth = depthQueue.get()

        cv2.imshow("left", leftSynced.getCvFrame())
        cv2.imshow("right", rightSynced.getCvFrame())

        colorizedDepth = dai.utility.colorizeDepthFrame(depth).getCvFrame()
        cv2.imshow("depth", colorizedDepth)

        # --- Coverage (non-blocking) ---
        coverage = dynCalibCoverageQueue.tryGet()
        if coverage is not None:
            print(f"2D Spatial Coverage = {coverage.meanCoverage} / 100 [%]")
            print(f"Data Acquired       = {coverage.dataAcquired} / 100 [%]")

        # --- Calibration result (non-blocking) ---
        dynCalibrationResult = dynCalibCalibrationQueue.tryGet()
        calibrationData = dynCalibrationResult.calibrationData if dynCalibrationResult is not None else None

        if dynCalibrationResult is not None:
            print(f"Dynamic calibration status: {dynCalibrationResult.info}")

        # --- Apply calibration if available, then reset+continue after a terminal result ---
        if calibrationData:
            print("Successfully calibrated")
            # Apply to device
            dynCalibInputControl.send(
                dai.DynamicCalibrationControl.applyCalibration(calibrationData.newCalibration)
            )

            q = calibrationData.calibrationDifference
            if q.pairwiseRotationDifference:
                pairwiseRotation = next(iter(q.pairwiseRotationDifference.values()))
                rotDiff = float(np.sqrt(sum(axis * axis for axis in pairwiseRotation)))
                print(f"Rotation difference: || r_current - r_new || = {rotDiff:.2f} deg")
            print(f"Mean Sampson error achievable = {q.sampsonErrorNew:.3f} px")
            print(f"Mean Sampson error current    = {q.sampsonErrorCurrent:.3f} px")

        invalidTranslation = (
            dynCalibrationResult is not None
            and dynCalibrationResult.info
            == "A multisensor pairwise recalibration changed the translation direction by 15 degrees or more"
        )
        if calibrationData or invalidTranslation:
            dynCalibInputControl.send(
                dai.DynamicCalibrationControl.resetData()
            )
            dynCalibInputControl.send(
                dai.DynamicCalibrationControl.startCalibration()
            )

        key = cv2.waitKey(1)
        if key == ord('q'):
            pipeline.stop()
            break

```

--------------------------------

### EfficientBBoxHead Initialization Example

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/base_detection_head.md

Demonstrates initializing an EfficientBBoxHead with two scales, reading from the last two of three provided feature maps.

```pycon
>>> from torch import Size
>>> from luxonis_train.nodes import EfficientBBoxHead
>>> sizes = [
...     Size([1, 4, 64, 64]),
...     Size([1, 8, 32, 32]),
...     Size([1, 16, 16, 16]),
... ]
>>> head = EfficientBBoxHead(
...     n_heads=2,
...     n_classes=3,
...     input_shapes=[{"features": sizes}],
...     original_in_shape=Size([3, 256, 256]),
... )
>>> head.attach_index, head.in_channels, head.stride.tolist()
((-3, -1), [8, 16], [8, 16])
```

--------------------------------

### Retrieve AnomalyDetectionModel variants

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/config/predefined_models/anomaly_detection/v1/model

Method signature and interactive example for getting the default variant name and available model configurations.

```python
def get_variants() -> tuple[str, dict[str, Params]]:
```

```pycon
>>> default, variants = AnomalyDetectionModel.get_variants()
>>> default
'light'
>>> variants["heavy"]
{'backbone_variant': 'l', 'head_variant': 'l'}
```

--------------------------------

### C++ PointCloud Visualizer Example

Source: https://docs.luxonis.com/software-v3/depthai/examples/pointcloud/point_cloud_visualizer.md

This example demonstrates how to set up a pipeline to generate and visualize a colorized point cloud. After running the code, access the visualizer at http://localhost:8082 to view the live output.

```cpp
/**
 * PointCloud Visualizer
 *
 * Live 3D visualization of colorized stereo-depth point clouds using the
 * built-in RemoteConnection (foxglove) visualizer.
 *
 * Run the example, then open the visualizer in a browser (default
 * http://localhost:8082) to see the live point cloud.
 * Press 'q' in the viewer or Ctrl-C in the terminal to quit.
 */

#include <csignal>
#include <iostream>

#include "depthai/depthai.hpp"
#include "depthai/remote_connection/RemoteConnection.hpp"

// ---------------------------------------------------------------------------
static volatile std::sig_atomic_t isRunning{1};
void signalHandler(int) {
    isRunning = 0;
}

// ---------------------------------------------------------------------------
int main() {
    std::signal(SIGINT, signalHandler);

    std::cout << "PointCloud Visualizer\n"
              << "=====================\n"
              << "Connecting to device...\n";

    dai::RemoteConnection remoteConnector;
    dai::Pipeline pipeline;

    auto device = pipeline.getDefaultDevice();
    std::cout << "Device: " << device->getDeviceName() << "  (ID: " << device->getDeviceId() << ")\n\n";

    const auto size = std::make_pair(640, 400);

    // ── Cameras ──────────────────────────────────────────────────────
    auto colorSockets = device->getConnectedCameras(dai::CameraSensorType::COLOR);
    auto colorSocket = colorSockets.empty() ? dai::CameraBoardSocket::CAM_A : colorSockets.front();
    auto color = pipeline.create<dai::node::Camera>();
    color->build(colorSocket);

    // ── Align depth to color camera ──────────────────────────────────
    auto colorOut = color->requestOutput(size, dai::ImgFrame::Type::RGB888i, dai::ImgResizeMode::CROP, std::nullopt, true);

    auto depth = pipeline.create<dai::node::Depth>();
    depth->build(dai::node::Depth::Algorithm::AUTO, std::nullopt, std::make_pair(640u, 400u));
    depth->setAlignTo(*colorOut);

    // ── PointCloud node ──────────────────────────────────────────────
    auto pc = pipeline.create<dai::node::PointCloud>();
    pc->setRunOnHost(true);
    depth->depth().link(pc->inputDepth);
    colorOut->link(pc->getColorInput());

    // Publish the point cloud to the remote visualizer
    remoteConnector.addTopic("pcl", pc->outputPointCloud);

    pipeline.start();
    remoteConnector.registerPipeline(pipeline);

    device->setIrLaserDotProjectorIntensity(0.7);

    std::cout << "Pipeline started.\n"
              << "Open the visualizer at http://localhost:8082 to see the point cloud.\n"
              << "Press 'q' in the viewer or Ctrl-C to quit.\n";

    while(isRunning != 0 && pipeline.isRunning()) {
        int key = remoteConnector.waitKey(1);
        if(key == 'q') {
            std::cout << "Got 'q' key from the remote connection.\n";
            break;
        }
    }

    std::cout << "Done.\n";
    return 0;
}
```

--------------------------------

### setup(stage: str)

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/luxonis_lightning.md

Makes the first epoch validate when the config asks for it. Lightning calls it at the start of every stage.

```APIDOC
## setup(stage: str)

### Description
Makes the first epoch validate when the config asks for it. Lightning calls it at the start of every stage. The method acts only when all of these hold: the stage is `fit`, `cfg.trainer.run_validation_after_first_epoch` is on, the current epoch is 0, and `trainer.check_val_every_n_epoch` is set and above 1. It acts once per run.

### Method
setup

### Parameters
- **stage** (`str`) - Required - The stage that starts: `"fit"`, `"validate"`, `"test"`, or `"predict"`.

### Returns
None
```

--------------------------------

### Example oakapp.toml configuration

Source: https://docs.luxonis.com/software-v3/oak-apps/configuration.md

A complete example of an oakapp.toml file showing required metadata, entrypoint, and optional container preparation steps.

```toml
# (Required) App Identifier
identifier = "com.luxonis.python_demo"
# (Required) App Entrypoint
entrypoint = ["bash", "-c", "python3 /app/main.py"]

# (Optional) Prepare container commands
# Here is the place where you can install all the dependencies that are needed at run-time
prepare_container = [
    { type = "COPY", source = "requirements.txt", target = "requirements.txt" },
    { type = "RUN", command = "apt-get update" },
    { type = "RUN", command = "apt-get install -y python3-pip" },
    { type = "RUN", command = "pip3 install -r /app/requirements.txt --break-system-packages" },
]

# (Optional) Prepare build dependencies
# Here is the place where you can install all the dependencies that are needed at build-time
prepare_build_container = [
    # Example: npm, gcc, ...
]

# (Optional) Additional commands after all the app files are copied to the container
build_steps = []
```

--------------------------------

### Initialize and run EfficientKeypointBBoxHead

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/efficient_keypoint_bbox_head.md

Example showing how to initialize the head and process input tensors to get raw keypoint output shapes.

```pycon
>>> import torch
>>> from torch import Size
>>> from luxonis_train.nodes import EfficientKeypointBBoxHead
>>> sizes = [Size([1, 8, 32, 32]), Size([1, 16, 16, 16])]
>>> head = EfficientKeypointBBoxHead(
...     n_heads=2,
...     n_classes=3,
...     n_keypoints=5,
...     input_shapes=[{"features": sizes}],
...     original_in_shape=Size([3, 256, 256]),
... )
>>> out = head([torch.zeros(size) for size in sizes])
>>> out["keypoints_raw"].shape
torch.Size([1, 1280, 15])
```

--------------------------------

### configure_callbacks

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/lightning/luxonis_lightning.md

Builds the callbacks that every run gets. Lightning calls it at the start of every fit, validate, test, or predict call.

```APIDOC
## configure_callbacks

### Description
Build the callbacks that every run gets. Lightning calls it at the start of every fit, validate, test, or predict call of the trainer.

### Method
configure_callbacks

### Signature
```python
def configure_callbacks(self) -> list[pl.Callback]:
```

### Returns
- **list[pl.Callback]** - The callbacks, in this order:
  1. TrainingManager
  2. LuxonisModelSummary
  3. ModelCheckpoint on val/loss, saved in save_dir / "min_val_loss"
  4. AIMETCallback, when cfg.exporter.aimet.active is on
  5. ModelCheckpoint on the main metric, saved in save_dir / "best_val_metric", when the config has one
  6. Every active callback of cfg.trainer.callbacks
  7. GradientAccumulationScheduler, when cfg.trainer.accumulate_grad_batches is set and no callback of that class is present yet
```

--------------------------------

### Install Skills Only

Source: https://docs.luxonis.com/software-v3/agent-tools/agent-skills.md

Alternative installation methods that provide skill instructions without the Luxonis MCP server.

```bash
npx skills@latest add luxonis/skills
```

```text
luxonis/skills
```

--------------------------------

### Copy prompt for agent installation

Source: https://docs.luxonis.com/software-v3/ai-inference/conversion/rvc-conversion/online?install-toolkit=1

Copy this prompt and paste it into your coding agent. It will follow the appropriate setup instructions and verify the connection.

```text
Copy this prompt and paste it into your coding agent. It will follow the appropriate setup instructions and verify the connection.
```

--------------------------------

### Start Llama.cpp server with CLI

Source: https://docs.luxonis.com/software-v3/ai-inference/inference/llamacpp.md

Starts a vision-capable server using local model files with specific hardware offloading settings for the Hexagon backend.

```bash
llama-server \
    --model /app/models/model.gguf \
    --mmproj /app/models/projector.gguf \
    --device HTP0 --mmproj-device HTP0 -ngl 99 \
    --host 127.0.0.1 --port 8081 \
    --ctx-size 2048 --parallel 1 \
    --threads 4 --threads-batch 4 \
    --batch-size 256 --ubatch-size 256 \
    --jinja
```

--------------------------------

### Initialize and use LuxonisTracker with multiple backends

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/tracker.md

Demonstrates how to instantiate the LuxonisTracker with TensorBoard and Weights & Biases enabled, log hyperparameters and metrics, and upload an artifact.

```python
from luxonis_ml.tracker import LuxonisTracker

tracker = LuxonisTracker(
    project_name="my-project",
    run_name="baseline",
    is_tensorboard=True,
    is_wandb=True,
    wandb_entity="my-entity",
)

tracker.log_hyperparams({"lr": 1e-3, "batch_size": 32})
tracker.log_metrics({"acc": 0.92, "loss": 0.18}, step=1)
tracker.upload_artifact("model.onnx", name="model", typ="model")
tracker.close()
```

--------------------------------

### C++: Initialize BenchmarkOut

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/benchmark_out.md

Basic setup for the BenchmarkOut node in C++, including setting the FPS and sending an initial frame to start the stream.

```cpp
#include "depthai/depthai.hpp"

dai::Pipeline pipeline;
auto benchmarkOut = pipeline.create<dai::node::BenchmarkOut>();

// Choose whether to run on host or device and set the output FPS
benchmarkOut->setRunOnHost(false);
benchmarkOut->setFps(30);

// To initialize the node, send a single frame
auto inputQueue = benchmarkOut->input.createInputQueue();
dai::ImgFrame initialFrame;
// Initialize initialFrame as needed...
inputQueue->send(initialFrame);
```

--------------------------------

### Initialize LuxonisLoader

Source: https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-ml/luxonis-ml-api-reference/data/loaders/luxonis_loader.md

Example of initializing the LuxonisLoader with a dataset, view, augmentation configuration, and target dimensions.

```python
loader = LuxonisLoader(
    dataset,
    view="train",
    augmentation_config=augmentation_config,
    height=640,
    width=640,
)
```

--------------------------------

### Tiling.build

Source: https://docs.luxonis.com/software-v3/ai-inference/inference/depthai-nodes/depthai-nodes-api-reference/node/tiling.md?install-toolkit=1

Stores the initial configuration for emission when the pipeline starts.

```APIDOC
## Tiling.build

### Description
Store the initial configuration for emission when the pipeline starts.

### Method
build

### Signature
```python
def build(overlap: float, gridSize: tuple[int, int], canvasShape: tuple[int, int], resizeShape: tuple[int, int], resizeMode: dai.ImageManipConfig.ResizeMode, globalDetection: bool = False, gridMatrix: np.ndarray | list | None = None) -> Tiling:
```

### Parameters
- **overlap** (float) - Required - Fractional overlap between adjacent cells, in [0, 1).
- **gridSize** (tuple[int, int]) - Required - Grid dimensions as (columns, rows).
- **canvasShape** (tuple[int, int]) - Required - Source canvas size as (width, height) pixels.
- **resizeShape** (tuple[int, int]) - Required - Output crop size as (width, height) pixels.
- **resizeMode** (dai.ImageManipConfig.ResizeMode) - Required - ImageManip resizing policy for each crop.
- **globalDetection** (bool) - Optional - Prepend a crop covering the entire canvas when true.
- **gridMatrix** (np.ndarray | list | None) - Optional - Optional row-major grid labels. Four-connected cells with equal labels are merged into their enclosing rectangle.

### Returns
- **Tiling**: This node. Crop generation and validation occur when it runs.

```

--------------------------------

### List installed apps with oakctl app list

Source: https://docs.luxonis.com/software-v3/oak-apps/oakctl.md

Shows installed apps with columns for enabled status, App ID, Identifier, Status, and Frontend URL. App IDs are truncated for readability; use --long to show full IDs. App states include installing, building, ready, starting, running, stopping, stopped, deleting, deleted, and unknown.

```bash
$ oakctl app list
+---+--------+---------------------------+---------+-------------------+
| E | App Id | Identifier                | Status  | Frontend URL      |
+====================================================================================================+
| * | f8da5  | com.luxonis.default:1.3.8 | running | 10.0.1.24:9000    |
+---+--------+---------------------------+---------+-------------------+
```

--------------------------------

### Configure DynamicCalibration for 3 sensors

Source: https://docs.luxonis.com/software-v3/depthai/depthai-components/host_nodes/dynamic_calibration.md

This example demonstrates connecting three camera sensors to the DynamicCalibration node, setting performance mode, and starting the calibration process.

```python
import depthai as dai

with dai.Pipeline() as pipeline:
    rgb = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A)
    monoLeft = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
    monoRight = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)

    rgbOut = rgb.requestOutput((1280, 800), fps=30)
    monoLeftOut = monoLeft.requestFullResolutionOutput()
    monoRightOut = monoRight.requestFullResolutionOutput()

    dynCalib = pipeline.create(dai.node.DynamicCalibration)
    rgbOut.link(dynCalib.rgb)
    monoLeftOut.link(dynCalib.left)
    monoRightOut.link(dynCalib.right)

    coverageQueue = dynCalib.coverageOutput.createOutputQueue()
    calibrationQueue = dynCalib.calibrationOutput.createOutputQueue()
    inputControl = dynCalib.inputControl.createInputQueue()

    inputControl.send(
        dai.DynamicCalibrationControl.setPerformanceMode(
            dai.DynamicCalibrationControl.PerformanceMode.OPTIMIZE_PERFORMANCE
        )
    )
    inputControl.send(
        dai.DynamicCalibrationControl.startCalibration(
            loadImagePeriod=0.5,
            calibrationPeriod=5.0,
            keepCameraCenters=True,
        )
    )

    coverage = coverageQueue.get()
    print(coverage.coveragePerCell.keys())

    result = calibrationQueue.get()
    if result.calibrationData is not None:
        print(result.calibrationData.calibrationDifference.pairwiseRotationDifference)
```
Tokens:
Raw