### Install Python Bindings Source: https://context7.com/uos/marina/llms.txt Install the Marina Python package using uv or pip. ```bash # Python API package uv add marina-py # or pip install marina-py ``` -------------------------------- ### Install Marina via PyPI Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Install the CLI tool or Python module using uv or pip. ```bash # CLI only uv tool install marina-cli # or pip install marina-cli # Python API pip install marina-py ``` -------------------------------- ### Full Marina Configuration Example Source: https://github.com/uos/marina/blob/main/docs/docs/config.md This example demonstrates the structure of the marina.rl configuration file, including settings, compression, and registry definitions. ```awk settings { time_display = relative completion_cache_ttl = 10min registry_timeout = 10s # default_registry = team-ssh } compression { pointcloud_mode = lossy pointcloud_accuracy = 1mm packed_mcap_compression = zstd packed_archive_compression = none unpacked_mcap_compression = lz4 } # Your own SSH registry registries { team_ssh { uri = "ssh://user@your-server.org:/data/marina-registry" } # Google Drive registry (authenticated) my_drive { uri = "gdrive://" kind = gdrive } # Local NFS mount nas { uri = "folder:///mnt/nas/marina" } } ``` -------------------------------- ### Install marina-py Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Installation commands for the marina Python package using uv or pip. ```bash uv add marina-py # or with pip pip install marina-py ``` -------------------------------- ### Install Marina on Ubuntu Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Install the package using apt after configuring the UOS Robotics PPA. ```bash sudo apt install marina ``` -------------------------------- ### Install Marina on macOS via Homebrew Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Add the UOS tap and install the precompiled binary. ```bash brew tap uos/marina brew install marina ``` -------------------------------- ### Install Marina on Arch Linux Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Use an AUR helper to install the package. ```bash paru -S marina # or maybe yay -S marina-bin ``` -------------------------------- ### Install Marina on macOS with Homebrew (Bash) Source: https://context7.com/uos/marina/llms.txt Installs Marina on macOS using Homebrew. This involves tapping the UOS Marina repository and then installing the `marina` formula. ```bash # macOS (Homebrew) brew tap uos/marina brew install marina ``` -------------------------------- ### Marina Configuration File Example Source: https://context7.com/uos/marina/llms.txt Example `marina.rl` configuration file in ratslang format. It sets display preferences, cache TTL, timeouts, default registry, and compression settings. ```awk settings { time_display = relative completion_cache_ttl = 10min registry_timeout = 10s default_registry = team-ssh } compression { pointcloud_mode = lossy pointcloud_accuracy = 1mm packed_mcap_compression = zstd packed_archive_compression = none unpacked_mcap_compression = lz4 } registries { team_ssh { uri = "ssh://user@your-server.org:/data/marina-registry" } my_drive { uri = "gdrive://" kind = gdrive } nas { uri = "folder:///mnt/nas/marina" } } ``` -------------------------------- ### Install Marina with Cargo (Bash) Source: https://context7.com/uos/marina/llms.txt Installs Marina using the Cargo package manager. This command compiles and installs the Marina executable from crates.io. ```bash # Cargo cargo install marina ``` -------------------------------- ### Configure Fish Completions Source: https://github.com/uos/marina/blob/main/docs/docs/features/shell/completions.md Install Fish completion scripts to the standard configuration directory. ```bash marina completions fish > ~/.config/fish/completions/marina.fish ``` ```bash marina completions fish > $(brew --prefix)/share/fish/completions/marina.fish ``` -------------------------------- ### Install Rust Compiler Source: https://github.com/uos/marina/blob/main/docs/docs/installation/ros.md Install a modern Rust compiler (1.85 or newer) using rustup, required for building Marina from source. ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` -------------------------------- ### Install Marina for ROS 2 Humble (Bash) Source: https://context7.com/uos/marina/llms.txt Installs the Marina ROS 2 package for the Humble distribution using `apt`. This command requires `sudo` privileges. ```bash # ROS 2 Humble sudo apt install ros-humble-marina ``` -------------------------------- ### Install Marina via Manual Binary Source: https://github.com/uos/marina/blob/main/docs/docs/installation/archives.md Download, extract, and install the static musl binary. This method is suitable for most Linux distributions. Ensure your ~/.local/bin directory is added to your PATH. ```bash export MARINA_VERSION="0.2.2" export MARINA_ARCH=$(uname -m) curl -L "https://codeberg.org/stelzo/marina/releases/download/v$MARINA_VERSION/marina-$MARINA_VERSION-$MARINA_ARCH-unknown-linux-musl.tar.gz" | tar xfzv - chmod +x marina-*/marina . mkdir -p ~/.local/bin mv marina-*/marina ~/.local/bin/ # Make sure ~/.local/bin is on your PATH echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc ``` ```zsh export MARINA_VERSION="0.2.2" export MARINA_ARCH=$(uname -m) curl -L "https://codeberg.org/stelzo/marina/releases/download/v$MARINA_VERSION/marina-$MARINA_VERSION-$MARINA_ARCH-unknown-linux-musl.tar.gz" | tar xfzv - chmod +x marina-*/marina . mkdir -p ~/.local/bin mv marina-*/marina ~/.local/bin/ # Make sure ~/.local/bin is on your PATH echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc source ~/.zshrc ``` ```fish set -gx MARINA_VERSION 0.2.2 set -gx MARINA_ARCH (uname -m) curl -L "https://codeberg.org/stelzo/marina/releases/download/v$MARINA_VERSION/marina-$MARINA_VERSION-$MARINA_ARCH-unknown-linux-musl.tar.gz" | tar xfzv - chmod +x marina-*/marina . mkdir -p ~/.local/bin mv marina-*/marina ~/.local/bin/ # Make sure ~/.local/bin is on your PATH fish_add_path ~/.local/bin ``` -------------------------------- ### Compile C Code with Marina Library Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md This command compiles a C file named `example.c` into an executable named `example`, linking against the `marina` library located in `./target/release`. ```bash gcc example.c -o example \ -L./target/release \ -lmarina \ -Wl,-rpath,./target/release ``` -------------------------------- ### Install Marina via Cargo Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Install directly from crates.io; requires a C compiler. ```bash cargo install marina ``` -------------------------------- ### Install Marina on Ubuntu (Bash) Source: https://context7.com/uos/marina/llms.txt Installs Marina on Ubuntu systems using `apt` from the UOS Robotics PPA. This command requires `sudo` privileges. ```bash # Ubuntu (via UOS Robotics PPA) sudo apt install marina ``` -------------------------------- ### Install Marina via AUR (Bash) Source: https://context7.com/uos/marina/llms.txt Installs Marina from the Arch User Repository (AUR) using `paru` or `yay`. These are AUR helpers for Arch Linux. ```bash # Arch Linux (AUR) paru -S marina yay -S marina-bin ``` -------------------------------- ### Install ROS 2 Marina Package Source: https://github.com/uos/marina/blob/main/docs/docs/installation/ros.md Install the pre-built Marina package for your ROS distribution from the UOS Robotics PPA. ```bash sudo apt install ros-jazzy-marina ``` ```bash sudo apt install ros-humble-marina ``` -------------------------------- ### Build Marina from Source Source: https://github.com/uos/marina/blob/main/docs/docs/installation/ros.md Clone the Marina repository into your ROS workspace, install dependencies, and build the package using colcon. ```bash cd ~/ros2_ws/src git clone https://codeberg.org/stelzo/marina.git cd .. rosdep install --from-paths src -y --ignore-src colcon build --packages-select marina source install/local_setup.bash ``` -------------------------------- ### Configure Bash Completions Source: https://github.com/uos/marina/blob/main/docs/docs/features/shell/completions.md Install Bash completion scripts for Linux or macOS environments. ```bash marina completions bash | sudo tee /etc/bash_completion.d/marina > /dev/null ``` ```bash marina completions bash > $(brew --prefix)/etc/bash_completion.d/marina ``` ```bash source /etc/bash_completion.d/marina ``` -------------------------------- ### Install Marina to Profile with Nix (Bash) Source: https://context7.com/uos/marina/llms.txt Installs Marina to the Nix profile, making it available system-wide without manual compilation. This command adds the package to your user environment. ```bash # Nix (install to profile) nix profile install git+https://codeberg.org/stelzo/marina ``` -------------------------------- ### Run or Install Marina with Nix Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Execute Marina directly via a flake or add it to your Nix profile. ```bash # Run without installing nix run git+https://codeberg.org/stelzo/marina -- ls --remote # Install into your profile nix profile install git+https://codeberg.org/stelzo/marina ``` -------------------------------- ### Marina Configuration for Google Drive Service Account in CI Source: https://context7.com/uos/marina/llms.txt Example `marina.rl` configuration file to be committed to a repository, setting up a 'prod' registry using Google Drive and specifying the `MARINA_SA_JSON` environment variable for authentication. ```awk # ci/marina/marina.rl - Commit this file to repo registries { prod { uri = "gdrive://" kind = "gdrive" auth_env = "MARINA_SA_JSON" } } ``` -------------------------------- ### Install Marina via cargo-binstall Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Use cargo-binstall to fetch precompiled binaries for supported targets. ```bash cargo binstall marina ``` -------------------------------- ### Install Marina for ROS 2 Jazzy (Bash) Source: https://context7.com/uos/marina/llms.txt Installs the Marina ROS 2 package for the Jazzy distribution using `apt`. This command requires `sudo` privileges. ```bash # ROS 2 Jazzy sudo apt install ros-jazzy-marina ``` -------------------------------- ### Configure Zsh Completions Source: https://github.com/uos/marina/blob/main/docs/docs/features/shell/completions.md Install Zsh completion scripts and update the fpath configuration in .zshrc. ```bash mkdir -p ~/.zsh/completions marina completions zsh > ~/.zsh/completions/_marina ``` ```bash marina completions zsh > $(brew --prefix)/share/zsh/site-functions/_marina ``` ```bash fpath=(~/.zsh/completions $fpath) autoload -U compinit && compinit ``` -------------------------------- ### Install Marina via Debian Package Source: https://github.com/uos/marina/blob/main/docs/docs/installation/archives.md Download and install the Marina Debian package. Ensure the MARINA_VERSION and MARINA_ARCH variables are set correctly for your system. ```bash export MARINA_VERSION="0.2.2" export MARINA_ARCH=$(dpkg --print-architecture) curl -Lo marina.deb "https://codeberg.org/uos-robotics/ppa/raw/branch/pages/ubuntu/pool/main/noble/marina_$MARINA_VERSION-1_$MARINA_ARCH.deb" sudo dpkg -i marina.deb ``` ```zsh export MARINA_VERSION="0.2.2" export MARINA_ARCH=$(dpkg --print-architecture) curl -Lo marina.deb "https://codeberg.org/uos-robotics/ppa/raw/branch/pages/ubuntu/pool/main/noble/marina_$MARINA_VERSION-1_$MARINA_ARCH.deb" sudo dpkg -i marina.deb ``` ```fish set -gx MARINA_VERSION 0.2.2 set -gx MARINA_ARCH (dpkg --print-architecture) curl -Lo marina.deb "https://codeberg.org/uos-robotics/ppa/raw/branch/pages/ubuntu/pool/main/noble/marina_$MARINA_VERSION-1_$MARINA_ARCH.deb" sudo dpkg -i marina.deb ``` -------------------------------- ### Pull Datasets in C Source: https://context7.com/uos/marina/llms.txt Download dataset and get local path with error handling. ```c #include "marina.h" #include int main(void) { char *path = marina_pull("outdoor-run:v2", NULL); if (!path) { char *err = marina_last_error_message(); fprintf(stderr, "pull failed: %s\n", err ? err : "unknown"); marina_free_string(err); return 1; } printf("cached at: %s\n", path); marina_free_string(path); return 0; } ``` -------------------------------- ### CI Pipeline for GDrive Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Example CI pipeline configuration for pulling and pushing datasets using a GDrive registry, with the service account JSON stored in a secret. ```yaml env: MARINA_SA_JSON: ${{ secrets.MARINA_SA_JSON }} steps: - run: marina pull namespace/dataset:tag - run: marina push namespace/dataset:tag ./data/ ``` -------------------------------- ### Run Marina with Nix (Bash) Source: https://context7.com/uos/marina/llms.txt Runs the Marina CLI without installation using `nix run`. This is useful for testing or temporary usage. The command lists remote repositories. ```bash # Nix (run without installing) nix run git+https://codeberg.org/stelzo/marina -- ls --remote ``` -------------------------------- ### GitHub Actions Workflow for Google Drive CI Integration Source: https://context7.com/uos/marina/llms.txt Example GitHub Actions workflow demonstrating unattended Google Drive access using a service account. It sets environment variables for configuration and cache directories, and uses secrets for service account JSON. ```yaml # GitHub Actions workflow example env: MARINA_CONFIG_DIR: ${{ github.workspace }}/ci/marina MARINA_SA_JSON: ${{ secrets.MARINA_SA_JSON }} MARINA_CACHE_DIR: /tmp/marina-cache steps: - run: marina pull namespace/dataset:tag - run: marina push namespace/dataset:tag ./data/ ``` -------------------------------- ### Build Marina from Source with Colcon (Bash) Source: https://context7.com/uos/marina/llms.txt Builds Marina as a ROS 2 package using `colcon`. This process involves installing the Rust compiler, cloning the repository, and then building the package within a ROS 2 workspace. ```bash # Install Rust compiler curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Clone and build cd ~/ros2_ws/src git clone https://codeberg.org/stelzo/marina.git cd .. rosdep install --from-paths src -y --ignore-src colcon build --packages-select marina source install/local_setup.bash ``` -------------------------------- ### GitHub Actions Workflow for SSH CI Integration Source: https://context7.com/uos/marina/llms.txt Example GitHub Actions workflow for configuring SSH registry access with key-based authentication. It securely handles the SSH private key, sets the `MARINA_SSH_KEY` environment variable, and then pulls a dataset. ```yaml # GitHub Actions workflow with SSH key steps: - run: | echo "$SSH_PRIVATE_KEY" > /tmp/marina_key chmod 600 /tmp/marina_key echo "MARINA_SSH_KEY=/tmp/marina_key" >> $GITHUB_ENV env: SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }} - run: marina pull namespace/dataset:tag ``` -------------------------------- ### CMake Build Configuration for Marina Source: https://github.com/uos/marina/blob/main/CMakeLists.txt Configures the project to build a Rust binary using Cargo and install it alongside a Python package. ```cmake cmake_minimum_required(VERSION 3.16) project(marina) find_package(ament_cmake REQUIRED) find_package(ament_cmake_python REQUIRED) set(CARGO_TARGET_DIR "${CMAKE_CURRENT_BINARY_DIR}/cargo-target") add_custom_command( OUTPUT "${CARGO_TARGET_DIR}/release/marina" COMMAND cargo build --release --target-dir "${CARGO_TARGET_DIR}" WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" COMMENT "Building marina with cargo" ) add_custom_target(marina_binary ALL DEPENDS "${CARGO_TARGET_DIR}/release/marina" ) install( PROGRAMS "${CARGO_TARGET_DIR}/release/marina" DESTINATION lib/${PROJECT_NAME} ) ament_python_install_package(ros2_marina) install( PROGRAMS "${CARGO_TARGET_DIR}/release/marina" DESTINATION "${PYTHON_INSTALL_DIR}/ros2_marina/_bin" ) ament_package() ``` -------------------------------- ### Environment Variables for Marina Configuration Source: https://context7.com/uos/marina/llms.txt Use environment variables to override default paths and provide credentials, especially for CI/CD pipelines. Examples include overriding config and cache directories, and providing Google Drive or SSH credentials. ```bash # Override config directory export MARINA_CONFIG_DIR=/path/to/custom/config ``` ```bash # Override cache directory export MARINA_CACHE_DIR=/tmp/marina-cache ``` ```bash # Google Drive OAuth credentials export MARINA_GDRIVE_CLIENT_ID=your-client-id export MARINA_GDRIVE_CLIENT_SECRET=your-client-secret ``` ```bash # SSH authentication (key path or password) export MARINA_SSH_KEY=/path/to/private/key export MARINA_SSH_KEY_PASSPHRASE=key-passphrase ``` -------------------------------- ### Download with Progress Callback (C) Source: https://context7.com/uos/marina/llms.txt Demonstrates how to download a dataset with custom progress reporting using the `marina_pull_with_callback` function. Requires the `marina.h` header and standard I/O libraries. ```c #include "marina.h" #include static void on_progress(const char *phase, const char *message, void *user_data) { (void)user_data; printf("[%-8s] %s\n", phase, message); } int main(void) { MarinaResolveDetailed r = marina_resolve_detailed("outdoor-run:v2", NULL); if (r.kind != MARINA_RESOLVE_REMOTE_AVAILABLE) { marina_free_resolve_detailed(&r); return 1; } char *path = marina_pull_with_callback(r.bag, r.registry, on_progress, NULL); if (!path) { char *err = marina_last_error_message(); fprintf(stderr, "pull failed: %s\n", err ? err : "unknown"); marina_free_string(err); marina_free_resolve_detailed(&r); return 1; } printf("done: %s\n", path); marina_free_string(path); marina_free_resolve_detailed(&r); return 0; } ``` -------------------------------- ### Prepare Recording Target with Marina Import Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Prepares a new cache directory for recording by 'ros2 bag record' and registers the dataset name in Marina. The path printed by 'marina import' is used as the output directory for the recording. ```bash ros2 bag record -o $(marina import my_recording:session1) /topic1 /topic2 ``` -------------------------------- ### Configure Marina as a Nix Flake Input Source: https://github.com/uos/marina/blob/main/docs/docs/installation/packages.md Add Marina as an input in your flake configuration. ```nix inputs.marina.url = "git+https://codeberg.org/stelzo/marina"; ``` -------------------------------- ### Build C/C++ Library Source: https://context7.com/uos/marina/llms.txt Compile the shared library from source using cargo. ```bash # Build release shared library cargo build --release # Output: target/release/libmarina.so (Linux) or libmarina.dylib (macOS) ``` -------------------------------- ### Retrieve and Free Marina Error Message Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md Call `marina_last_error_message` to get the last error string. Remember to free the returned string using `marina_free_string` if an error occurred. ```c char *err = marina_last_error_message(); // NULL if no error if (err) { fprintf(stderr, "marina error: %s\n", err); marina_free_string(err); } ``` -------------------------------- ### Configure Compression in marina.rl Source: https://github.com/uos/marina/blob/main/docs/docs/features/compression.md Define global compression defaults in the ~/.config/marina/marina.rl configuration file using the ratslang format. ```awk compression { pointcloud_mode = lossy pointcloud_accuracy = 1mm # any length unit: mm, cm, m packed_mcap_compression = zstd packed_archive_compression = none unpacked_mcap_compression = lz4 } ``` -------------------------------- ### Add Local or Network-Mounted Directory Registry Source: https://context7.com/uos/marina/llms.txt Configure a local filesystem directory or a network-mounted directory as a Marina registry. ```bash # Add local or network-mounted directory marina registry add local folder://./local-registry marina registry add nas folder:///mnt/nas/marina ``` -------------------------------- ### Mirror Registry Contents Source: https://context7.com/uos/marina/llms.txt Copies all datasets from a source registry to a target registry using `marina registry mirror` for backup or migration. ```bash marina registry mirror source_registry target_registry ``` -------------------------------- ### Set up Passwordless SSH Login Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Copies your public SSH key to the server to enable passwordless authentication for SSH registries. ```bash ssh-copy-id -i ~/.ssh/id_ed25519.pub user@your-server.org ``` -------------------------------- ### Pull with progress output Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md Configure progress reporting to stdout during the download process. ```c // Print progress events to stdout char *path = marina_pull_with_progress("outdoor-run:v2", NULL, MARINA_PROGRESS_MODE_STDOUT); ``` -------------------------------- ### Minimal GDrive Registry Configuration Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md A minimal `marina.rl` configuration for a GDrive registry, specifying the URI, kind, and the environment variable holding the service account JSON. ```yaml registries { prod { uri = "gdrive://" kind = "gdrive" auth_env = "MARINA_SA_JSON" } } ``` -------------------------------- ### List All Registries Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Use this command to view all configured registries in Marina. ```bash marina registry ls ``` -------------------------------- ### Build the Marina library Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md Compile the library to generate the necessary shared object or dynamic library file. ```bash cargo build --release # outputs: target/release/libmarina.so (Linux) or libmarina.dylib (macOS) ``` -------------------------------- ### Add Private Google Drive Registry and Authenticate Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Adds a private Google Drive registry and initiates the authentication process. This will open a browser for OAuth. ```bash # Private or writable. Authenticate after adding marina registry add my_drive gdrive:// marina registry auth my_drive ``` -------------------------------- ### Manage datasets using Standalone CLI Source: https://github.com/uos/marina/blob/main/README.md Commands for pulling, resolving, listing, and pushing datasets using the standalone marina binary. ```bash # Pull a dataset from any configured registry. marina pull dlg_feldtage_24:cut # Get and use the data from the local cache. cat $(marina resolve dlg_feldtage_24:cut)/metadata.yaml # See available bags from configured remotes... marina list --remote # or use the alias. marina ls --remote # Push any folder or bags to your registry. marina bag push my-run:v1 /path/to/dataset/ --registry team_ssh ``` -------------------------------- ### Import with Move to Cache Option Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Moves the source bag directory into the Marina cache instead of copying it. This is efficient when the source and cache are on the same filesystem. ```bash marina --move-to-cache import outdoor-run:v2 /path/to/outdoor-run/ ``` -------------------------------- ### Record and push datasets using ROS 2 CLI Source: https://github.com/uos/marina/blob/main/README.md Commands for recording data directly to the local cache and pushing it to a remote registry. ```bash # Record to local cache... ros2 bag record --all -o $(ros2 bag import my_recording:session1) # and push to your registry. ros2 bag push my-run:v1 /path/to/bag/ --registry team_ssh ``` -------------------------------- ### List Local Datasets Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Lists all datasets currently stored in your local Marina cache. ```bash marina ls ``` -------------------------------- ### Pull and Play Datasets with ROS 2 CLI Source: https://github.com/uos/marina/blob/main/docs/docs/index.md Use the ROS 2 CLI to pull a dataset from a configured registry and pass it directly to playback. Ensure the dataset is available in the registry. ```bash # Pull a dataset from any configured registry... ros2 bag pull dlg_feldtage_24:cut # and pass it straight to playback ros2 bag play $(ros2 bag resolve dlg_feldtage_24:cut) ``` -------------------------------- ### Add a New Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Command to add a new registry with a specified name and URI. Replace and with your desired values. ```bash marina registry add ``` -------------------------------- ### Push Datasets with Standalone CLI Source: https://github.com/uos/marina/blob/main/docs/docs/index.md Push any local folder or bag files to a specified registry using the standalone Marina CLI. Ensure the `--registry` flag points to a valid, configured registry. ```bash # Push any folder or bags to your registry. marina bag push my-run:v1 /path/to/dataset/ --registry team_ssh ``` -------------------------------- ### Pull Datasets by Name and Tags Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Use these commands to download datasets. You can specify the dataset by name only, or include version and descriptive tags. Namespaces can also be included for more specific targeting. ```bash marina pull outdoor-run ``` ```bash marina pull outdoor-run:v2 ``` ```bash marina pull outdoor-run:rainy ``` ```bash marina pull outdoor-run:2024-06-10 ``` ```bash marina pull teamA/outdoor-run:v2 ``` ```bash marina pull teamA/outdoor-run:v2:ouster ``` -------------------------------- ### Add HTTP Registry Source: https://context7.com/uos/marina/llms.txt Configure an HTTP registry for read-only access to datasets hosted online. ```bash # Add HTTP registry (read-only) marina registry add web_main https://datasets.example.org/marina ``` -------------------------------- ### Import Local Bag Directory Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Registers an existing local bag directory in the Marina catalog without uploading it to a registry. The bag is copied into the Marina cache. ```bash marina import outdoor-run:v2 /path/to/outdoor-run/ ``` -------------------------------- ### Push a Dataset to a Registry Source: https://context7.com/uos/marina/llms.txt Uploads a local bagfile directory to a registry with automatic compression. Marina compresses PointCloud2 messages with millimeter accuracy by default and packs bags into optimized archives. ```bash marina push outdoor-run:v2 /path/to/outdoor-run/ ``` ```bash marina push outdoor-run:v2 /path/to/outdoor-run/ --registry team-ssh ``` ```bash marina push outdoor-run:v2 ./bag/ --pointcloud-mode lossless ``` ```bash marina push outdoor-run:v2 ./bag/ \ --pointcloud-mode lossy \ --pointcloud-accuracy-mm 0.1 ``` ```bash marina push outdoor-run:v2 ./bag/ --pointcloud-mode off ``` ```bash marina push outdoor-run:v2 ./bag/ --dry-run ``` ```bash marina push outdoor-run:v2 ./bag/ --move-to-cache ``` ```bash marina push outdoor-run:v2 ./bag/ --write-http-index ``` ```bash marina -y push outdoor-run:v2 /path/to/bag/ --registry team-ssh ``` -------------------------------- ### Pull a Dataset by Name and Tag Source: https://context7.com/uos/marina/llms.txt Downloads a dataset from configured registries into the local cache. Marina searches all configured registries by default and caches downloads for instant subsequent access. ```bash marina pull outdoor-run:v2 ``` ```bash marina pull outdoor-run:v2 --registry team-ssh ``` ```bash marina pull outdoor-run:v2 --force ``` ```bash ros2 bag pull dlg_feldtage_24:cut ``` ```bash ros2 bag play $(ros2 bag resolve dlg_feldtage_24:cut) ``` -------------------------------- ### Import Existing Bag into Cache Source: https://context7.com/uos/marina/llms.txt Registers an existing local bag directory in the Marina catalog without pushing to a registry. Can also prepare a new cache directory for recording. ```bash marina import outdoor-run:v2 /path/to/outdoor-run/ ``` ```bash marina import outdoor-run:v2 /path/to/outdoor-run/ --move-to-cache ``` ```bash ros2 bag record -o $(marina import my_recording:session1) --all ``` ```bash ros2 bag record -o $(ros2 bag import my-dataset:v1) /topic1 /topic2 ``` -------------------------------- ### Add Registry with Authentication Environment Variable Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Use the `--auth-env` flag when adding a registry to specify an environment variable that will hold the credentials at runtime. This is crucial for headless CI operations. ```bash marina registry add --auth-env MY_SECRET_VAR ``` -------------------------------- ### Mirror a Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Copies all datasets from a source registry to a target registry. Useful for backups or migrating between storage backends. ```bash marina registry mirror ``` -------------------------------- ### Configure Marina Directories via Environment Variables Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Override default Marina configuration and cache directories using `MARINA_CONFIG_DIR` and `MARINA_CACHE_DIR` environment variables. This allows shipping configuration with the repository. ```yaml env: MARINA_CONFIG_DIR: ${{ github.workspace }}/ci/marina MARINA_SA_JSON: ${{ secrets.MARINA_SA_JSON }} MARINA_CACHE_DIR: /tmp/marina-cache steps: - run: marina pull namespace/dataset:tag ``` -------------------------------- ### Pull with progress output Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Track download progress via stdout or a custom writer. ```python # Print progress to stdout path = marina.pull_with_progress("outdoor-run:v2", progress=True) # Redirect progress to a custom writer import sys path = marina.pull_with_progress("outdoor-run:v2", writer=sys.stderr) ``` -------------------------------- ### Manage datasets using ROS 2 CLI Source: https://github.com/uos/marina/blob/main/README.md Commands for pulling, playing, and listing datasets via the ROS 2 CLI interface. ```bash # Pull a dataset from any configured registry... ros2 bag pull dlg_feldtage_24:cut # and pass it straight to playback. ros2 bag play $(ros2 bag resolve dlg_feldtage_24:cut) # See available bags from configured remotes... ros2 bag datasets --remote # or use the alias. ros2 bag ds --remote ``` -------------------------------- ### Add GDrive Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Register a Google Drive registry using its URI and specifying the environment variable that contains the service account JSON credentials. ```bash marina registry add prod gdrive:// --auth-env MARINA_SA_JSON ``` -------------------------------- ### Add and Remove Marina Registries Source: https://github.com/uos/marina/blob/main/docs/docs/config.md Use these commands to manage registry configurations from the command line. The `add` command takes a registry name and its URI, while `rm` removes a specified registry. ```bash marina registry add team_ssh ssh://user@host:/path ``` ```bash marina registry rm team_ssh ``` -------------------------------- ### Complete Python Workflow Source: https://context7.com/uos/marina/llms.txt Typical pattern for resolving and pulling datasets in data pipelines. ```python import marina r = marina.resolve_detailed("outdoor-run:v2") if r.should_pull: path = marina.pull_with_progress(r.bag, r.registry, progress=True) else: path = r.path print("dataset ready at:", path) # Use the dataset import os for f in os.listdir(path): print(f) ``` -------------------------------- ### Record ROS 2 Data Directly to Cache Source: https://github.com/uos/marina/blob/main/docs/docs/installation/ros.md Record ROS 2 data directly into the local cache using `ros2 bag record` and specifying the output destination via `ros2 bag import`. ```bash ros2 bag record -o $(ros2 bag import my-dataset:v1) --all ``` -------------------------------- ### Configure Shell Completions Source: https://context7.com/uos/marina/llms.txt Enable tab completion for Marina commands in Bash, Zsh, and Fish shells. ```bash # Bash on Linux marina completions bash | sudo tee /etc/bash_completion.d/marina > /dev/null source /etc/bash_completion.d/marina # Bash on macOS (Homebrew) marina completions bash > $(brew --prefix)/etc/bash_completion.d/marina # Zsh on Linux mkdir -p ~/.zsh/completions marina completions zsh > ~/.zsh/completions/_marina # Add to ~/.zshrc before compinit: # fpath=(~/.zsh/completions $fpath) # autoload -U compinit && compinit # Zsh on macOS (Homebrew) marina completions zsh > $(brew --prefix)/share/zsh/site-functions/_marina # Fish on Linux marina completions fish > ~/.config/fish/completions/marina.fish # Fish on macOS (Homebrew) marina completions fish > $(brew --prefix)/share/fish/completions/marina.fish # Manual refresh of completion index marina complete-refresh & ``` -------------------------------- ### Push Local Bagfile to Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Uploads a local bagfile directory to a specified registry. You can also specify the target registry using the --registry flag. ```bash marina push outdoor-run:v2 /path/to/outdoor-run/ ``` ```bash marina push outdoor-run:v2 /path/to/outdoor-run/ --registry team-ssh ``` -------------------------------- ### Pull Datasets in Python Source: https://context7.com/uos/marina/llms.txt Download datasets to the local cache and manage progress output. ```python import marina # Simple pull path = marina.pull("outdoor-run:v2") print("cached at:", path) # Pull from specific registry path = marina.pull("outdoor-run:v2", registry="team-ssh") # Pull with progress output to stdout path = marina.pull_with_progress("outdoor-run:v2", progress=True) # Redirect progress to custom writer import sys path = marina.pull_with_progress("outdoor-run:v2", writer=sys.stderr) ``` -------------------------------- ### Add SSH Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md Adds a registry accessible via SSH. Ensure passwordless SSH login is set up first. ```bash marina registry add team_ssh ssh://user@your-server.org:/path/to/registry ``` -------------------------------- ### Pull a dataset Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md Download a dataset and retrieve its local path. The returned string must be freed using marina_free_string. ```c char *path = marina_pull("outdoor-run:v2", NULL); if (!path) { char *err = marina_last_error_message(); fprintf(stderr, "pull failed: %s\n", err ? err : "unknown"); marina_free_string(err); return 1; } printf("cached at: %s\n", path); marina_free_string(path); ``` -------------------------------- ### Override Compression via Command Line Source: https://github.com/uos/marina/blob/main/docs/docs/features/compression.md Apply temporary compression settings for specific push or pull operations without modifying the global configuration file. ```bash # Push with lossless point cloud compression marina push outdoor-run:v2 ./bag/ --pointcloud-mode lossless # Push with higher accuracy (0.1 mm) marina push outdoor-run:v2 ./bag/ \ --pointcloud-mode lossy \ --pointcloud-accuracy-mm 0.1 # Push without any point cloud compression marina push outdoor-run:v2 ./bag/ --pointcloud-mode off # Change MCAP chunk compression marina push outdoor-run:v2 ./bag/ \ --packed-mcap-compression lz4 \ --packed-archive-compression gzip # Pull with no recompression of local cache marina pull outdoor-run:v2 --unpacked-mcap-compression none ``` -------------------------------- ### Pull Specific Dataset from Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Downloads a specific dataset from a registry into the local cache. Use the --registry flag to target a particular registry. ```bash marina pull outdoor-run:v2 --registry team-ssh ``` -------------------------------- ### List Available Datasets with Standalone CLI Source: https://github.com/uos/marina/blob/main/docs/docs/index.md Check for available datasets in configured remote registries using the standalone Marina CLI. The `--remote` flag is necessary for remote queries. ```bash # See available bags from configured remotes marina list --remote # or use the alias marina ls --remote ``` -------------------------------- ### SSH Registry Configuration with Key Path Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Configure an SSH registry using a private key stored in a CI secret. The environment variable points to the path of the key file. ```yaml env: MARINA_SSH_KEY: ${{ secrets.MARINA_SSH_KEY }} # path to key file MARINA_SSH_KEY_PASSPHRASE: ${{ secrets.SSH_PASSPHRASE }} # omit if unencrypted ``` -------------------------------- ### Resolve Dataset Path Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Prints the local filesystem path to a cached dataset. This is useful for piping the path to other tools, such as 'ros2 bag play'. If the dataset is not cached, Marina will attempt to pull it first. ```bash marina resolve outdoor-run:v2 ``` ```bash ros2 bag play $(marina resolve outdoor-run:v2) ``` -------------------------------- ### Python: Pull Dataset Source: https://context7.com/uos/marina/llms.txt Download a dataset to the local cache using the Python library. ```APIDOC ## marina.pull(dataset_id, registry=None) ### Description Download a dataset to the local cache and return the local path. ### Parameters #### Request Body - **dataset_id** (string) - Required - The identifier of the dataset. - **registry** (string) - Optional - The specific registry to pull from. ``` -------------------------------- ### Resolve Dataset Path to Local Filesystem Source: https://context7.com/uos/marina/llms.txt Returns the local filesystem path to a cached dataset. If the dataset is not cached, Marina pulls it first. Useful for piping into other tools. ```bash marina resolve outdoor-run:v2 ``` ```bash ros2 bag play $(marina resolve outdoor-run:v2) ``` ```bash cat $(marina resolve outdoor-run:v2)/metadata.yaml ``` -------------------------------- ### marina.pull Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Downloads a dataset to the local cache if it is not already present. ```APIDOC ## marina.pull(dataset_ref, registry=None) ### Description Downloads a dataset if it is not already cached and returns the local path. ### Parameters #### Arguments - **dataset_ref** (str) - Required - The dataset identifier. - **registry** (str) - Optional - The specific registry to pull from. ### Response - **path** (str) - The local path where the dataset is cached. ``` -------------------------------- ### List Available Datasets with ROS 2 CLI Source: https://github.com/uos/marina/blob/main/docs/docs/index.md View available datasets from configured remote registries using the ROS 2 CLI. The `--remote` flag is required to query remotes. ```bash # See available bags from configured remotes ros2 bag datasets --remote # or use the alias ros2 bag ds --remote ``` -------------------------------- ### C: Pull Dataset Source: https://context7.com/uos/marina/llms.txt Download a dataset using the C library. ```APIDOC ## marina_pull(dataset_id, registry) ### Description Download dataset and get local path. ### Parameters #### Request Body - **dataset_id** (string) - Required - The identifier of the dataset. - **registry** (string) - Optional - The specific registry to pull from (pass NULL for automatic selection). ### Response - **path** (string) - The local path to the cached dataset, or NULL on failure. ``` -------------------------------- ### Push with Write HTTP Index Option Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Updates the HTTP index file for paired HTTP registries after a push operation. This is relevant for registries served over HTTP. ```bash marina --write-http-index push outdoor-run:v2 /path/to/outdoor-run/ ``` -------------------------------- ### marina.pull_with_progress Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Downloads a dataset with progress reporting to stdout or a custom writer. ```APIDOC ## marina.pull_with_progress(dataset_ref, progress=False, writer=None) ### Description Downloads a dataset and provides progress feedback during the operation. ### Parameters #### Arguments - **dataset_ref** (str) - Required - The dataset identifier. - **progress** (bool) - Optional - If True, prints phase events to stdout. - **writer** (object) - Optional - A custom writer object for redirecting output. ``` -------------------------------- ### List Datasets in Cache or Remote Registries Source: https://context7.com/uos/marina/llms.txt Lists datasets in the local cache or from remote registries. Dataset names support namespaces and tags in the format `[namespace/]name[:tag[:tag...]]`. ```bash marina ls ``` ```bash marina ls --remote ``` ```bash marina ls --remote --registry team-ssh ``` ```bash ros2 bag datasets --remote ``` ```bash ros2 bag ds --remote --registry osnabotics-public ``` -------------------------------- ### Add SSH Registry Source: https://context7.com/uos/marina/llms.txt Configure a new SSH registry. Supports key authentication and environment variables for explicit key paths. ```bash # Add SSH registry (uses key authentication) marina registry add team_ssh ssh://user@your-server.org:/path/to/registry ``` ```bash # Add SSH registry with explicit key path via environment variable marina registry add team_ssh ssh://user@your-server.org:/path/to/reg \ --auth-env MARINA_SSH_KEY ``` -------------------------------- ### Search Datasets by Pattern Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Searches for datasets in the configured registries using a pattern that is matched against dataset names. ```bash marina search "*feldtage*" ``` -------------------------------- ### Pull a dataset Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Download a dataset if not cached, returning the local path. ```python path = marina.pull("outdoor-run:v2") print("cached at:", path) ``` ```python path = marina.pull("outdoor-run:v2", registry="team-ssh") ``` -------------------------------- ### Add Google Drive Registry Source: https://context7.com/uos/marina/llms.txt Configure Google Drive registries. Supports public read-only access or private access requiring authentication. ```bash # Add public Google Drive registry (read-only) marina registry add public_drive gdrive:// ``` ```bash # Add private Google Drive registry and authenticate marina registry add my_drive gdrive:// marina registry auth my_drive ``` -------------------------------- ### Add SSH Registry Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Register an SSH registry, specifying the connection URI and the environment variable that will hold the authentication credentials (key path or password). ```bash marina registry add prod ssh://user@host:22/path/to/registry --auth-env MARINA_SSH_KEY ``` -------------------------------- ### List Remote Datasets Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Lists all datasets available in configured remote registries. You can filter this list to a specific registry using the --registry flag. ```bash marina ls --remote ``` ```bash marina ls --remote --registry team-ssh ``` -------------------------------- ### SSH Registry Configuration with Password Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Configure an SSH registry using a password directly provided as the environment variable value. Marina treats non-file-path values as passwords. ```yaml env: MARINA_SSH_KEY: ${{ secrets.SSH_PASSWORD }} ``` -------------------------------- ### Push with Dry Run Option Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Performs a dry run of the push operation, executing the full pipeline (read, compress, pack) but skipping the actual upload. This is useful for testing the process without transferring data. ```bash marina --dry-run push outdoor-run:v2 /path/to/outdoor-run/ ``` -------------------------------- ### Pull with a custom progress callback Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/c.md Implement a custom callback function to handle progress events during dataset retrieval. ```c #include "marina.h" #include static void on_progress(const char *phase, const char *message, void *user_data) { (void)user_data; printf("[%-8s] %s\n", phase, message); } int main(void) { MarinaResolveDetailed r = marina_resolve_detailed("outdoor-run:v2", NULL); if (r.kind != MARINA_RESOLVE_REMOTE_AVAILABLE) { marina_free_resolve_detailed(&r); return 1; } char *path = marina_pull_with_callback(r.bag, r.registry, on_progress, NULL); if (!path) { char *err = marina_last_error_message(); fprintf(stderr, "pull failed: %s\n", err ? err : "unknown"); marina_free_string(err); marina_free_resolve_detailed(&r); return 1; } printf("done: %s\n", path); marina_free_string(path); marina_free_resolve_detailed(&r); return 0; } ``` ```c typedef void (*MarinaProgressCallback)( const char *phase, // short phase name, e.g. "download", "unpack" const char *message, // human-readable description void *user_data // pointer passed through from the call site ); ``` -------------------------------- ### Resolve dataset with registry Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Target a specific registry during resolution. ```python r = marina.resolve_detailed("outdoor-run:v2", registry="team-ssh") ``` -------------------------------- ### Resolve dataset path Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md Simple resolution returning a local path or a remote reference string. ```python result = marina.resolve("outdoor-run:v2") print(result) ``` -------------------------------- ### Refresh Completion Cache Source: https://github.com/uos/marina/blob/main/docs/docs/features/shell/completions.md Manually trigger a background refresh of the dataset name completion cache. ```bash marina complete-refresh & ``` -------------------------------- ### Write SSH Private Key to Temporary File in CI Source: https://github.com/uos/marina/blob/main/docs/docs/features/ci.md Dynamically write an SSH private key from a CI secret to a temporary file and set the `MARINA_SSH_KEY` environment variable to its path. Ensure correct file permissions. ```yaml steps: - run: | echo "$SSH_PRIVATE_KEY" > /tmp/marina_key chmod 600 /tmp/marina_key echo "MARINA_SSH_KEY=/tmp/marina_key" >> $GITHUB_ENV env: SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }} - run: marina pull namespace/dataset:tag ``` -------------------------------- ### List Remote Datasets Source: https://github.com/uos/marina/blob/main/docs/docs/installation/ros.md Access Marina's dataset listing functionality via `ros2 bag datasets` or its alias `ros2 bag ds`. Use the `--remote` flag to query remote sources. ```bash ros2 bag datasets --remote ``` ```bash ros2 bag ds --remote --registry osnabotics-public ``` -------------------------------- ### Add SSH Key to SSH Agent Source: https://github.com/uos/marina/blob/main/docs/docs/features/registries.md If your SSH key has a non-standard name, add it to the SSH agent to allow Marina to use it for authentication. ```bash eval "$(ssh-agent -s)" ssh-add ~/.ssh/privatekey ``` -------------------------------- ### Pull and Resolve Datasets with Standalone CLI Source: https://github.com/uos/marina/blob/main/docs/docs/index.md Use the standalone Marina CLI to pull a dataset from a registry and then resolve its local cache path. The `marina resolve` command outputs the path to the dataset's metadata. ```bash # Pull a dataset from any configured registry marina pull dlg_feldtage_24:cut # Get and use the data from the local cache cat $(marina resolve dlg_feldtage_24:cut)/metadata.yaml ``` -------------------------------- ### Force Re-download of Dataset Source: https://github.com/uos/marina/blob/main/docs/docs/features/datasets.md Forces a re-download of a dataset even if it is already present in the local cache. This is useful for ensuring you have the latest version. ```bash marina pull outdoor-run:v2 --force ``` -------------------------------- ### Resolve and pull a dataset Source: https://github.com/uos/marina/blob/main/docs/docs/libraries/python.md A typical workflow for resolving a dataset and pulling it if necessary. ```python import marina r = marina.resolve_detailed("outdoor-run:v2") if r.should_pull: path = marina.pull_with_progress(r.bag, r.registry, progress=True) else: path = r.path print("dataset ready at:", path) ```