### Create conda environment and install dependencies Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Sets up a conda environment named 'abot-recon' with Python 3.11, installs PyTorch 2.5.1 with CUDA 12.1 support, and installs the project in editable mode. Requires Linux, Python 3.10+, and CUDA 12.1. ```bash conda create -n abot-recon python=3.11 -y conda activate abot-recon pip install torch==2.5.1 torchvision==0.20.1 \ --index-url https://download.pytorch.org/whl/cu121 pip install -e . ``` -------------------------------- ### Install optional loop closure dependencies and download retrieval checkpoints Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Install the optional loop closure dependencies and download the DINOv2-SALAD retrieval checkpoints. Requires the 'loop' extra and outputs files to the specified directory. ```bash pip install -e ".[loop]" python scripts/download_loop_assets.py --output-dir checkpoints/loop ``` -------------------------------- ### Install visualization dependencies Source: https://github.com/amap-cvlab/abot-recon/blob/main/visualization/README.md Install required Python packages for visualization. Use onnxruntime instead of onnxruntime-gpu for CPU execution. ```bash pip install 'opencv-python>=4.8' 'scipy>=1.10' pip install onnxruntime-gpu # For sky masks; use onnxruntime instead for CPU. ``` -------------------------------- ### Run abot-recon CLI example Source: https://github.com/amap-cvlab/abot-recon Example command that runs abot-recon with an image directory, output directory, attention backend, no loop closure, world points saving, and a confidence threshold. ```bash abot-recon \ --image-dir examples/images \ --output-dir outputs/demo \ --attention-backend auto \ --no-loop-closure \ --save-world-points \ --confidence-threshold 0.3 ``` -------------------------------- ### RoPE3D usage example Source: https://github.com/amap-cvlab/abot-recon Shows how to instantiate RoPE3D with specific parameters and call it to get frequencies. The example prints the shape of the frequencies and demonstrates applying them to a query tensor using apply_rotary_emb. Note that in the network, the caller broadcasts frequencies per head before applying. ```python import torch from abot_recon.modeling.streaming.rope3d import RoPE3D, apply_rotary_emb rope = RoPE3D(attention_head_dim=64, patch_size=(1, 14, 14), max_seq_len=22000, theta=10000.0, fhw_dim=(20, 22, 22)) freqs = rope(ppf=2, pph=20, ppw=36, patch_start_idx=5, device=torch.device("cpu"), f_start=100, f_end=102) print(freqs.shape) # [1, 1, 2*(5+720), 32] # In the network, the caller broadcasts frequencies per head before applying. # freqs is [1, 1, L, head_dim//2]; q is [B, heads, L, head_dim]: q = torch.rand(1, 16, 1450, 64) freqs_g = freqs.repeat(1, 16, 1, 1) # [1, 16, 1450, 32] — same length as q out = apply_rotary_emb(q, freqs_g) print(out.shape == q.shape) # True ``` -------------------------------- ### Install FlashInfer and compile cuRoPE Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Installs FlashInfer for paged KV-cache operators and compiles cuRoPE to accelerate rotary position encoding. Falls back to PyTorch SDPA if FlashInfer is unavailable. ```bash pip install flashinfer-python flashinfer show-config cd abot_recon/modeling/pi3/models/curope pip install ninja python setup.py build_ext --inplace cd - ``` -------------------------------- ### Example: from_pretrained with local checkpoint Source: https://github.com/amap-cvlab/abot-recon Loads the model from a local checkpoint file path, specifying the device as CUDA. ```python model = ABotRecon.from_pretrained("checkpoints/abot_recon.safetensors", device="cuda") ``` -------------------------------- ### Example: retrieve_candidates with RetrievalConfig Source: https://github.com/amap-cvlab/abot-recon Shows how to configure retrieval and obtain candidate loop closures. Requires a torch device and checkpoint paths; prints the number of candidates and the first score. ```python import numpy as np from abot_recon.sparse_loop.retrieval import RetrievalConfig, retrieve_candidates from abot_recon.sparse_loop.graph import build_odometry_edges, optimize_pose_graph, PoseGraphConfig cfg = RetrievalConfig( salad_checkpoint="checkpoints/loop/dino_salad.ckpt", dino_checkpoint="checkpoints/loop/dinov2_vitb14_pretrain.pth", faiss_use_gpu=False, batch_size=8, ) rgb = np.random.randint(0, 255, (60, 336, 336, 3), dtype=np.uint8) candidates = retrieve_candidates(rgb, cfg, torch.device("cpu")) print(len(candidates), candidates[0].score if candidates else 0) ``` -------------------------------- ### Example usage of transform_local_points Source: https://github.com/amap-cvlab/abot-recon Shows how to import and use transform_local_points with sample tensors. Prints the shape of the result. ```python import torch from abot_recon.geometry import transform_local_points points = torch.rand(4, 280, 504, 3) poses = torch.eye(4).unsqueeze(0).repeat(4, 1, 1) world = transform_local_points(points, poses) print(world.shape) # torch.Size([4, 280, 504, 3]) ``` -------------------------------- ### Example usage of relative_from_c2w Source: https://github.com/amap-cvlab/abot-recon Shows how to import and use relative_from_c2w with a sample tensor. Prints the shape of the result. ```python import torch from abot_recon.geometry import relative_from_c2w poses = torch.eye(4).unsqueeze(0).repeat(3, 1, 1) # [3,4,4] poses[1, 0, 3] = 1.0 rel = relative_from_c2w(poses) # [2,4,4] print(rel.shape) ``` -------------------------------- ### Using build_model with InferenceConfig Source: https://github.com/amap-cvlab/abot-recon Example usage of build_model with an InferenceConfig override. Configures checkpoint and device, then infers camera poses from image paths. ```python from abot_recon.config import InferenceConfig from abot_recon.model import build_model config = InferenceConfig().override(checkpoint="checkpoints/abot_recon.safetensors", device="cuda") model = build_model(config) poses = model.infer_paths(["a.jpg", "b.jpg"], output_local_points=False, output_world_points=False, output_confidence=False)["camera_poses"] ``` -------------------------------- ### apply_loop_closure usage example Source: https://github.com/amap-cvlab/abot-recon Shows how to use apply_loop_closure with a built model and initial poses. The example prints the refined shape and number of loop edges. ```python import torch from abot_recon.loop_closure import apply_loop_closure, LoopClosureConfig from abot_recon.model import build_model from abot_recon.config import InferenceConfig config = InferenceConfig().override(checkpoint="checkpoints/abot_recon.safetensors", loop_closure=False) model = build_model(config) poses = torch.eye(4).repeat(50, 1, 1) refined, info = apply_loop_closure( model, [f"frames/{i:06d}.jpg" for i in range(50)], poses, device=torch.device("cuda"), loop_cfg=LoopClosureConfig(enabled=True, salad_checkpoint="checkpoints/loop/dino_salad.ckpt", dino_checkpoint="checkpoints/loop/dinov2_vitb14_pretrain.pth"), save_dir="outputs/loop", ) print(refined.shape, info["num_loop_edges"]) ``` -------------------------------- ### Example: from_pretrained with HF repo Source: https://github.com/amap-cvlab/abot-recon Loads the model from the Hugging Face repository 'acvlab/ABot-Recon' with device, attention backend, and loop closure overrides. ```python from abot_recon import ABotRecon model = ABotRecon.from_pretrained("acvlab/ABot-Recon", device="cuda", attention_backend="auto", loop_closure=False) ``` -------------------------------- ### Minimal streaming inference example Source: https://github.com/amap-cvlab/abot-recon Shows the minimal usage of infer() with loop_closure disabled. Sorts image paths, loads the model, and prints the shapes of camera poses and local points. ```python from pathlib import Path from abot_recon import ABotRecon images = sorted(Path("examples/images").glob("*.jpg")) model = ABotRecon.from_pretrained("acvlab/ABot-Recon", device="cuda", loop_closure=False) result = model.infer(images) print(result.camera_poses.shape) # [N, 4, 4] print(result.local_points.shape) # [N, H, W, 3] ``` -------------------------------- ### Python API inference with ABotRecon Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Loads the ABot-Recon model from Hugging Face and runs inference on a list of image paths. The checkpoint is cached locally; for offline use, pass a local path instead of the repository ID. Set output_world_points=True to get global point maps and dense_output_indices for selective dense outputs. ```python from pathlib import Path from abot_recon import ABotRecon images = sorted(Path("examples/images").glob("*.jpg")) model = ABotRecon.from_pretrained( "acvlab/ABot-Recon", device="cuda", attention_backend="auto", loop_closure=False, ) result = model.infer(images) trajectory = result.camera_poses relative_poses = result.relative_poses local_points = result.local_points confidence = result.confidence ``` -------------------------------- ### preprocess_image usage example Source: https://github.com/amap-cvlab/abot-recon Loads a JPEG with PIL and preprocesses it to a [3, 280, 504] tensor, printing shape and valid_rows slice. ```python from PIL import Image from abot_recon.preprocessing import preprocess_image with Image.open("frame_000001.jpg") as im: tensor, fov = preprocess_image(im) # tensor: [3, 280, 504] print(tensor.shape, fov.valid_rows) ``` -------------------------------- ### start_descriptor_worker function signature Source: https://github.com/amap-cvlab/abot-recon Defines the start_descriptor_worker function with keyword-only parameters for SALAD and DINOv2 checkpoints, device, batch size, and queue size. Returns a started SaladDescriptorWorker using auto_download=False, so assets must exist. ```python def start_descriptor_worker( *, salad_checkpoint: str | Path, dino_checkpoint: str | Path, device: str | torch.device, batch_size: int = 32, queue_size: int = 64, ): ``` -------------------------------- ### Usage example for AdjacentPoseHead, TransformerDecoder, and LinearPts3d Source: https://github.com/amap-cvlab/abot-recon Shows how to instantiate and run the AdjacentPoseHead, TransformerDecoder, and LinearPts3d layers. The head processes a feature tensor of shape [B, F, tokens, C] and returns poses and state; the decoder and point head produce a per-pixel output of shape [2, 280, 504, 3]. ```python import torch from abot_recon.modeling.pi3.models.layers.adjacent_pose_head import AdjacentPoseHead from abot_recon.modeling.pi3.models.layers.transformer_head import TransformerDecoder, LinearPts3d head = AdjacentPoseHead( dim=512, hidden_dim=512, pair_hidden_dim=512, num_pose_tokens=5, rotation_format="quat", translation_param="vector", rot_correction_mode="temporal_rotation_refinement", rot_correction_kernel=10, rot_correction_max_deg=2.0, ) feat = torch.rand(1, 2, 5 + 720, 512) # [B, F, tokens, C] poses, state = head(feat, return_state=True) print(poses.shape, state["previous_pose"].shape) dec = TransformerDecoder(in_dim=2048, out_dim=1024, dec_embed_dim=1024) pts = LinearPts3d(patch_size=14, dec_embed_dim=1024, output_dim=3) hidden = torch.rand(2, 725, 2048) out = pts(dec(hidden), (280, 504)) print(out.shape) # [2, 280, 504, 3] ``` -------------------------------- ### Run demo.py with minimal options Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Runs a causal pass without loop closure on images in examples/images, writing raw trajectory, poses, point maps, confidence maps, and metadata to outputs/demo. Input frames must be zero-padded and sorted lexicographically. ```bash python demo.py \ --image-dir examples/images \ --output-dir outputs/demo \ --attention-backend auto \ --no-loop-closure ``` -------------------------------- ### Expected files after downloading loop closure assets Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Shows the expected directory structure after downloading the loop closure assets, including the ABot-Recon model and DINOv2-SALAD checkpoints. ```text checkpoints/ ├── abot_recon.safetensors └── loop/ ├── dino_salad.ckpt └── dinov2_vitb14_pretrain.pth ``` -------------------------------- ### iter_preprocessed usage example Source: https://github.com/amap-cvlab/abot-recon Iterates over two image paths, printing tensor shape and crop/pad amounts for each frame. ```python from abot_recon.preprocessing import iter_preprocessed for tensor, fov in iter_preprocessed(["a.jpg", "b.jpg"]): print(tensor.shape, fov.crop_top, fov.pad_top) ``` -------------------------------- ### Render with upsampling to source resolution Source: https://github.com/amap-cvlab/abot-recon/blob/main/visualization/README.md Upsamples point clouds to original image size and exports matching RGB video. Source image order and --image-start/--image-end/--image-stride must match inference exactly. ```bash python visualization/render.py \ --result-dir outputs/inference --output-dir outputs/dense_visualization \ --image-dir examples/images --image-stride 5 \ --resolution source --upsample bilinear --pixel-step 1 \ --outputs video trajectory pointcloud rgb --device cuda:0 ``` -------------------------------- ### Output files structure Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Shows the exact set of output files generated based on selected output options. Some files are only created with loop closure or specific flags. ```text outputs/demo/ ├── camera_poses.npy ├── relative_poses.npy ├── camera_poses_noloop.npy ├── relative_poses_noloop.npy ├── camera_poses_loop.npy # only with loop closure ├── relative_poses_loop.npy # only with loop closure ├── local_points.pt # enabled by default ├── world_points.pt # with --save-world-points ├── colors.pt # RGB aligned with saved point maps ├── confidence.pt # enabled by default ├── confidence_mask.pt # enabled by default └── metadata.json ``` -------------------------------- ### Download loop assets Source: https://github.com/amap-cvlab/abot-recon Downloads the loop-closure assets (dino_salad.ckpt and dinov2_vitb14_pretrain.pth) to the specified output directory. The script returns 0 on success or dry-run, 1 on failure. ```bash python scripts/download_loop_assets.py --output-dir checkpoints/loop ``` -------------------------------- ### ABotRecon.__init__(model, config) Source: https://github.com/amap-cvlab/abot-recon Constructor for the user-facing streaming reconstruction model. Wraps a constructed torch.nn.Module plus an InferenceConfig and orchestrates frame preprocessing, streaming inference, optional loop closure, pose conversion, point-map post-processing, and confidence masking. ```APIDOC ## ABotRecon.__init__(model, config) ### Description Constructor for the user-facing streaming reconstruction model. Wraps a constructed torch.nn.Module plus an InferenceConfig and orchestrates frame preprocessing, streaming inference, optional loop closure, pose conversion, point-map post-processing, and confidence masking. ### Parameters #### Parameters - **model** (torch.nn.Module) - Required - Constructed inference network (usually a ReleasedABotReconModel) - **config** (InferenceConfig) - Required - Frozen inference configuration ### Response #### Return - **ABotRecon** - A new instance of the ABotRecon class ### Notes Public attributes: model, config. Normally constructed via from_pretrained, not directly. ``` -------------------------------- ### checkpoint_has_prefix(path, prefixes) Source: https://github.com/amap-cvlab/abot-recon Inspect the released state dict without constructing the model. Returns True if any state-dict key starts with any prefix. ```APIDOC ## `checkpoint_has_prefix(path, prefixes)` ### Description Inspect the released state dict without constructing the model. ### Method Function call ### Signature ```python def checkpoint_has_prefix(path: str | Path, prefixes: tuple[str, ...]) -> bool: ``` ### Parameters - `path` (str | Path) - Required - Checkpoint path. - `prefixes` (tuple[str, ...]) - Required - Tuple of prefixes to check. ### Return `True` if any state-dict key starts with any prefix — used by `ReleasedABotReconModel` to decide whether the checkpoint contains a confidence decoder (`("conf_decoder.", "conf_head.")`). ``` -------------------------------- ### reset() method implementation Source: https://github.com/amap-cvlab/abot-recon Resets streaming state by delegating to the model's reset method if present. Called automatically at the start of every infer. ```python def reset(self) -> None: reset = getattr(self.model, "reset", None) if reset is not None: reset() ``` -------------------------------- ### set_local_window_frames(w) Source: https://github.com/amap-cvlab/abot-recon Sets the sliding temporal window for streaming and streaming bias modes, updating the local window frames and all installed StreamingFlashAttention instances. ```APIDOC ## set_local_window_frames(w) ### Description Sets the sliding temporal window (streaming + streaming bias); reference unchanged. Updates `local_window_frames` (clamped to ≥ 1) and each installed `StreamingFlashAttention.local_window_frames`. ### Method function ### Parameters - **w** (int) - Required - The new local window frame count. ``` -------------------------------- ### Define main function Source: https://github.com/amap-cvlab/abot-recon Entry point for the abot-recon CLI. Parses arguments, builds the model, runs inference, and saves outputs. Raises ValueError for invalid stride or dense-stride values. ```python def main(argv=None) -> None: ``` -------------------------------- ### set_local_window_frames method Source: https://github.com/amap-cvlab/abot-recon Sets the sliding temporal window for streaming and streaming bias modes. Updates local_window_frames (clamped to ≥ 1) and each installed StreamingFlashAttention.local_window_frames. ```python def set_local_window_frames(self, w: int) -> None: """Set sliding temporal window (streaming + streaming bias); reference unchanged.""" ``` -------------------------------- ### Asset download URLs for SALAD and DINOv2 Source: https://github.com/amap-cvlab/abot-recon Constants for the SALAD checkpoint and DINOv2 ViT-B/14 pretrained weights used by the loop-closure assets module. ```python SALAD_CHECKPOINT_URL = "https://github.com/serizba/salad/releases/download/v1.0.0/dino_salad.ckpt" DINO_VITB14_URL = "https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth" ``` -------------------------------- ### reset() Source: https://github.com/amap-cvlab/abot-recon Resets streaming state (paged KV manager and camera-head state) by delegating to the model's reset method if present. Called automatically at the start of every infer. ```APIDOC ## reset() ### Description Resets streaming state (paged KV manager and camera-head state) by delegating to the model's reset method if present. Called automatically at the start of every infer. ### Method reset ### Signature ```python def reset(self) -> None: reset = getattr(self.model, "reset", None) if reset is not None: reset() ``` ### Parameters None ### Response #### Success Response - **Return**: `None` ``` -------------------------------- ### Export reconstruction to PLY and BEV trajectory Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Export an RGB point-cloud PLY and a separate BEV trajectory PNG from the saved outputs. Requires the poses, points, and colors files. ```bash python scripts/export_reconstruction_ply.py \ --poses outputs/demo/camera_poses.npy \ --points outputs/demo/local_points.pt \ --colors outputs/demo/colors.pt \ --output outputs/demo/reconstruction.ply \ --bev-output outputs/demo/trajectory_bev.png ``` -------------------------------- ### Using resolve_checkpoint, sha256, and checkpoint_has_prefix Source: https://github.com/amap-cvlab/abot-recon Shows how to resolve a checkpoint path, compute its SHA-256 hash, and check if it has a given prefix. ```python from abot_recon.checkpoint import resolve_checkpoint, sha256, checkpoint_has_prefix ckpt = resolve_checkpoint("checkpoints/abot_recon.safetensors") print(ckpt, sha256(ckpt)) print(checkpoint_has_prefix(ckpt, ("conf_decoder.", "conf_head."))) ``` -------------------------------- ### start_descriptor_worker(...) Source: https://github.com/amap-cvlab/abot-recon Starts a descriptor worker that computes SALAD and DINOv2 descriptors concurrently with reconstruction. The worker consumes CPU tensors sent to the stream via the model's `_image_observer` hook. ```APIDOC ## start_descriptor_worker(...) ### Description Starts a descriptor worker that computes SALAD and DINOv2 descriptors concurrently with reconstruction. The worker consumes CPU tensors sent to the stream via the model's `_image_observer` hook. ### Method Function call ### Signature ```python def start_descriptor_worker( *, salad_checkpoint: str | Path, dino_checkpoint: str | Path, device: str | torch.device, batch_size: int = 32, queue_size: int = 64, ): ``` ### Parameters - `salad_checkpoint` (str | Path) - Required - SALAD checkpoint path - `dino_checkpoint` (str | Path) - Required - DINOv2 checkpoint path - `device` (str | torch.device) - Required - Inference device - `batch_size` (int) - Optional (default: `32`) - Descriptor batch size - `queue_size` (int) - Optional (default: `64`) - Worker queue capacity ### Return A started `SaladDescriptorWorker`. Uses `auto_download=False` (assets must exist). ``` -------------------------------- ### Instantiate and run Pi3 model Source: https://github.com/amap-cvlab/abot-recon Constructs a Pi3 model with rope100 positional encoding, large decoder, frozen encoder, confidence enabled, and relative adjacent camera pose mode. The model expects a batch of images with shape (1, 2, 3, 280, 504) and returns local points with shape [1, 2, 280, 504, 3]. ```python from abot_recon.modeling.pi3.models.pi3 import Pi3 model = Pi3( pos_type="rope100", decoder_size="large", load_vggt=False, freeze_encoder=True, enable_confidence=True, camera_pose_mode="relative_adjacent", relative_camera_head_cfg={ "head_type": "token_pair", "rotation_format": "quat", "hidden_dim": 512, "pair_hidden_dim": 512, "num_pose_tokens": 5, "init_std": 1e-4, "rot_correction_mode": "temporal_rotation_refinement", "rot_correction_kernel": 10, "rot_correction_max_deg": 2.0, "rot_correction_use_age_embed": True, "translation_param": "vector", }, point_z_log_max=10.0, ) imgs = torch.rand(1, 2, 3, 280, 504) # normalized internally with image_mean/image_std out = model(imgs) print(out["local_points"].shape) # [1, 2, 280, 504, 3] ``` -------------------------------- ### Run real-checkpoint integration test Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Runs the real-checkpoint integration test. Requires environment variables for checkpoint path, image directory, and device (CUDA). ```bash ABOT_RECON_CHECKPOINT=checkpoints/abot_recon.safetensors \ ABOT_RECON_IMAGE_DIR=examples/images \ ABOT_RECON_DEVICE=cuda \ pytest -q tests/integration/test_real_checkpoint.py ``` -------------------------------- ### Run inference with loop closure Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Run ABot-Recon inference with loop closure enabled. When enabled, 'camera_poses' stores the refined trajectory, while 'camera_poses_noloop' preserves the raw streaming prediction. ```bash python demo.py \ --image-dir examples/images \ --output-dir outputs/demo_loop \ --attention-backend auto \ --loop-closure ``` -------------------------------- ### Run test suite with pytest Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Runs the full test suite. Use this command to execute all tests. ```bash pytest -q ``` -------------------------------- ### ABotRecon.from_pretrained(...) Source: https://github.com/amap-cvlab/abot-recon Classmethod to load ABot-Recon from a local checkpoint or Hugging Face repository. Builds a fresh InferenceConfig(), resolves the checkpoint, applies overrides, then constructs the heavy network through build_model(config). ```APIDOC ## ABotRecon.from_pretrained(...) ### Description Classmethod to load ABot-Recon from a local checkpoint or Hugging Face repository. Builds a fresh InferenceConfig(), resolves the checkpoint, applies overrides, then constructs the heavy network through build_model(config). ### Parameters #### Parameters - **pretrained_model_name_or_path** (str | Path) - Optional - Default: "acvlab/ABot-Recon" - Hugging Face repo ID or local file/dir checkpoint path - **filename** (str) - Optional - Default: "abot_recon.safetensors" - File name inside the HF repo - **cache_dir** (str | Path | None) - Optional - Default: None - HF cache dir override - **revision** (str | None) - Optional - Default: None - HF repo revision - **token** (str | bool | None) - Optional - Default: None - HF auth token - **local_files_only** (bool) - Optional - Default: False - Never touch the network in HF resolution - **overrides** (Any) - Optional - Any InferenceConfig field, applied via InferenceConfig.override(); "checkpoint" is set automatically to the resolved path ### Response #### Return - **ABotRecon** - A fully constructed ABotRecon (checkpoint weights loaded strictly, model in eval() mode) ### Errors - FileNotFoundError / ValueError from resolve_pretrained_checkpoint - ValueError from InferenceConfig.__post_init__ if an override is invalid ### Request Example ```python from abot_recon import ABotRecon model = ABotRecon.from_pretrained("acvlab/ABot-Recon", device="cuda", attention_backend="auto", loop_closure=False) ``` ### Request Example ```python model = ABotRecon.from_pretrained("checkpoints/abot_recon.safetensors", device="cuda") ``` ``` -------------------------------- ### RoPE3D.__init__ Source: https://github.com/amap-cvlab/abot-recon Initializes the RoPE3D module with attention head dimension, patch size, and optional frequency dimension split. Precomputes bulk frequencies for (t, h, w) as complex values in float64. ```APIDOC ## RoPE3D.__init__ ### Description Initializes the RoPE3D module with attention head dimension, patch size, and optional frequency dimension split. Precomputes bulk frequencies for (t, h, w) as complex values in float64. ### Method Constructor ### Signature ```python def __init__( self, attention_head_dim: int, patch_size: Tuple[int, int, int], max_seq_len: int = 1024, theta: float = 10000.0, fhw_dim: Optional[Tuple[int, int, int]] = None, *, rope_fwd_cache_max: int = 32, ): ``` ### Parameters - `attention_head_dim` (int) - Required - Head dim; must equal `sum(fhw_dim)` - `patch_size` (tuple[int,int,int]) - Required - `(frame, h, w)` patch size (used only for bookkeeping) - `max_seq_len` (int) - Optional - Max frames for frequency tables (default: `1024`) - `theta` (float) - Optional - RoPE base (default: `10000.0`) - `fhw_dim` (tuple[int,int,int] | None) - Optional - Explicit split `(t, h, w)`; all even; default `h=w=2·(d//6)`, `t = d - h - w` (default: `None`) - `rope_fwd_cache_max` (int) - Optional - LRU cache size for `forward` outputs keyed by `(ppf, pph, ppw, patch_start_idx, device, f_start, f_end)` (default: `32`) ### Throws - Asserts: `attention_head_dim == sum(fhw_dim)`; `fhw_dim` all even. ``` -------------------------------- ### ReleasedABotReconModel.__init__(config) Source: https://github.com/amap-cvlab/abot-recon Constructor for the ReleasedABotReconModel class. Builds the inference network with a fixed architecture and loads checkpoint weights. ```APIDOC ## ReleasedABotReconModel.__init__(config) ### Description Thin runtime wrapper around the checkpoint-exact inference network. Determines confidence from checkpoint, builds the network with fixed released architecture, resolves attention backend, loads checkpoint strictly, and sets eval mode. ### Parameters - **config** (InferenceConfig) - Required - Configuration object containing checkpoint path, device, attention backend, max frames, local window frames, and amp dtype. ### Throws - **RuntimeError** - If `attention_backend="paged"` without FlashInfer, or from strict checkpoint load mismatch. - **ValueError** / **TypeError** - From `ABotReconNetwork.__init__` for invalid kwargs. ### Public Attributes - **network** (ABotReconNetwork) - The inference network. - **attention_backend** (str) - `"paged"` or `"sdpa"`. - **disabled_packaged_flash_modules** (int) - Number of disabled packaged flash attention modules. - **config** (InferenceConfig) - The configuration object. - **device_name** (str) - The device name. - **compute_dtype** (torch.dtype) - The compute dtype from `amp_dtype`. ``` -------------------------------- ### Switch to eval branch for evaluation protocols Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Use this command to switch to the `eval` branch, which maintains camera-pose and dense-reconstruction protocols. Dense reconstruction is evaluated without loop closure to match the paper protocol. ```bash git switch eval ``` -------------------------------- ### prepare_sparse_graph Source: https://github.com/amap-cvlab/abot-recon Prepares a sparse graph representation from poses and loop edges. Returns keyframes, their corresponding poses, and remapped edges renumbered to node indices. ```APIDOC ## prepare_sparse_graph(poses, loop_edges, stride) ### Description Prepares a sparse graph representation from poses and loop edges. ### Parameters - **poses** (array) - Required - Trajectory poses. - **loop_edges** (list) - Required - List of loop closure edges. - **stride** (int) - Required - Interval for selecting keyframes. ### Returns - **tuple** - A tuple containing `(keyframes, poses[keyframes], remapped_edges)` where edges are renumbered to node indices. ``` -------------------------------- ### IMAGE_SUFFIXES constant Source: https://github.com/amap-cvlab/abot-recon Set of image file extensions accepted when collecting input frames. ```python IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png", ".bmp", ".webp"} ``` -------------------------------- ### Render ground-follow point-cloud video Source: https://github.com/amap-cvlab/abot-recon/blob/main/visualization/README.md Renders point-cloud video and bird's-eye images. Sky filtering is on by default; missing or incompatible masks cause an error. Use --no-sky to disable. ```bash python visualization/render.py \ --result-dir outputs/inference --output-dir outputs/visualization \ --device cuda:0 ``` -------------------------------- ### PagedKVCacheManager.__init__() Source: https://github.com/amap-cvlab/abot-recon Initializes a PagedKVCacheManager instance with the given configuration parameters. Validates the provided arguments and sets up the page pool layout and FlashInfer wrapper. ```APIDOC ## PagedKVCacheManager.__init__() ### Description Initializes the paged KV cache manager with the specified configuration. Validates parameters and sets up the page pool layout and FlashInfer wrapper. ### Parameters #### Keyword Arguments - **num_layers** (int) - Required - Number of global decoder layers - **tpf** (int) - Required - Tokens per frame (`patch_h*patch_w + patch_start_idx`) - **num_heads** (int) - Required - Attention heads - **head_dim** (int) - Required - Head dimension - **num_reference_frames** (int) - Required - Reference pages always retained - **local_window_frames** (int) - Required - Window capacity **including the current frame**; must be ≥ 1 - **num_summary_tokens** (int) - Required - Summary tokens per evicted frame when `memory_mode="streaming"` - **memory_mode** (str) - Required - `"streaming"` or `"window"` - **max_total_frames** (int) - Required - Maximum stream length (pools `max_total_frames - ka - W + 1` summary frames when no FIFO cap) - **max_summary_frames** (int) - Optional, default `0` - Summary FIFO frame cap; `0` = no cap - **dtype** (torch.dtype) - Required - Storage dtype (fp16/bf16 for FlashInfer; bf16 fallback handled by caller) - **device** (torch.device) - Required - CUDA device - **force_fp32** (bool) - Optional, default `False` - Debug path: gather visible pages and run fp32 PyTorch SDPA instead of FlashInfer - **fa3** (bool) - Optional, default `False` - Select FlashInfer `fa3` backend instead of `fa2` ### Throws - `ValueError` if `memory_mode` is not `"streaming"` or `"window"` - `ValueError` if `local_window_frames` < 1 - `ValueError` if `num_reference_frames` < 0 - `ValueError` if `num_summary_tokens` not in `[0, tpf]` - `RuntimeError` if FlashInfer is not installed and `force_fp32` is `False` ``` -------------------------------- ### Checkpoint file path Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Place the manually downloaded checkpoint at this path for offline inference. The Python API and demo download it automatically from Hugging Face and reuse the local cache. ```text checkpoints/abot_recon.safetensors ``` -------------------------------- ### flashinfer_available() function Source: https://github.com/amap-cvlab/abot-recon Returns True if the flashinfer package imported successfully at module load, else False. ```python def flashinfer_available() -> bool: return _FLASHINFER_AVAILABLE ``` -------------------------------- ### Package __init__.py exports Source: https://github.com/amap-cvlab/abot-recon Defines the public inference API for ABot-Recon, re-exporting ABotRecon, InferenceConfig, ReconstructionResult, and setting __version__ to "0.1.0". ```python """Public inference API for ABot-Recon.""" from .api import ABotRecon from .config import InferenceConfig from .types import ReconstructionResult __all__ = ["ABotRecon", "InferenceConfig", "ReconstructionResult"] __version__ = "0.1.0" ``` -------------------------------- ### optimize_pose_graph_gpu_sparse Source: https://github.com/amap-cvlab/abot-recon Optimizes a pose graph on GPU using sparse PGO with SE3/Sim3 residuals. Returns optimized camera-to-world poses. ```APIDOC ## optimize_pose_graph_gpu_sparse ### Description Optimizes a pose graph on GPU using sparse PGO with SE3/Sim3 residuals. Returns optimized camera-to-world poses. ### Method Function call ### Signature ```python def optimize_pose_graph_gpu_sparse( keyframe_c2w_init: np.ndarray, edge_src: Sequence[int], edge_dst: Sequence[int], edge_measurements: Sequence[np.ndarray], edge_weights: Sequence[float], *, model: str = "se3", update_mode: str = "all", trans_weight: float = 1.0, rot_weight: float = 1.0, scale_weight: float = 1.0, max_iterations: int = 30, lambda_init: float = 1e-4, device: str = "cuda", pcg_max_iterations: int = 256, pcg_tolerance: float = 1e-5, pcg_check_interval: int = 8, coarse_group_size: int = 64, solve_dtype: str = "float64", outer_relative_tolerance: float = 0.0, verbose: bool = True, log_fn: Optional[Callable[[str], None]] = None, runtime_stats: Optional[Dict[str, Any]] = None, ) -> np.ndarray: ``` ### Parameters #### Required Parameters - **keyframe_c2w_init** (np.ndarray) - Required - `[N,4,4]` initial c2w poses - **edge_src** (Sequence[int]) - Required - Edge endpoints (node indices) - **edge_dst** (Sequence[int]) - Required - Edge endpoints (node indices) - **edge_measurements** (Sequence[np.ndarray]) - Required - Relative transforms - **edge_weights** (Sequence[float]) - Required - Per-edge weights #### Optional Parameters - **model** (str) - Optional - `"se3"` or `"sim3"` (default: `"se3"`) - **update_mode** (str) - Optional - `"all"`/`"translation_only"` (+`"translation_scale"` for sim3) (default: `"all"`) - **trans_weight** (float) - Optional - Residual component weight (default: `1.0`) - **rot_weight** (float) - Optional - Residual component weight (default: `1.0`) - **scale_weight** (float) - Optional - Residual component weight (default: `1.0`) - **max_iterations** (int) - Optional - Outer LM iterations (default: `30`) - **lambda_init** (float) - Optional - Initial damping (default: `1e-4`) - **device** (str) - Optional - Must be an available CUDA device (default: `"cuda"`) - **pcg_max_iterations** (int) - Optional - Inner PCG / preconditioner settings (default: `256`) - **pcg_tolerance** (float) - Optional - Inner PCG / preconditioner settings (default: `1e-5`) - **pcg_check_interval** (int) - Optional - Inner PCG / preconditioner settings (default: `8`) - **coarse_group_size** (int) - Optional - Inner PCG / preconditioner settings (default: `64`) - **solve_dtype** (str) - Optional - Inner PCG / preconditioner settings (default: `"float64"`) - **outer_relative_tolerance** (float) - Optional - Inner PCG / preconditioner settings (default: `0.0`) - **verbose** (bool) - Optional - Logging (default: `True`) - **log_fn** (Optional[Callable[[str], None]]) - Optional - Logging (default: `None`) - **runtime_stats** (Optional[Dict[str, Any]]) - Optional - Stats (filled with `gpu_pgo_*` keys) (default: `None`) ### Response #### Success Response - **np.ndarray** - Optimized `[N,4,4]` array. No edges or ≤ 1 node → copy of input. ### Errors - `ValueError(f"Unsupported pose graph model: {model}")` - `RuntimeError(f"GPU sparse PGO requires an available CUDA device, got {device!r}")` - `ValueError(f"Unsupported update_mode={update_mode!r} for model={model!r}; expected one of {sorted(valid_modes)}")` ``` -------------------------------- ### optimize_pose_graph Source: https://github.com/amap-cvlab/abot-recon Optimizes a pose graph using odometry and loop edges. ```APIDOC ## optimize_pose_graph ### Description Optimizes a pose graph using odometry and loop edges. ### Method Function call ### Signature ```python optimize_pose_graph(camera_poses, odometry_edges, loop_edges, config, *, runtime_stats=None) -> np.ndarray ``` ### Parameters - **camera_poses** - Initial camera poses - **odometry_edges** - Odometry edges - **loop_edges** - Loop edges - **config** - Configuration object - **runtime_stats** (optional) - Runtime statistics ### Response - **np.ndarray** - Optimized poses ### Notes - Forwards to `optimize_keyframe_pose_graph_gpu_sparse`. ``` -------------------------------- ### optimize_keyframe_pose_graph_gpu_sparse Source: https://github.com/amap-cvlab/abot-recon Convenience wrapper for pose-graph optimization that takes odometry and loop edges and a configuration object. Computes loop-edge weights based on inlier counts. ```APIDOC ## optimize_keyframe_pose_graph_gpu_sparse ### Description Convenience wrapper for pose-graph optimization that takes odometry and loop edges and a configuration object. Computes loop-edge weights based on inlier counts. ### Method Function call ### Signature ```python def optimize_keyframe_pose_graph_gpu_sparse(...) ``` ### Parameters #### Required Parameters - **odom_edges** - Objects with `method`, `src_pos`, `dst_pos`, `transform_ji`, `inliers` - **loop_edges** - Objects with `method`, `src_pos`, `dst_pos`, `transform_ji`, `inliers` - **cfg** (PoseGraphConfig-like) - Configuration object ### Response #### Success Response - **np.ndarray** - Optimized poses (same as `optimize_pose_graph_gpu_sparse`) ### Notes - Loop-edge weights = `pose_graph_loop_weight * sqrt(max(inliers,1) / max(rigid_min_inliers,1))`; odometry weight 1.0. ``` -------------------------------- ### reset method implementation Source: https://github.com/amap-cvlab/abot-recon Resets the paged KV manager of the network when present. Uses getattr to safely access the manager and calls its reset method if it exists. ```python def reset(self) -> None: manager = getattr(self.network, "_paged_manager", None) if manager is not None: manager.reset() ``` -------------------------------- ### _load_salad_checkpoint Source: https://github.com/amap-cvlab/abot-recon Loads a SALAD checkpoint into a model, tolerantly stripping `model.`/`module.` prefixes. ```APIDOC ## _load_salad_checkpoint ### Description Tolerant state-dict load stripping `model.`/`module.` prefixes. ### Parameters - `model` - The model to load into. - `checkpoint` - The checkpoint to load. ### Exceptions - `RuntimeError("SALAD checkpoint is missing aggregator parameters: ...")` when aggregator weights are missing. ``` -------------------------------- ### Using override with validation Source: https://github.com/amap-cvlab/abot-recon Shows how to use override to change fields, that unlisted fields keep values, and that invalid values raise ValueError with the documented message. ```python from abot_recon.config import InferenceConfig base = InferenceConfig() assert base.loop_closure is True and base.width == 504 config = base.override(device="cpu", attention_backend="sdpa", output_world_points=True) print(config.device, config.width) # cpu 504 # Invalid values fail fast: try: base.override(local_window_frames=24) except ValueError as exc: print(exc) # The released model requires a 12-frame local window ``` -------------------------------- ### ensure_loop_assets Source: https://github.com/amap-cvlab/abot-recon Ensures both the SALAD checkpoint and DINO checkpoint files are available, downloading them if necessary. Returns a tuple of the resolved paths. ```APIDOC ## ensure_loop_assets ### Description Ensures both the SALAD checkpoint and DINO checkpoint files are available, downloading them if necessary. Returns a tuple of the resolved paths. ### Method Function ### Signature `ensure_loop_assets(salad_checkpoint, dino_checkpoint, *, auto_download) -> tuple[Path, Path]` ### Parameters - **salad_checkpoint** (Path) - Required - The path to the SALAD checkpoint file. - **dino_checkpoint** (Path) - Required - The path to the DINO checkpoint file. - **auto_download** (bool) - Required - If True, downloads missing files; otherwise raises FileNotFoundError. ### Returns - **tuple[Path, Path]** - A tuple containing the resolved paths to the SALAD and DINO checkpoints. ``` -------------------------------- ### Release config.json metadata Source: https://github.com/amap-cvlab/abot-recon HF metadata at repository root; not read by the runtime dataclass. Specifies model type, architecture, checkpoint file, input dimensions, local window frames, AMP dtype, and attention backend. ```json { "model_type": "abot_recon", "architecture": "ABotRecon", "checkpoint_file": "abot_recon.safetensors", "input_height": 280, "input_width": 504, "local_window_frames": 12, "amp_dtype": "bf16", "attention_backend": "auto" } ``` -------------------------------- ### SaladDescriptorWorker class constructor Source: https://github.com/amap-cvlab/abot-recon Constructor for the SaladDescriptorWorker class. Initializes with a RetrievalConfig, a torch device, and an optional queue size. ```python class SaladDescriptorWorker: def __init__(self, config: RetrievalConfig, device: torch.device, queue_size: int = 64): ``` -------------------------------- ### Run CUDA-specific parity test Source: https://github.com/amap-cvlab/abot-recon/blob/main/README.md Runs the CUDA-specific parity test. Requires the `ABOT_RECON_REQUIRE_CUROPE` environment variable set to `1`. ```bash ABOT_RECON_REQUIRE_CUROPE=1 pytest -q tests/test_curope_parity.py ```