Efficient LoFTR

https://github.com/zju3dv/efficientloftr
Admin
Efficient LoFTR is a PyTorch implementation for semi-dense local feature matching between image...
Tokens:10,936
Snippets:55
Trust Score:7.9
Update:4 days ago
Show doc for...
Tokens:
Raw
### Set up training environment and start outdoor training

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Creates a conda environment from environment_training.yaml, installs requirements, activates the environment, and runs the outdoor training script.

```shell
conda env create -f environment_training.yaml  # used a different version of pytorch, maybe slightly different from the inference environment
pip install -r requirements.txt
conda activate eloftr_training
bash scripts/reproduce_train/eloftr_outdoor.sh eloftr_outdoor
```

--------------------------------

### Setup and imports for EfficientLoFTR demo

Source: https://github.com/zju3dv/efficientloftr/blob/main/notebooks/demo_single_pair.ipynb

Imports required libraries and changes working directory to project root. Needed before running the matching examples.

```python
import os
os.chdir("..")
from copy import deepcopy

import torch
import cv2
import numpy as np
import matplotlib.cm as cm
from src.utils.plotting import make_matching_figure
```

--------------------------------

### Basic EfficientLoFTR Matching with Pretrained Weights

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Loads the pretrained outdoor model, reparameterizes it (essential for performance), resizes images to multiples of 32, and runs inference to get matched keypoints and confidence scores.

```python
import torch
import cv2
import numpy as np
from copy import deepcopy
from src.loftr import LoFTR, full_default_cfg, reparameter

# Initialize the matcher with default settings
_default_cfg = deepcopy(full_default_cfg)
matcher = LoFTR(config=_default_cfg)

# Load pretrained weights
matcher.load_state_dict(torch.load("weights/eloftr_outdoor.ckpt")['state_dict'])
matcher = reparameter(matcher)  # Essential for good performance
matcher = matcher.eval().cuda()

# Load and preprocess images
img0_raw = cv2.imread("path/to/image0.jpg", cv2.IMREAD_GRAYSCALE)
img1_raw = cv2.imread("path/to/image1.jpg", cv2.IMREAD_GRAYSCALE)

# Resize images to be divisible by 32
img0_raw = cv2.resize(img0_raw, (img0_raw.shape[1]//32*32, img0_raw.shape[0]//32*32))
img1_raw = cv2.resize(img1_raw, (img1_raw.shape[1]//32*32, img1_raw.shape[0]//32*32))

# Convert to tensors
img0 = torch.from_numpy(img0_raw)[None][None].cuda() / 255.
img1 = torch.from_numpy(img1_raw)[None][None].cuda() / 255.
batch = {'image0': img0, 'image1': img1}

# Inference
with torch.no_grad():
    matcher(batch)
    mkpts0 = batch['mkpts0_f'].cpu().numpy()  # Matched keypoints in image0
    mkpts1 = batch['mkpts1_f'].cpu().numpy()  # Matched keypoints in image1
    mconf = batch['mconf'].cpu().numpy()      # Matching confidence scores
```

--------------------------------

### Install EfficientLoFTR with Conda and pip

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Sets up the conda environment, activates it, installs PyTorch 2.0.0 with CUDA 11.8, and installs the project requirements.

```shell
conda env create -f environment.yaml
conda activate eloftr
pip install torch==2.0.0+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt 
```

--------------------------------

### Set up symlinks for ScanNet and MegaDepth test subsets

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Creates symlinks from previously downloaded datasets to the test directories. Replace /path/to/... with actual paths.

```shell
ln -s /path/to/scannet-1500-testset/* /path/to/EfficientLoFTR/data/scannet/test
ln -s /path/to/megadepth-1500-testset/* /path/to/EfficientLoFTR/data/megadepth/test
```

--------------------------------

### Measure inference time for indoor models

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Activates the eloftr environment and runs the full and optimized inference timing scripts.

```shell
conda activate eloftr
bash scripts/reproduce_test/indoor_full_time.sh
bash scripts/reproduce_test/indoor_opt_time.sh
```

--------------------------------

### Post-process and visualize EfficientLoFTR matches

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Converts raw model outputs into readable keypoints and matches using a threshold of 0.2. Prints each match with its score and visualizes the matches on the images.

```python
# Post-process to get keypoints and matches in a readable format
image_sizes = [[(image.height, image.width) for image in images]]
outputs = processor.post_process_keypoint_matching(outputs, image_sizes, threshold=0.2)

# Print matching results
for i, output in enumerate(outputs):
    print(f"Image pair {i}:")
    print(f"Found {len(output['keypoints0'])} matches")
    for keypoint0, keypoint1, matching_score in zip(
            output["keypoints0"], output["keypoints1"], output["matching_scores"]
    ):
        print(
            f"Keypoint {keypoint0.numpy()} ↔ {keypoint1.numpy()} (score: {matching_score:.3f})"
        )

# Visualize matches
processor.visualize_keypoint_matching(images, outputs)
```

--------------------------------

### Advanced EfficientLoFTR Matching with Model Type and Precision Options

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Allows choosing between 'full' (best quality) and 'opt' (best efficiency) models, and precision modes 'fp32', 'mp' (mixed precision), or 'fp16'. Applies precision settings, runs inference with autocast for mixed precision, and visualizes matches.

```python
import torch
import cv2
import numpy as np
import matplotlib.cm as cm
from copy import deepcopy
from src.loftr import LoFTR, full_default_cfg, opt_default_cfg, reparameter
from src.utils.plotting import make_matching_figure

# Model configuration options
model_type = 'full'  # Choose: 'full' for best quality, 'opt' for best efficiency
precision = 'fp32'   # Choose: 'fp32', 'mp' (mixed precision), 'fp16' for best efficiency

# Load appropriate config
if model_type == 'full':
    _default_cfg = deepcopy(full_default_cfg)
elif model_type == 'opt':
    _default_cfg = deepcopy(opt_default_cfg)

# Set precision options
if precision == 'mp':
    _default_cfg['mp'] = True
elif precision == 'fp16':
    _default_cfg['half'] = True

# Initialize matcher
matcher = LoFTR(config=_default_cfg)
matcher.load_state_dict(torch.load("weights/eloftr_outdoor.ckpt")['state_dict'])
matcher = reparameter(matcher)

# Apply precision settings
if precision == 'fp16':
    matcher = matcher.half()

matcher = matcher.eval().cuda()

# Load and preprocess images
img0_raw = cv2.imread("path/to/image0.jpg", cv2.IMREAD_GRAYSCALE)
img1_raw = cv2.imread("path/to/image1.jpg", cv2.IMREAD_GRAYSCALE)
img0_raw = cv2.resize(img0_raw, (img0_raw.shape[1]//32*32, img0_raw.shape[0]//32*32))
img1_raw = cv2.resize(img1_raw, (img1_raw.shape[1]//32*32, img1_raw.shape[0]//32*32))

# Convert to tensors with appropriate precision
if precision == 'fp16':
    img0 = torch.from_numpy(img0_raw)[None][None].half().cuda() / 255.
    img1 = torch.from_numpy(img1_raw)[None][None].half().cuda() / 255.
else:
    img0 = torch.from_numpy(img0_raw)[None][None].cuda() / 255.
    img1 = torch.from_numpy(img1_raw)[None][None].cuda() / 255.

batch = {'image0': img0, 'image1': img1}

# Inference with different precision modes
with torch.no_grad():
    if precision == 'mp':
        with torch.autocast(enabled=True, device_type='cuda'):
            matcher(batch)
    else:
        matcher(batch)
    
    mkpts0 = batch['mkpts0_f'].cpu().numpy()
    mkpts1 = batch['mkpts1_f'].cpu().numpy()
    mconf = batch['mconf'].cpu().numpy()

# Post-process confidence scores for 'opt' model
if model_type == 'opt':
    mconf = (mconf - min(20.0, mconf.min())) / (max(30.0, mconf.max()) - min(20.0, mconf.min()))

# Visualize matches
color = cm.jet(mconf)
text = ['EfficientLoFTR', 'Matches: {}'.format(len(mkpts0))]
fig = make_matching_figure(img0_raw, img1_raw, mkpts0, mkpts1, color, text=text)
```

--------------------------------

### Configure and load EfficientLoFTR model

Source: https://github.com/zju3dv/efficientloftr/blob/main/notebooks/demo_single_pair.ipynb

Sets model type ('full' or 'opt') and precision ('fp32', 'mp', 'fp16'), loads pretrained outdoor weights, and applies reparameterization. For fp16, converts model to half precision. Output shows the default configuration dictionary.

```python
from src.loftr import LoFTR, full_default_cfg, opt_default_cfg, reparameter

# You can choose model type in ['full', 'opt']
model_type = 'full' # 'full' for best quality, 'opt' for best efficiency

# You can choose numerical precision in ['fp32', 'mp', 'fp16']. 'fp16' for best efficiency
precision = 'fp32' # Enjoy near-lossless precision with Mixed Precision (MP) / FP16 computation if you have a modern GPU (recommended NVIDIA architecture >= SM_70).

# You can also change the default values like thr. and npe (based on input image size)

if model_type == 'full':
    _default_cfg = deepcopy(full_default_cfg)
elif model_type == 'opt':
    _default_cfg = deepcopy(opt_default_cfg)
    
if precision == 'mp':
    _default_cfg['mp'] = True
elif precision == 'fp16':
    _default_cfg['half'] = True
    
print(_default_cfg)
matcher = LoFTR(config=_default_cfg)

matcher.load_state_dict(torch.load("weights/eloftr_outdoor.ckpt")['state_dict'])
matcher = reparameter(matcher) # no reparameterization will lead to low performance

if precision == 'fp16':
    matcher = matcher.half()

matcher = matcher.eval().cuda()
```

```text
Output:
{'backbone_type': 'RepVGG', 'align_corner': False, 'resolution': (8, 1), 'fine_window_size': 8, 'mp': False, 'replace_nan': True, 'half': False, 'backbone': {'block_dims': [64, 128, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'agg_size0': 4, 'agg_size1': 4, 'no_flash': False, 'rope': True, 'npe': [832, 832, 832, 832]}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'dsmax_temperature': 0.1, 'skip_softmax': False, 'fp16matmul': False, 'train_coarse_percent': 0.2, 'train_pad_num_gt_min': 200}, 'match_fine': {'local_regress_temperature': 10.0, 'local_regress_slicedim': 8}}
```

--------------------------------

### Load and run EfficientLoFTR inference with Transformers

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Uses AutoImageProcessor and AutoModel from the zju-community/efficientloftr checkpoint. The default processor resizes images to 480x640; modify config for high-resolution matching. Outputs include keypoints, matches, and matching scores.

```python
from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests

# Load example images (same as in the original paper)
url_image1 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg"
image1 = Image.open(requests.get(url_image1, stream=True).raw)
url_image2 = "https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/refs/heads/master/assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg"
image2 = Image.open(requests.get(url_image2, stream=True).raw)

images = [image1, image2]

# Load processor and model
processor = AutoImageProcessor.from_pretrained("zju-community/efficientloftr")
model = AutoModel.from_pretrained("zju-community/efficientloftr")

# Process images and run inference
inputs = processor(images, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
# keypoints = outputs.keypoints        # Keypoints in both images
# matches = outputs.matches            # Matching indices 
# matching_scores = outputs.matching_scores  # Confidence scores
```

--------------------------------

### Measure accuracy for outdoor and indoor models

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Activates the eloftr environment and runs the AUC evaluation scripts for both full and optimized models.

```shell
conda activate eloftr
bash scripts/reproduce_test/outdoor_full_auc.sh
bash scripts/reproduce_test/outdoor_opt_auc.sh
bash scripts/reproduce_test/indoor_full_auc.sh
bash scripts/reproduce_test/indoor_opt_auc.sh
```

--------------------------------

### Load images and run inference

Source: https://github.com/zju3dv/efficientloftr/blob/main/notebooks/demo_single_pair.ipynb

Loads two grayscale images, resizes them to multiples of 32, and runs EfficientLoFTR inference. For fp16 precision, images are converted to half; for mp, uses torch.autocast. Extracts matched keypoints and confidence scores.

```python
# Load example images
img0_pth = "assets/phototourism_sample_images/united_states_capitol_26757027_6717084061.jpg"
img1_pth = "assets/phototourism_sample_images/united_states_capitol_98169888_3347710852.jpg"
img0_raw = cv2.imread(img0_pth, cv2.IMREAD_GRAYSCALE)
img1_raw = cv2.imread(img1_pth, cv2.IMREAD_GRAYSCALE)
img0_raw = cv2.resize(img0_raw, (img0_raw.shape[1]//32*32, img0_raw.shape[0]//32*32))  # input size shuold be divisible by 32
img1_raw = cv2.resize(img1_raw, (img1_raw.shape[1]//32*32, img1_raw.shape[0]//32*32))

if precision == 'fp16':
    img0 = torch.from_numpy(img0_raw)[None][None].half().cuda() / 255.
    img1 = torch.from_numpy(img1_raw)[None][None].half().cuda() / 255.
else:
    img0 = torch.from_numpy(img0_raw)[None][None].cuda() / 255.
    img1 = torch.from_numpy(img1_raw)[None][None].cuda() / 255.
batch = {'image0': img0, 'image1': img1}

# Inference with EfficientLoFTR and get prediction
with torch.no_grad():
    if precision == 'mp':
        with torch.autocast(enabled=True, device_type='cuda'):
            matcher(batch)
    else:
        matcher(batch)
    mkpts0 = batch['mkpts0_f'].cpu().numpy()
    mkpts1 = batch['mkpts1_f'].cpu().numpy()
    mconf = batch['mconf'].cpu().numpy()
```

--------------------------------

### BibTeX citation for EfficientLoFTR

Source: https://github.com/zju3dv/efficientloftr/blob/main/README.md

Citation entry for the Efficient LoFTR paper.

```bibtex
@inproceedings{wang2024eloftr,
  title={{Efficient LoFTR}: Semi-Dense Local Feature Matching with Sparse-Like Speed},
  author={Wang, Yifan and He, Xingyi and Peng, Sida and Tan, Dongli and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2024}
}
```

--------------------------------

### Draw matching figure

Source: https://github.com/zju3dv/efficientloftr/blob/main/notebooks/demo_single_pair.ipynb

Visualizes the matched keypoints using matplotlib colormap. For the 'opt' model, normalizes confidence scores. Output is a figure with size 750x450.

```python
# Draw
if model_type == 'opt':
    print(mconf.max())
    mconf = (mconf - min(20.0, mconf.min())) / (max(30.0, mconf.max()) - min(20.0, mconf.min()))

color = cm.jet(mconf)
text = [
    'LoFTR',
    'Matches: {}'.format(len(mkpts0)),
]
fig = make_matching_figure(img0_raw, img1_raw, mkpts0, mkpts1, color, text=text)
```

```text
Result:
<Figure size 750x450 with 2 Axes>
```
Tokens:
Raw