integrate-matcher

Integrate Matcher into image-matching-webui

This skill automates the integration of a new local feature matching method into the image-matching-webui project. Given a GitHub repository URL, it follows the project's established patterns to add the matcher as a fully functional option in the WebUI.

Prerequisites

• The target repository must be a local feature matching method (sparse or standalone)
• You must be working inside the image-matching-webui project root

Step-by-Step Integration Guide

Step 1: Analyze the Target Repository

1. Clone the repo to /tmp/<repo-name> for analysis (do NOT add as submodule yet)
2. Identify the matcher type:
   • Dense/Standalone matcher: Takes raw images as input, performs detect+describe+match internally (e.g., LoMa, RoMa, LoFTR). required_inputs = ["image0", "image1"]
   • Sparse matcher: Takes keypoints+descriptors as input, performs matching only (e.g., LightGlue, SuperGlue). required_inputs includes keypoints0, descriptors0, etc.
3. Find the core model class and its API:
   • How to initialize the model (constructor args, config options)
   • How to run inference (forward method signature)
   • What the model outputs (keypoints, matches, scores, etc.)
   • Model weight download URLs
4. Check dependencies in pyproject.toml or requirements.txt
5. Identify model variants (e.g., different sizes: B/L/G/R)
6. Check for device/amp issues:
   • Does the model use torch.autocast or mixed precision (mp, amp)?
   • Does it manage its own device placement (like LoMa's loma.device)?
   • On MPS/CPU, does it produce dtype mismatches?

Step 2: Add Git Submodule

git submodule add <repo-url> imcui/third_party/<RepoName>

• Use the original repo name (PascalCase) as the submodule directory name
• If the repo has a fork in the project's org (e.g., Vincentqyw/xxx or agipro/xxx), prefer the fork

⚠️ CRITICAL: Never modify third_party code directly

imcui/third_party/ contains pinned third-party submodules — you are NOT the owner of this code. If a dependency needs a compatibility fix (e.g., API changes in PyTorch/kornia, import path changes):

1. Fork the original repo to the agipro GitHub account
2. Apply the fix in the fork and push
3. Replace the submodule URL in the main repo to point to the fork:

   git submodule deinit -f imcui/third_party/<RepoName>
   git rm -f imcui/third_party/<RepoName>
   rm -rf .git/modules/imcui/third_party/<RepoName>
   git submodule add https://github.com/agipro/<RepoName>.git imcui/third_party/<RepoName>
   

4. Update .gitmodules — the URL must point to the fork

Example: EfficientLoFTR was forked to agipro/EfficientLoFTR to fix a kornia.utils.grid → kornia.utils import change required by kornia 0.8+.

Step 3: Create Matcher Implementation

Create imcui/hloc/matchers/<matcher-name>.py following these rules:

3.1 Import Pattern

import sys
from pathlib import Path

from .. import logger
from ..utils.base_model import BaseModel

# Add third_party to sys.path for import
<matcher>_path = Path(__file__).parent / "../../third_party/<RepoName>/src"
sys.path.append(str(<matcher>_path))

# Import the model class
from <module> import <ModelClass>

• If the third_party repo has src/ as the package root, append <path>/src
• Some repos put the package directly at root level — adjust accordingly

3.2 Class Definition

The class MUST:

• Inherit from BaseModel
• Be the only BaseModel subclass in the file (the dynamic_load function asserts this)
• Define default_conf dict with all configurable parameters
• Define required_inputs list

class MatcherName(BaseModel):
    default_conf = {
        "name": "two_view_pipeline",
        "model_name": "<default_variant>",
        "max_keypoints": 2048,
        # ... other config
    }
    required_inputs = [
        "image0",
        "image1",
        # For sparse matchers, also include:
        # "keypoints0", "scores0", "descriptors0",
        # "keypoints1", "scores1", "descriptors1",
    ]

3.3 _init Method

• Use self.conf (not the conf parameter) to access merged config
• Log model loading with logger.info()
• Handle model weight downloading:
  • If weights are on HuggingFace: use self._download_model(repo_id=MODEL_REPO_ID, filename=...)
  • If weights are auto-downloaded by the model (e.g., torch.hub), let it handle
  • If weights need manual download, document the URL
• Platform compatibility: On non-CUDA devices, disable mixed precision:

  if not torch.cuda.is_available():
      # Disable mp/amp for CPU/MPS compatibility
  

3.4 _forward Method

For standalone matchers (input: raw images):

The data dict contains preprocessed tensors:

• data["image0"]: shape (1, C, H, W), float32, range [0, 1]
• data["image1"]: shape (1, C, H, W)`, float32, range [0, 1]

If the model expects PIL images or file paths, convert:

img0 = data["image0"].cpu().numpy().squeeze() * 255
img0 = img0.transpose(1, 2, 0)  # CHW -> HWC
img0 = Image.fromarray(img0.astype("uint8"))

For sparse matchers (input: keypoints + descriptors):

# Repackage data for the model's expected format
input = {
    "image0": {"image": data["image0"], "keypoints": data["keypoints0"], ...},
    "image1": {"image": data["image1"], "keypoints": data["keypoints1"], ...},
}
return self.net(input)

3.5 Output Format

The _forward method MUST return a dict with these keys:

Dense matchers that output matched keypoints only:

pred = {
    "keypoints0": kpts0,      # torch.Tensor, shape (N, 2), pixel coords in resized image
    "keypoints1": kpts1,      # torch.Tensor, shape (N, 2)
    "mconf": confidence,      # torch.Tensor, shape (N,), match confidence scores
}

Dense matchers that can separate detected vs matched keypoints (PREFERRED):

pred = {
    "keypoints0": all_kpts0,     # All detected keypoints (for UI "Keypoints" display)
    "keypoints1": all_kpts1,     # All detected keypoints
    "mkeypoints0": matched_kpts0, # Matched keypoints (for UI match lines)
    "mkeypoints1": matched_kpts1, # Matched keypoints
    "mconf": confidence,          # Match confidence scores
}

Sparse matchers (LightGlue-style):

# Return the model's raw output — match_dense.py handles the rest
return self.net(input)

Key output rules:

• keypoints0/1: ALL detected keypoints (shown in UI "Open for More: Keypoints")
• mkeypoints0/1: Only MATCHED keypoints (shown as match lines in UI)
• If mkeypoints0/1 is missing, match_dense.py falls back to keypoints0/1
• Coordinates must be in pixel space of the resized image (not normalized [-1,1])
• mconf should be real confidence scores (not all-ones)

Step 4: Add Matcher Configuration

4.1 imcui/hloc/configs/matchers.py

Add a configuration entry for each model variant:

"<matcher-name>": {
    "output": "matches-<matcher-name>",
    "model": {
        "name": "<matcher-module-name>",  # Must match the .py filename in matchers/
        "model_name": "<variant>",         # Passed as conf["model_name"]
        "max_keypoints": 2048,
        # ... other model-specific config
    },
    "preprocessing": {
        "grayscale": False,     # True for LoFTR-style; False for most modern matchers
        "force_resize": True,
        "resize_max": 1024,
        "width": 640,
        "height": 480,
        "dfactor": 8,          # Image dimensions must be divisible by this
    },
},

Key rules:

• "name" in model must match the Python filename (e.g., "loma" → loma.py)
• Each model variant gets its own top-level entry (e.g., loma-b, loma-l, loma-g)
• preprocessing.grayscale: Set True only for models that expect 1-channel input (e.g., LoFTR)
• preprocessing.dfactor: Ensure resized dimensions are divisible by this value

4.2 config/app.yaml (local dev) AND imcui/config/app.yaml (package default)

BOTH files must be updated identically. Add an entry under matcher_zoo:

<DisplayName>:
  matcher: <matcher-config-name>    # Must match key in matchers.py
  standalone: true                   # true = takes two images directly (no separate extractor needed)
  skip_ci: false                     # DEFAULT: false. Only set to true for heavy matchers (see rule below)
  info:
    name: <DisplayName>              # Display name in WebUI dropdown
    source: "Venue Year"             # e.g., "ECCV 2026", "ICCV 2023"
    paper: <paper-url>               # arXiv or published paper URL
    github: <github-url>             # Official GitHub repo
    display: true                    # Whether to show in WebUI
    efficiency: medium               # low (heavy), medium, high (fast)

Key rules:

• standalone: true means the matcher takes two raw images directly (no separate feature extractor needed)
• standalone: false means the matcher requires a feature extractor (e.g., SuperPoint+LightGlue)
• For feature+matcher combos, use <extractor>+<matcher> format (e.g., superpoint+lightglue)
• Set enable: false for very heavy models that most users won't use by default
• CRITICAL: Both config/app.yaml and imcui/config/app.yaml must be kept in sync

skip_ci rule

skip_ci defaults to false — you can omit the field entirely for most matchers. Only set skip_ci: true when the model is too heavy to run in CI (GitHub Actions CPU-only Ubuntu runner with 7GB RAM):

Condition	skip_ci
Lightweight matcher (e.g., LightGlue, SuperGlue, XFeat, ALIKED-based)	false (or omit)
Heavy dense matcher prone to CI OOM/timeout (e.g., RoMa, DKM, GIM, Mast3R, LoMa-L/G/R, MINIMA large)	true

Rule of thumb: if the model has variants like B/L/G/R, the "B" (base) variant usually passes CI, larger ones may not. When unsure, omit skip_ci (defaults to false) and let CI tell you — if it OOMs, set it to true in a follow-up commit.

Step 5: Handle Platform Compatibility

Common issues and fixes:

5.1 Mixed Precision (MP/AMP) Issues

On MPS/CPU, torch.autocast with float16/bfloat16 causes:

• RuntimeError: Input type (c10::Half) and bias type (float) should be the same
• RuntimeError: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same

Fix pattern (before importing third-party code):

# Patch module-level amp_dtype before import
import <module>.device as _device
if not torch.cuda.is_available():
    _device.amp_dtype = torch.float32

# Also patch submodule local bindings
import <module>.submod as _submod
if not torch.cuda.is_available():
    _submod.amp_dtype = torch.float32

After model construction:

if not torch.cuda.is_available():
    cfg = dataclasses.replace(cfg, mp=False)
    for module in self.net.modules():
        if hasattr(module, "amp"):
            module.amp = False

5.2 Device Mismatch Issues

If the model manages its own device (like LoMa's loma.device):

# Override .to() to keep model on its expected device
def to(self, device=None, **kwargs):
    return super().to(<model_expected_device>, **kwargs)

5.3 Inference Mode vs No Grad

If a model uses @torch.inference_mode() but you need to pass its outputs to another module that requires grad tracking:

with torch.no_grad():
    output = model.detect_and_describe(...)
    output = output.clone()  # Detach from inference mode graph

Step 6: Update README.md Algorithm Table

The README.md contains a "The tool currently supports..." table listing all algorithms with their support status.

6.1 Table Format

| Algorithm        | Supported | Conference/Journal | Year | GitHub Link |
|------------------|-----------|--------------------|------|-------------|
| LoMa             | ✅ | ECCV    | 2026 | [Link](https://github.com/davnords/LoMa) |
| RIPE             | ✅ | ICCV    | 2025 | [Link](https://github.com/fraunhoferhhi/RIPE) |

Rows are sorted by year descending, then alphabetically by algorithm name within the same year.

6.2 Rules

Scenario	Action
Algorithm exists with ❌	Change to ✅
Algorithm not in table	Add new row, maintaining sort order
Algorithm already has ✅	Skip (no change needed)

6.3 Example

After integrating "DaD" published at ARXIV 2025:

| DaD            | ✅ | ARXIV   | 2025 | [Link](https://github.com/Parskatt/dad) |

Insert between RIPE (ICCV 2025) and MINIMA (ARXIV 2024).

Step 7: Pre-Commit Check (MANDATORY)

Before committing, pre-commit MUST pass. This is non-negotiable — commits that fail pre-commit will be rejected at the PR stage.

# Run ALL pre-commit hooks on all files
pre-commit run -a

# Or run specific hooks if you only changed certain files
pre-commit run ruff --all-files       # Python linting
pre-commit run mypy --all-files       # Type checking (if new Python code)

Common issues and fixes:

Hook	Common failure	Fix
ruff	unused imports, line too long	Remove unused imports, wrap long lines
ruff-format	inconsistent indentation	Let ruff format: ruff format <file>
mypy	missing type annotations	Add type hints to new functions
trailing-whitespace	blank lines with spaces	Strip trailing whitespace
check-yaml	invalid YAML syntax	Fix indentation/quoting in app.yaml
end-of-file-fixer	missing newline at EOF	Add trailing newline

If pre-commit run -a fails, fix all errors before proceeding to commit. Do NOT skip hooks with --no-verify.

Step 8: Verification Checklist

After integration and pre-commit pass, verify:

1. Import test: python -c "from imcui.hloc.matchers import <name>"
2. Model loading: The model loads without errors on CPU/MPS/CUDA
3. Inference test: Run matching on a test image pair
4. WebUI test: python app.py → select the matcher → run matching
5. Keypoints display: UI "Open for More: Keypoints" shows detected keypoints
6. Match lines display: UI shows correct match lines between images
7. Both config files: config/app.yaml and imcui/config/app.yaml are identical for the new matcher
8. Pre-commit passes: pre-commit run -a exits with zero

Step 9: Commit and Create PR

Once pre-commit passes and all verification checks are green:

# Stage the changes
git add .gitmodules imcui/third_party/<RepoName>
git add imcui/hloc/matchers/<name>.py
git add imcui/hloc/configs/matchers.py
git add imcui/config/app.yaml
git add config/app.yaml
git add README.md

# Commit with a descriptive message following convention
git commit -m "feat: integrate <MatcherName>

Add <MatcherName> matcher from <venue> <year>.

- Add submodule: imcui/third_party/<RepoName>
- Add matcher implementation: imcui/hloc/matchers/<name>.py
- Add matcher config entries
- Update app.yaml matcher zoo (both package and user configs)
- Update README.md algorithm table

Co-Authored-By: Claude <[email protected]>"

# Push and create PR
git push origin main
gh pr create --title "feat: integrate <MatcherName>" \
  --body "Integrate <MatcherName> matcher from <paper-link>.

## Changes
- [x] Submodule added
- [x] Matcher implementation
- [x] Config entries in both app.yaml files
- [x] README.md updated
- [x] Pre-commit passes

## Tested on
- [ ] CPU
- [ ] CUDA
- [ ] MPS

🤖 Generated with [Claude Code](https://claude.com/claude-code)"

Important notes:

• Use feat: prefix for new matcher integrations (following conventional commits)
• Always include Co-Authored-By: Claude <[email protected]> in commit messages
• End PR body with 🤖 Generated with [Claude Code](https://claude.com/claude-code)
• If CI fails after pushing, check the logs — if it's an OOM on your new matcher, set skip_ci: true in a follow-up commit

File Change Summary

For each integration, these files are typically modified:

File	Action	Description
.gitmodules	Modify	Add submodule entry
imcui/third_party/<RepoName>	Add	Git submodule
imcui/hloc/matchers/<name>.py	Create	Matcher implementation
imcui/hloc/configs/matchers.py	Modify	Add matcher config entries
config/app.yaml	Modify	Add WebUI display config
imcui/config/app.yaml	Modify	Add WebUI display config (must match config/app.yaml)
README.md	Modify	Update supported algorithms table (add row or flip ❌→✅)

Before committing all changes, run pre-commit run -a — all hooks must pass.

Reference Implementations

Pattern	File	Description
Dense standalone	imcui/hloc/matchers/roma.py	RoMa — raw images → matched keypoints
Dense standalone + separate detected/matched kpts	imcui/hloc/matchers/loma.py	LoMa — separates all detected from matched keypoints
Sparse matcher	imcui/hloc/matchers/lightglue.py	LightGlue — keypoints+descriptors → matches
Detector-only + external descriptor	imcui/hloc/extractors/raco.py	RaCo — detects keypoints, delegates descriptor to ALIKED

RaCo pattern: detector that needs a descriptor extractor

Some models are keypoint detectors only — they output keypoints/scores but no descriptors. The RaCo integration demonstrates chaining RaCo detection with ALIKED description:

1. Create an extractor (not matcher) in imcui/hloc/extractors/raco.py
2. In _forward, first run RaCo detection → keypoints, then run ALIKED descriptor on those keypoints
3. Register in configs/extractors.py with max_num_keypoints / nms_radius parameters
4. Register a matcher config in configs/matchers.py with features: "raco-aliked" — a custom LightGlue+ checkpoint trained for RaCo+ALIKED features
5. In app.yaml, the entry uses feature: raco (the extractor) + matcher: raco-lightglue (the LightGlue variant)

Fork priority for third-party fixes

When a submodule needs a compatibility fix:

Repo owner	Action
Vincentqyw/*	Push fix directly to the Vincentqyw fork
agipro/*	Push fix directly to the agipro fork
Anyone else	Fork to agipro, apply fix, switch submodule URL

Always verify push succeeded with git log --oneline -1 in the submodule directory.

Kornia compatibility (kornia >= 0.8)

Kornia 0.8.0 removed the kornia.utils.grid submodule. create_meshgrid moved to kornia.utils (0.8.0-0.8.2), then to kornia.geometry (0.8.3+). When fixing submodule imports, use this future-proof pattern:

try:
    from kornia.geometry import create_meshgrid  # kornia >= 0.8.3
except ImportError:
    from kornia.utils import create_meshgrid  # kornia < 0.8.3