Detail_HomographyBasedEstimator Core dumped

[dominikheinz]

Hello,

I'm implementing a custom image stitching pipeline using opencv-rust, specifically working with feature matching between two images and camera parameter estimation from homography.

Important Notes:

• I'm aware OpenCV provides a stiching API which handles what I am doing automatically. I intentionally don't use it, I am trying to create my own pipeline using the same modules that are used by the official stiching API.
• I am deliberately avoiding whole-image feature detection with masks to ensure uniform feature distribution in specific overlap regions

Description of the Problem:

Witt that in mind, I threw together a quick selfcontained example that showcases the crash I am encountering.

1. Load two adjacent images
2. Detect features in overlap regions:
   a. Right side of image 1
   b. Left side of image 2
3. Compute feature matches
4. Apply RANSAC filtering
5. Estimate camera parameters using Detail_HomographyBasedEstimator
   a. Crashes during apply() function call

Runtime crash message:

opencvtest on  master 
❯ cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/opencvtest`
Matches after RANSAC: 340/1000
/usr/include/c++/14.2.1/bits/stl_vector.h:1149: std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](size_type) const [with _Tp = cv::detail::MatchesInfo; _Alloc = std::allocator<cv::detail::MatchesInfo>; const_reference = const cv::detail::MatchesInfo&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
zsh: IOT instruction (core dumped)  cargo run

Example code:

use opencv::{
    calib3d::{find_homography, RANSAC},
    core::{no_array, KeyPoint, KeyPointTrait, Mat, Point2f, Vector, NORM_L2},
    features2d::{BFMatcher, SIFT},
    imgcodecs::{imread, IMREAD_COLOR},
    prelude::*,
    stitching::{Detail_HomographyBasedEstimator, Detail_ImageFeatures, Detail_MatchesInfo},
};

fn main() -> anyhow::Result<()> {
    // Load images and setup SIFT
    let img1 = imread("1.jpg", IMREAD_COLOR)?;
    let img2 = imread("2.jpg", IMREAD_COLOR)?;
    let mut sift = SIFT::create(1000, 3, 0.04, 10.0, 1.6, false)?;

    // Process right half of img1
    let roi1 = Mat::roi(
        &img1,
        opencv::core::Rect::new(img1.cols() / 2, 0, img1.cols() / 2, img1.rows()),
    )?;
    let mut kp1 = Vector::new();
    let mut desc1 = Mat::default();
    sift.detect_and_compute(&roi1, &no_array(), &mut kp1, &mut desc1, false)?;

    // Adjust keypoints to full image space
    let adj_kp1: Vector<KeyPoint> = kp1
        .iter()
        .map(|kp| {
            let mut new_kp = kp.clone();
            let pt = kp.pt();
            new_kp.set_pt(Point2f::new(pt.x + img1.cols() as f32 / 2.0, pt.y));
            new_kp
        })
        .collect();

    // Process left half of img2
    let roi2 = Mat::roi(
        &img2,
        opencv::core::Rect::new(0, 0, img2.cols() / 2, img2.rows()),
    )?;
    let mut kp2 = Vector::new();
    let mut desc2 = Mat::default();
    sift.detect_and_compute(&roi2, &no_array(), &mut kp2, &mut desc2, false)?;

    // Create ImageFeatures objects
    let mut features1 = Detail_ImageFeatures::default();
    features1.set_img_size(opencv::core::Size::new(img1.cols(), img1.rows()));
    features1.set_keypoints(adj_kp1);
    features1.set_descriptors(desc1.get_umat(
        opencv::core::AccessFlag::ACCESS_READ,
        opencv::core::UMatUsageFlags::USAGE_DEFAULT,
    )?);

    let mut features2 = Detail_ImageFeatures::default();
    features2.set_img_size(opencv::core::Size::new(img2.cols(), img2.rows()));
    features2.set_keypoints(kp2);
    features2.set_descriptors(desc2.get_umat(
        opencv::core::AccessFlag::ACCESS_READ,
        opencv::core::UMatUsageFlags::USAGE_DEFAULT,
    )?);

    // Match features
    let mut matches = Vector::new();
    BFMatcher::new(NORM_L2, true)?.train_match(
        &features1.descriptors(),
        &features2.descriptors(),
        &mut matches,
        &no_array(),
    )?;

    // Convert matches to points for RANSAC
    let points: (Vec<Point2f>, Vec<Point2f>) = matches
        .iter()
        .map(|m| {
            (
                features1
                    .keypoints()
                    .get(m.query_idx as usize)
                    .unwrap()
                    .pt(),
                features2
                    .keypoints()
                    .get(m.train_idx as usize)
                    .unwrap()
                    .pt(),
            )
        })
        .unzip();

    // Find homography using RANSAC
    let mut mask = Mat::default();
    let h = find_homography(
        &Mat::from_slice(&points.0)?,
        &Mat::from_slice(&points.1)?,
        &mut mask,
        RANSAC,
        3.0,
    )?;

    // Create MatchesInfo with filtered matches
    let mut matches_info = Detail_MatchesInfo::default()?;
    let mut inlier_matches = Vector::new();

    // Filter matches using mask
    let mask_slice = mask.data_typed::<u8>()?;
    for (i, m) in matches.iter().enumerate() {
        if mask_slice[i] != 0 {
            inlier_matches.push(m.clone());
        }
    }

    matches_info.set_matches(inlier_matches.clone());
    matches_info.set_num_inliers(inlier_matches.len() as i32);
    matches_info.set_h(h);
    matches_info.set_confidence(inlier_matches.len() as f64 / matches.len() as f64);

    println!(
        "Matches after RANSAC: {}/{}",
        inlier_matches.len(),
        matches.len()
    );

    // Prepare for camera estimation
    let mut features_vec = Vector::new();
    features_vec.push(features1);
    features_vec.push(features2);

    let mut pairwise_matches = Vector::new();
    pairwise_matches.push(matches_info);

    // Estimate cameras
    let mut cameras = Vector::new();
    Detail_HomographyBasedEstimator::new_def()?.apply(
        &features_vec,
        &pairwise_matches,
        &mut cameras,
    )?;

    // Print camera parameters
    let camera = cameras.get(0)?;
    println!(
        "Camera: focal={}, ppx={}, ppy={}, aspect={}",
        camera.focal(),
        camera.ppx(),
        camera.ppy(),
        camera.aspect()
    );

    Ok(())
}

I would like to know what exactly causes the crash? I assume it has to do with how I feed the matches/keypoints to the HomographyBasedEstimator?

System Information:

1. Operating System: Arch Linux 6.11.1-arch1-1
2. OpenCV Installation Method: Simply via pacman.
3. OpenCV version: 0.93.4
4. Rustc version: rustc 1.80.0 (051478957 2024-07-21)
5. RUST BACKTRACE:

❯ RUST_BACKTRACE=full cargo build -vv 
       Fresh libc v0.2.162
       Fresh glob v0.3.1
       Fresh memchr v2.7.4
       Fresh regex-syntax v0.8.5
       Fresh shlex v1.3.0
       Fresh jobserver v0.1.32
       Fresh autocfg v1.4.0
       Fresh percent-encoding v2.3.1
       Fresh once_cell v1.20.2
       Fresh dunce v1.0.5
       Fresh vcpkg v0.2.15
       Fresh pkg-config v0.3.31
       Fresh aho-corasick v1.1.3
       Fresh cc v1.2.1
       Fresh regex-automata v0.4.9
       Fresh semver v1.0.23
       Fresh anyhow v1.0.93
       Fresh clang-sys v1.8.1
       Fresh regex v1.11.1
       Fresh num-traits v0.2.19
       Fresh clang v2.0.0
       Fresh opencv-binding-generator v0.93.0
       Fresh opencv v0.93.4
       Fresh opencvtest v0.1.0 (/home/void/projects/opencvtest)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s