Metashape equirectangular issue

[gradeeterna]

Hey,

I was just testing the updated Metashape support with equirectangular images, but it doesn't seem to be working correctly.

#1605 @f-dy

Here is the point cloud in Metashape. camera type set to spherical:

And the NeRF from the same viewpoint after around 5K steps (I also trained for 30K but looked similar):

[f-dy]

I didn't have a dataset to test it, so this was just guessing from the doc. Can you share a sample dataset?

[gradeeterna]

Hey Frédéric,

Sure, I will send you a DM on Discord with the dataset.

Thanks!

[f-dy]

ok! my ID on discord is the same as on github (f-dy)

[gradeeterna]

I sent yesterday, but it's probably gone in to your message requests folder. Thanks!

[f-dy]

I still don't see it, so maybe I don't know how to use discord. Can you send me a link by email (frederic dot devernay at m4x dot org) or maybe post a link here to a public equirectangular dataset, if such a think exists (I couldn't find one).

My guess is that there's a 90° or 180° rotation issue between nerfstudio and metashape.

It would be nice to have a freely available multi-view panoramic dataset for nerfstudio 😉

[gradeeterna]

I just emailed it to you, thanks!

[aaronhdgolf]

Any update on this?

[aaronhdgolf]

Yah can't get this to work I am able to convert the metashape file but its almost like its not knowing its a 360 image

[aaronhdgolf]

Bumping this to see if we can get anything?

[f-dy]

I'm looking at this. I have to replicate the way it is done for COLMAP (ns-process-data creates multiple perspective views per equirectangular image).

[f-dy]

OK here's the solution. TL;DR, nerfstudio doesnt't support the equirectangular sensor, but knows how to make planar projections from these images.

Here's my recipe, tell me if you can reproduce it. I'll submit a PR with documentation for that.

dataset="WaterlilyPath"
NERFSTUDIO="$HOME/data/nerfstudio"
PROCESSED_DATA_DIR="data/custom/${dataset}"
IMAGES_PATH="${PROCESSED_DATA_DIR}/raw_images"

cd "${NERFSTUDIO}"
mkdir "${PROCESSED_DATA_DIR}"
mkdir "${IMAGES_PATH}"
# upload images in $nerfstudio/data_$dataset/raw_images

# then preprocess:
cd "${NERFSTUDIO}"
# Generate planar projections, but don't run colmap
ns-process-data images --camera-type equirectangular --data "${IMAGES_PATH}" --output-dir "${PROCESSED_DATA_DIR}" \
  --skip-colmap --skip-image-processing --images-per-equirect 14
# Align planar projections with metashape
time ./run_metashape.py -i "${IMAGES_PATH}/planar_projections" -o "${PROCESSED_DATA_DIR}"
ns-process-data metashape --data "${IMAGES_PATH}/planar_projections" \
  --xml "${PROCESSED_DATA_DIR}/cameras.xml" --output-dir "${PROCESSED_DATA_DIR}" \
  --max-dataset-size -1

[f-dy]

ok there are a few mis-calibrated planes and a couple of "sky" floaters, but overall it works

[gradeeterna]

Thanks for looking in to this. If I understand correctly, this wouldn't use the equirectangular alignment from Metashape at all, and would instead just use ns-process-data to slice the equirectangular images into 8 or 14 perspective views, and align those in Metashape with the regular "frame" camera type?

[tancik]

OK here's the solution. TL;DR, nerfstudio doesnt't support the equirectangular sensor, but knows how to make planar projections from these images.

Nerfstudio does support equirectangular sensors -

nerfstudio/nerfstudio/cameras/cameras.py

Line 47 in 7887b14

EQUIRECTANGULAR = auto()

[aaronhdgolf]

OK here's the solution. TL;DR, nerfstudio doesnt't support the equirectangular sensor, but knows how to make planar projections from these images.

Nerfstudio does support equirectangular sensors -

nerfstudio/nerfstudio/cameras/cameras.py

Line 47 in 7887b14

EQUIRECTANGULAR = auto()

How can we get this to work? is there a command to let nerf know its a EQUIRECTANGULAR when we are processing it.

[tancik]

The camera type can be specified in the json - https://docs.nerf.studio/en/latest/quickstart/data_conventions.html
You can specify ns-process-data images --camera-type equirectangular but this will spilt up the images into multiple perspective images since colmap does not support equirectangular.

[aaronhdgolf]

Yes but when we do ns-train it doesn't know the camera-type

[tancik]

Assuming you are using the nerfstudio dataset format, ns-train takes in the transforms.json which contain the field camera_model. The value is read here -

nerfstudio/nerfstudio/data/dataparsers/nerfstudio_dataparser.py

Line 255 in 7887b14

camera_type = CAMERA_MODEL_TO_TYPE[meta["camera_model"]]

[gradeeterna]

Do you have any examples of a nerfstudio transforms.json with equirectangular poses?

Attached an xml exported from Metashape using the spherical camera model below.

[gradeeterna]

Attaching as txt file, as can't upload xml here
equi.txt

[f-dy]

@tancik setting the camera_model is exactly what I did previously, here:

nerfstudio/nerfstudio/process_data/metashape_utils.py

Line 76 in 7887b14

data["camera_model"] = CAMERA_MODELS["equirectangular"].value

But unfortunately, CAMERA_MODELS["equirectangular"] == CameraModel.OPENCV, so in the end the camera was treated as perspective.

Should we just add EQUIRECTANGULAR to the CameraModel Enum?

[f-dy]

Should we just add EQUIRECTANGULAR to the CameraModel Enum?

I'm testing this.

[tancik]

Yea, the reason it was like that before was because the COLMAP scripts use a perspective model when you specify --camera-type equirectangular. That logic should really exist in the colmap scripts, not the process_data_utils.py.

[f-dy]

It works!

[f-dy]

OK so I updated #1841 with the solution for Metashape, but it probably broke the COLMAP solution to equirectangular, so I converted it to draft in the mean time

[f-dy]

@tancik what if I simply add self.camera_type = "perspective" after each call to generate_planar_projections_from_equirectangular? Looks like it would do the job

[f-dy]

ok this works. COLMAP works again:

Updated #1841 - ready to review

[aaronhdgolf]

It works!

Tried with the latest and still don't look clear at all
am I missing something?

[tancik]

Is the camera path what you expect?

[rlesac]

Is the camera path what you expect?

Yes, the visualization in the viewer is correct as are the images loaded.

[tancik]

Hmm, I'm at a bit of a loss then.

[aaronhdgolf]

What version of metashape are you using?

If you are using 2.0 and above you need to remove some lines from the xml I believe

I think if was lines 14-17
Remove this
calibration type="spherical" class="adjusted">
resolution width="5888" height="2944"/>
937.10430492507987
/calibration>

Then process the xml.

[rlesac]

What version of metashape are you using?

If you are using 2.0 and above you need to remove some lines from the xml I believe

I think if was lines 14-17 Remove this calibration type="spherical" class="adjusted"> resolution width="5888" height="2944"/> 937.10430492507987 /calibration>

Then process the xml.

Yes, I'm using 2.0 Metashape.

Wow! I'm only at 250 steps, but I can already see some shapes.
How did you find out that the calibration tag was the issue?

Also, this should be fixed in the ns-process-data script.

[aaronhdgolf]

What version of metashape are you using?
If you are using 2.0 and above you need to remove some lines from the xml I believe
I think if was lines 14-17 Remove this calibration type="spherical" class="adjusted"> resolution width="5888" height="2944"/> 937.10430492507987 /calibration>
Then process the xml.

Yes, I'm using 2.0 Metashape.

Wow! I'm only at 250 steps, but I can already see some shapes. How did you find out that the calibration tag was the issue?

Also, this should be fixed in the ns-process-data script.

gradeeterna actually figured it out because all his stuff was for 1.8 and I was using 2.0 so we knew something was different.

[devernay]

Can you share the camera.xml file and the edit you did? That's weird cause I wrote and tested the code with Metashape 2.0

[gradeeterna]

Can you share the camera.xml file and the edit you did? That's weird cause I wrote and tested the code with Metashape 2.0

Hey, it turned out to not be related to Metashape versions, as I had the issue with both 1.8 and 2.0.

The issue is that some spherical camera.xml files contain these "calibration" lines 14-17, where the "f" number messes things up. I'm not sure why some have this, and some don't. The sample data I sent you didn't have this.

<sensors next_id="1">
  <sensor id="0" label="unknown" type="spherical">
    <resolution width="5888" height="2944"/>
    <property name="layer_index" value="0"/>
    <bands>
      <band label="Red"/>
      <band label="Green"/>
      <band label="Blue"/>
    </bands>
    <data_type>uint8</data_type>
    <calibration type="spherical" class="adjusted">
      <resolution width="5888" height="2944"/>
      <f>937.10493518152987</f>
    </calibration>
    <black_level>0 0 0</black_level>
    <sensitivity>1 1 1</sensitivity>
  </sensor>
</sensors>

For now, I'm just manually deleting these lines before running ns-process metashape

[antoinebio]

just a follow up on that equirectangular export from metashape
most recent metashape version doesn't include calibration type field in the xml

but ns-process-data doesn't work correctly.

images are parse but nothing (extrinsic, intrisic cam model) is derived from the xml to the transform.json file which is empty.
images and downscaled images folder are well generated.

[f-dy]

can you share cameras.xml and transforms.json?

spherical sensors don't have a calibration, and this is already handled here

so the error must be somewhere else.

Note that I don't recommend building a NeRF from omni images, unless your scene is far from the camera: those cameras do not have a single optical center, and this creates blurry radiance fields if the scene is too close

[antoinebio]

Hi @devernay

please find the tranforms.json attached. As a result of the command it's an empty json.
transforms.zip

and please also find xml generated by metashape (here with a version 2.0.2 so with the most recent format , not in version 1.8 or earlier).
cameras_pose.zip

you will notice there is not longer calibration label in the xml.

source project comes from metashape.

as you mentioned

spherical sensors don't have a calibration

but this should be handle either by https://github.com/nerfstudio-project/nerfstudio/blob/main/nerfstudio/process_data/metashape_utils.py#L62 or an alternative way if using gaussian splatting (that script > https://github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py )

here is what gives the command into nerfstudio that convert metashape xml info into COLMAP format
ns-process-data metashape --data {data directory} --xml {xml file} --output-dir {output directory}

so first part of the script (dowscaled images to 2, 4, 8) is working
second part is skipped when camera pose has to be stored in transform.json.

However from my understanding both scripts (resp the one used in NERFSTUDIO and the other used to prepare dataset for Gaussian Splatting) don't manage cam model in equirectangular format (spherical).
Both need pinhole cam model to feed the transforms.json (for NERF) or images.bin (in sparse folder for Gaussian Splatting).

so both methods generated intermediate cube map models to get pinhole images. This is time consuming and also leads to more images to train (x 6 with cube map model).

so when one wants to work with a 360° cam such as INSTA360 RS ONE, there is 2 solutions to use such imageries.
A/ work with fisheye cam model into metashape because such 360° camera are designed with 2 fisheye lens => 2 pinhole cam model, therefore extrinsic + intrisic values should be exported correctly and you benefit from fast SFM of metashape to get tiepoints cloud (initial alignment).

B/ use equirectangular source imageries from 360°cam, and use generic command from NERF that will convert equirect model into cube map model then launch COLMAP for SFM preprocess (initial alignment). this is longer to execute (data preparation , intermediate images x6 ), hence because of last step , SFM step takes a while vs metashape.

ns-process-data images --camera-type equirectangular --images-per-equirect {8, or 14} --crop-factor {top bottom left right} --data {data directory} --output-dir {output directory}

[f-dy]

ok this happens because all your cameras are in a group called "insta360" and groups are not handled. Just remove the line <group id="0" label="insta360" type="folder"> and the line </group> from the xml and it should work. I'll push a fix to handle groups.

[antoinebio]

ok this happens because all your cameras are in a group called "insta360" and groups are not handled. Just remove the line <group id="0" label="insta360" type="folder"> and the line </group> from the xml and it should work. I'll push a fix to handle groups.

I will test it .

[antoinebio]

ok this happens because all your cameras are in a group called "insta360" and groups are not handled. Just remove the line <group id="0" label="insta360" type="folder"> and the line </group> from the xml and it should work. I'll push a fix to handle groups.

[antoinebio]

@f-dy
I was able to reproduce the same issue with fisheye cam model

ok this happens because all your cameras are in a group called "insta360" and groups are not handled. Just remove the line <group id="0" label="insta360" type="folder"> and the line </group> from the xml and it should work. I'll push a fix to handle groups.

and if I apply your solution it doesn't work neither
so there seems to be an issue with fisheye cam model ?

xml is available here

cam_pose.zip

[f-dy]

It works for me:
94 images were skipped because they were missing camera poses.
Final dataset is 906 frames.

My guess is that the images are not available in the images directory you're pointing to? Verbose execution will tell you more.

In the meantime, I made PR: #2626

[f-dy]

If you're trying to calibrate the raw fisheye images, you should check out:
https://www.agisoft.com/pdf/metashape-pro_2_0_en.pdf p 28 Rigid camera rig data

[antoinebio]

If you're trying to calibrate the raw fisheye images, you should check out: https://www.agisoft.com/pdf/metashape-pro_2_0_en.pdf p 28 Rigid camera rig data

not sure I understand why we should absolutly use CAM RIG method. Is it because cam group concept from metashape is not well handled by ns-process-data metashape method ?

[antoinebio]

It works for me: 94 images were skipped because they were missing camera poses. Final dataset is 906 frames.

My guess is that the images are not available in the images directory you're pointing to? Verbose execution will tell you more.

In the meantime, I made PR: #2626

still....

and json is empty

[f-dy]

the camera rig means that the pose of each slave camera within the rig is fix with respect to the master camera, so that if you have a rig with 6 cameras, and take pictures from n rig positions, theres only 5+n sets of extrinsics to compute, instead of 6*n.

Please execute ns-process-data with the --verbose flag. as I said above, my guess is that it doesn't find the raw images, and thus skips them

[antoinebio]

Please execute ns-process-data with the --verbose flag. as I said above, my guess is that it doesn't find the raw images, and thus skips them

here it is (images are written and renamed into frame_xxx , that's it)

[antoinebio]

the camera rig means that the pose of each slave camera within the rig is fix with respect to the master camera, so that if you have a rig with 6 cameras, and take pictures from n rig positions, theres only 5+n sets of extrinsics to compute, instead of 6*n.

Please execute ns-process-data with the --verbose flag. as I said above, my guess is that it doesn't find the raw images, and thus skips them

which means a faster SFM processing step with colmap ? what is the benefit for NERF to get less extrinsics to compute ?

[f-dy]

which means a faster SFM processing step with colmap ? what is the benefit for NERF to get less extrinsics to compute ?

You mean metashape?
Less parameters = more constraints = images are easier and faster to match.

[f-dy]

in metashape_utils.py, after the line

            if camera_label not in image_filename_map:

add:

                if verbose:
                    CONSOLE.print(f"Missing image for {camera.get('label')}, Skipping")

You can also print image_filename_map to see what it contains. My guess is that it can't find the images

[antoinebio]

which means a faster SFM processing step with colmap ? what is the benefit for NERF to get less extrinsics to compute ?

You mean metashape? Less parameters = more constraints = images are easier and faster to match.

this is true.
but except if you design your own camera rig set up, relative cam position is not always known accurately (even for 2 opposite fisheye cams like the below screencaps).
you need to calcul in a euclidian ref fram the position of entrance pupil of slave camera(s) vs master camera.

in outdoor conditions you can also geotag each camera (with an gnss device). that was our scenario here. SFM is also speed up when you give pre-position during initial alignement. Here again you have to take into account a leverarm between gnss and optical center or entrance pupil.
but cam rig is a suitable option too.... as long as you know the geometry of your rig.

[f-dy]

You don't have to know the exact geometry: Metashape optimizes the intra-rig camera poses too

[f-dy]

btw @antoinebio I noticed that in your metashape project you used the same sensor for the "AV" and "AR" groups, but from the photos above, I guess these are actually different sensors. You should have as many sensors as there are physical cameras in your metashape project

[antoinebio]

btw @antoinebio I noticed that in your metashape project you used the same sensor for the "AV" and "AR" groups, but from the photos above, I guess these are actually different sensors. You should have as many sensors as there are physical cameras in your metashape project

Correct
Let me test again the right way with 2 cam models.

I will also try your strategy

in metashape_utils.py, after the line

            if camera_label not in image_filename_map:

add:

                if verbose:
                    CONSOLE.print(f"Missing image for {camera.get('label')}, Skipping")

You can also print image_filename_map to see what it contains. My guess is that it can't find the images

without group label in the xml (as suggested)

it's copying the source images to the destination folder

but at the end of the process...

and transforms.json is still empty...

[ariannaferraris]

in metashape_utils.py, after the line

            if camera_label not in image_filename_map:

add:

                if verbose:
                    CONSOLE.print(f"Missing image for {camera.get('label')}, Skipping")

You can also print image_filename_map to see what it contains. My guess is that it can't find the images

@f-dy Hello, I have the same issue and, by adding this, I get a "Missing image" line for every camera. I don't get how I should be able to fix it though. For more info regarding my issue, you can find it here #3573