Poor convergence when using non centered principal point

[btoussai]

Hello,
First of all, congratulation for this very good paper!

I would like to test your reconstruction method on my own dataset but my cameras have a non-centered principal point.
I have followed the instructions here to modify the projection matrix:
graphdeco-inria/gaussian-splatting#144 (comment)
With these modifications, the original 3D gaussian splatting implementation is able to converge nicely.
However, using the same modifications, your method does not converge well.
Is there anything else that I should change in your code so that I can use non-centered cameras ?

I have also changed the code so that I can use the remote SIBR gaussian viewer to follow the optimization interactively. It appears that some gaussians are added very close to the cameras instead of being added further away and thus preventing a good convergence. I've also tried to increase the distance of the near plane but its not effective. Do you have any tips ?

Thanks for your time.

[niujinshuchong]

Hi, the current code base assumes the principal point the the image is centered and is hard coded here https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/submodules/diff-gaussian-rasterization/cuda_rasterizer/forward.cu#L448. You could change the code to accept non-centered cameras and generate rays based on the camera parameters.