Error in vtk_aruco_app.py

[rohang1411]

I was trying out the tutorial files and was successful with the first two files but the file vtk_aruco_app.py is showing some error(attached below). Not sure why?

Can someone pls help!

[rohang1411]

I tried to debug and observed that tag2camera list is empty. That's why calling the move_camera(tag2camera[0]) produces out of range error as there is no element in the tag2camera list. Can someone pls tell why tag2camera list is coming empty?

[thompson318]

Hello, thanks for raising this. tag2camera will be empty when there are no ArUco tags visible in the image. I think this is a bug at line 74 of vtk_aruco_app.py. Instead of doing

if tag2camera is not None:

we should do
if tag2camera:
as an empty list will return false.

Can you please try changing line 74 as above and let me know if that works for you?

[rohang1411]

Hey Thanks for replying. Yeah I tried the change you mentioned above but as soon as I bring the aruco tag in frame the model disappears instead of tracking it.

Before the change I was not able to see the camera feed, now I'm able to see it but as soon as I bring aruco tag in frame the model disappears.

[thompson318]

It might be todo with the clipping range if you're quite far from the camera. On line 61 we set the clipping range to be 10 - 800

self.vtk_overlay_window.set_camera_state({"ClippingRange": [10, 800]})

try increasing the 800 to 2000 or more and move your tag around. The model won't be exactly on the tag so you might have to move the tag to see the model.

[rohang1411]

Yeah I was able to make it work by changing the clipping range. I actually have a bunch of models to test so can you please guide me about clipping range. Like what does it signify and what does changing the range actually do like what happens when I increase the range or decrease the range. Although I'm testing it out on my side too through trial and error but it would be great if you could provide an insight.

[thompson318]

The clipping range in the distance in mm from the camera that VTK will render models. So 10:800 will render anything from 10 mm to 800 mm from the camera. Distance from camera is estimated by the ArUco tracker, however if the camera isn't calibrated properly or the tags are the wrong size, the estimated distance could be wrong, which will impact your display.

[rohang1411]

Oh Okay got it, thanks. So is there a proper way to determine clipping range or just trial and error.

[thompson318]

Probably trial and error, although it depends on your application. For my applications (key hole surgery) things are usually within 30 cm of the camera. But if you're doing something on a larger scale things might be metres away. I think increasing the clipping range leads to more processing power, but I haven't tested it much.

[rohang1411]

Thanks for all the help. Actually I converted .obj models into .vtk format but they all were augmenting very close to the camera even after experimenting with multiple clipping ranges. I just have a few questions in my mind.

1. Can we use any other type of 3D model files instead of vtk (ex. .obj). Is there any specific reason or advantage of .vtk format?
2. Can we use anything other than aruco tags or follow a markerless approach?
3. If I add a depth camera then will it be any helpful or is there any provision for using to a depth camera to increase the accuracy? Personally I don;t think that it would be helpful since it's a marker based approach so depth data wouldn'tbe used.

Sorry for disturbing you again and again but you've been kind enough to explain everything in depth. Can you pls share your email id (if you don't mind) where I can get in touch with you if have any doubts in the future.

[thompson318]

Hello. Answers below:

1. We're just using the file formats supported by VTK by default. At line 102 of vtk_aruco_app.py we call viewer.add_vtk_models_from_dir which eventually calls VTKSurfaceModel to try and read any files in the directory. Checking that function tells me it should work for .vtk .stl .ply and .vtp formats. Ultimately any 3D model you want to use needs to be converted into a vtkActor to be displayed in our overlay window. Provided someone has implemented a file reader that does that it should be possible.
2. The tutorial only covers using the ArUco tracker as it is usable on just about any system. We use NDI trackers clinically (Optical and electromagnetic) so we have SciKit-SurgeryNDITracker which could be easily swapped into the tutorial code. Tracking with different technology should be possible, you'd need to implement a Python interface that supplies the functions (a get_frame function that returns the tracking matrices). We're looking at doing this for some other tracking systems but it's not a priority at the moment.
3. We've got no plans for depth cameras at the moment. If we want high accuracy we tend to use the NDI tracking systems.

If you want to get in touch for more general queries I'm trying to get people to use the discussion board at https://github.com/UCL/scikit-surgery/discussions which should get a pretty quick response and means that other people can help out too.