Projection issue with drawing 2D lines in 3D

[GBagley]

Related to this discussion: asny/three-d#310

We've been using the new pixel_at_position method to convert 3D world coordinate lines into 2D screen coordinates updated every frame, and then drawing them with a 2D camera. For example, to draw a reference grid on the ground, and keep line width the same regardless of distance from camera. This works fine when the camera is relatively far away from the lines, but when the camera gets closer, the projection starts to get wacky. Some example snapshots below. Not sure if we're just not using the feature correctly, or if something else is going on.

Examples that look ok, with the camera further back:

Examples that get weird, with the camera closer in:

I made some minimal changes to the shapes example that show the issue. Here's the diff. Basically just creating some 3d lines (start and end points in 3d, where I want them) that form a uniform grid at Z=0. And then in each frame of the render loop converting that into a set of 2d Lines, calling the pixel_at_position method on each to get the screenspace 2D start and end coordinates, and then rendering those.

 
diff --git a/examples/shapes/src/main.rs b/examples/shapes/src/main.rs
index 671bdb1d..d213d2f1 100644
--- a/examples/shapes/src/main.rs
+++ b/examples/shapes/src/main.rs
@@ -18,7 +18,7 @@ pub fn main() {
         0.1,
         1000.0,
     );
-    let mut control = OrbitControl::new(*camera.target(), 1.0, 100.0);
+    let mut control = OrbitControl::new(*camera.target(), 1.0, 1000.0);
 
     let mut sphere = Gm::new(
         Mesh::new(&context, &CpuMesh::sphere(16)),
@@ -95,7 +95,48 @@ pub fn main() {
     let light0 = DirectionalLight::new(&context, 1.0, Srgba::WHITE, &vec3(0.0, -0.5, -0.5));
     let light1 = DirectionalLight::new(&context, 1.0, Srgba::WHITE, &vec3(0.0, 0.5, 0.5));
 
+
+    struct Line3d {
+        world_start: Vec3,
+        world_end: Vec3,
+    }
+    let mut lines_3d: Vec<Line3d> = Vec::new();
+    let grid_spacing = 10.0;
+    let grid_length = 1000.0;
+    for i in 0..100 {
+        let y = i as f32 * grid_spacing;
+        lines_3d.push(Line3d {
+            world_start: vec3(0.0, y, 0.0),
+            world_end: vec3(grid_length, y, 0.0),
+        });
+    }
+    for i in 0..100 {
+        let x = i as f32 * grid_spacing;
+        lines_3d.push(Line3d {
+            world_start: vec3(x, 0.0, 0.0),
+            world_end: vec3(x, grid_length, 0.0),
+        });
+    }
+
+
+
     window.render_loop(move |mut frame_input| {
+
+        let mut lines_2d: Vec<Gm<Line, ColorMaterial>> = lines_3d.iter().map(|line| {
+            Gm::new(
+                Line::new(
+                    &context,
+                    camera.pixel_at_position(line.world_start),
+                    camera.pixel_at_position(line.world_end),
+                    5.0,
+                ),
+                ColorMaterial {
+                    color: Srgba::BLACK,
+                    ..Default::default()
+                },
+            )
+        }).collect();
+
         camera.set_viewport(frame_input.viewport);
         control.handle_events(&mut camera, &mut frame_input.events);
 
@@ -114,6 +155,14 @@ pub fn main() {
                     .chain(&bounding_box_cylinder),
                 &[&light0, &light1],
             );
+  
+            frame_input
+                .screen()
+                .render(
+                    &Camera::new_2d(frame_input.viewport),
+                    &lines_2d,
+                    &[],
+                );
 
         FrameOutput::default()
     });

[asny]

Hi.

Thanks for the report and thanks for the detailed description 🙏 I think it makes sense, therefore I didn't debug it, but I'll do that if you don't buy my explanation 🙂

So pixel_at_position returns the pixel in the viewport of the Camera and works just fine if the position is actually inside the viewport, ie. if you can actually see the point from the view of the Camera. However, what happens when the position is outside of the Cameras viewport? Well, it probably works ok just outside of the viewport, it will just return a pixel that is outside of the viewport, but you don't really care about that since it's ok if the line end point is outside of the viewport. However, if the point is close to being in the Cameras image plane, numerical issues are bound to arise.

So that's what I think happens, to verify, you can print the pixel positions and see if they go crazy when you have issues. You can see the math here if you're interested, the problem arrises when you try to project the point onto the image plane.

To fix it, I think you need to ensure that your 3D end points are relatively inside the viewport of the camera before using the pixel_at_position method.

[GBagley]

Yeah, that makes sense. I did see some pixel position values going crazy while debugging, like you mention. We were trying to use relatively long lines in 3D space to make a grid covering the 'ground' over a large space. But that means that the endpoints of any given line are usually far outside of the viewport. Maybe I can flip things around a bit and use the methods you have to first get the extents of the 3d locations currently in the viewport, and then just draw the relevant lines based on those extents, so they don't extend too far past what's visible

[asny]

I think that sounds like a good plan 👍 Do you have any luck so far?

[GBagley]

This wasn't quite as straightforward as I had thought. I think I would need to clip the 3d lines to the extents of the view frustum, but not sure offhand how to do that. I see there is an in_frustum method in the three_d camera, but that's just a true/false on whether a bounding box is in the frustum. I would need something that gets the 3d intersection point(s) of a line and any of the frustum planes. And then would need to update those and re-draw the lines every frame when the camera moves. At that point I'm not sure we're in a better place than drawing 3d cylinders and updating the width every frame based on camera distance to approximate fixed-width.

I see from: asny/three-d#310 that there is an option in OpenGL for doing these kinds of fixed-width lines. Any possibility of adding that as an option to three-d?

[asny]

This wasn't quite as straightforward as I had thought.

Yeah, graphics can be hard 🙂

I think I would need to clip the 3d lines to the extents of the view frustum, but not sure offhand how to do that

It's just line/plane intersection and you don't need to do it for all 6 sides of the frustum, the only plane that matters is the near plane. So just stop your line where the line intersects the near plane and you should be good 🙂

At that point I'm not sure we're in a better place than drawing 3d cylinders and updating the width every frame based on camera distance to approximate fixed-width.

That's also an option, but I don't think it will perform better if that's what you think and getting the screen space width of the line correct is not easy.

I see from: asny/three-d#310 that there is an option in OpenGL for doing these kinds of fixed-width lines. Any possibility of adding that as an option to three-d?

Oh I get that question a lot 😆 But it's still no. Drawing lines with OpenGL is legacy functionality so it's not necessarily supported by all hardware and even if it is, it might be poorly implemented.