feat: migrate to vite. add support for ESM

[timothydang]

• Upgrade to latest mind-ar release (1.2.3)
• Bump versions of other dependencies
• Migrate to vite from webpack in sync with mind-ar
• Add support for ESM

[tommasoturchi]

The issue with FaceTracking is sorted, just needed to add controller.onInputResized(webcamRef.current.video); before await controller.setup(flipUserCamera);. ImageTracking is a little more difficult (that's where I got stuck in my old porting to vitejs too): mind-ar uses a WebWorker to analyse the image and vite doesn't import the worker used by submodules. I'm trying to look for a way around this, but if someone is more experienced with vite and could point me in the right direction it would be much appreciated 😅

[timothydang]

much thanks @tommasoturchi, face tracking example is now working for me with the changes you suggested. I'll spend some time later this week to look into the image tracking issue. i think it might be something else related to the recent changes from mind-ar. from what i can see, vite is importing webworker correctly. controller.worker.js is loaded and responding to messages sent from image controller class.

[timothydang]

hi @tommasoturchi , i think i've managed to get image tracking to display, by moving this block

          const ARprojectionMatrix = controller.getProjectionMatrix();
          camera.fov = (2 * Math.atan(1 / ARprojectionMatrix[5]) * 180) / Math.PI;
          camera.near = ARprojectionMatrix[14] / (ARprojectionMatrix[10] - 1.0);
          camera.far = ARprojectionMatrix[14] / (ARprojectionMatrix[10] + 1.0);
          camera.updateProjectionMatrix();

to before await controller.addImageTargets(imageTargets);. also noticed that location of anchor plane is not correctly positioned and the image detection only work intermittently. I'm quite new to both mind-ar and R3F so will definitely need your help to find what went wrong there.

[Darkensses]

@timothydang I just found that detection works good if you change the input width and height to the 'real' sizesof your screen. I also noticed that if I had the Devtools open (small viewport) the model rendered in the expected position.

screen-capture.webm

I use your modified version of AR.js https://github.com/timothydang/react-three-mind/blob/feat/vite-esm-migration/src/AR.jsx#L87 so if we adjust those lines, you can see interesting results ;)

controller = new ImageTargetController({
inputWidth: webcamRef.current.video.clientWidth,
inputHeight: webcamRef.current.video.clientHeight,
...
});

That explains why detection sometimes worked and why the position of the models were not the expected. However, it needs more investigation IMO.

Also, mind-ar just dropped a new release 2 days ago, so the changes that you and @tommasoturchi suggested need to be in a PR to release a new version of react-three-mind. I did another example of the facemesh, so let me know if you need help to put this togheter or make anoter PR.

Thanks to both of you for your effort here!

[Darkensses]

Ok, after deep dive into this, I finally made it work. I figured out that the webcam component was inside a drei html, however I suggest to not put it in that way becuase that component is for adding annotations to the 3D models in the scene, makes it hard to get and set sizes like the width and height of the video element. So, the first thing i made was to move the webcam element outside of the Canvas component.

Then, I updated the startTracking method by taking the latest version of https://github.com/hiukim/mind-ar-js/blob/master/src/image-target/three.js#L141

I also refactored the code and components by follow the style of this amazing repo https://github.com/krsbx/mind-ar-react
Now, we can use the image-tracking even if the viewport is resized!

Here's the result 🎉

screen-capture.webm

I only tested this using 1 anchor so far. I hope to have a chance to experiments with others scenarios.

I don't know if I'm following the best practices of react, so if anyone knows how to improve this, please let us know :)

[Darkensses]

Repo: https://github.com/Darkensses/react-three-mind-experiment

[timothydang]

thanks so much for picking this up @Darkensses I'll try to resume working on this over the next couple of days. will report back if we could update it to the latest version of mind-ar along with merging in your suggested changes

[tommasoturchi]

The problem I keep battling against is how to package this so that it correctly exports everything as a library that can be embedded on other react-three projects, given that it's using web workers. I'm not an expert on vite, but last time I tried I had that problem. It's "easy enough" to implement it in one's own repo and use it, since you just import mind-ar as an external dependency. Here what would be neat is for people to just import this library and it takes care of importing mind-ar implementing an interface react-three-compliant. At least that was my original idea, but now that mind-ar moved to vitejs I found it quite difficult. If you can find a way to sort this out that would be amazing! I'll look into this once more during the weekend :-)
Cheers!

[iuvivo]

This would be much appreciated, it'd be super helpful to use MindAR within R3F! Thanks for your work!

[chillbert]

@Darkensses thx for your repo! I noticed even a better performance for multi image trackers with your implementation.

What I can't understand is how would I implement Anchor found and lost events? I can't understand the logic in the ARAnchor component:

  useEffect(() => {
    if (ref.current) {
      if (controller.inputWidth === 0) {
        return;
      }
      // console.log("anchor target", anchor[target])
      if (anchor[target]) { // L#159
        //if (ref.current.visible !== true && onAnchorFound) onAnchorFound();
        ref.current.visible = true;
        ref.current.matrix = new Matrix4().fromArray(anchor[target]);
        setVisible(true)
      } else {
        //if (ref.current.visible !== false && onAnchorLost) onAnchorLost();
        ref.current.visible = false;
        setVisible(false)
        console.log("we lost",target); //all those trackers which are not visible?
      }
    }
  }, [controller, anchor, target])



 <ARAnchor target={0}
          onAnchorFound={()=>{console.log("Image 0 found!")}} //fires only once
          onAnchorLost={()=>{console.log("Image 0 lost!)}} // never fires!
        >
          {/* <Plane /> */}
        </ARAnchor>

[Darkensses]

@Darkensses thx for your repo! I noticed even a better performance for multi image trackers with your implementation.

What I can't understand is how would I implement Anchor found and lost events? I can't understand the logic in the ARAnchor component:

Hi @chillbert, the code you looked is very experimental, so I ommited the events that you mentioned, however you can implemented based on the code of this repo:

https://github.com/tommasoturchi/react-three-mind/blob/main/src/AR.js#L302C3-L302C16

[chillbert]

@Darkensses thx for your repo! I noticed even a better performance for multi image trackers with your implementation.
What I can't understand is how would I implement Anchor found and lost events? I can't understand the logic in the ARAnchor component:

Hi @chillbert, the code you looked is very experimental, so I ommited the events that you mentioned, however you can implemented based on the code of this repo:

https://github.com/tommasoturchi/react-three-mind/blob/main/src/AR.js#L302C3-L302C16

I actually did exactly that, uncommented the lines:

 320: if (ref.current.visible !== false && onAnchorLost) onAnchorLost();
 325: if (ref.current.visible !== true && onAnchorFound) onAnchorFound();

but as mentioned:

          onAnchorFound={()=>{console.log("Image 0 found!")}} //fires only once
          onAnchorLost={()=>{console.log("Image 0 lost!)}} // never fires!

"the code you looked is very experimental" - I see; did you also change the logic somehow or why is your code so much faster (not the vite build process, the actuall app, recognition and tracking).

[chillbert]

aaah now I see, you moved the webcam view from the html drei component in its own canvas - this explains the performance boost much likely.

[Darkensses]

aaah now I see, you moved the webcam view from the html drei component in its own canvas - this explains the performance boost much likely.

That's right, html drei component is only recommended for text or small element, not for video

[chillbert]

I still didn't manage to fix the onAnchorLost trigger...

In this code the visibility of the anchor children works, but the onAnchorLost callback is not triggered:

function ARAnchor({
  children,
  target = 0,
  onAnchorFound,
  onAnchorLost,
}) {
  const { controller } = useAR();
  const ref = useRef();
  const anchor = useAtomValue(anchorsAtom);

  useEffect(() => {
    if(ref.current){
      if(controller.inputWidth === 0) {
        return;
      }
      if(anchor[target]) { // Anchor found
        // If the anchor is not visible, call onAnchorFound
        if (ref.current.visible !== true && onAnchorFound) onAnchorFound();
        ref.current.visible = true;
        ref.current.matrix = new Matrix4().fromArray(anchor[target]);
      } else { // Anchor lost
        // If the anchor is visible and now lost, call onAnchorLost
        if (ref.current.visible !== false && onAnchorLost) onAnchorLost();
        ref.current.visible = false;
      }
    }
  }, [controller, anchor, target, onAnchorFound, onAnchorLost]);

  return (
    <group ref={ref} visible={false} matrixAutoUpdate={false}>
      {children}
    </group>
  );
}

in this case, the onAnchorLost is triggered, but the elements on the anchor stays visible even when the anchor is lost:

function ARAnchor({
  children,
  target = 0,
  onAnchorFound,
  onAnchorLost,
}) {
  const { controller } = useAR();
  const ref = useRef();
  const anchor = useAtomValue(anchorsAtom);

  useEffect(() => {
    if (ref.current) {
      if (controller.inputWidth === 0) {
        console.log("Controller input width is 0, returning early");
        return;
      }

      const isAnchorLost = (arr) => Array.isArray(arr) && arr.every(val => val === 0 || val === 1);

      if (anchor[target] && !isAnchorLost(anchor[target])) { // Anchor found
        console.log("Anchor found for target:", target);
        
        // If the anchor was not visible, call onAnchorFound
        if (!ref.current.visible) {
          if (onAnchorFound) {
            console.log("Calling onAnchorFound()");
            onAnchorFound();
          }
          ref.current.visible = true;
        }

        // Update the matrix for the current anchor
        ref.current.matrix = new Matrix4().fromArray(anchor[target]);

      } else { // Anchor lost
        console.log("Anchor lost for target:", target);

        // If the anchor was visible and now lost, call onAnchorLost
        if (ref.current.visible) {
          if (onAnchorLost) {
            console.log("Calling onAnchorLost()");
            onAnchorLost();
          }
          ref.current.visible = false;
        }
      }
    }

  }, [controller, anchor, target, onAnchorFound, onAnchorLost]);

  return (
    <group ref={ref} visible={false} matrixAutoUpdate={false}>
      {children}
    </group>
  );
}

I didn't manage to have both working in the same code... strange!

[chillbert]

finally:

function ARAnchor({
  children,
  target = 0,
  onAnchorFound,
  onAnchorLost,
}) {
  const { controller } = useAR();
  const ref = useRef();
  const anchor = useAtomValue(anchorsAtom);
  const prevAnchorState = useRef(false); // Track the previous anchor state (found or lost)

  // 1st useEffect: Handle visibility
  useEffect(() => {
    if (ref.current) {
      if (controller.inputWidth === 0) {
        return;
      }

      // Directly manage visibility based on anchor truthiness
      if (anchor[target]) {
        ref.current.visible = true;
        ref.current.matrix = new Matrix4().fromArray(anchor[target]);
      } else {
        ref.current.visible = false;
      }
    }
  }, [controller, anchor, target]);

  // 2nd useEffect: Handle callbacks for anchor found and lost
  useEffect(() => {
    if (ref.current) {
      if (controller.inputWidth === 0) {
        return;
      }

      // Helper function to detect anchor loss (e.g., array of zeros)
      const isAnchorLost = (arr) => Array.isArray(arr) && arr.every(val => val === 0 || val === 1);

      // Anchor found condition
      if (anchor[target] && !isAnchorLost(anchor[target])) {
        // Only trigger onAnchorFound if the previous state was "lost"
        if (!prevAnchorState.current) {
          console.log("Anchor found, calling onAnchorFound");
          if (onAnchorFound) {
            onAnchorFound();
          }
        }
        prevAnchorState.current = true; // Update previous anchor state to "found"

      } else {
        // Anchor lost condition (either falsy or an array of zeros)
        if (prevAnchorState.current) {
          console.log("Anchor lost, calling onAnchorLost");
          if (onAnchorLost) {
            onAnchorLost();
          }
        }
        prevAnchorState.current = false; // Update previous anchor state to "lost"
      }
    }
  }, [controller, anchor, target, onAnchorFound, onAnchorLost]);

  return (
    <group ref={ref} visible={false} matrixAutoUpdate={false}>
      {children}
    </group>
  );
}