Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to preserve aspect ratio and flip on VirtualBackground? #48

Open
sgrund14 opened this issue Sep 30, 2024 · 0 comments
Open

Option to preserve aspect ratio and flip on VirtualBackground? #48

sgrund14 opened this issue Sep 30, 2024 · 0 comments

Comments

@sgrund14
Copy link

sgrund14 commented Sep 30, 2024

Hello!

Could this support a preserveAspectRatio and flipImageHorizontal option to the VideoBackground transformer?

Getting squashed and flipped images when I try to simply pass through a remote URL to the constructor
VirtualBackground('https://i0.wp.com/apartmentapothecary.com/wp-content/uploads/2021/10/shoe-storage-for-very-narrow-hallway.webp?ssl=1')

Screenshot 2024-09-30 at 7 32 36 PM

^ that's using this image fwiw, notice how it's taller and oriented the other way horizontally

Was messing with the sample, and think this can be achieved pretty simply in BackgroundTransformer

something like this preserves the aspect ratio and flips the image (which is getting flipped by default in the current implementation, for some reason)

async drawVirtualBackground(frame: VideoFrame) {
    if (!this.canvas || !this.ctx || !this.segmentationResults || !this.inputVideo) return;
    if (this.segmentationResults?.categoryMask) {
      this.ctx.filter = 'blur(10px)';
      this.ctx.globalCompositeOperation = 'copy';
      const bitmap = await maskToBitmap(
        this.segmentationResults.categoryMask,
        this.segmentationResults.categoryMask.width,
        this.segmentationResults.categoryMask.height,
      );
      this.ctx.drawImage(bitmap, 0, 0, this.canvas.width, this.canvas.height);
      this.ctx.filter = 'none';
      this.ctx.globalCompositeOperation = 'source-in';
      if (this.backgroundImage) {
        this.drawImagePreserveAspectRatio(this.backgroundImage, this.canvas.width, this.canvas.height);
      } else {
        this.ctx.fillStyle = '#00FF00';
        this.ctx.fillRect(0, 0, this.canvas.width, this.canvas.height);
      }

      this.ctx.globalCompositeOperation = 'destination-over';
    }
    // Draw the frame without flipping
    this.ctx.drawImage(frame, 0, 0, this.canvas.width, this.canvas.height);
  }

drawImagePreserveAspectRatio(image: ImageBitmap, canvasWidth: number, canvasHeight: number) {
    const imgRatio = image.width / image.height;
    const canvasRatio = canvasWidth / canvasHeight;
    let drawWidth, drawHeight, x, y;

    if (imgRatio > canvasRatio) {
      drawHeight = canvasHeight;
      drawWidth = image.width * (drawHeight / image.height);
      x = (canvasWidth - drawWidth) / 2;
      y = 0;
    } else {
      drawWidth = canvasWidth;
      drawHeight = image.height * (drawWidth / image.width);
      x = 0;
      y = (canvasHeight - drawHeight) / 2;
    }
    
    // Original flipped version (commented out)
    // Apply horizontal flip transformation for background image
    this.ctx!.save();
    this.ctx!.scale(-1, 1);
    this.ctx!.drawImage(image, -x - drawWidth, y, drawWidth, drawHeight);
    this.ctx!.restore();
  }

Might be nice to do something like

const virtualBackground = VirtualBackground('/background.png', { preserveAspectRatio: true, flipHorizontal: true })
@sgrund14 sgrund14 changed the title Preserve aspect ratio and flip on VirtualBackground? Option to preserve aspect ratio and flip on VirtualBackground? Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant