Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Organised README.md #3

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Organised README.md #3

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,18 @@

![teaser](assets/teaser.jpg)

Many video editing tasks such as rotoscoping or object removal require the propagation of context across frames. While transformers and other attention-based approaches that aggregate features globally have demonstrated great success at propagating object masks from keyframes to the whole video, they struggle to propagate high-frequency details such as textures faithfully. We hypothesize that this is due to an inherent bias of global attention towards low-frequency features. To overcome this limitation, we present a two-stream approach, where high-frequency features interact locally and low-frequency features interact globally. The global interaction stream remains robust in difficult situations such as large camera motions, where explicit alignment fails. The local interaction stream propagates high-frequency details through deformable feature aggregation and, informed by the global interaction stream, learns to detect and correct errors of the deformation field. We evaluate our two-stream approach for inpainting tasks, where experiments show that it improves both the propagation of features within a single frame as required for image inpainting, as well as their propagation from keyframes to target frames. Applied to video inpainting, our approach leads to 44% and 26% improvements in FID and LPIPS scores.
### How Guided Inpainting is different

Video editing tasks such as rotoscoping and object removal require context propagation across frames. Transformers and other attention-based approaches that aggregate features globally work great at propagating object masks from keyframes to the whole video. However, they struggle to propagate high-frequency details such as textures properly. We hypothesize this is due to an inherent bias of global attention toward low-frequency features. To overcome this limitation we used a two-stream approach where high-frequency features and low-frequency features interact globally.

### How Guided Inpainting works

Guided Inpainting uses a two-stream approach that used high-frequency features and low-frequency features to interact globally. The global interaction stream remains robust in difficult situations such as large camera motions, where explicit alignment fails. The local interaction stream propagates high-frequency details through deformable feature aggregation and, informed by the global interaction stream, learns to detect and correct errors of the deformation field.

### Evaluation

We evaluate our two-stream approach for inpainting tasks, where experiments show that it improves both the propagation of features within a single frame (required for image inpainting) and their propagation from keyframes to target frames. Applied to video inpainting, our approach leads to 44% and 26% improvements in FID and LPIPS scores.


[***Towards Unified Keyframe Propagation Models***](https://arxiv.org/abs/2205.09731)<br/>
Patrick Esser, Peter Michael, Soumyadip Sengupta
Expand Down