[Community] AnimateDiff + Controlnet Pipeline

[a-r-r-o-w]

What does this PR do?

Community pipeline version for #5866.

Colab Notebook

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@yiyixuxu @patrickvonplaten

[a-r-r-o-w]

Hi @charchit7 @staghado @EdoardoBotta. I've opened this PR since I had some free time today and was trying to implement the community version of this as Yiyi suggested. I was only able to get in touch with charchit7 on discord and it seems like he's a little busy currently. If you're still interested, and if anyone else is looking to help out with the implementation, please DM on wandereronarock#0000 (Arrow#1334) so I can add you as a collaborator to my fork and colab notebook.

[charchit7]

Sure, go ahead @a-r-r-o-w I'll hope in 5 days. You can add me if that'll not be too late. (Ref PR : #5872)
Also, please check the issue No. is 5866.

[EdoardoBotta]

@a-r-r-o-w I'm interested in being added. I have also been experimenting with the pipelines at https://github.com/EdoardoBotta/diffusers/blob/controlanimate/examples/community/animatediff_controlnet.py. It would be great to merge the changes.

[a-r-r-o-w]

Sure, go ahead @a-r-r-o-w I'll hope in 5 days. You can add me if that'll not be too late. (Ref PR : #5872)
Also, please check the issue No. is 5866.

@charchit7 Sure! Thanks for pointing out the incorrect issue number - fixed.

@a-r-r-o-w I'm interested in being added. I have also been experimenting with the pipelines at https://github.com/EdoardoBotta/diffusers/blob/controlanimate/examples/community/animatediff_controlnet.py. It would be great to merge the changes.

@EdoardoBotta Sending a collaborator invite to my fork. After taking a quick look, it seems like we've both adopted code from existing pipelines and have similar changes for the most part. I've currently commented out all MultiControlNet related stuff since I wanted a single one to work first. Would be great if you could add in those changes!

[a-r-r-o-w]

@yiyixuxu Seems like it works, however I'm not too sure about the quality. Will run a few experiments to see if this can be improved.

Edit: Referencing #5881 here because of the similarity in output quality.

Results

SG161222/Realistic_Vision_V5.1_noVAE
CardosAnime

[yiyixuxu]

cc @DN6 here

[patrickvonplaten]

Super cool pipeline!
Could you also add a short section to https://github.com/huggingface/diffusers/blob/main/examples/community/README.md ?

[a-r-r-o-w]

Ah, forgot doing that. Doing so now

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[loboere]

How can I make the controlnet only affect the first frame or certain frames?
I want animatediff to generate the animation based on the frames affected by controlnet
I want to use a single image canny ,but currently it affects all frames, animatediff can't do much since controlnet canny affects all frames

[patrickvonplaten]

Good to merge for me once all tests are green

[yiyixuxu]

@a-r-r-o-w
merged! 🥳

[a-r-r-o-w]

@a-r-r-o-w merged! 🥳

wow, awesome ❤️ this is probably the first PR I consider as actual good work from my end haha.

Also big thanks to @EdoardoBotta. He had something working as well on his fork which helped with the multicontrolnet support that I couldn't get working initially.

How can I make the controlnet only affect the first frame or certain frames?
I want animatediff to generate the animation based on the frames affected by controlnet
I want to use a single image canny ,but currently it affects all frames, animatediff can't do much since controlnet canny affects all frames

@loboere I definitely think this is possible. Let me try to experiment a bit and I'll add to this pipeline if I'm able to get it to work.

[charchit7]

@a-r-r-o-w, @EdoardoBotta nicely done!

[rmasiso]

This is incredible work! Well done!!! This is very exciting. Thank you for setting this up for all of us. I was wondering if you guys knew of a strategy to have consistent animations past the 16 frame limit? Say for example, a video of 32 frames, I run this for the first 16, then I would run this again for the next 16 frames, but when I stitch it together, it's obvious when the first 16 frames end. I've tried setting the seed manually, but that doesn't seem to work. The weirdest part is that the same seed works to generate the same 16 frames whether its the first half or the second half. I've tried including the first frame and the last 16 frames to see if it had to do with the interpolation that included the first frame, but that doesn't seem to work. This is for the pose or canny controlnets. I'm on a m2 macbook pro, so I can't really run things in a single python script before I run out of memory. My strategy so far has been running python animatediff.py 0 16 and then python aniamtediff.py 16 32 -- any ideas? Thank you!

Edit: I just really have to emphasize, how amazing it is that this was set up by all of you. Thanks again!
Edit 2: My guess is that this is a torch seed issue? It's hard to be deterministic with torch when I'm running the script twice, but separately, correct? In a perfect world, I would have a for loop and just do 16 frames at a time within the same script, but as I mentioned (memory issues).
Edit 3: seems like doing a rolling window works-ish -- with color artifacts (discoloration). will continue to test.

[a-r-r-o-w]

Thanks for the kind words @rmasiso! This is indeed possible and I'm experimenting on adding support for arbitrary number of frames to both animatediff and this community pipeline. I have a non-functional notebook at the moment but I'm getting there slowly by reversing this comfy-ui code that does support longer generations.

[madriss]

Amazing work @a-r-r-o-w ! I was wondering if the current animatediff pipeline would work with SDXL and its related controlnets and motion module ?

Best

[a-r-r-o-w]

Amazing work @a-r-r-o-w ! I was wondering if the current animatediff pipeline would work with SDXL and its related controlnets and motion module ?

Best

Thanks! It is indeed possible. See here: https://github.com/guoyww/AnimateDiff/blob/sdxl/. I'll try to make a notebook and share it here soon.

@yiyixuxu @patrickvonplaten @sayakpaul AnimateDiff with SDXL seems like a great addition as the quality looks really cool. Would you be open to PRs adding support for it? I'm not sure if the current diffusers implementation will work since I see a few implementation differences in the AnimateDiff repo and here. Will test more and get back.

[a-r-r-o-w]

Just a quick notebook update before I go sleep: Colab. I'm yet to replicate the results of the sdxl branch as demonstrated in the repo. This notebook is able to get the inference process up and running.

Current Results

1.mp4	2.mp4
f.mp4

Also, it probably makes sense to create a new issue asking for support of these features at this point :)

[rmasiso]

Thanks for the kind words @rmasiso! This is indeed possible and I'm experimenting on adding support for arbitrary number of frames to both animatediff and this community pipeline. I have a non-functional notebook at the moment but I'm getting there slowly by reversing this comfy-ui code that does support longer generations.

This is exciting! Is the hack they did in comfy-ui, the idea that one converts the entire video into latents first, and then sends them through animate-diff for the 16 frame interpolation in a sort of rolling context window? Similar to my hack, but it would end up more consistent, because it's in latent space from the get-go?

[fkjkey]

Thank you for the pipe.Can't this pipe use multiple ControlNets like other pipes？

[a-r-r-o-w]

Thank you for the pipe.Can't this pipe use multiple ControlNets like other pipes？

It does support multiple controlnets. Does it error out when you use it with multiple? If so, I'll look into fixing it quickly.

[fkjkey]

when I do this：

    self.tile = ControlNetModel.from_pretrained(
        "/home/img2video/models/control/control_v11f1e_sd15_tile", torch_dtype=torch.float16
    )
    self.pose = ControlNetModel.from_pretrained(
        "/home/lora_test/model/controlnet_pose", torch_dtype=torch.float16
    )
    self.pose_processor = OpenposeDetector.from_pretrained('/home/lora_test/model/pose_dete').to('cuda')

    self.adapter = MotionAdapter.from_pretrained("/home/AnimateDiff-main/models/Motion_Module",
                                            torch_dtype=torch.float16)
    self.pipe = AnimateDiffControlNetPipeline.from_pretrained(
        "/home/lora_test/model/AWPainting_v1.2",
        motion_adapter=self.adapter,
        controlnet=[self.pose, self.tile],
        torch_dtype=torch.float16)

I get ：

Traceback (most recent call last):
File "/home/re_sd/dance_test.py", line 163, in
M = Dance()
File "/home/re_sd/dance_test.py", line 51, in init
self.pipe = AnimateDiffControlNetPipeline.from_pretrained(
File "/root/anaconda3/envs/stable_fast/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/root/anaconda3/envs/stable_fast/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1363, in from_pretrained
model = pipeline_class(**init_kwargs)
File "/home/re_sd/pipeline_animatediff_controlnet.py", line 170, in init
self.register_modules(
File "/root/anaconda3/envs/stable_fast/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 571, in register_modules
library = not_compiled_module.module.split(".")[0]
AttributeError: 'list' object has no attribute 'module'. Did you mean: 'mul'?

[a-r-r-o-w]

Okay, I see the issue. Not sure why/how I removed the line to handle multicontrolnet. In the __init__ function, the following line seems to be missing:

        if isinstance(controlnet, (list, tuple)):
            controlnet = MultiControlNetModel(controlnet)

There might be other things I've missed. I'll test and open a PR fixing it in a bit. Thanks for reporting this!