kolors support

[dribnet]

FYI: I took a first stab at adding support for the new Kolors model - you can find that work in my kolors branch. Currently it loads the kolors model and can create samples which follow regions, however there are lingering issues / bugs in the generation. Here's a sample output following the "1girl, looking at viewer, pink hair, leather jacket" region example in the README:

Pausing my efforts for now, but happy to combine efforts to finish this or check out my region_kolors.py script if anyone is interested in working on this more.

[ironjr]

Thank you for trying to add support for Kolors model! I will check this out.

[dribnet]

I've made some progress on this and now have a version that can generate kolors panoramas, including with ip-adapter support. For example, here is a kolors "photo of the dolomites" panorama with style applied via ip-adapter (input image shown in bottom corner)

and here's a version with the same prompt, but different ip-adpater style input image (again in the bottom corner)

I've again pushed these changes to my kolors branch. The code is in the demos subdirectory - I subclass the diffusers KolorsPipeline class and add a single method sample_panorama (which is based very closely on the call method of the superclass). The only hacky thing I had to do was directly manipulate the scheduler via self.scheduler._step_index = cached_scheduler_step for each panorama tile to keep it from overlfowing, etc (not sure what the better way would be).

Hope this is a helpful forward step. I think next would be cleaning this up a bit and then adding region support.

[dqj5182]

@dribnet Really appreciate sharing your code on trying Kolors on our StableMultiDiffusion! I have just pushed Kolors pipeline for StableMultiDiffusion with simple notebook demo (sadly not including LoRA, as there is no Kolors model trained in LoRA for now). Please feel free to check them and play around. Let us know if there is any problem :) Also, I'll try to look into your panorama code for further development on Kolors implementation and its extension with IP-Adapter.

Background prompt: 'deep black space with stars, galaxies, spaceships, floating spaceman, aliens',
Foreground prompts: 1) 'a photo of yellow nebula, masterpiece, absurd quality, background, no humans', 2) 'a photo of a smiling polar bear in space suit holding coca-cola bottle gazing at a spaceship',