Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error using Latent Couple w/ ControlNet Reference #55

Open
TheZAbides opened this issue Jun 6, 2023 · 0 comments
Open

Error using Latent Couple w/ ControlNet Reference #55

TheZAbides opened this issue Jun 6, 2023 · 0 comments

Comments

@TheZAbides
Copy link

Bit of an edge case, I'm sure — and possibly an issue with ControlNet (I'll report there too), but thought I'd mention it...
Trying to create a 910x512 image using:
Clip Skip: 2
Lora: 1
Steps: 15
CFG Scale: 8
ControlNet - Reference - reference_only - My prompt is more important - Resize and Fill - reference image is 910x512
ControlNet - Canny - canny - Pixel Perfect - ControlNet is more important - Resize and Fill - reference image is 910x512
ControlNet - Depth - depth_midas - Pixel Perfect - CN is more important - Resize and Fill - reference image is 910x512
Latent Couple - 4 sections - reference image is 910x512

I'm able to get this setup to run perfectly, only if:

  1. I remove the Latent Couple "AND" parts from the prompt
    OR
  2. Turn OFF ControNet Reference

Here is the error:

Loading preprocessor: reference_only
preprocessor resolution = 512
locon load lora method
  0%|                                                                                                      | 0/15 [00:00<?, ?it/s]ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 113]).
100%|█████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:05<00:00,  2.80it/s]
Total progress: 100%|█████████████████████████████████████████████████████████████████████████████| 15/15 [00:04<00:00,  3.04it/s]
Loading preprocessor: reference_only██████████████████████████████████████████████████████████████| 15/15 [00:04<00:00,  3.28it/s]
preprocessor resolution = 512
Loading model: control_v11f1p_sd15_depth [cfd03158]
Loaded state_dict from [C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\models\ControlNet\ControlNet-v1-1\control_v11f1p_sd15_depth.pth]
Loading config: C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\models\ControlNet\ControlNet-v1-1\control_v11f1p_sd15_depth.yaml
ControlNet model control_v11f1p_sd15_depth [cfd03158] loaded.
Loading preprocessor: depth
Pixel Perfect Computation:
resize_mode = ResizeMode.OUTER_FIT
raw_H = 512
raw_W = 910
target_H = 512
target_W = 904
estimation = 508.62417582417584
preprocessor resolution = 509
Loading model: control_v11p_sd15_canny [d14c016b]
Loaded state_dict from [C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\models\ControlNet\ControlNet-v1-1\control_v11p_sd15_canny.pth]
Loading config: C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\models\ControlNet\ControlNet-v1-1\control_v11p_sd15_canny.yaml
ControlNet model control_v11p_sd15_canny [d14c016b] loaded.
Loading preprocessor: canny
Pixel Perfect Computation:
resize_mode = ResizeMode.OUTER_FIT
raw_H = 512
raw_W = 910
target_H = 512
target_W = 904
estimation = 508.62417582417584
preprocessor resolution = 509
locon load lora method
  0%|                                                                                                      | 0/15 [00:00<?, ?it/s]ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 64, 113]).
  0%|                                                                                                      | 0/15 [00:01<?, ?it/s]
Error completing request
Arguments: ('task(ghn1dalb79o5w32)', '(forest sunrise with stream winding through:1.2) <lora:detailmaker:1>\nAND (giant old craggy stone with waterfalls pouring down it:0.9) (with overgrown moss hanging-vines wildflowers growing on it:1.3)\nAND (giant old tree stumps stone:0.9) (with overgrown clumps of soft green moss and creeping-vines and wildflowers growing on it:1.3) \nAND (giant cluster of vines and flowers:0.9) (with butterflies and humming birds fluttering around it it:1.3)', '(bad-artist:0.25) (EasyNegative:1) (low quality, worst quality:1.3) (text, signature, watermark:1.2) (people, person, structure, building, window, house:1.5), fantasy (fire:1.3)', [], 15, 0, False, False, 1, 1, 8, -1.0, -1.0, 0, 0, 0, False, 512, 910, False, 0.7, 2, 'R-ESRGAN 4x+ Anime6B', 0, 0, 0, 0, '', '', [], 0, '\n    <div style="padding: 10px">\n      <div>Estimated VRAM usage: <span style="color: rgb(255.00, 31.35, 4.62)">7891.27 MB / 10240 MB (77.06%)</span></div>\n      <div>(5679 MB system + 2011.16 MB used)</div>\n    </div>\n    ', False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1}, False, 'MultiDiffusion', False, True, 1024, 1024, 128, 128, 84, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 1536, 128, True, True, True, False, False, '', 0, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000010B3F83AB90>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000010DB67F48E0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000010B3F79E230>, False, False, 'Matrix', 'Horizontal', 'Mask', 'Prompt', '1,1', '0.2', False, False, False, 'Attention', False, '0', '0', '0.4', None, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 150, 0.2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', 5, 24, 12.5, 1000, '', 'DDIM', 0, 64, 64, '', 64, 7.5, 0.42, 'DDIM', 64, 64, 1, 0, 92, True, True, True, False, False, False, 'midas_v21_small', None, None, False, None, None, False, None, None, False, 50) {}
Traceback (most recent call last):
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\txt2img.py", line 57, in txt2img
    processed = processing.process_images(p)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 611, in process_images
    res = process_images_inner(p)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 729, in process_images_inner
    samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 293, in process_sample
    return process.sample_before_CN_hack(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\processing.py", line 977, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 383, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 257, in launch_sampling
    return func()
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 383, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 159, in forward
    x_out[-uncond.shape[0]:] = self.inner_model(x_in[-uncond.shape[0]:], sigma_in[-uncond.shape[0]:], cond=make_condition_dict([uncond], image_cond_in[-uncond.shape[0]:]))
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 628, in forward_webui
    return forward(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 531, in forward
    outer.original_forward(
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
    h = module(h, emb, context)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
    x = layer(x, context)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
    x = block(x, context=context[i])
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 664, in hacked_basic_transformer_inner_forward
    x = self.attn2(self.norm2(x), context=context) + x
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 515, in scaled_dot_product_no_mem_attention_forward
    return scaled_dot_product_attention_forward(self, x, context, mask)
  File "C:\Stable_Diffusion\SD-WebUI_02\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 490, in scaled_dot_product_attention_forward
    k = k_in.view(batch_size, -1, h, head_dim).transpose(1, 2)
RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant