Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

[Lexxnorth]

Decided to start coaching LoRA.
I got an error: 'FluxUpperLowerWrapper' object has no attribute 'prepare_block_swap_before_forward'
Then I updated the FluxTrainer node and started getting another error:
`

ComfyUI Error Report

Error Details

• Node ID: 4
• Node Type: FluxTrainLoop
• Exception Type: RuntimeError
• Exception Message: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

Stack Trace

  File "F:\ComfyUI\ComfyUI\execution.py", line 327, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\execution.py", line 202, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\execution.py", line 174, in _map_node_over_list
    process_inputs(input_dict, i)

  File "F:\ComfyUI\ComfyUI\execution.py", line 163, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\nodes.py", line 859, in train
    steps_done = training_loop(
                 ^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\train_network.py", line 1247, in training_loop
    noise_pred, target, timesteps, weighting = self.get_noise_pred_and_target(
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\flux_train_network_comfy.py", line 360, in get_noise_pred_and_target
    model_pred = call_dit(
                 ^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\flux_train_network_comfy.py", line 312, in call_dit
    model_pred = unet(
                 ^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\accelerate\utils\operations.py", line 823, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\accelerate\utils\operations.py", line 811, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 1039, in forward
    img, txt = block(img=img, txt=txt, vec=vec, pe=pe, txt_attention_mask=txt_attention_mask)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 736, in forward
    return checkpoint(self._forward, img, txt, vec, pe, txt_attention_mask, use_reentrant=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\_compile.py", line 32, in inner
    return disable_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\_dynamo\eval_frame.py", line 632, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\utils\checkpoint.py", line 496, in checkpoint
    ret = function(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 685, in _forward
    img_mod1, img_mod2 = self.img_mod(vec)
                         ^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 634, in forward
    out = self.lin(nn.functional.silu(vec))[:, None, :].chunk(self.multiplier, dim=-1)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\networks\lora_flux.py", line 103, in forward
    org_forwarded = self.org_forward(x)
                    ^^^^^^^^^^^^^^^^^^^

  File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

System Information

• ComfyUI Version: 0.3.10
• Arguments: ComfyUI\main.py --windows-standalone-build
• OS: nt
• Python Version: 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
• Embedded Python: true
• PyTorch Version: 2.5.1+cu121

Devices

• Name: cuda:0 NVIDIA GeForce RTX 4060 Ti : cudaMallocAsync
  • Type: cuda
  • VRAM Total: 17175150592
  • VRAM Free: 10534542176
  • Torch VRAM Total: 5469372416
  • Torch VRAM Free: 87579488

Logs

2025-01-12T15:06:04.749568 - 2025-01-12T15:06:05.074316 - ok2025-01-12T15:06:05.075317 - 
2025-01-12T15:06:05.097100 - [Moondream] found model versions: 2024-03-04, 2024-03-06, 2024-03-13, 2024-04-02, 2024-05-08, 2024-05-20, 2024-07-23 (Thundermoon), 2024-08-26, 2025-01-092025-01-12T15:06:05.097100 - 
2025-01-12T15:06:05.155793 - ### Loading: ComfyUI-Impact-Pack (V8.3)2025-01-12T15:06:05.155793 - 
2025-01-12T15:06:05.413662 - [Impact Pack] Wildcards loading done.2025-01-12T15:06:05.413662 - 
2025-01-12T15:06:05.475607 - ### Loading: ComfyUI-Inspire-Pack (V1.9.1)2025-01-12T15:06:05.475607 - 
2025-01-12T15:06:05.671245 - Total VRAM 16380 MB, total RAM 32608 MB
2025-01-12T15:06:05.671245 - pytorch version: 2.5.1+cu121
2025-01-12T15:06:05.672257 - Set vram state to: NORMAL_VRAM
2025-01-12T15:06:05.672257 - Device: cuda:0 NVIDIA GeForce RTX 4060 Ti : cudaMallocAsync
2025-01-12T15:06:05.923490 - ### Loading: ComfyUI-Manager (V3.6.5)
2025-01-12T15:06:06.147528 - ### ComfyUI Version: v0.3.10-51-g90f349f9 | Released on '2025-01-12'
2025-01-12T15:06:06.614062 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-01-12T15:06:06.654704 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-01-12T15:06:06.706018 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
2025-01-12T15:06:06.745228 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
2025-01-12T15:06:06.796521 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
2025-01-12T15:06:06.797518 - FETCH DATA from: F:\ComfyUI\ComfyUI\user\default\ComfyUI-Manager\cache\2233941102_nodes_page_1_limit_1000.json2025-01-12T15:06:06.797518 - 2025-01-12T15:06:06.858326 -  [DONE]2025-01-12T15:06:06.858326 - 
2025-01-12T15:06:06.893079 - nightly_channel: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/cache
2025-01-12T15:06:06.893079 - FETCH DATA from: F:\ComfyUI\ComfyUI\user\default\ComfyUI-Manager\cache\1514988643_custom-node-list.json2025-01-12T15:06:06.893079 - 2025-01-12T15:06:06.896080 -  [DONE]2025-01-12T15:06:06.896080 - 
2025-01-12T15:06:06.939050 - 
### [Old-Photo-Restoration] Verifying Models...2025-01-12T15:06:06.940058 - 
2025-01-12T15:06:08.786195 - �[0;33m[ReActor]�[0m - �[38;5;173mSTATUS�[0m - �[0;32mRunning v0.5.2-b1 in ComfyUI�[0m2025-01-12T15:06:08.786195 - 
2025-01-12T15:06:09.572474 - Torch version: 2.5.1+cu1212025-01-12T15:06:09.573476 - 
2025-01-12T15:06:09.699051 - [rvtools �[0;32mINFO�[0m] RvTools Version: 2.1.0
2025-01-12T15:06:15.220222 - no module 'xformers'. Processing without...2025-01-12T15:06:15.220222 - 
2025-01-12T15:06:15.222260 - no module 'xformers'. Processing without...2025-01-12T15:06:15.222260 - 
2025-01-12T15:06:16.777413 - (pysssss:WD14Tagger) [DEBUG] Available ORT providers: TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider2025-01-12T15:06:16.777413 - 
2025-01-12T15:06:16.777413 - (pysssss:WD14Tagger) [DEBUG] Using ORT providers: CUDAExecutionProvider, CPUExecutionProvider2025-01-12T15:06:16.777413 - 
2025-01-12T15:06:16.826565 - ------------------------------------------2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.826565 - �[34mComfyroll Studio v1.76 : �[92m 175 Nodes Loaded�[0m2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.826565 - ------------------------------------------2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.826565 - ** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.826565 - ** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.826565 - ------------------------------------------2025-01-12T15:06:16.826565 - 
2025-01-12T15:06:16.861050 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using ckpts path: F:\ComfyUI\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts�[0m
2025-01-12T15:06:16.861050 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using symlinks: False�[0m
2025-01-12T15:06:16.861050 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']�[0m
2025-01-12T15:06:16.941537 - DWPose: Onnxruntime with acceleration providers detected2025-01-12T15:06:16.941537 - 
2025-01-12T15:06:18.531582 - �[92m[tinyterraNodes] �[32mLoaded�[0m2025-01-12T15:06:18.531582 - 
2025-01-12T15:06:18.681470 - All packages from requirements.txt are installed and up to date.2025-01-12T15:06:18.681470 - 
2025-01-12T15:06:18.697418 - llama-cpp installed2025-01-12T15:06:18.697418 - 
2025-01-12T15:06:18.698421 - All packages from requirements.txt are installed and up to date.2025-01-12T15:06:18.698421 - 
2025-01-12T15:06:19.414860 - �[36;20m[comfy_mtb] | INFO -> loaded �[96m94�[0m nodes successfuly�[0m
2025-01-12T15:06:19.414860 - �[36;20m[comfy_mtb] | INFO -> Some nodes (2) could not be loaded. This can be ignored, but go to http://127.0.0.1:8188/mtb if you want more information.�[0m
2025-01-12T15:06:19.441038 - 
�[32mInitializing ControlAltAI Nodes�[0m2025-01-12T15:06:19.442048 - 
2025-01-12T15:06:19.612197 - 
�[36mEfficiency Nodes:�[0m Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...�[92mSuccess!�[0m2025-01-12T15:06:19.613207 - 
2025-01-12T15:06:19.823889 - 
2025-01-12T15:06:19.823889 - �[92m[rgthree-comfy] Loaded 42 extraordinary nodes. 🎉�[00m2025-01-12T15:06:19.823889 - 
2025-01-12T15:06:19.823889 - 
2025-01-12T15:06:20.208947 - Searge-SDXL v4.3.1 in F:\ComfyUI\ComfyUI\custom_nodes\SeargeSDXL2025-01-12T15:06:20.208947 - 
2025-01-12T15:06:21.665301 - �[34mWAS Node Suite: �[0mOpenCV Python FFMPEG support is enabled�[0m2025-01-12T15:06:21.665645 - 
2025-01-12T15:06:21.665645 - �[34mWAS Node Suite �[93mWarning: �[0m`ffmpeg_bin_path` is not set in `F:\ComfyUI\ComfyUI\custom_nodes\was-node-suite-comfyui\was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.�[0m2025-01-12T15:06:21.665645 - 
2025-01-12T15:06:22.369339 - �[34mWAS Node Suite: �[0mFinished.�[0m �[32mLoaded�[0m �[0m220�[0m �[32mnodes successfully.�[0m2025-01-12T15:06:22.369339 - 
2025-01-12T15:06:22.369339 - 
	�[3m�[93m"The purpose of art is washing the dust of daily life off our souls."�[0m�[3m - Pablo Picasso�[0m
2025-01-12T15:06:22.369339 - 
2025-01-12T15:06:22.389577 - 
Import times for custom nodes:
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\sd-dynamic-thresholding
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-PixelResolutionCalculator
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_UNet_bitsandbytes_NF4
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Detail-Daemon
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-saveimage-plus
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\masquerade-nodes-comfyui
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-image-round
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_CreaPrompt
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\Comfyui_TTP_Toolset
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_CatVTON_Wrapper
2025-01-12T15:06:22.389577 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\rk-comfy-nodes
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-mxToolkit
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_JPS-Nodes
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\cg-use-everywhere
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\websocket_image_save.py
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Inpaint-CropAndStitch
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyLiterals
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Lora-Auto-Trigger-Words
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Image-Saver
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-WD14-Tagger
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-inpaint-nodes
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AutomaticCFG
2025-01-12T15:06:22.391118 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-SuperBeasts
2025-01-12T15:06:22.392122 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus
2025-01-12T15:06:22.392122 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_Comfyroll_CustomNodes
2025-01-12T15:06:22.392122 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfy-nodes
2025-01-12T15:06:22.392122 -    0.0 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-LTXTricks
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Florence-2
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfy-image-saver
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Florence2
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-RMBG
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_essentials
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_UltimateSDUpscale
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Custom-Scripts
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\OneButtonPrompt
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-GGUF
2025-01-12T15:06:22.392122 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-various
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\efficiency-nodes-comfyui
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ControlAltAI-Nodes
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Chibi-Nodes
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_ExtraModels
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\rgthree-comfy
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_tinyterraNodes
2025-01-12T15:06:22.393124 -    0.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-old-photo-restoration
2025-01-12T15:06:22.393124 -    0.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-KJNodes
2025-01-12T15:06:22.393124 -    0.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui_controlnet_aux
2025-01-12T15:06:22.393124 -    0.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Inspire-Pack
2025-01-12T15:06:22.393124 -    0.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_LayerStyle
2025-01-12T15:06:22.393124 -    0.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_LayerStyle_Advance
2025-01-12T15:06:22.393124 -    0.3 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-dream-project
2025-01-12T15:06:22.393124 -    0.3 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack
2025-01-12T15:06:22.393124 -    0.3 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-MediaMixer
2025-01-12T15:06:22.394123 -    0.3 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui_segment_anything
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfy_mtb
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Hangover-Moondream
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-RvTools
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Fluxpromptenhancer
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_VLM_nodes
2025-01-12T15:06:22.394123 -    0.4 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer
2025-01-12T15:06:22.394123 -    0.5 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI_FaceAnalysis
2025-01-12T15:06:22.394123 -    0.5 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Crystools
2025-01-12T15:06:22.394123 -    0.5 seconds: F:\ComfyUI\ComfyUI\custom_nodes\SeargeSDXL
2025-01-12T15:06:22.394123 -    0.6 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Manager
2025-01-12T15:06:22.394123 -    0.6 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-sound-lab
2025-01-12T15:06:22.394123 -    0.7 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-VideoHelperSuite
2025-01-12T15:06:22.394123 -    0.7 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-tensorops
2025-01-12T15:06:22.394123 -    0.8 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Easy-Use
2025-01-12T15:06:22.394123 -    0.9 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-ollama
2025-01-12T15:06:22.394123 -    0.9 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-EasyCivitai-XTNodes
2025-01-12T15:06:22.394123 -    1.2 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Bringing-Old-Photos-Back-to-Life
2025-01-12T15:06:22.394123 -    1.7 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AdvancedLivePortrait
2025-01-12T15:06:22.395340 -    1.8 seconds: F:\ComfyUI\ComfyUI\custom_nodes\comfyui-reactor-node
2025-01-12T15:06:22.395340 -    2.1 seconds: F:\ComfyUI\ComfyUI\custom_nodes\was-node-suite-comfyui
2025-01-12T15:06:22.395340 -    2.5 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-CogVideoXWrapper
2025-01-12T15:06:22.395340 -    4.5 seconds: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-SUPIR
2025-01-12T15:06:22.395340 - 
2025-01-12T15:06:22.413171 - Starting server

2025-01-12T15:06:22.414171 - To see the GUI go to: http://127.0.0.1:8188
2025-01-12T15:06:25.108369 - FETCH DATA from: F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json2025-01-12T15:06:25.108369 - 2025-01-12T15:06:25.121411 -  [DONE]2025-01-12T15:06:25.121411 - 
2025-01-12T15:06:27.255291 - �[36;20m[comfy_mtb] | INFO -> Found multiple match, we will pick the last f:/stable-diffusion-webui/models/SwinIR
['F:\\ComfyUI\\ComfyUI\\models\\upscale_models', 'f:/stable-diffusion-webui/models/ESRGAN', 'f:/stable-diffusion-webui/models/RealESRGAN', 'f:/stable-diffusion-webui/models/SwinIR']�[0m
2025-01-12T15:06:56.909464 - got prompt
2025-01-12T15:06:56.921127 - queue_counter: 02025-01-12T15:06:56.922135 - 
2025-01-12T15:06:56.967524 - {'steps': 20, 'width': 1024, 'height': 1024, 'guidance_scale': 3.0, 'seed': 42, 'shift': True, 'base_shift': 0.5, 'max_shift': 1.15}2025-01-12T15:06:56.967524 - 
2025-01-12T15:06:56.979038 - additional_args: 2025-01-12T15:06:56.979038 - 
2025-01-12T15:06:56.988160 - highvram is enabled / highvramが有効です
2025-01-12T15:06:57.124468 - 2025-01-12 15:06:57 INFO     Checking the state dict: Diffusers or BFL, dev or schnell                  flux_utils.py:40
2025-01-12T15:06:57.126463 -                     INFO     t5xxl_max_token_length: 512                                 flux_train_network_comfy.py:111
2025-01-12T15:06:57.855223 -                     INFO     Loading dataset config from [[datasets]]                               train_network.py:342
                             resolution = [ 512, 512,]                                                                  
                             batch_size = 2                                                                             
                             enable_bucket = true                                                                       
                             bucket_no_upscale = false                                                                  
                             min_bucket_reso = 256                                                                      
                             max_bucket_reso = 1024                                                                     
                             [[datasets.subsets]]                                                                       
                             image_dir = "f:\\ComfyUI\\Dataset\\"                                                       
                             class_tokens = ""                                                                          
                             num_repeats = 10                                                                           
                                                                                                                        
                                                                                                                        
                             [general]                                                                                  
                             shuffle_caption = false                                                                    
                             caption_extension = ".txt"                                                                 
                             keep_tokens_separator = "|||"                                                              
                             caption_dropout_rate = 0.0                                                                 
                             color_aug = false                                                                          
                             flip_aug = false                                                                           
                                                                                                                        
2025-01-12T15:06:57.857224 -                     INFO     prepare images.                                                          train_util.py:1955
2025-01-12T15:06:57.859925 -                     INFO     get image size from name of cache files                                  train_util.py:1864
2025-01-12T15:06:57.862446 - 
  0%|                                                                                           | 0/26 [00:00<?, ?it/s]2025-01-12T15:06:57.863444 - 
100%|███████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 26026.71it/s]2025-01-12T15:06:57.863444 - 
2025-01-12T15:06:57.864445 -                     INFO     set image size from cache files: 26/26                                   train_util.py:1894
2025-01-12T15:06:57.865444 -                     INFO     found directory f:\ComfyUI\Dataset\ contains 26 image files              train_util.py:1896
2025-01-12T15:06:57.865444 - 
read caption:   0%|                                                                             | 0/26 [00:00<?, ?it/s]2025-01-12T15:06:57.867443 - 
read caption: 100%|█████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 13010.25it/s]2025-01-12T15:06:57.868442 - 
2025-01-12T15:06:57.868442 -                     INFO     Found captions for 26 images.                                            train_util.py:1926
2025-01-12T15:06:57.869442 -                     INFO     260 train images with repeating.                                         train_util.py:1996
2025-01-12T15:06:57.870442 -                     INFO     0 reg images.                                                            train_util.py:1999
2025-01-12T15:06:57.871442 -                     WARNING  no regularization images / 正則化画像が見つかりませんでした              train_util.py:2004
2025-01-12T15:06:57.874666 -                     INFO     [Dataset 0]                                                              config_util.py:567
                               batch_size: 2                                                                            
                               resolution: (512, 512)                                                                   
                               enable_bucket: True                                                                      
                               network_multiplier: 1.0                                                                  
                               min_bucket_reso: 256                                                                     
                               max_bucket_reso: 1024                                                                    
                               bucket_reso_steps: 64                                                                    
                               bucket_no_upscale: False                                                                 
                                                                                                                        
                               [Subset 0 of Dataset 0]                                                                  
                                 image_dir: "f:\ComfyUI\Dataset\"                                                       
                                 image_count: 26                                                                        
                                 num_repeats: 10                                                                        
                                 shuffle_caption: False                                                                 
                                 keep_tokens: 0                                                                         
                                 keep_tokens_separator: |||                                                             
                                 caption_separator: ,                                                                   
                                 secondary_separator: None                                                              
                                 enable_wildcard: False                                                                 
                                 caption_dropout_rate: 0.0                                                              
                                 caption_dropout_every_n_epochs: 0                                                      
                                 caption_tag_dropout_rate: 0.0                                                          
                                 caption_prefix: None                                                                   
                                 caption_suffix: None                                                                   
                                 color_aug: False                                                                       
                                 flip_aug: False                                                                        
                                 face_crop_aug_range: None                                                              
                                 random_crop: False                                                                     
                                 token_warmup_min: 1                                                                    
                                 token_warmup_step: 0.0                                                                 
                                 alpha_mask: False                                                                      
                                 custom_attributes: {}                                                                  
                                 is_reg: False                                                                          
                                 class_tokens:                                                                          
                                 caption_extension: .txt                                                                
                                                                                                                        
                                                                                                                        
2025-01-12T15:06:57.875675 -                     INFO     [Dataset 0]                                                              config_util.py:573
2025-01-12T15:06:57.875675 -                     INFO     loading image sizes.                                                      train_util.py:918
2025-01-12T15:06:57.876664 - 
  0%|                                                                                           | 0/26 [00:00<?, ?it/s]2025-01-12T15:06:57.876664 - 
100%|██████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<?, ?it/s]2025-01-12T15:06:57.876664 - 
2025-01-12T15:06:57.876664 -                     INFO     make buckets                                                              train_util.py:924
2025-01-12T15:06:57.878667 -                     INFO     number of images (including repeats) /                                    train_util.py:970
                             各bucketの画像枚数（繰り返し回数を含む）                                                   
2025-01-12T15:06:57.879666 -                     INFO     bucket 0: resolution (512, 512), count: 260                               train_util.py:975
2025-01-12T15:06:57.881171 -                     INFO     mean ar error (without repeats): 0.0                                      train_util.py:980
2025-01-12T15:06:57.883176 -                     INFO     preparing accelerator                                                  train_network.py:407
2025-01-12T15:06:57.950111 - accelerator device:2025-01-12T15:06:57.950111 -  2025-01-12T15:06:57.950111 - cuda2025-01-12T15:06:57.950111 - 
2025-01-12T15:06:57.951797 -                     INFO     Checking the state dict: Diffusers or BFL, dev or schnell                  flux_utils.py:40
2025-01-12T15:06:57.953805 -                     INFO     Building Flux model dev from BFL checkpoint                                flux_utils.py:97
2025-01-12T15:06:57.999872 -                     INFO     Loading state dict from                                                   flux_utils.py:114
                             F:\ComfyUI\ComfyUI\models\unet\flux1-dev-fp8.safetensors                                   
2025-01-12T15:06:58.031574 - 2025-01-12 15:06:58 INFO     Loaded Flux: <All keys matched successfully>                              flux_utils.py:132
2025-01-12T15:06:58.033578 -                     INFO     Loaded torch.float8_e4m3fn FLUX model                        flux_train_network_comfy.py:69
2025-01-12T15:06:58.034587 -                     INFO     enable block swap: blocks_to_swap=1                          flux_train_network_comfy.py:74
2025-01-12T15:06:58.034587 - FLUX: Block swap enabled. Swapping 1 blocks, double blocks: 0, single blocks: 2.2025-01-12T15:06:58.034587 - 
2025-01-12T15:06:58.035573 -                     INFO     Building CLIP-L                                                           flux_utils.py:158
2025-01-12T15:06:58.069156 -                     INFO     Loading state dict from F:\ComfyUI\ComfyUI\models\clip\clip_l.safetensors flux_utils.py:254
2025-01-12T15:06:58.103424 -                     INFO     Loaded CLIP-L: <All keys matched successfully>                            flux_utils.py:257
2025-01-12T15:06:58.154525 -                     INFO     Loading state dict from                                                   flux_utils.py:309
                             F:\ComfyUI\ComfyUI\models\clip\t5xxl_fp16.safetensors                                      
2025-01-12T15:06:58.204146 -                     INFO     Loaded T5xxl: <All keys matched successfully>                             flux_utils.py:312
2025-01-12T15:06:58.206147 -                     INFO     Building AutoEncoder                                                      flux_utils.py:139
2025-01-12T15:06:58.225740 -                     INFO     Loading state dict from F:\ComfyUI\ComfyUI\models\vae\ae.safetensors      flux_utils.py:144
2025-01-12T15:06:58.257509 -                     INFO     Loaded AE: <All keys matched successfully>                                flux_utils.py:147
2025-01-12T15:06:58.257509 - import network module:2025-01-12T15:06:58.257509 -  2025-01-12T15:06:58.258510 - .networks.lora_flux2025-01-12T15:06:58.258510 - 
2025-01-12T15:06:59.252257 - 2025-01-12 15:06:59 INFO     [Dataset 0]                                                              train_util.py:2479
2025-01-12T15:06:59.253262 -                     INFO     caching latents with caching strategy.                                   train_util.py:1026
2025-01-12T15:06:59.253262 -                     INFO     caching latents...                                                       train_util.py:1075
2025-01-12T15:06:59.733306 - 
100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 55.36it/s]2025-01-12T15:06:59.733306 - 
100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 54.27it/s]2025-01-12T15:06:59.733306 - 
2025-01-12T15:07:00.069487 - 2025-01-12 15:07:00 INFO     move vae and unet to cpu to save memory                     flux_train_network_comfy.py:164
2025-01-12T15:07:00.277121 -                     INFO     move text encoders to gpu                                   flux_train_network_comfy.py:172
2025-01-12T15:07:18.615740 - 2025-01-12 15:07:18 INFO     [Dataset 0]                                                              train_util.py:2501
2025-01-12T15:07:18.616764 -                     INFO     caching Text Encoder outputs with caching strategy.                      train_util.py:1209
2025-01-12T15:07:18.616764 -                     INFO     checking cache validity...                                               train_util.py:1220
2025-01-12T15:07:18.617763 - 
  0%|                                                                                           | 0/26 [00:00<?, ?it/s]2025-01-12T15:07:18.623293 - 
100%|████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 4701.73it/s]2025-01-12T15:07:18.623293 - 
2025-01-12T15:07:18.624297 -                     INFO     no Text Encoder outputs to cache                                         train_util.py:1247
2025-01-12T15:07:18.625288 -                     INFO     cache Text Encoder outputs for sample prompt: ['portrait of flux_train_network_comfy.py:188
                             a young woman']                                                                            
2025-01-12T15:07:18.626292 -                     INFO     cache Text Encoder outputs for prompt: portrait of a young  flux_train_network_comfy.py:218
                             woman                                                                                      
2025-01-12T15:07:19.028479 - 2025-01-12 15:07:19 INFO     cache Text Encoder outputs for prompt:                      flux_train_network_comfy.py:218
2025-01-12T15:07:19.239665 -                     INFO     move CLIP-L back to cpu                                     flux_train_network_comfy.py:229
2025-01-12T15:07:19.470416 -                     INFO     move t5XXL back to cpu                                      flux_train_network_comfy.py:231
2025-01-12T15:07:22.117461 - 2025-01-12 15:07:22 INFO     move vae and unet back to original device                   flux_train_network_comfy.py:236
2025-01-12T15:07:22.122983 -                     INFO     create LoRA network. base dim (rank): 16, alpha: 16.0                      lora_flux.py:491
2025-01-12T15:07:22.124986 -                     INFO     neuron dropout: p=None, rank dropout: p=None, module dropout: p=None       lora_flux.py:492
2025-01-12T15:07:22.124986 -                     INFO     train all blocks only                                                      lora_flux.py:502
2025-01-12T15:07:22.125986 -                     INFO     create LoRA for Text Encoder 1:                                            lora_flux.py:589
2025-01-12T15:07:22.152514 -                     INFO     create LoRA for Text Encoder 1: 72 modules.                                lora_flux.py:592
2025-01-12T15:07:22.647710 -                     INFO     create LoRA for FLUX all blocks: 304 modules.                              lora_flux.py:606
2025-01-12T15:07:22.649708 -                     INFO     enable LoRA for U-Net: 304 modules                                         lora_flux.py:755
2025-01-12T15:07:22.653214 - FLUX: Gradient checkpointing enabled. CPU offload: False2025-01-12T15:07:22.653214 - 
2025-01-12T15:07:22.653214 - prepare optimizer, data loader etc.2025-01-12T15:07:22.653214 - 
2025-01-12T15:07:22.657218 -                     INFO     use 8-bit AdamW optimizer | {}                                           train_util.py:4642
2025-01-12T15:07:22.660216 - enable full bf16 training.2025-01-12T15:07:22.660216 - 
2025-01-12T15:07:22.693333 - enable fp8 training for U-Net.2025-01-12T15:07:22.693333 - 
2025-01-12T15:07:22.693333 - unet_weight_dtype: torch.float8_e4m3fn2025-01-12T15:07:22.693333 - 
2025-01-12T15:07:22.694332 -                     INFO     set U-Net weight dtype to torch.float8_e4m3fn                          train_network.py:638
2025-01-12T15:07:34.136908 - running training2025-01-12T15:07:34.136908 - 
2025-01-12T15:07:34.136908 -   num train images * repeats: 2602025-01-12T15:07:34.137907 - 
2025-01-12T15:07:34.137907 -   num reg images: 02025-01-12T15:07:34.137907 - 
2025-01-12T15:07:34.137907 -   num batches per epoch: 1302025-01-12T15:07:34.137907 - 
2025-01-12T15:07:34.137907 -   num epochs: 82025-01-12T15:07:34.137907 - 
2025-01-12T15:07:34.137907 -   batch size per device: 22025-01-12T15:07:34.137907 - 
2025-01-12T15:07:34.137907 -   gradient accumulation steps: 12025-01-12T15:07:34.138907 - 
2025-01-12T15:07:34.138907 -   total optimization steps: 10002025-01-12T15:07:34.138907 - 
2025-01-12T15:07:51.317908 - 2025-01-12 15:07:51 INFO     text_encoder is not needed for training. deleting to save memory.     train_network.py:1107
2025-01-12T15:07:51.325411 -                     INFO     unet dtype: torch.float8_e4m3fn, device: cuda:0                       train_network.py:1124
2025-01-12T15:07:51.527278 - 
Epoch 1/8 - steps:   0%|                                                                      | 0/1000 [00:00<?, ?it/s]2025-01-12T15:07:53.660223 - 2025-01-12 15:07:53 ERROR    !!! Exception during processing !!! Expected all tensors to be on the same execution.py:396
                             device, but found at least two devices, cpu and cuda:0! (when checking                     
                             argument for argument mat1 in method wrapper_CUDA_addmm)                                   
2025-01-12T15:07:53.761519 -                     ERROR    Traceback (most recent call last):                                         execution.py:397
                               File "F:\ComfyUI\ComfyUI\execution.py", line 327, in execute                             
                                 output_data, output_ui, has_subgraph = get_output_data(obj,                            
                             input_data_all, execution_block_cb=execution_block_cb,                                     
                             pre_execute_cb=pre_execute_cb)                                                             
                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^                                                                                          
                               File "F:\ComfyUI\ComfyUI\execution.py", line 202, in get_output_data                     
                                 return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION,                 
                             allow_interrupt=True, execution_block_cb=execution_block_cb,                               
                             pre_execute_cb=pre_execute_cb)                                                             
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^^^^^^^^^^^^^^^                                                                         
                               File "F:\ComfyUI\ComfyUI\execution.py", line 174, in _map_node_over_list                 
                                 process_inputs(input_dict, i)                                                          
                               File "F:\ComfyUI\ComfyUI\execution.py", line 163, in process_inputs                      
                                 results.append(getattr(obj, func)(**inputs))                                           
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                            
                               File "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\nodes.py",                     
                             line 859, in train                                                                         
                                 steps_done = training_loop(                                                            
                                              ^^^^^^^^^^^^^^                                                            
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\train_network.py",                    
                             line 1247, in training_loop                                                                
                                 noise_pred, target, timesteps, weighting =                                             
                             self.get_noise_pred_and_target(                                                            
                                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^                                                                                       
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\flux_train_network_co                 
                             mfy.py", line 360, in get_noise_pred_and_target                                            
                                 model_pred = call_dit(                                                                 
                                              ^^^^^^^^^                                                                 
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\flux_train_network_co                 
                             mfy.py", line 312, in call_dit                                                             
                                 model_pred = unet(                                                                     
                                              ^^^^^                                                                     
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1736, in _wrapped_call_impl                                                           
                                 return self._call_impl(*args, **kwargs)                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1747, in _call_impl                                                                   
                                 return forward_call(*args, **kwargs)                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\accelerate\utils\operations.p                 
                             y", line 823, in forward                                                                   
                                 return model_forward(*args, **kwargs)                                                  
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                  
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\accelerate\utils\operations.p                 
                             y", line 811, in __call__                                                                  
                                 return convert_to_fp32(self.model_forward(*args, **kwargs))                            
                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                             
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py",                  
                             line 44, in decorate_autocast                                                              
                                 return func(*args, **kwargs)                                                           
                                        ^^^^^^^^^^^^^^^^^^^^^                                                           
                               File                                                                                     
2025-01-12T15:07:53.762931 -                              "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.p                 
                             y", line 1039, in forward                                                                  
                                 img, txt = block(img=img, txt=txt, vec=vec, pe=pe,                                     
                             txt_attention_mask=txt_attention_mask)                                                     
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^^^^^^^^^^^^^^^^                                                                        
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1736, in _wrapped_call_impl                                                           
                                 return self._call_impl(*args, **kwargs)                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1747, in _call_impl                                                                   
                                 return forward_call(*args, **kwargs)                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.p                 
                             y", line 736, in forward                                                                   
                                 return checkpoint(self._forward, img, txt, vec, pe,                                    
                             txt_attention_mask, use_reentrant=False)                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 
                             ^^^^^^^^^^^^^^^^^^^^^^                                                                     
                               File "F:\ComfyUI\python_embeded\Lib\site-packages\torch\_compile.py",                    
                             line 32, in inner                                                                          
                                 return disable_fn(*args, **kwargs)                                                     
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\_dynamo\eval_frame.py",                 
                             line 632, in _fn                                                                           
                                 return fn(*args, **kwargs)                                                             
                                        ^^^^^^^^^^^^^^^^^^^                                                             
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\utils\checkpoint.py",                   
                             line 496, in checkpoint                                                                    
                                 ret = function(*args, **kwargs)                                                        
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^                                                        
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.p                 
                             y", line 685, in _forward                                                                  
                                 img_mod1, img_mod2 = self.img_mod(vec)                                                 
                                                      ^^^^^^^^^^^^^^^^^                                                 
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1736, in _wrapped_call_impl                                                           
                                 return self._call_impl(*args, **kwargs)                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1747, in _call_impl                                                                   
                                 return forward_call(*args, **kwargs)                                                   
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.p                 
                             y", line 634, in forward                                                                   
                                 out = self.lin(nn.functional.silu(vec))[:, None,                                       
                             :].chunk(self.multiplier, dim=-1)                                                          
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1736, in _wrapped_call_impl                                                           
                                 return self._call_impl(*args, **kwargs)                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\module.py",                  
                             line 1747, in _call_impl                                                                   
                                 return forward_call(*args, **kwargs)                                                   
2025-01-12T15:07:53.762931 -                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
                               File                                                                                     
                             "F:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\networks\lora_flux.py                 
                             ", line 103, in forward                                                                    
                                 org_forwarded = self.org_forward(x)                                                    
                                                 ^^^^^^^^^^^^^^^^^^^                                                    
                               File                                                                                     
                             "F:\ComfyUI\python_embeded\Lib\site-packages\torch\nn\modules\linear.py",                  
                             line 125, in forward                                                                       
                                 return F.linear(input, self.weight, self.bias)                                         
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                         
                             RuntimeError: Expected all tensors to be on the same device, but found at                  
                             least two devices, cpu and cuda:0! (when checking argument for argument                    
                             mat1 in method wrapper_CUDA_addmm)                                                         
                                                                                                                        
2025-01-12T15:07:53.765933 -                     INFO     Prompt executed in 56.85 seconds                                                main.py:186

Attached Workflow

Please make sure that workflow does not contain any sensitive information such as API keys or passwords.

Workflow too large. Please manually upload the workflow from local file system.

Additional Context

(Please add any additional context or steps to reproduce the error here)
`

[Lexxnorth]

The problem was solved by installing a second instance of ComfyUI v.0.3.10 specifically for training with four extensions of the latest versions of fluxtrainer, rgthree, kjnodes and mtb.
Errors
'Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)'
and
'FluxUpperLowerWrapper' object has no attribute 'prepare_block_swap_before_forward'
disappeared