Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: RuntimeError: none of output has requires_grad=True, this checkpoint() is not necessary #549

Open
hacknullchainik opened this issue Nov 7, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@hacknullchainik
Copy link

What happened?

Im finetunning sdxl and hitting 30 epoch i twice had faced this issue on 1st step 30 epoch

What did you expect would happen?

Id should't crash i guess

Relevant log output

step:   3%|█▋                                                           | 1/37 [00:19<11:42, 19.52s/it, loss=0.0329, smooth loss=0.0306]
epoch:  38%|████████████████████████████████▋                                                      | 30/80 [3:16:44<5:27:54, 393.49s/it]
Traceback (most recent call last):
  File "D:\Models\Training\OneTrainer\modules\ui\TrainUI.py", line 558, in __training_thread_function
    trainer.train()
  File "D:\Models\Training\OneTrainer\modules\trainer\GenericTrainer.py", line 662, in train
    loss.backward()
  File "D:\Models\Training\OneTrainer\venv\lib\site-packages\torch\_tensor.py", line 525, in backward
    torch.autograd.backward(
  File "D:\Models\Training\OneTrainer\venv\lib\site-packages\torch\autograd\__init__.py", line 267, in backward
    _engine_run_backward(
  File "D:\Models\Training\OneTrainer\venv\lib\site-packages\torch\autograd\graph.py", line 744, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "D:\Models\Training\OneTrainer\venv\lib\site-packages\torch\autograd\function.py", line 301, in apply
    return user_fn(self, *args)
  File "D:\Models\Training\OneTrainer\venv\lib\site-packages\torch\utils\checkpoint.py", line 316, in backward
    raise RuntimeError(
RuntimeError: none of output has requires_grad=True, this checkpoint() is not necessary
Saving D:/Models/Training/OneFolders/dreamboth with florence med/models/result2222

Output of pip freeze

No response

@hacknullchainik hacknullchainik added the bug Something isn't working label Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant