Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: how print training resume logs. #5117

Merged
merged 27 commits into from
Oct 2, 2023
Merged

fix: how print training resume logs. #5117

merged 27 commits into from
Oct 2, 2023

Conversation

sayakpaul
Copy link
Member

What does this PR do?

Addresses #5005 and #5034.

Incorporates how @williamberman implemented it for ControlNet:

progress_bar = tqdm(
range(0, args.max_train_steps),
initial=initial_global_step,
desc="Steps",
# Only show the progress bar once on each machine.
disable=not accelerator.is_local_main_process,
)
image_logs = None
for epoch in range(first_epoch, args.num_train_epochs):
for step, batch in enumerate(train_dataloader):
with accelerator.accumulate(controlnet):
# Convert images to latent space

If okay, happy to propagate the changes to the other examples.

Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that's indeed cleaner

@bghira
Copy link
Contributor

bghira commented Sep 20, 2023

fwiw you can also use set_description to display current / expected num epochs. i really like this change. thank you!

@sayakpaul
Copy link
Member Author

sayakpaul commented Sep 20, 2023

Needs to be propagated to:

  • DreamBooth
  • Custom Diffusion
  • Kandinsky
  • Textual inversion

@sayakpaul sayakpaul marked this pull request as ready for review September 23, 2023 15:55
@sayakpaul sayakpaul requested a review from yiyixuxu September 23, 2023 15:55
@sayakpaul
Copy link
Member Author

@patrickvonplaten @yiyixuxu if you could lend your 👁️ here and share your comments, that would be great!

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Sep 23, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me!

@sayakpaul
Copy link
Member Author

Will fix the failing test (for examples) and then merge.

@sayakpaul
Copy link
Member Author

Failing tests are unrelated.

@sayakpaul sayakpaul merged commit d56825e into main Oct 2, 2023
@sayakpaul sayakpaul deleted the fix/training-resume branch October 2, 2023 16:29
chuzhdontcode pushed a commit to chuzhdontcode/diffusers that referenced this pull request Oct 4, 2023
* fix: how print training resume logs.

* propagate changes to text-to-image scripts.

* propagate changes to instructpix2pix.

* propagate changes to dreambooth

* propagate changes to custom diffusion and instructpix2pix

* propagate changes to kandinsky

* propagate changes to textual inv.

* debug

* fix: checkpointing.

* debug

* debug

* debug

* back to the square

* debug

* debug

* change condition order.

* debug

* debug

* debug

* debug

* revert to original

* clean

---------

Co-authored-by: Patrick von Platen <[email protected]>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
* fix: how print training resume logs.

* propagate changes to text-to-image scripts.

* propagate changes to instructpix2pix.

* propagate changes to dreambooth

* propagate changes to custom diffusion and instructpix2pix

* propagate changes to kandinsky

* propagate changes to textual inv.

* debug

* fix: checkpointing.

* debug

* debug

* debug

* back to the square

* debug

* debug

* change condition order.

* debug

* debug

* debug

* debug

* revert to original

* clean

---------

Co-authored-by: Patrick von Platen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants