Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement of Visualize Loss, backup the loss list and create a new node without validation #113

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

stephanbrunker
Copy link

Within this PR are three commits:

  • 1cc2554 improves the Visualize Loss function. If you save states, the loss is not saved too. So when the occasional crashes happen, you lose the loss plot which is quite useful in determining which epoch to pick. So I saved the loss list to the states folder. Then, the shift because of the running average in the loss plot was not considered, the plot is now shifted for the window_size to the right. I also saved a list of savepoints with the states and marked these savepoints with vertical lines in the plot that you can see where exactly your savepoints are.
  • 51aa87f simply changes the epoch in the progress bar to the total number if you resume from a saved state.
  • 785bc6d adds a new loop node without validation. Validation gets an OOM when training with 16GB VRAM. So this node is a copy of the normal loop node, without validation. I also made the prompt input to the init node optional to create a cleaner workflow.

@stephanbrunker stephanbrunker changed the title Improvement von Visualize Loss, backup the loss list and create a new node without validation Improvement of Visualize Loss, backup the loss list and create a new node without validation Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant