Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Train text encoder #84

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

raulc0399
Copy link
Contributor

added code to train the text encoder, with options for lora rank, learning rate and early stopping

@linnanwang
Copy link

@raulc0399 just to be curious, why would you want to train text encoders?

@raulc0399
Copy link
Contributor Author

@linnanwang there are several papers that explore training the text encoder for sdxl. so i thought to try it with pixart - but this trains the textencoder it is no reproduction of any of those papers.

also some posts on reddit show some improvements while training for the text encoder for sdxl: https://www.reddit.com/r/StableDiffusion/comments/17xdb3m/text_encoder_onoff/

my tests show that with smaller data sets it converges sooner and it improves the quality of the generation.
also if using the prompts without the trigger words, the generation was closer to the original basemodel.
but i did run only a few numbers of tests ... but would be interesting to see what others find out

i will also try it with some regularization images to try to reproduce dreambooth training.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants