Skip to content

Commit

Permalink
don't resize embeddings if it's already large enough (axolotl-ai-clou…
Browse files Browse the repository at this point in the history
…d#577)

* don't resize embeddings if it's already large enough

* make sure to tie weights, even if we aren't resizing
  • Loading branch information
winglian authored Sep 15, 2023
1 parent 67370a5 commit 89e0eed
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion src/axolotl/utils/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,10 @@ def load_model(
if cfg.resize_token_embeddings_to_32x
else len(tokenizer)
)
model.resize_token_embeddings(embeddings_len)
if model.get_input_embeddings().num_embeddings < embeddings_len:
model.resize_token_embeddings(embeddings_len)
else:
model.tie_weights()

if (
hasattr(model.config, "max_position_embeddings")
Expand Down

0 comments on commit 89e0eed

Please sign in to comment.