Skip to content

Commit

Permalink
Fix: Cast prefetch_bucket_size to integer for deepspeed >= 0.15 (hugg…
Browse files Browse the repository at this point in the history
…ingface#33402)

Fix: Cast prefetch bucket size to integer in zero_optimization
  • Loading branch information
kiddj authored Sep 11, 2024
1 parent 7a51cbc commit ecf7024
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/integrations/deepspeed.py
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ def trainer_config_finalize(self, args, model, num_training_steps):
# automatically assign the optimal config values based on model config
self.fill_only(
"zero_optimization.stage3_prefetch_bucket_size",
0.9 * hidden_size * hidden_size,
int(0.9 * hidden_size * hidden_size),
)
self.fill_only(
"zero_optimization.stage3_param_persistence_threshold",
Expand Down

0 comments on commit ecf7024

Please sign in to comment.