Skip to content

Commit

Permalink
fix
Browse files Browse the repository at this point in the history
  • Loading branch information
flybird11111 committed Sep 11, 2024
1 parent e57cf7e commit dbb4982
Show file tree
Hide file tree
Showing 4 changed files with 4 additions and 2 deletions.
1 change: 1 addition & 0 deletions docs/source/en/concepts/paradigms_of_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,4 @@ Related paper:
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/abs/2104.07857)
- [PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management](https://arxiv.org/abs/2108.05818)
<!-- doc-test-command: echo -->
2 changes: 1 addition & 1 deletion docs/source/en/features/sequence_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ plugin = HybridParallelPlugin(
)
```
#### Using Booster
```
```python
booster = Booster(plugin=plugin)
dataloader = plugin.prepare_dataloader(dataset, batch_size=args.batch_size, shuffle=True, drop_last=True, seed=42)
model, optimizer, _, dataloader, _ = booster.boost(model, optimizer, dataloader=dataloader)
Expand Down
1 change: 1 addition & 0 deletions docs/source/zh-Hans/concepts/paradigms_of_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,4 @@ ring attention思路类似于flash attention,每个GPU只计算一个局部的
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/abs/2104.07857)
- [PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management](https://arxiv.org/abs/2108.05818)
<!-- doc-test-command: echo -->
2 changes: 1 addition & 1 deletion docs/source/zh-Hans/features/sequence_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ plugin = HybridParallelPlugin(
)
```
#### 使用booster
```
```python
booster = Booster(plugin=plugin)
dataloader = plugin.prepare_dataloader(dataset, batch_size=args.batch_size, shuffle=True, drop_last=True, seed=42)
model, optimizer, _, dataloader, _ = booster.boost(model, optimizer, dataloader=dataloader)
Expand Down

0 comments on commit dbb4982

Please sign in to comment.