-
|
I have only seen the |
Beta Was this translation helpful? Give feedback.
Answered by
terrykong
Jan 7, 2025
Replies: 4 comments 8 replies
-
|
sorry that's not supported at this moment. do you have a particular use case for it? if yes I'm happy to look into adding it. |
Beta Was this translation helpful? Give feedback.
8 replies
-
|
hey @alex-ht I checked and it turns out that unfortunately MoE + CP is not a combination that is supported at the moment in the nemo ecosystem. CP is only supported when we have no MoE(that part I will try to add asap). |
Beta Was this translation helpful? Give feedback.
-
|
I am trying to train Llama3 model, but I have encountered another problem. First, I added CMD="python -u examples/nlp/gpt/train_gpt_sft.py \
trainer.precision=16-mixed \
trainer.num_nodes=${SLURM_JOB_NUM_NODES} \
trainer.devices=-1 \
trainer.sft.limit_val_batches=1 \
trainer.sft.val_check_interval=12 \
trainer.sft.save_interval=12 \
model.megatron_amp_O2=True \
model.restore_from_path=${PRETRAINED_ACTOR_NEMO_FILE} \
model.optim.lr=5e-6 \
model.optim.name=distributed_fused_adam \
model.answer_only_loss=True \
model.data.num_workers=0 \
model.data.train_ds.micro_batch_size=1 \
model.data.train_ds.global_batch_size=8 \
model.data.train_ds.file_path=${TRAIN_DATA_PATH} \
model.data.train_ds.max_seq_length=8192 \
model.data.train_ds.add_eos=True \
model.data.train_ds.add_bos=True \
+model.data.train_ds.packed_sequence=True \
model.data.validation_ds.max_seq_length=8192 \
model.data.validation_ds.micro_batch_size=1 \
model.data.validation_ds.global_batch_size=128 \
model.data.validation_ds.file_path=${VALID_DATA_PATH} \
model.data.validation_ds.add_bos=True \
model.data.validation_ds.add_eos=True \
+model.data.validation_ds.packed_sequence=True \
model.tensor_model_parallel_size=8 \
model.pipeline_model_parallel_size=4 \
model.sequence_parallel=False \
model.activations_checkpoint_granularity=selective \
model.activations_checkpoint_method=uniform \
+model.context_parallel_size=2 \
exp_manager.create_wandb_logger=True \
exp_manager.explicit_log_dir=${RESULTS_DIR} \
exp_manager.wandb_logger_kwargs.project=${PROJECT} \
exp_manager.wandb_logger_kwargs.name=dolly_sft_run_tp8 \
exp_manager.resume_if_exists=True \
exp_manager.resume_ignore_no_checkpoint=True \
exp_manager.create_checkpoint_callback=True \
exp_manager.checkpoint_callback_params.save_nemo_on_train_end=True \
exp_manager.checkpoint_callback_params.monitor=validation_loss \
"I ran into the following error when starting the training besides, the TransformerEngine I am using has been modified to incorporate xformers to support the V100 GPU. This program has passed the test at I can't find examples of using context parallel, and any would be very helpful. |
Beta Was this translation helpful? Give feedback.
-
|
hello, i talked to the CP expert here and they told me this seems more like a torch compile issue. To help rule it out can you let me know if llama3 works if you don't use CP at all? Unfortunately the only docs we have for CP is here: https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it can run normally without CP. |
Beta Was this translation helpful? Give feedback.
-
|
I have placed the log messages and scripts here. The first experiment (without CP) encountered an error while saving the checkpoint, and the second experiment (with CP) had an error right at the start of training. |
Beta Was this translation helpful? Give feedback.
-
|
any update? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
any update?
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Context parallel for SFT is supported in the 24.12 release container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags. Please see these release notes for instructions: https://github.com/NVIDIA/NeMo-Aligner/releases/tag/v0.6.0 CP support for other algorithms is coming soon. In particular, for DPO |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
alex-ht
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context parallel for SFT is supported in the 24.12 release container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags. Please see these release notes for instructions: https://github.com/NVIDIA/NeMo-Aligner/releases/tag/v0.6.0
CP support for other algorithms is coming soon. In particular, for DPO