Add LLaMA2 model to pt nightly tests

[ManfeiBai]

Description

Add LLaMA2 model to xl-ml-dashboard to track

Tests

./scripts/run-oneshot.sh -t pt-nightly-l2-n-i-func-v4-8-1vm
./scripts/run-oneshot.sh -t pt-nightly-l2-t-h-f-func-v4-8-1vm

(l2 means llama2, -e means -eager, -c means -chat, -t means -tocken, -q means -quantization, -w means -without-download, -n-i means -next-inference, -s means -stable, -xla means -xla)

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run one-shot tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[miladm]

Thanks @ManfeiBai

We need a metric to detect regressions. For inference, we can define a latency/token threshold that we know is reachable. For training, we can define a MFU threshold.

Let's use our benchmark results to define the right threshold values. Wdyt?

[miladm]

@ManfeiBai wdyt we submit a follow up PR for GPU testing as well? (after this work successfully lands)

[ManfeiBai]

We need a metric to detect regressions. For inference, we can define a latency/token threshold that we know is reachable. For training, we can define a MFU threshold.

Let's use our benchmark results to define the right threshold values. Wdyt?

For metric to detect regression, agree to add threshold for inference, do we want to add latency/token threshold by using it to mark test success or failed?

for training, we didn't find training in ReadME yet, so we don't have training for LLaMA2 in this PR, or we mean training for other models?

For using benchmark results to define the right threshold values, sure, will do

[ManfeiBai]

@ManfeiBai wdyt we submit a follow up PR for GPU testing as well? (after this work successfully lands)

sure, we have GPU test on XL-ML-dashboard already, let's add LLaMA2 on GPU test in the follow-up PR

[miladm]

Bunch of comments - for the record

[ManfeiBai]

local test failed with ModuleNotFoundError: No module named 'torch.distributed._functional_collectives'

[ManfeiBai]

tested locally and output expected

[ManfeiBai]

Thanks @ManfeiBai

We need a metric to detect regressions. For inference, we can define a latency/token threshold that we know is reachable. For training, we can define a MFU threshold.

Let's use our benchmark results to define the right threshold values. Wdyt?

will try to latency/token for inference accoridng to pytorch-tpu/llama#35 or calculate directly

[ManfeiBai]

@ManfeiBai wdyt we submit a follow up PR for GPU testing as well? (after this work successfully lands)

will create new PRs for GPU tests

[ManfeiBai]

will ad threshold in the follow up PRs

[ManfeiBai]

has added latency/token as a threshold for inference

so current PR contain inference + llama2 + v4-8 + threshold

Hi, @miladm, would you mind review this PR again and mark the change request fixed when this PR is ready to merge or need new feature?

[miladm]

I don't see the inference latency threshold and train thresholds. Will that be a follow up PR?

we need to push this change asap. Though, it seems we will need a follow up PR that would target more accurate thresholds, different model sizes, on GPU and TPU. Let's push this in for now. Thank you. @wonjoolee95 to take a second look as well.

[ManfeiBai]

I don't see the inference latency threshold and train thresholds. Will that be a follow up PR?

we need to push this change asap. Though, it seems we will need a follow up PR that would target more accurate thresholds, different model sizes, on GPU and TPU. Let's push this in for now. Thank you. @wonjoolee95 to take a second look as well.

yes, we have threshold for inference and will have a followup PR for threshold of training

will merge this PR now