[Llama] first run with generating positional rotation matrix caches segfaults and OOMs #9837
Assignees
Labels
Comments
|
I was able to repro this segfault on sjc-snva-t3002. Config:
|
|
When I run with async disabled, I see a variety of errors. Case 1: |
|
case 2: deallocate |
|
Case 3: search_first |
|
I was not able to repro this segfault with async queues disabled. In one of the This made me wonder if this code pattern is the culprit: rot_mats = ttnn.as_tensor(
rot_mat,
dtype=ttnn.bfloat16,
layout=ttnn.TILE_LAYOUT,
device=self.device_mesh,
cache_file_name=cache_name(f"rot_mat_decode_{start_pos}"),
memory_config=self.model_config["DRAM_MEMCFG"],
mesh_mapper=ReplicateTensorToMesh(self.device_mesh),
)
rot_mats = ttnn.to_device(rot_mats, self.device_mesh)The |
cglagovichTT
added a commit
that referenced
this issue
Jul 3, 2024
|
Repro instructions:
gdb --args python -m pytest -svv models/demos/t3000/llama3_70b/demo/demo.py::test_LlamaModel_demo[wormhole_b0-True-check_disabled-greedy-tt-70b-T3000-80L-decode_only-text_completion-llama3]Expected output: |
|
On a new T3000 machine to get the first run to 2816 tokens generated in a single sequence I got 6 crashes: 
I did a soft reset crash 1: 2: crash 3 (same stack trace as above): crash 4 (same stack trace as above): crash 5: crash 6 (hang) Rerunning after this crash got to 2816 tokens and gets to the known issue #9839. This completes the first run and generation for 2k context is relatively reliable. |
tt-asaigal
added a commit
that referenced
this issue
Jul 3, 2024
- This handles cases where a device tensor is reassigned to a host tensor
- Exposed during model cache generation which uses the following pattern:
device_tensor = device_tensor.cpu()
|
Hey @tstescoTT would you mind running with this commit cherry-picked: 4558673. It resolved the segfault for me locally |
3 tasks
tt-asaigal
added a commit
that referenced
this issue
Jul 4, 2024
- This handles cases where a device tensor is reassigned to a host tensor
- Exposed during model cache generation which uses the following pattern:
device_tensor = device_tensor.cpu()
tt-asaigal
added a commit
that referenced
this issue
Jul 4, 2024
- This handles cases where a device tensor is reassigned to a host tensor
- Exposed during model cache generation which uses the following pattern:
device_tensor = device_tensor.cpu()
|
@tstescoTT - can you help repro and confirm? |
|
@mbahnasTT confirms tested - can be closed |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
With a fresh tt-metal weights cache for llama2 and llama3 on first run the rotation matrices (rot mats) are cached for later use. For example:
During this first run caching segfaults typically occur as the token position is increased.
Workaround: a soft reset (tt-smi -r 0,1,2,3) can be used to reset the device and run again to generate caches for a higher token position until the entire max seq len is reached.
Without doing first-run generation for the entire max seq len, the segfaults or hangs may occur during applications if the current seq len does not have cached rot mats. Ideally this would be part of the inital model set up for applications to avoid unpredictable caching during application runtime.
To Reproduce
Steps to reproduce the behavior:
demo_first_run_4k.pyscript (https://gist.github.com/tstescoTT/86e31370590666e0edb920bd6bf615aa#file-demo_first_run_4k-py) forcing 4k token generation.pytest -svv demo_first_run_4k.py::test_LlamaModel_demo[wormhole_b0-True-check_disabled-greedy-tt-70b-T3000-80L-decode_only-text_completion-llama3]Expected behavior
The rot mat cache generation should not cause segfaults or OOMs.
Ideally there should be a way to optionally pre-compute all the rot mats ahead of application runtime to avoid unexpected caching and resulting issues, e.g. with read-only file systems.
Example traces
Example segfault:
At higher token positions DRAM OOM occured:
Please complete the following environment information:
The text was updated successfully, but these errors were encountered: