Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

tgg-frequent-tests / TGG frequent tests

Process completed with exit code 1.

tgg-frequent-tests / TGG frequent tests

The action 'Run frequent regression tests' has timed out after 90 minutes.

pcie-cards-are-being-used-cleanup

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

tg-frequent-tests / tg-frequent-tests (TG Llama3-70B (old) frequent tests, wormhole_b0, llama3-70b-old, 90, U03FJB5TM5Y)

Process completed with exit code 1.

tg-frequent-tests / tg-frequent-tests (TG Llama3-70B (old) frequent tests, wormhole_b0, llama3-70b-old, 90, U03FJB5TM5Y)

The action 'Run frequent regression tests' has timed out after 90 minutes.

tg-frequent-tests / tg-frequent-tests (TG resnet50 frequent tests, wormhole_b0, resnet50, 90, U013121KDH9)

Process completed with exit code 1.

tg-unit-tests / TG Llama3-70b unit tests

Process completed with exit code 1.

tgg-model-perf-tests / TGG CNN model perf tests

The action 'Run model perf regression tests' has timed out after 60 minutes.

tgg-model-perf-tests / TGG CNN model perf tests

Process completed with exit code 2.

pcie-cards-are-being-used-cleanup

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

tgg-model-perf-tests / TGG CNN model perf tests

Process completed with exit code 1.

tg-frequent-tests / tg-frequent-tests (TG unit/distributed frequent tests, wormhole_b0, unit, 90, XXXXX)

The action 'Run frequent regression tests' has timed out after 90 minutes.

pcie-cards-are-being-used-cleanup

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

tg-frequent-tests / tg-frequent-tests (TG unit/distributed frequent tests, wormhole_b0, unit, 90, XXXXX)

Process completed with exit code 1.

tg-unit-tests / TG Llama3-small unit tests

Process completed with exit code 1.

tg-unit-tests / TG unit tests

The action 'Run unit regression tests' has timed out after 30 minutes.

pcie-cards-are-being-used-cleanup

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

tg-unit-tests / TG unit tests

Process completed with exit code 1.

disk-usage-before-startup

Disk usage is 39 %

disk-usage-after-startup

Disk usage is 39 %

printing-smi-info-startup

Touching and printing out SMI info

reset-successful-startup

tt-smi reset was successful

disk-usage-before-startup

Disk usage is 39 %