The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Process completed with exit code 2.

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

Unable to reset board successfully, rebooting

The operation was canceled.

The action 'Run model perf regression tests' has timed out after 60 minutes.

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Touching and printing out SMI info

Attempting to reset card(s).

tt-smi reset was successful