The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.

Unable to reset board successfully, rebooting

The operation was canceled.

Process completed with exit code 2.

The action 'Run model perf regression tests' has timed out after 60 minutes.

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Unsuccessful board reset, trying again in 1 minute ...

Touching and printing out SMI info

Attempting to reset card(s).

tt-smi reset was successful