Skip to content

(Single-card) Nightly model and ttnn tests #6338

(Single-card) Nightly model and ttnn tests

(Single-card) Nightly model and ttnn tests #6338

Manually triggered December 9, 2024 12:10
Status Failure
Total duration 1h 52m 13s
Artifacts 35
build-artifact  /  ...  /  build-docker-image
22s
build-artifact / build-docker-image / build-docker-image
Matrix: build-artifact / build-artifact
Matrix: fd-nightly / fd-nightly
Matrix: fd-nightly / nightly-wh-models
Matrix: fd-nightly / nightly-wh-unstable-models
Fit to window
Zoom out
Zoom in

Annotations

10 errors, 2 warnings, and 222 notices
fd-nightly / Nightly N150 mistral7b_eth
Process completed with exit code 1.
fd-nightly / [Unstable] Nightly N150 mamba 2
Process completed with exit code 1.
fd-nightly / FD GS ttnn nightly grayskull
Process completed with exit code 1.
fd-nightly / FD WH N300 pgm dispatch nightly wormhole_b0
Process completed with exit code 1.
pcie-cards-are-being-used-cleanup
Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.
fd-nightly / FD WH N300 ttnn nightly wormhole_b0
Process completed with exit code 1.
fd-nightly / FD WH N300 ttnn nightly wormhole_b0
The action 'Run frequent reg tests scripts' has timed out after 70 minutes.
fd-nightly / FD WH N150 ttnn nightly wormhole_b0
The self-hosted runner: tt-metal-ci-vm-39 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
fd-nightly / FD WH N150 ttnn nightly wormhole_b0
The action 'Run frequent reg tests scripts' has timed out after 70 minutes.
pcie-cards-are-being-used-cleanup
Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.
fd-nightly / FD WH N300 pgm dispatch nightly wormhole_b0
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
fd-nightly / FD WH N300 ttnn nightly wormhole_b0
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 51 %
disk-usage-after-startup
Disk usage is 51 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 65 %
disk-usage-after-startup
Disk usage is 65 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 73 %
disk-usage-after-startup
Disk usage is 73 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 66 %
disk-usage-after-startup
Disk usage is 66 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 66 %
disk-usage-after-startup
Disk usage is 66 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 77 %
disk-usage-after-startup
Disk usage is 77 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 53 %
disk-usage-after-startup
Disk usage is 53 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 40 %
disk-usage-after-startup
Disk usage is 40 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 51 %
disk-usage-after-startup
Disk usage is 51 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 73 %
disk-usage-after-startup
Disk usage is 73 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 69 %
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 70 %
disk-usage-after-startup
Disk usage is 70 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 76 %
disk-usage-after-startup
Disk usage is 76 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 65 %
disk-usage-after-startup
Disk usage is 65 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 72 %
disk-usage-after-startup
Disk usage is 72 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 71 %
disk-usage-after-startup
Disk usage is 71 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 70 %
disk-usage-after-startup
Disk usage is 70 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 34 %
disk-usage-after-startup
Disk usage is 34 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 69 %
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 69 %
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 78 %
disk-usage-after-startup
Disk usage is 78 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 69 %
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 65 %
disk-usage-after-startup
Disk usage is 65 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 67 %
disk-usage-after-startup
Disk usage is 67 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 76 %
disk-usage-after-startup
Disk usage is 76 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 72 %
disk-usage-after-startup
Disk usage is 72 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 61 %
disk-usage-after-startup
Disk usage is 61 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 69 %
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 63 %
disk-usage-after-startup
Disk usage is 63 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 22 %
disk-usage-after-startup
Disk usage is 22 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 67 %
disk-usage-after-startup
Disk usage is 67 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
disk-usage-before-startup
Disk usage is 63 %
disk-usage-after-startup
Disk usage is 63 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info

Artifacts

Produced during runtime
Name Size
TTMetal_build_grayskull
303 MB
TTMetal_build_wormhole_b0
304 MB
test_reports_092f4e1f-ee9c-4796-8373-c99430ed8fec
429 Bytes
test_reports_0dc0a2a7-f709-4afe-bdfd-bcf8bef78587
1.33 KB
test_reports_15ddd582-4566-4dcc-b81a-6c0243bd7661
733 Bytes
test_reports_18678d79-f82f-4058-aada-a1c55346fb0b
475 Bytes
test_reports_19b8b98a-11fb-4c99-ada3-ce1001830f39
860 Bytes
test_reports_1d224738-4fc5-4f2b-bd47-c52558d0d31a
721 Bytes
test_reports_1eeefbe1-b70f-4393-a2db-2a10bb1ee02a
744 Bytes
test_reports_2c4fb087-afe5-494f-b238-38c40052c15b
969 Bytes
test_reports_318570cd-b83f-4ccc-a414-f6b30f9270ae
11.5 KB
test_reports_355107cd-984e-4d4f-815d-20c7a6d31ad8
652 Bytes
test_reports_3cca9fba-2e6d-4e49-96ba-b7ae81e27a56
430 Bytes
test_reports_4532b80d-b9c5-4e05-be9e-f55a70e892d7
4.59 KB
test_reports_46a605d6-61fe-44fa-be98-d036760862c6
561 Bytes
test_reports_47543f2d-0582-475e-8cb6-63ee963bf152
1.08 KB
test_reports_548f297c-f47f-417d-b2c8-c88e224db1db
485 Bytes
test_reports_55a436c2-910b-4c75-a131-4a2cd1877c27
604 Bytes
test_reports_56779685-ac48-45d4-8754-e4c20b2512f1
819 Bytes
test_reports_56ba5831-4620-4215-aaec-0b36ef31f786
708 Bytes
test_reports_60204cf1-5de4-4adc-98c1-5eba512f2ea3
545 Bytes
test_reports_69f8e6dc-18b8-4c32-b974-067ebfcd0a4b
624 Bytes
test_reports_7e8340f6-1207-4bf2-8200-9475de1eeb61
809 Bytes
test_reports_87139c4b-a439-486a-877b-65ee73f44aa3
476 Bytes
test_reports_8c392624-8392-44d1-919c-7db75335102a
610 Bytes
test_reports_9012300f-355e-4102-951e-0645c55f9e62
4.62 KB
test_reports_9576ef8c-707b-4b0e-a7e4-64574b8cce2a
1.36 KB
test_reports_a317b630-2616-4153-a5da-69554e5d1109
655 Bytes
test_reports_ad2c7535-b8d2-414a-b6c4-ca8ea93dbb14
658 Bytes
test_reports_bd1a0d99-834d-4ab1-860d-f718c01c30a6
711 Bytes
test_reports_cec45222-09f1-4f06-a33f-4458d54eda1b
724 Bytes
test_reports_d3e6e7f2-6368-4b5b-aec6-5fb7e0d564f6
621 Bytes
test_reports_db2b28d2-1d4d-4971-adeb-4f54377cd3f8
626 Bytes
test_reports_eca08e7d-fad7-4774-afa2-a3733bc5c4c3
860 Bytes
test_reports_f649ae7b-f020-4831-9e6b-d95c19b72c2d
1.32 KB