Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate test executables per architecture #16594

Merged
merged 29 commits into from
Jan 13, 2025
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
46c7bf2
Generate unit test executable per architecture
blozano-tt Jan 10, 2025
c72dafc
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 10, 2025
b4ea273
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 10, 2025
160fbe1
Update unit tests exe name
blozano-tt Jan 10, 2025
e209653
Fix some stuff
blozano-tt Jan 10, 2025
2d80b1e
Generate more EXE
blozano-tt Jan 11, 2025
33ef200
Clean
blozano-tt Jan 11, 2025
f002353
Delete useless script
blozano-tt Jan 11, 2025
df06f83
Update scripts
blozano-tt Jan 11, 2025
f5d1dae
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 11, 2025
a17a147
Adjust perf_microbenchmark
blozano-tt Jan 12, 2025
45b1759
Merge branch 'blozano-test-per-arch' of https://github.com/tenstorren…
blozano-tt Jan 12, 2025
13c725c
Fix
blozano-tt Jan 12, 2025
c97c2aa
Update test_pgm_dispatch references
blozano-tt Jan 12, 2025
54d54b4
Update test_bw_and_latency references
blozano-tt Jan 12, 2025
d5f4b4c
Update references to test_dispatcher
blozano-tt Jan 12, 2025
a3dfdc2
Update references to test_ethernet_read_and_send_data
blozano-tt Jan 12, 2025
365d7bf
Update references to test_workers_and_erisc_datamover_unidirectional
blozano-tt Jan 12, 2025
abefbfd
Update references to test_ethernet_bidirectional_bandwidth_no_edm
blozano-tt Jan 12, 2025
ce4bdbe
Update references to test_ethernet_link_ping_latency_no_edm
blozano-tt Jan 12, 2025
5211fc2
Update references to test_ethernet_hop_latencies_no_edm
blozano-tt Jan 12, 2025
7d6b45e
Update ref to test_vc_uni_tunnel and related
blozano-tt Jan 12, 2025
050995c
Update references to test_noc_unicast_vs_multicast_to_single_core_lat…
blozano-tt Jan 12, 2025
5517d5c
Update references
blozano-tt Jan 12, 2025
2d33409
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 12, 2025
3e66442
Update CMakeLists.txt
blozano-tt Jan 12, 2025
948fe94
Update tests/tt_metal/tt_metal/api/CMakeLists.txt
blozano-tt Jan 13, 2025
3f1ea2c
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 13, 2025
413801a
Merge branch 'main' into blozano-test-per-arch
blozano-tt Jan 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/build-and-unit-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,12 @@ jobs:
{name: All C++, cmd: ./tests/scripts/run_cpp_unit_tests.sh},
{name: tools, cmd: ./tests/scripts/run_tools_tests.sh},

{name: user kernel path, cmd: "rm -rf /tmp/kernels && TT_METAL_KERNEL_PATH=/tmp/kernels ./build/test/tt_metal/unit_tests_api --gtest_filter=CompileProgramWithKernelPathEnvVarFixture.*"},
{name: api, cmd: "./build/test/tt_metal/unit_tests_api"},
{name: debug_tools, cmd: "./build/test/tt_metal/unit_tests_debug_tools"},
{name: user kernel path, cmd: "rm -rf /tmp/kernels && TT_METAL_KERNEL_PATH=/tmp/kernels ./build/test/tt_metal/unit_tests_api_${{ inputs.arch }} --gtest_filter=CompileProgramWithKernelPathEnvVarFixture.*"},
{name: api, cmd: "./build/test/tt_metal/unit_tests_api_${{ inputs.arch }}"},
{name: debug_tools, cmd: "./build/test/tt_metal/unit_tests_debug_tools_${{ inputs.arch }}"},
{name: device, cmd: "./build/test/tt_metal/unit_tests_device"},
{name: dispatch, cmd: "./build/test/tt_metal/unit_tests_dispatch"},
{name: eth, cmd: "./build/test/tt_metal/unit_tests_eth"},
{name: dispatch, cmd: "./build/test/tt_metal/unit_tests_dispatch_${{ inputs.arch }}"},
{name: eth, cmd: "./build/test/tt_metal/unit_tests_eth_${{ inputs.arch }}"},
{name: llk, cmd: "./build/test/tt_metal/unit_tests_llk"},
{name: stl, cmd: "./build/test/tt_metal/unit_tests_stl"},
{name: distributed, cmd: "./build/test/tt_metal/distributed/distributed_unit_tests --gtest_filter=MeshDeviceSuite.*"},
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/cpp-post-commit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,17 @@ jobs:
{name: All C++, cmd: ./tests/scripts/run_cpp_unit_tests.sh},
{name: tools, cmd: ./tests/scripts/run_tools_tests.sh},

{name: user kernel path, cmd: "rm -rf /tmp/kernels && TT_METAL_KERNEL_PATH=/tmp/kernels ./build/test/tt_metal/unit_tests_api --gtest_filter=CompileProgramWithKernelPathEnvVarFixture.*"},
{name: api, cmd: "./build/test/tt_metal/unit_tests_api"},
{name: debug_tools, cmd: "./build/test/tt_metal/unit_tests_debug_tools"},
{name: user kernel path, cmd: "rm -rf /tmp/kernels && TT_METAL_KERNEL_PATH=/tmp/kernels ./build/test/tt_metal/unit_tests_api_${{ inputs.arch }} --gtest_filter=CompileProgramWithKernelPathEnvVarFixture.*"},
{name: api, cmd: "./build/test/tt_metal/unit_tests_api_${{ inputs.arch }}"},
{name: debug_tools, cmd: "./build/test/tt_metal/unit_tests_debug_tools_${{ inputs.arch }}"},
{name: device, cmd: "./build/test/tt_metal/unit_tests_device"},
{name: dispatch, cmd: "./build/test/tt_metal/unit_tests_dispatch"},
{name: eth, cmd: "./build/test/tt_metal/unit_tests_eth"},
{name: dispatch, cmd: "./build/test/tt_metal/unit_tests_dispatch_${{ inputs.arch }}"},
{name: eth, cmd: "./build/test/tt_metal/unit_tests_eth_${{ inputs.arch }}"},
{name: llk, cmd: "./build/test/tt_metal/unit_tests_llk"},
{name: stl, cmd: "./build/test/tt_metal/unit_tests_stl"},
{name: distributed, cmd: "./build/test/tt_metal/distributed/distributed_unit_tests --gtest_filter=MeshDeviceSuite.*"},

{name: dispatch multicmd queue, cmd: "TT_METAL_GTEST_NUM_HW_CQS=2 ./build/test/tt_metal/unit_tests_dispatch --gtest_filter=MultiCommandQueue*Fixture.*"},
{name: dispatch multicmd queue, cmd: "TT_METAL_GTEST_NUM_HW_CQS=2 ./build/test/tt_metal/unit_tests_dispatch_${{ inputs.arch }} --gtest_filter=MultiCommandQueue*Fixture.*"},

{name: ttnn cpp unit tests, cmd: ./build/test/ttnn/unit_tests_ttnn},
{name: ttnn ccl cpp unit tests, cmd: ./build/test/ttnn/unit_tests_ttnn_ccl},
Expand Down
9 changes: 0 additions & 9 deletions tests/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,3 @@
if(NOT DEFINED ENV{ARCH_NAME})
message(FATAL_ERROR "Please set ARCH_NAME to grayskull, wormhole_b0, or blackhole")
endif(NOT DEFINED ENV{ARCH_NAME})

set(currentArch "$ENV{ARCH_NAME}")
string(REPLACE "wormhole_b0" "wormhole" currentArch "${currentArch}")

include_directories(${PROJECT_SOURCE_DIR}/tt_metal/hw/inc/${currentArch})

enable_testing()
include(GoogleTest)
add_library(test_common_libs INTERFACE)
Expand Down
161 changes: 83 additions & 78 deletions tests/scripts/run_cpp_fd2_tests.sh

Large diffs are not rendered by default.

7 changes: 6 additions & 1 deletion tests/scripts/run_cpp_unit_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

set -eo pipefail

if [[ -z "$ARCH_NAME" ]]; then
echo "Must provide ARCH_NAME in environment" 1>&2
exit 1
fi

if [[ -z "$TT_METAL_HOME" ]]; then
echo "Must provide TT_METAL_HOME in environment" 1>&2
exit 1
Expand All @@ -13,7 +18,7 @@ if [[ ! -z "$TT_METAL_SLOW_DISPATCH_MODE" ]]; then
else
# Enable this on BH after #14613
if [[ "$ARCH_NAME" == "wormhole_b0" ]]; then
TT_METAL_GTEST_ETH_DISPATCH=1 ./build/test/tt_metal/unit_tests_dispatch
TT_METAL_GTEST_ETH_DISPATCH=1 ./build/test/tt_metal/unit_tests_dispatch_${ARCH_NAME}
fi
env python3 tests/scripts/run_tt_eager.py --dispatch-mode fast
env python3 tests/scripts/run_tt_metal.py --dispatch-mode fast
Expand Down
96 changes: 0 additions & 96 deletions tests/scripts/run_testpoint_perprocess.py
blozano-tt marked this conversation as resolved.
Show resolved Hide resolved

This file was deleted.

2 changes: 1 addition & 1 deletion tests/scripts/run_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ run_frequent_api_pipeline_tests() {
local dispatch_mode=$3

if [[ $dispatch_mode == "slow" ]]; then
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_dispatch --gtest_filter=DispatchStress.TensixRunManyTimes
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_dispatch_${ARCH_NAME} --gtest_filter=DispatchStress.TensixRunManyTimes
echo "Running Python API unit tests in SD for frequent..."
./tests/scripts/run_python_api_unit_tests.sh
fi
Expand Down
11 changes: 8 additions & 3 deletions tests/scripts/run_tools_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,17 @@ if [[ -z "$TT_METAL_HOME" ]]; then
exit 1
fi

if [ -z "${ARCH_NAME}" ]; then
echo "Error: ARCH_NAME is not set. Exiting." >&2
exit 1
fi

if [[ -z "$TT_METAL_SLOW_DISPATCH_MODE" ]] ; then
# Watcher dump tool testing
echo "Running watcher dump tool tests..."

# Run a test that populates basic fields but not watcher fields
./build/test/tt_metal/unit_tests_debug_tools --gtest_filter=*PrintHanging
./build/test/tt_metal/unit_tests_debug_tools_${ARCH_NAME} --gtest_filter=*PrintHanging

# Run dump tool w/ minimum data - no error expected.
./build/tools/watcher_dump -d=0 -w -c
Expand All @@ -22,15 +27,15 @@ if [[ -z "$TT_METAL_SLOW_DISPATCH_MODE" ]] ; then
echo "Watcher dump minimal test - Pass"

# Now run with all watcher features, expect it to throw.
./build/test/tt_metal/unit_tests_debug_tools --gtest_filter=*WatcherAssertBrisc
./build/test/tt_metal/unit_tests_debug_tools_${ARCH_NAME} --gtest_filter=*WatcherAssertBrisc
./build/tools/watcher_dump -d=0 -w &> tmp.log || { echo "Above failure is expected."; }

# Verify the error we expect showed up in the program output.
grep "brisc tripped an assert" tmp.log > /dev/null || { echo "Error: couldn't find expected string in command output:" ; cat tmp.log; exit 1; }
echo "Watcher dump all data test - Pass"

# Check that stack dumping is working
./build/test/tt_metal/unit_tests_debug_tools --gtest_filter=*TestWatcherRingBufferBrisc
./build/test/tt_metal/unit_tests_debug_tools_${ARCH_NAME} --gtest_filter=*TestWatcherRingBufferBrisc
./build/tools/watcher_dump -d=0 -w
grep "brisc highest stack usage:" generated/watcher/watcher.log > /dev/null || { echo "Error: couldn't find stack usage in watcher log after dump." ; exit 1; }
echo "Watcher stack usage test - Pass"
Expand Down
11 changes: 8 additions & 3 deletions tests/scripts/run_tunneler_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ if [[ -z "$TT_METAL_SLOW_DISPATCH_MODE" ]]; then
exit 1
fi

if [[ -z "$ARCH_NAME" ]]; then
echo "Must provide ARCH_NAME in environment" 1>&2
exit 1
fi

export TT_METAL_CLEAR_L1=1

echo "Running tunneler tests now...";
Expand Down Expand Up @@ -46,9 +51,9 @@ main() {

if [[ $ARCH_NAME == "wormhole_b0" && $machine_type != "N150" ]]; then
for max_packet_size_words in 256 512 1024 2048; do
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_uni_tunnel --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 32768 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_bi_tunnel_2ep --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 32768 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_bi_tunnel_4ep --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 16384 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_uni_tunnel_${ARCH_NAME} --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 32768 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_bi_tunnel_2ep_${ARCH_NAME} --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 32768 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
run_test "./build/test/tt_metal/perf_microbenchmark/routing/test_vc_bi_tunnel_4ep_${ARCH_NAME} --tx_x 4 --tx_y 7 --mux_x 0 --mux_y 7 --demux_x 0 --demux_y 0 --rx_x 0 --rx_y 1 --max_packet_size_words $max_packet_size_words --tx_skip_pkt_content_gen 1 --rx_disable_data_check 1 --rx_disable_header_check 1 --tx_pkt_dest_size_choice 1 --check_txrx_timeout 1 --data_kb_per_tx 1048576 --tunneler_queue_size_bytes 16384 --tx_queue_size_bytes 65536 --rx_queue_size_bytes 131072 --mux_queue_size_bytes 65536 --demux_queue_size_bytes 65536"
done
fi

Expand Down
20 changes: 13 additions & 7 deletions tests/scripts/t3000/run_t3000_unit_tests.sh
Original file line number Diff line number Diff line change
@@ -1,20 +1,26 @@
#!/bin/bash
set -eo pipefail

# Exit immediately if ARCH_NAME is not set or empty
if [ -z "${ARCH_NAME}" ]; then
echo "Error: ARCH_NAME is not set. Exiting." >&2
exit 1
fi

run_t3000_ttmetal_tests() {
# Record the start time
fail=0
start_time=$(date +%s)

echo "LOG_METAL: Running run_t3000_ttmetal_tests"
./build/test/tt_metal/distributed/distributed_unit_tests
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth --gtest_filter="DeviceFixture.ActiveEthKernelsDirectSendAllConnectedChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth --gtest_filter="DeviceFixture.ActiveEthKernelsSendInterleavedBufferAllConnectedChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth --gtest_filter="DeviceFixture.ActiveEthKernelsDirectRingGatherAllChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth --gtest_filter="DeviceFixture.ActiveEthKernelsInterleavedRingGatherAllChips" ; fail+=$?
TT_METAL_ENABLE_REMOTE_CHIP=1 ./build/test/tt_metal/unit_tests_dispatch --gtest_filter="CommandQueueSingleCard*Fixture.*" ; fail+=$?
./build/test/tt_metal/unit_tests_dispatch --gtest_filter="CommandQueueMultiDevice*Fixture.*" ; fail+=$?
./build/test/tt_metal/unit_tests_debug_tools --gtest_filter="DPrintFixture.*:WatcherFixture.*" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth_${ARCH_NAME} --gtest_filter="DeviceFixture.ActiveEthKernelsDirectSendAllConnectedChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth_${ARCH_NAME} --gtest_filter="DeviceFixture.ActiveEthKernelsSendInterleavedBufferAllConnectedChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth_${ARCH_NAME} --gtest_filter="DeviceFixture.ActiveEthKernelsDirectRingGatherAllChips" ; fail+=$?
TT_METAL_SLOW_DISPATCH_MODE=1 ./build/test/tt_metal/unit_tests_eth_${ARCH_NAME} --gtest_filter="DeviceFixture.ActiveEthKernelsInterleavedRingGatherAllChips" ; fail+=$?
TT_METAL_ENABLE_REMOTE_CHIP=1 ./build/test/tt_metal/unit_tests_dispatch_${ARCH_NAME} --gtest_filter="CommandQueueSingleCard*Fixture.*" ; fail+=$?
./build/test/tt_metal/unit_tests_dispatch_${ARCH_NAME} --gtest_filter="CommandQueueMultiDevice*Fixture.*" ; fail+=$?
./build/test/tt_metal/unit_tests_debug_tools_${ARCH_NAME} --gtest_filter="DPrintFixture.*:WatcherFixture.*" ; fail+=$?

# Record the end time
end_time=$(date +%s)
Expand Down
Loading
Loading