Resnet matmul data mismatch

[dgolubovicTT]

This error was caught while running ResNet unique op tests.

Repro is available on branch dgolubovic/add-resnet-unique-op-tests-that-fail, and will hopefully be in main soon.

Run: 'pytest -svv forge/test/mlir/test_model_unique_ops.py::test_matmul_resnet'

Error:

Tensor mismatch. PCC = 0.9425581505871167, but required = 0.99

I am attaching ttir and ttnnir:

Matmul_resnet_ttir.txt
Matmul_resnet_ttnn.txt

Bug tracking on metal side:
tenstorrent/tt-metal#16366

[dgolubovicTT]

This might be a metal issue so I will now try to repro it in ttnn test.

[dgolubovicTT]

I've tried to repro mismatch on ttnn:

def test_matmul_resnet(device):
    torch.manual_seed(0)
    pcc = 0.99
    torch_input_tensor_a = torch.randn([1, 2048])
    torch_input_tensor_b = torch.randn([2048, 1000])
    torch_output_tensor = torch.matmul(torch_input_tensor_a, torch_input_tensor_b)

    input_tensor_a = ttnn.from_torch(torch_input_tensor_a, layout=ttnn.TILE_LAYOUT, device=device)
    input_tensor_b = ttnn.from_torch(torch_input_tensor_b, layout=ttnn.TILE_LAYOUT, device=device)

    output1 = ttnn.matmul(input_tensor_a, input_tensor_b)
    output_tensor = ttnn.to_torch(output1)
    assert_with_pcc(torch_output_tensor, output_tensor, pcc=pcc)

But the test passes with pcc=0.99995...

[mtopalovicTT]

@dgolubovicTT did you derive this example from forge IR?

[dgolubovicTT]

I've examined ttnn IR and saw that inputs are in tile layout, the rest is default. I matched the dimensions and that is all. I've provided ttnn IR in issue description. Is there anything else that could influence the discrepancy?

[mtopalovicTT]

Bug tracking on metal side:
tenstorrent/tt-metal#16366