ttnn.hypot_bw low PCC

[amalbasaTT]

Describe the bug
ttnn.hypot_bw fails with low PCC when input_tensor_a and/or input_tensor_b has bfloat8_b dtype

To Reproduce
Steps to reproduce the behavior:
Sweep test for hypot_bw is located in 'tests/sweep_framework/sweeps/eltwise/binary_backward/hypot_bw/hypot_bw.py'

1. Go to 'tests/sweep_framework/sweeps/eltwise/binary_backward/hypot_bw/hypot_bw.py'
2. Generate new parameter vectors and run the sweep test

python3 tests/sweep_framework/sweeps_parameter_generator.py --elastic cloud --module-name eltwise.binary_backward.hypot_bw.hypot_bw
python3 tests/sweep_framework/sweeps_runner.py --elastic cloud --module-name eltwise.binary_backward.hypot_bw.hypot_bw --suite-name xfail

5. See the error. Results can be found on elastic cloud as explained here: https://github.com/tenstorrent/tt-metal/tree/main/tests/sweep_framework

[KalaivaniMCW]

ttnn.hypot_bw uses ttnn.reciprocal in its implementation which has this issue #14672
This is causing the PCC drop when using bfloat8_b

[KalaivaniMCW]

Hi @amalbasaTT ,
When using bfloat8_b dtype, could you try with non-zero inputs to see if these cases pass ?
You can find more information on this here #14672

When involving bfloat8_b and ttnn.reciprocal related ops we need separate implementations for zero- and non-zero inputs.
Handling these at kernel level will make the op slower, so we'll discuss what the best approach will be - whether to do it at composite level. For now could we test on non-zero inputs to identify any other issues other than this.

[amalbasaTT]

@KalaivaniMCW ,
Of course, no problem. Do you want a unit test or a sweep test for each of ttnn.reciprocal related ops? Also is there a list of ttnn.reciprocal related ops somehwere?

[KalaivaniMCW]

For now, the below issues seems to be related.
If these op sweeps pass for non-zero inputs, we can rule out these to be related to bfloat8_b behaviour.
#13776
#13937
#13975

[amalbasaTT]

• ttnn.hypot passes in all cases when all inputs (input_tensor_a and input_tensor_b) are with non-zero elements and with bfloat8_b datatype
• ttnn.rdiv_bw (issue) passes in 4.167% cases when all inputs (grad_tensor, input_tensor_a and factor) are with non-zero elements and when grad_tensor and input_tensor_a have bfloat8_b datatype

I will need some additional time to analyse div_bw since it has way more parameters.

checkout branch amalbasaTT/reciprocal-related-nonzero-sweeps to check the tests (hypot_nonzero.py and rdiv_bw_nonzero.py)

[amalbasaTT]

@KalaivaniMCW

I have tested rdiv_bw, hypot_bw and div_bw with inputs outside of range [-1, 1]. Before running the ops, i have converted inputs to ttnn tensor with dtype bfloat8_b and back to torch before running ops with them, so that i could ensure that all the inputs are non-zero after any eventual cut-off when converting them to ttnn tensor with bfloat8_b. Issue i found is that when input_tensor has bfloat8_b and "certain" shape, it produces an all-zero tensor (or tensors for binary ops). You can check details in issues i found bellow, i have provided a unit test and a sweep test for each op:

#16305
#16304
#16306

Besides the issue i reported, all three ops work fine