Revert "Avoid fallback for avg_pool -"

[yeounoh]

Reverts #6409 to test if this is related to

======================================================================
[2703](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2704)FAIL: test_pooling_shape_xla (__main__.TestPoolingNNDeviceTypeXLA)
[2704](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2705)Test the output shape calculation for pooling functions
[2705](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2706)----------------------------------------------------------------------
[2706](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2707)Traceback (most recent call last):
[2707](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2708)  File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 2697, in wrapper
[2708](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2709)    method(*args, **kwargs)
[2709](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2710)  File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_device_type.py", line 418, in instantiated_test
[2710](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2711)    result = test(self, **param_kwargs)
[2711](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2712)  File "/tmp/pytorch/xla/test/../../test/nn/test_pooling.py", line 705, in test_pooling_shape
[2712](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2713)    check((1, 1, 3, 3, 4), (1, 1, 5, 6, 7), kernel_size=1, stride=2, padding=0, ceil_mode=True)
[2713](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2714)  File "/tmp/pytorch/xla/test/../../test/nn/test_pooling.py", line 703, in check
[2714](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2715)    self.assertEqual(op(t, *args, **kwargs).shape, expected_out_shape[:i + 2])
[2715](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2716)  File "/tmp/pytorch/xla/test/pytorch_test_base.py", line 693, in assertEqual
[2716](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2717)    return DeviceTypeTestBase.assertEqual(self, x, y, *args, **kwargs)
[2717](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2718)  File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 3574, in assertEqual
[2718](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2719)    raise error_metas.pop()[0].to_error(
[2719](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2720)AssertionError: Scalars are not close!
[2720](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2721)
[2721](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2722)Expected 3 but got 4.
[2722](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2723)Absolute difference: 1 (up to 0.001 allowed)
[2723](https://github.com/pytorch/xla/actions/runs/7747175825/job/21129058373#step:7:2724)Relative difference: 0.3333333333333333 (up to 0.001 allowed)

on master

[yeounoh]

Ok, confirmed -- it passed locally after revert

test_pooling_shape_xla (__main__.TestPoolingNNDeviceTypeXLA)
Test the output shape calculation for pooling functions ... ok