Make torch xla available on GPU #3963

Job	Run time
run-tests (latest, test_prod)	3m 41s
run-tests (latest, test_core)	15m 27s
run-tests (latest, test_cli)	5m 42s
run-tests (latest, test_big_modeling)	4m 22s
run-tests (latest, test_deepspeed)	7m 52s
run-tests (latest, test_fsdp)	4m 25s
run-tests (latest, test_example_differences)	6m 7s
run-tests (latest, test_checkpoint_step)	15m 29s
run-tests (latest, test_checkpoint_epoch)	5m 43s
run-tests (latest, test_rest)	10m 5s
run-tests (minimum, test_prod)	3m 55s
run-tests (minimum, test_core)	5m 6s
run-tests (minimum, test_cli)	6m 24s
run-tests (minimum, test_big_modeling)	4m 28s
run-tests (minimum, test_deepspeed)	4m 10s
run-tests (minimum, test_fsdp)	4m 38s
run-tests (minimum, test_example_differences)	4m 49s
run-tests (minimum, test_checkpoint_step)	6m 21s
run-tests (minimum, test_checkpoint_epoch)	6m 9s
run-tests (minimum, test_rest)	15m 27s
	2h 20m 20s

Provide feedback