Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update xla_docker_build.tf #6488

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Update xla_docker_build.tf

87b15ca
Select commit
Loading
Failed to load commit list.
Draft

Update xla_docker_build.tf #6488

Update xla_docker_build.tf
87b15ca
Select commit
Loading
Failed to load commit list.
Google Cloud Build / ci-tpu-test-trigger (tpu-pytorch) failed Feb 7, 2024 in 57m 34s

Summary

Build Information

Trigger ci-tpu-test-trigger
Build 8130a5d0-05d5-461d-8322-6daad591031c
Start 2024-02-06T16:41:23-08:00
Duration 56m41.411s
Status FAILURE

Steps

Step Status Duration
git_fetch SUCCESS 9.527s
git_checkout SUCCESS 686ms
build_pytorch-xla-test_docker_image SUCCESS 40m38.071s
push_pytorch-xla-test_docker_image SUCCESS 1m3.919s
run_e2e_tests FAILURE 14m45.281s

Details

starting build "8130a5d0-05d5-461d-8322-6daad591031c"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
 * branch            87b15ca3c6c87bac3b8e958551b143dff2552529 -> FETCH_HEAD
HEAD is now at 87b15ca Update xla_docker_build.tf
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch":  * [new branch]          01172023_pin_update     -> origin/01172023_pin_update
Step #0 - "git_fetch":  * [new branch]          2-0-python-3-10         -> origin/2-0-python-3-10
Step #0 - "git_fetch":  * [new branch]          4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch":  * [new branch]          5d-unit-test            -> origin/5d-unit-test
Step #0 - "git_fetch":  * [new branch]          FixTransferToServer     -> origin/FixTransferToServer
Step #0 - "git_fetch":  * [new branch]          JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch":  * [new branch]          JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch":  * [new branch]          JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch":  * [new branch]          JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch":  * [new branch]          JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch":  * [new branch]          JackCao/update_to_2.0   -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch":  * [new branch]          JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-1        -> origin/JackCaoG-patch-1
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-2        -> origin/JackCaoG-patch-2
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-3        -> origin/JackCaoG-patch-3
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-4        -> origin/JackCaoG-patch-4
Step #0 - "git_fetch":  * [new branch]          JackCaoG-patch-5        -> origin/JackCaoG-patch-5
Step #0 - "git_fetch":  * [new branch]          JackCaoG-remove-pin     -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch":  * [new branch]          JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch":  * [new branch]          JackCaoG/2.0_patch      -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch":  * [new branch]          JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch":  * [new branch]          JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch":  * [new branch]          JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_2_1_py311_build -> origin/JackCaoG/add_2_1_py311_build
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_gpu_doc    -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch":  * [new branch]          JackCaoG/add_test_print -> origin/JackCaoG/add_test_print
Step #0 - "git_fetch":  * [new branch]          JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backout_22_mesh_spmd -> origin/JackCaoG/backout_22_mesh_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_allgather_out -> origin/JackCaoG/backport_22_allgather_out
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_diagonal_scatter -> origin/JackCaoG/backport_22_diagonal_scatter
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_doc -> origin/JackCaoG/backport_22_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_reduce_scatter -> origin/JackCaoG/backport_22_reduce_scatter
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_reducescatter_out -> origin/JackCaoG/backport_22_reducescatter_out
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_scalar -> origin/JackCaoG/backport_22_scalar
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_22_transfer_gpu_scalar -> origin/JackCaoG/backport_22_transfer_gpu_scalar
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_dynamo_cpu_move -> origin/JackCaoG/backport_dynamo_cpu_move
Step #0 - "git_fetch":  * [new branch]          JackCaoG/backport_frame_count -> origin/JackCaoG/backport_frame_count
Step #0 - "git_fetch":  * [new branch]          JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch":  * [new branch]          JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch":  * [new branch]          JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch":  * [new branch]          JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch":  * [new branch]          JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cherry_pick_frame_count -> origin/JackCaoG/cherry_pick_frame_count
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/clean_up_gettensor -> origin/JackCaoG/clean_up_gettensor
Step #0 - "git_fetch":  * [new branch]          JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch":  * [new branch]          JackCaoG/cleanup_docs   -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch":  * [new branch]          JackCaoG/colab_whl_abi  -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch":  * [new branch]          JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch":  * [new branch]          JackCaoG/debug_fix      -> origin/JackCaoG/debug_fix
Step #0 - "git_fetch":  * [new branch]          JackCaoG/debug_tool     -> origin/JackCaoG/debug_tool
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_flaky_test_2 -> origin/JackCaoG/disable_flaky_test_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_flaky_test_3 -> origin/JackCaoG/disable_flaky_test_3
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_pjrt_computation_client_test -> origin/JackCaoG/disable_pjrt_computation_client_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_doc     -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd    -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch":  * [new branch]          JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_cumsum_long -> origin/JackCaoG/enable_cumsum_long
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_mp_test_on_tpuci -> origin/JackCaoG/enable_mp_test_on_tpuci
Step #0 - "git_fetch":  * [new branch]          JackCaoG/enable_reduce_scatter_test_gpu -> origin/JackCaoG/enable_reduce_scatter_test_gpu
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_diagonal_scatter_dim -> origin/JackCaoG/fix_diagonal_scatter_dim
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo     -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_input_sharding_test -> origin/JackCaoG/fix_dynamo_input_sharding_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_hlo_dump   -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_typo       -> origin/JackCaoG/fix_typo
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch":  * [new branch]          JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch":  * [new branch]          JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch":  * [new branch]          JackCaoG/graph_hash_debug -> origin/JackCaoG/graph_hash_debug
Step #0 - "git_fetch":  * [new branch]          JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch":  * [new branch]          JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch":  * [new branch]          JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch":  * [new branch]          JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_pin_2_0   -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch":  * [new branch]          JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch":  * [new branch]          JackCaoG/new_alias_api  -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/nightly_build_python -> origin/JackCaoG/nightly_build_python
Step #0 - "git_fetch":  * [new branch]          JackCaoG/parallel_spmd  -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pin_expecttest -> origin/JackCaoG/pin_expecttest
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pt_xla_debug_5 -> origin/JackCaoG/pt_xla_debug_5
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pt_xla_debug_dynamo_execute -> origin/JackCaoG/pt_xla_debug_dynamo_execute
Step #0 - "git_fetch":  * [new branch]          JackCaoG/pt_xla_debug_max_frame -> origin/JackCaoG/pt_xla_debug_max_frame
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_ABI_0     -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch":  * [new branch]          JackCaoG/remove_pin     -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch":  * [new branch]          JackCaoG/remove_reshape_lowering -> origin/JackCaoG/remove_reshape_lowering
Step #0 - "git_fetch":  * [new branch]          JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch":  * [new branch]          JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch":  * [new branch]          JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch":  * [new branch]          JackCaoG/skip_partitioner -> origin/JackCaoG/skip_partitioner
Step #0 - "git_fetch":  * [new branch]          JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_aliasing  -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch":  * [new branch]          JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch":  * [new branch]          JackCaoG/tracing_timer  -> origin/JackCaoG/tracing_timer
Step #0 - "git_fetch":  * [new branch]          JackCaoG/trouble_shooting_update -> origin/JackCaoG/trouble_shooting_update
Step #0 - "git_fetch":  * [new branch]          JackCaoG/truncate_python_stack -> origin/JackCaoG/truncate_python_stack
Step #0 - "git_fetch":  * [new branch]          JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch":  * [new branch]          JackCaoG/updateDocker   -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_doc1    -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_doc_remove_xrt -> origin/JackCaoG/update_doc_remove_xrt
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_docker_instruction -> origin/JackCaoG/update_docker_instruction
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_backend_name -> origin/JackCaoG/update_dynamo_backend_name
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_backend_nograd -> origin/JackCaoG/update_dynamo_backend_nograd
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_doc -> origin/JackCaoG/update_dynamo_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_number_2.1 -> origin/JackCaoG/update_dynamo_number_2.1
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_readme -> origin/JackCaoG/update_dynamo_readme
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_save_graph_path -> origin/JackCaoG/update_dynamo_save_graph_path
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_dynamo_torch_equal -> origin/JackCaoG/update_dynamo_torch_equal
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_doc -> origin/JackCaoG/update_gpu_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_wheel_link -> origin/JackCaoG/update_gpu_wheel_link
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_gpu_wheellink -> origin/JackCaoG/update_gpu_wheellink
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_pjrt_future -> origin/JackCaoG/update_pjrt_future
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_readme_2 -> origin/JackCaoG/update_readme_2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_set_default_tensor_type -> origin/JackCaoG/update_set_default_tensor_type
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_tf_2023018 -> origin/JackCaoG/update_tf_2023018
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_to_2.2  -> origin/JackCaoG/update_to_2.2
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_torch_compile_usage -> origin/JackCaoG/update_torch_compile_usage
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_trouble_shotting_doc -> origin/JackCaoG/update_trouble_shotting_doc
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_troubleshotting_with_sanitycheck -> origin/JackCaoG/update_troubleshotting_with_sanitycheck
Step #0 - "git_fetch":  * [new branch]          JackCaoG/update_wheel_docker_format -> origin/JackCaoG/update_wheel_docker_format
Step #0 - "git_fetch":  * [new branch]          JackCaoG/use_new_alias_api -> origin/JackCaoG/use_new_alias_api
Step #0 - "git_fetch":  * [new branch]          JackCaoG/xrt_cherry_pick_07_10 -> origin/JackCaoG/xrt_cherry_pick_07_10
Step #0 - "git_fetch":  * [new branch]          JackCaoG/xrt_cherrypick_0721 -> origin/JackCaoG/xrt_cherrypick_0721
Step #0 - "git_fetch":  * [new branch]          JackCaog/dynamo_input_sharding -> origin/JackCaog/dynamo_input_sharding
Step #0 - "git_fetch":  * [new branch]          JackCaog/quick_spmd_fix -> origin/JackCaog/quick_spmd_fix
Step #0 - "git_fetch":  * [new branch]          JackCaog/spmd_device_default -> origin/JackCaog/spmd_device_default
Step #0 - "git_fetch":  * [new branch]          Liyang90-patch-1        -> origin/Liyang90-patch-1
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-1       -> origin/ManfeiBai-patch-1
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-10      -> origin/ManfeiBai-patch-10
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-11      -> origin/ManfeiBai-patch-11
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-12      -> origin/ManfeiBai-patch-12
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-13      -> origin/ManfeiBai-patch-13
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-14      -> origin/ManfeiBai-patch-14
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-15      -> origin/ManfeiBai-patch-15
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-16      -> origin/ManfeiBai-patch-16
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-17      -> origin/ManfeiBai-patch-17
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-18      -> origin/ManfeiBai-patch-18
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-19      -> origin/ManfeiBai-patch-19
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-2       -> origin/ManfeiBai-patch-2
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-20      -> origin/ManfeiBai-patch-20
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-21      -> origin/ManfeiBai-patch-21
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-22      -> origin/ManfeiBai-patch-22
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-23      -> origin/ManfeiBai-patch-23
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-24      -> origin/ManfeiBai-patch-24
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-25      -> origin/ManfeiBai-patch-25
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-26      -> origin/ManfeiBai-patch-26
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-27      -> origin/ManfeiBai-patch-27
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-28      -> origin/ManfeiBai-patch-28
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-29      -> origin/ManfeiBai-patch-29
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-3       -> origin/ManfeiBai-patch-3
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-30      -> origin/ManfeiBai-patch-30
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-31      -> origin/ManfeiBai-patch-31
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-32      -> origin/ManfeiBai-patch-32
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-33      -> origin/ManfeiBai-patch-33
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-34      -> origin/ManfeiBai-patch-34
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-35      -> origin/ManfeiBai-patch-35
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-36      -> origin/ManfeiBai-patch-36
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-37      -> origin/ManfeiBai-patch-37
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-38      -> origin/ManfeiBai-patch-38
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-39      -> origin/ManfeiBai-patch-39
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-4       -> origin/ManfeiBai-patch-4
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-40      -> origin/ManfeiBai-patch-40
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-41      -> origin/ManfeiBai-patch-41
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-41-test -> origin/ManfeiBai-patch-41-test
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-42      -> origin/ManfeiBai-patch-42
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-43      -> origin/ManfeiBai-patch-43
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-44      -> origin/ManfeiBai-patch-44
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-45      -> origin/ManfeiBai-patch-45
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-46      -> origin/ManfeiBai-patch-46
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-47      -> origin/ManfeiBai-patch-47
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-48      -> origin/ManfeiBai-patch-48
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-49      -> origin/ManfeiBai-patch-49
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-5       -> origin/ManfeiBai-patch-5
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-50      -> origin/ManfeiBai-patch-50
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-51      -> origin/ManfeiBai-patch-51
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-52      -> origin/ManfeiBai-patch-52
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-53      -> origin/ManfeiBai-patch-53
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-54      -> origin/ManfeiBai-patch-54
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-55      -> origin/ManfeiBai-patch-55
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-56      -> origin/ManfeiBai-patch-56
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-57      -> origin/ManfeiBai-patch-57
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-58      -> origin/ManfeiBai-patch-58
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-59      -> origin/ManfeiBai-patch-59
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-6       -> origin/ManfeiBai-patch-6
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-60      -> origin/ManfeiBai-patch-60
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-61      -> origin/ManfeiBai-patch-61
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-62      -> origin/ManfeiBai-patch-62
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-63      -> origin/ManfeiBai-patch-63
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-64      -> origin/ManfeiBai-patch-64
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-65      -> origin/ManfeiBai-patch-65
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-7       -> origin/ManfeiBai-patch-7
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-8       -> origin/ManfeiBai-patch-8
Step #0 - "git_fetch":  * [new branch]          ManfeiBai-patch-9       -> origin/ManfeiBai-patch-9
Step #0 - "git_fetch":  * [new branch]          XLASymNodeImplSub       -> origin/XLASymNodeImplSub
Step #0 - "git_fetch":  * [new branch]          XLA_HLO_DEBUG_METADATA  -> origin/XLA_HLO_DEBUG_METADATA
Step #0 - "git_fetch":  * [new branch]          a_tensor_cleanup        -> origin/a_tensor_cleanup
Step #0 - "git_fetch":  * [new branch]          accu_rat
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/8130a5d0-05d5-461d-8322-6daad591031c?project=164006649440.]
...
e_tests": Epoch 0 step 4220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8190 loss 0.0
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/spmd/test_xla_spmd_python_api_interaction.py
Step #4 - "run_e2e_tests": WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148304   49093 pjrt_api.cc:100] GetPjrtApi was found for tpu at /usr/local/lib/python3.10/site-packages/torch_xla/lib/libtpu.so
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148385   49093 pjrt_api.cc:79] PJRT_Api is set for device type tpu
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148399   49093 pjrt_api.cc:146] The PJRT plugin has PJRT API version 0.40. The framework PJRT API version is 0.40.
Step #4 - "run_e2e_tests": ./usr/local/lib/python3.10/site-packages/torch_xla/runtime.py:234: UserWarning: XLA_USE_SPMD is being deprecated. Use torch_xla.runtime.use_spmd() without setting XLA_USE_SPMD env-var.
Step #4 - "run_e2e_tests":   warnings.warn("XLA_USE_SPMD is being deprecated. "
Step #4 - "run_e2e_tests": .............../usr/local/lib/python3.10/site-packages/torch_xla/core/xla_model.py:104: UserWarning: `devkind` argument is deprecated and will be removed in a future release.
Step #4 - "run_e2e_tests":   warnings.warn("`devkind` argument is deprecated and will be removed in a "
Step #4 - "run_e2e_tests": .......
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 23 tests in 13.798s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": OK
Step #4 - "run_e2e_tests": + XLA_EXPERIMENTAL=nonzero:masked_select
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/ds/test_dynamic_shape_models.py -v
Step #4 - "run_e2e_tests": WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.445957   49845 pjrt_api.cc:100] GetPjrtApi was found for tpu at /usr/local/lib/python3.10/site-packages/torch_xla/lib/libtpu.so
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.446034   49845 pjrt_api.cc:79] PJRT_Api is set for device type tpu
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.446048   49845 pjrt_api.cc:146] The PJRT plugin has PJRT API version 0.40. The framework PJRT API version is 0.40.
Step #4 - "run_e2e_tests": /usr/local/lib/python3.10/site-packages/torch_xla/core/xla_model.py:104: UserWarning: `devkind` argument is deprecated and will be removed in a future release.
Step #4 - "run_e2e_tests":   warnings.warn("`devkind` argument is deprecated and will be removed in a "
Step #4 - "run_e2e_tests": test_backward_pass_with_dynamic_input (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests": test_forward_pass_dynamic_input_compile_once (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests": test_forward_pass_dynamic_input_correctness (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_backward_pass_with_dynamic_input (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests":   File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 114, in test_backward_pass_with_dynamic_input
Step #4 - "run_e2e_tests":     loss = criterion(y_pred.squeeze(), y_training)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests":     return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests":     return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests":     return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests":     if target.size() != input.size():
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests":     return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests":     raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_forward_pass_dynamic_input_compile_once (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests":   File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 91, in test_forward_pass_dynamic_input_compile_once
Step #4 - "run_e2e_tests":     criterion(y_pred.squeeze(), y_test)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests":     return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests":     return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests":     return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests":     if target.size() != input.size():
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests":     return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests":     raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_forward_pass_dynamic_input_correctness (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests":   File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 68, in test_forward_pass_dynamic_input_correctness
Step #4 - "run_e2e_tests":     before_train = criterion(y_pred.squeeze(), y_test)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests":     return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests":     return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests":     return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests":     if target.size() != input.size():
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests":     return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests":   File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests":     raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 3 tests in 0.023s
Step #4 - "run_e2e_tests": 
Step #4 - "run_e2e_tests": FAILED (failures=3)
Step #4 - "run_e2e_tests": ++ kubectl get pod/xla-test-job-kvpcs -o 'jsonpath={.status.containerStatuses[?(@.name=="xla-test")].state.terminated.exitCode}'
Step #4 - "run_e2e_tests": + exit 1
Finished Step #4 - "run_e2e_tests"
ERROR
ERROR: build step 4 "google/cloud-sdk" failed: step exited with non-zero status: 1

Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/8130a5d0-05d5-461d-8322-6daad591031c?project=164006649440