Update xla_docker_build.tf #6488
Draft
Google Cloud Build / ci-tpu-test-trigger (tpu-pytorch)
failed
Feb 7, 2024 in 57m 34s
Summary
Build Information
Trigger | ci-tpu-test-trigger |
Build | 8130a5d0-05d5-461d-8322-6daad591031c |
Start | 2024-02-06T16:41:23-08:00 |
Duration | 56m41.411s |
Status | FAILURE |
Steps
Step | Status | Duration |
---|---|---|
git_fetch | SUCCESS | 9.527s |
git_checkout | SUCCESS | 686ms |
build_pytorch-xla-test_docker_image | SUCCESS | 40m38.071s |
push_pytorch-xla-test_docker_image | SUCCESS | 1m3.919s |
run_e2e_tests | FAILURE | 14m45.281s |
Details
starting build "8130a5d0-05d5-461d-8322-6daad591031c"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
* branch 87b15ca3c6c87bac3b8e958551b143dff2552529 -> FETCH_HEAD
HEAD is now at 87b15ca Update xla_docker_build.tf
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch": * [new branch] 01172023_pin_update -> origin/01172023_pin_update
Step #0 - "git_fetch": * [new branch] 2-0-python-3-10 -> origin/2-0-python-3-10
Step #0 - "git_fetch": * [new branch] 4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch": * [new branch] 5d-unit-test -> origin/5d-unit-test
Step #0 - "git_fetch": * [new branch] FixTransferToServer -> origin/FixTransferToServer
Step #0 - "git_fetch": * [new branch] JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch": * [new branch] JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch": * [new branch] JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch": * [new branch] JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCao/update_to_2.0 -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch": * [new branch] JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-1 -> origin/JackCaoG-patch-1
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-2 -> origin/JackCaoG-patch-2
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-3 -> origin/JackCaoG-patch-3
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-4 -> origin/JackCaoG-patch-4
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-5 -> origin/JackCaoG-patch-5
Step #0 - "git_fetch": * [new branch] JackCaoG-remove-pin -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch": * [new branch] JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/2.0_patch -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch": * [new branch] JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch": * [new branch] JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch": * [new branch] JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch": * [new branch] JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch": * [new branch] JackCaoG/add_2_1_py311_build -> origin/JackCaoG/add_2_1_py311_build
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_print -> origin/JackCaoG/add_test_print
Step #0 - "git_fetch": * [new branch] JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch": * [new branch] JackCaoG/backout_22_mesh_spmd -> origin/JackCaoG/backout_22_mesh_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_allgather_out -> origin/JackCaoG/backport_22_allgather_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_diagonal_scatter -> origin/JackCaoG/backport_22_diagonal_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_doc -> origin/JackCaoG/backport_22_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reduce_scatter -> origin/JackCaoG/backport_22_reduce_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reducescatter_out -> origin/JackCaoG/backport_22_reducescatter_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_scalar -> origin/JackCaoG/backport_22_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_transfer_gpu_scalar -> origin/JackCaoG/backport_22_transfer_gpu_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_dynamo_cpu_move -> origin/JackCaoG/backport_dynamo_cpu_move
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_frame_count -> origin/JackCaoG/backport_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch": * [new branch] JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch": * [new branch] JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_frame_count -> origin/JackCaoG/cherry_pick_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_gettensor -> origin/JackCaoG/clean_up_gettensor
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch": * [new branch] JackCaoG/cleanup_docs -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch": * [new branch] JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_fix -> origin/JackCaoG/debug_fix
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_tool -> origin/JackCaoG/debug_tool
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_2 -> origin/JackCaoG/disable_flaky_test_2
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_3 -> origin/JackCaoG/disable_flaky_test_3
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_pjrt_computation_client_test -> origin/JackCaoG/disable_pjrt_computation_client_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch": * [new branch] JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_doc -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_cumsum_long -> origin/JackCaoG/enable_cumsum_long
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_mp_test_on_tpuci -> origin/JackCaoG/enable_mp_test_on_tpuci
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_reduce_scatter_test_gpu -> origin/JackCaoG/enable_reduce_scatter_test_gpu
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_diagonal_scatter_dim -> origin/JackCaoG/fix_diagonal_scatter_dim
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_input_sharding_test -> origin/JackCaoG/fix_dynamo_input_sharding_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_typo -> origin/JackCaoG/fix_typo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch": * [new branch] JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch": * [new branch] JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch": * [new branch] JackCaoG/graph_hash_debug -> origin/JackCaoG/graph_hash_debug
Step #0 - "git_fetch": * [new branch] JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch": * [new branch] JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch": * [new branch] JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch": * [new branch] JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch": * [new branch] JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pin_2_0 -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch": * [new branch] JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch": * [new branch] JackCaoG/new_alias_api -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/nightly_build_python -> origin/JackCaoG/nightly_build_python
Step #0 - "git_fetch": * [new branch] JackCaoG/parallel_spmd -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_expecttest -> origin/JackCaoG/pin_expecttest
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_5 -> origin/JackCaoG/pt_xla_debug_5
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_dynamo_execute -> origin/JackCaoG/pt_xla_debug_dynamo_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_max_frame -> origin/JackCaoG/pt_xla_debug_max_frame
Step #0 - "git_fetch": * [new branch] JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_ABI_0 -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_pin -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_reshape_lowering -> origin/JackCaoG/remove_reshape_lowering
Step #0 - "git_fetch": * [new branch] JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_partitioner -> origin/JackCaoG/skip_partitioner
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_aliasing -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch": * [new branch] JackCaoG/tracing_timer -> origin/JackCaoG/tracing_timer
Step #0 - "git_fetch": * [new branch] JackCaoG/trouble_shooting_update -> origin/JackCaoG/trouble_shooting_update
Step #0 - "git_fetch": * [new branch] JackCaoG/truncate_python_stack -> origin/JackCaoG/truncate_python_stack
Step #0 - "git_fetch": * [new branch] JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCaoG/updateDocker -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch": * [new branch] JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch": * [new branch] JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc1 -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc_remove_xrt -> origin/JackCaoG/update_doc_remove_xrt
Step #0 - "git_fetch": * [new branch] JackCaoG/update_docker_instruction -> origin/JackCaoG/update_docker_instruction
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_name -> origin/JackCaoG/update_dynamo_backend_name
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_nograd -> origin/JackCaoG/update_dynamo_backend_nograd
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_doc -> origin/JackCaoG/update_dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_number_2.1 -> origin/JackCaoG/update_dynamo_number_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_readme -> origin/JackCaoG/update_dynamo_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_save_graph_path -> origin/JackCaoG/update_dynamo_save_graph_path
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_torch_equal -> origin/JackCaoG/update_dynamo_torch_equal
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_doc -> origin/JackCaoG/update_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheel_link -> origin/JackCaoG/update_gpu_wheel_link
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheellink -> origin/JackCaoG/update_gpu_wheellink
Step #0 - "git_fetch": * [new branch] JackCaoG/update_pjrt_future -> origin/JackCaoG/update_pjrt_future
Step #0 - "git_fetch": * [new branch] JackCaoG/update_readme_2 -> origin/JackCaoG/update_readme_2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_set_default_tensor_type -> origin/JackCaoG/update_set_default_tensor_type
Step #0 - "git_fetch": * [new branch] JackCaoG/update_tf_2023018 -> origin/JackCaoG/update_tf_2023018
Step #0 - "git_fetch": * [new branch] JackCaoG/update_to_2.2 -> origin/JackCaoG/update_to_2.2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_torch_compile_usage -> origin/JackCaoG/update_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/update_trouble_shotting_doc -> origin/JackCaoG/update_trouble_shotting_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_troubleshotting_with_sanitycheck -> origin/JackCaoG/update_troubleshotting_with_sanitycheck
Step #0 - "git_fetch": * [new branch] JackCaoG/update_wheel_docker_format -> origin/JackCaoG/update_wheel_docker_format
Step #0 - "git_fetch": * [new branch] JackCaoG/use_new_alias_api -> origin/JackCaoG/use_new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherry_pick_07_10 -> origin/JackCaoG/xrt_cherry_pick_07_10
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherrypick_0721 -> origin/JackCaoG/xrt_cherrypick_0721
Step #0 - "git_fetch": * [new branch] JackCaog/dynamo_input_sharding -> origin/JackCaog/dynamo_input_sharding
Step #0 - "git_fetch": * [new branch] JackCaog/quick_spmd_fix -> origin/JackCaog/quick_spmd_fix
Step #0 - "git_fetch": * [new branch] JackCaog/spmd_device_default -> origin/JackCaog/spmd_device_default
Step #0 - "git_fetch": * [new branch] Liyang90-patch-1 -> origin/Liyang90-patch-1
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-1 -> origin/ManfeiBai-patch-1
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-10 -> origin/ManfeiBai-patch-10
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-11 -> origin/ManfeiBai-patch-11
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-12 -> origin/ManfeiBai-patch-12
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-13 -> origin/ManfeiBai-patch-13
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-14 -> origin/ManfeiBai-patch-14
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-15 -> origin/ManfeiBai-patch-15
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-16 -> origin/ManfeiBai-patch-16
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-17 -> origin/ManfeiBai-patch-17
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-18 -> origin/ManfeiBai-patch-18
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-19 -> origin/ManfeiBai-patch-19
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-2 -> origin/ManfeiBai-patch-2
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-20 -> origin/ManfeiBai-patch-20
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-21 -> origin/ManfeiBai-patch-21
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-22 -> origin/ManfeiBai-patch-22
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-23 -> origin/ManfeiBai-patch-23
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-24 -> origin/ManfeiBai-patch-24
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-25 -> origin/ManfeiBai-patch-25
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-26 -> origin/ManfeiBai-patch-26
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-27 -> origin/ManfeiBai-patch-27
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-28 -> origin/ManfeiBai-patch-28
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-29 -> origin/ManfeiBai-patch-29
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-3 -> origin/ManfeiBai-patch-3
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-30 -> origin/ManfeiBai-patch-30
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-31 -> origin/ManfeiBai-patch-31
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-32 -> origin/ManfeiBai-patch-32
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-33 -> origin/ManfeiBai-patch-33
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-34 -> origin/ManfeiBai-patch-34
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-35 -> origin/ManfeiBai-patch-35
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-36 -> origin/ManfeiBai-patch-36
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-37 -> origin/ManfeiBai-patch-37
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-38 -> origin/ManfeiBai-patch-38
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-39 -> origin/ManfeiBai-patch-39
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-4 -> origin/ManfeiBai-patch-4
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-40 -> origin/ManfeiBai-patch-40
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-41 -> origin/ManfeiBai-patch-41
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-41-test -> origin/ManfeiBai-patch-41-test
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-42 -> origin/ManfeiBai-patch-42
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-43 -> origin/ManfeiBai-patch-43
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-44 -> origin/ManfeiBai-patch-44
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-45 -> origin/ManfeiBai-patch-45
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-46 -> origin/ManfeiBai-patch-46
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-47 -> origin/ManfeiBai-patch-47
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-48 -> origin/ManfeiBai-patch-48
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-49 -> origin/ManfeiBai-patch-49
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-5 -> origin/ManfeiBai-patch-5
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-50 -> origin/ManfeiBai-patch-50
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-51 -> origin/ManfeiBai-patch-51
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-52 -> origin/ManfeiBai-patch-52
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-53 -> origin/ManfeiBai-patch-53
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-54 -> origin/ManfeiBai-patch-54
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-55 -> origin/ManfeiBai-patch-55
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-56 -> origin/ManfeiBai-patch-56
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-57 -> origin/ManfeiBai-patch-57
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-58 -> origin/ManfeiBai-patch-58
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-59 -> origin/ManfeiBai-patch-59
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-6 -> origin/ManfeiBai-patch-6
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-60 -> origin/ManfeiBai-patch-60
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-61 -> origin/ManfeiBai-patch-61
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-62 -> origin/ManfeiBai-patch-62
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-63 -> origin/ManfeiBai-patch-63
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-64 -> origin/ManfeiBai-patch-64
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-65 -> origin/ManfeiBai-patch-65
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-7 -> origin/ManfeiBai-patch-7
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-8 -> origin/ManfeiBai-patch-8
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-9 -> origin/ManfeiBai-patch-9
Step #0 - "git_fetch": * [new branch] XLASymNodeImplSub -> origin/XLASymNodeImplSub
Step #0 - "git_fetch": * [new branch] XLA_HLO_DEBUG_METADATA -> origin/XLA_HLO_DEBUG_METADATA
Step #0 - "git_fetch": * [new branch] a_tensor_cleanup -> origin/a_tensor_cleanup
Step #0 - "git_fetch": * [new branch] accu_rat
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/8130a5d0-05d5-461d-8322-6daad591031c?project=164006649440.]
...
e_tests": Epoch 0 step 4220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 4990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 5990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 6990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7190 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7200 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7210 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7220 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7230 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7240 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7250 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7260 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7270 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7280 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7290 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7300 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7310 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7320 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7330 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7340 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7350 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7360 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7370 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7380 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7390 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7400 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7410 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7420 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7430 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7440 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7450 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7460 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7470 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7480 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7490 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7500 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7510 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7520 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7530 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7540 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7550 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7560 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7570 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7580 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7590 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7600 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7610 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7620 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7630 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7640 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7650 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7660 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7670 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7680 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7690 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7700 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7710 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7720 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7730 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7740 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7750 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7760 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7770 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7780 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7790 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7800 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7810 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7820 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7830 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7840 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7850 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7860 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7870 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7880 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7890 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7900 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7910 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7920 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7930 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7940 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7950 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7960 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7970 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7980 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 7990 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8000 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8010 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8020 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8030 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8040 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8050 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8060 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8070 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8080 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8090 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8100 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8110 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8120 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8130 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8140 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8150 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8160 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8170 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8180 loss 0.0
Step #4 - "run_e2e_tests": Epoch 0 step 8190 loss 0.0
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/spmd/test_xla_spmd_python_api_interaction.py
Step #4 - "run_e2e_tests": WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148304 49093 pjrt_api.cc:100] GetPjrtApi was found for tpu at /usr/local/lib/python3.10/site-packages/torch_xla/lib/libtpu.so
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148385 49093 pjrt_api.cc:79] PJRT_Api is set for device type tpu
Step #4 - "run_e2e_tests": I0000 00:00:1707269855.148399 49093 pjrt_api.cc:146] The PJRT plugin has PJRT API version 0.40. The framework PJRT API version is 0.40.
Step #4 - "run_e2e_tests": ./usr/local/lib/python3.10/site-packages/torch_xla/runtime.py:234: UserWarning: XLA_USE_SPMD is being deprecated. Use torch_xla.runtime.use_spmd() without setting XLA_USE_SPMD env-var.
Step #4 - "run_e2e_tests": warnings.warn("XLA_USE_SPMD is being deprecated. "
Step #4 - "run_e2e_tests": .............../usr/local/lib/python3.10/site-packages/torch_xla/core/xla_model.py:104: UserWarning: `devkind` argument is deprecated and will be removed in a future release.
Step #4 - "run_e2e_tests": warnings.warn("`devkind` argument is deprecated and will be removed in a "
Step #4 - "run_e2e_tests": .......
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 23 tests in 13.798s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": OK
Step #4 - "run_e2e_tests": + XLA_EXPERIMENTAL=nonzero:masked_select
Step #4 - "run_e2e_tests": + python3 /src/pytorch/xla/test/ds/test_dynamic_shape_models.py -v
Step #4 - "run_e2e_tests": WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.445957 49845 pjrt_api.cc:100] GetPjrtApi was found for tpu at /usr/local/lib/python3.10/site-packages/torch_xla/lib/libtpu.so
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.446034 49845 pjrt_api.cc:79] PJRT_Api is set for device type tpu
Step #4 - "run_e2e_tests": I0000 00:00:1707269874.446048 49845 pjrt_api.cc:146] The PJRT plugin has PJRT API version 0.40. The framework PJRT API version is 0.40.
Step #4 - "run_e2e_tests": /usr/local/lib/python3.10/site-packages/torch_xla/core/xla_model.py:104: UserWarning: `devkind` argument is deprecated and will be removed in a future release.
Step #4 - "run_e2e_tests": warnings.warn("`devkind` argument is deprecated and will be removed in a "
Step #4 - "run_e2e_tests": test_backward_pass_with_dynamic_input (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests": test_forward_pass_dynamic_input_compile_once (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests": test_forward_pass_dynamic_input_correctness (__main__.TestDynamicShapeModels) ... FAIL
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_backward_pass_with_dynamic_input (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests": File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 114, in test_backward_pass_with_dynamic_input
Step #4 - "run_e2e_tests": loss = criterion(y_pred.squeeze(), y_training)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests": return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests": return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests": return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests": if target.size() != input.size():
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests": return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests": raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_forward_pass_dynamic_input_compile_once (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests": File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 91, in test_forward_pass_dynamic_input_compile_once
Step #4 - "run_e2e_tests": criterion(y_pred.squeeze(), y_test)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests": return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests": return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests": return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests": if target.size() != input.size():
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests": return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests": raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ======================================================================
Step #4 - "run_e2e_tests": FAIL: test_forward_pass_dynamic_input_correctness (__main__.TestDynamicShapeModels)
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Traceback (most recent call last):
Step #4 - "run_e2e_tests": File "/src/pytorch/xla/test/ds/test_dynamic_shape_models.py", line 68, in test_forward_pass_dynamic_input_correctness
Step #4 - "run_e2e_tests": before_train = criterion(y_pred.squeeze(), y_test)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
Step #4 - "run_e2e_tests": return self._call_impl(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
Step #4 - "run_e2e_tests": return forward_call(*args, **kwargs)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 618, in forward
Step #4 - "run_e2e_tests": return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/nn/functional.py", line 3144, in binary_cross_entropy
Step #4 - "run_e2e_tests": if target.size() != input.size():
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1226, in binary_magic_impl
Step #4 - "run_e2e_tests": return (method_to_operator(method))(get_constant(self), other)
Step #4 - "run_e2e_tests": File "/usr/local/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 1186, in get_constant
Step #4 - "run_e2e_tests": raise AssertionError("expect to be called with constant SymBools")
Step #4 - "run_e2e_tests": AssertionError: expect to be called with constant SymBools
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": ----------------------------------------------------------------------
Step #4 - "run_e2e_tests": Ran 3 tests in 0.023s
Step #4 - "run_e2e_tests":
Step #4 - "run_e2e_tests": FAILED (failures=3)
Step #4 - "run_e2e_tests": ++ kubectl get pod/xla-test-job-kvpcs -o 'jsonpath={.status.containerStatuses[?(@.name=="xla-test")].state.terminated.exitCode}'
Step #4 - "run_e2e_tests": + exit 1
Finished Step #4 - "run_e2e_tests"
ERROR
ERROR: build step 4 "google/cloud-sdk" failed: step exited with non-zero status: 1
Loading