Add torch_pin to 2.5 release branch #7979
Closed
Google Cloud Build / r2-5-0-rc1-3-9-cuda-12-4 (tpu-pytorch-releases)
failed
Sep 9, 2024 in 2m 58s
Summary
Build Information
| Trigger | r2-5-0-rc1-3-9-cuda-12-4 |
| Build | c2d2a107-ea03-4b01-8b18-99b97899d555 |
| Start | 2024-09-09T14:42:59-07:00 |
| Duration | 1m49.984s |
| Status | FAILURE |
Steps
| Step | Status | Duration |
|---|---|---|
| git_fetch | SUCCESS | 10.44s |
| git_checkout | SUCCESS | 652ms |
| build_xla_docker_image | FAILURE | 1m35.409s |
| push_xla_docker_image | QUEUED | 0s |
| copy_wheels_to_volume | QUEUED | 0s |
| upload_wheels_to_storage_bucket | QUEUED | 0s |
Details
starting build "c2d2a107-ea03-4b01-8b18-99b97899d555"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
* branch 87304aa05a478d32a27b20ee471576230954f30c -> FETCH_HEAD
HEAD is now at 87304aa Add torch_pin to 2.5 release branch
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch": * [new branch] 01172023_pin_update -> origin/01172023_pin_update
Step #0 - "git_fetch": * [new branch] 2-0-python-3-10 -> origin/2-0-python-3-10
Step #0 - "git_fetch": * [new branch] 2.4_triton_doc -> origin/2.4_triton_doc
Step #0 - "git_fetch": * [new branch] 4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch": * [new branch] 5d-unit-test -> origin/5d-unit-test
Step #0 - "git_fetch": * [new branch] 7511-op-info-test-for-maskedsoftmax-maskedvar -> origin/7511-op-info-test-for-maskedsoftmax-maskedvar
Step #0 - "git_fetch": * [new branch] AddVimToReleasePackage -> origin/AddVimToReleasePackage
Step #0 - "git_fetch": * [new branch] FixTransferToServer -> origin/FixTransferToServer
Step #0 - "git_fetch": * [new branch] JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch": * [new branch] JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch": * [new branch] JackCao/Dynamo_sync_input_counter -> origin/JackCao/Dynamo_sync_input_counter
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch": * [new branch] JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch": * [new branch] JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCao/update_to_2.0 -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch": * [new branch] JackCao/update_to_223 -> origin/JackCao/update_to_223
Step #0 - "git_fetch": * [new branch] JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-1 -> origin/JackCaoG-patch-1
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-2 -> origin/JackCaoG-patch-2
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-3 -> origin/JackCaoG-patch-3
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-4 -> origin/JackCaoG-patch-4
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-5 -> origin/JackCaoG-patch-5
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-6 -> origin/JackCaoG-patch-6
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-7 -> origin/JackCaoG-patch-7
Step #0 - "git_fetch": * [new branch] JackCaoG-remove-pin -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch": * [new branch] JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/2.0_patch -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/24_backport_7256 -> origin/JackCaoG/24_backport_7256
Step #0 - "git_fetch": * [new branch] JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch": * [new branch] JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch": * [new branch] JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch": * [new branch] JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch": * [new branch] JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch": * [new branch] JackCaoG/XLAShardedTensor_torch_dispatch -> origin/JackCaoG/XLAShardedTensor_torch_dispatch
Step #0 - "git_fetch": * [new branch] JackCaoG/add_2_1_py311_build -> origin/JackCaoG/add_2_1_py311_build
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_print -> origin/JackCaoG/add_test_print
Step #0 - "git_fetch": * [new branch] JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch": * [new branch] JackCaoG/auto_wrap_spmd -> origin/JackCaoG/auto_wrap_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/backout_22_mesh_spmd -> origin/JackCaoG/backout_22_mesh_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/backport2_4_eager_compile -> origin/JackCaoG/backport2_4_eager_compile
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_allgather_out -> origin/JackCaoG/backport_22_allgather_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_diagonal_scatter -> origin/JackCaoG/backport_22_diagonal_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_doc -> origin/JackCaoG/backport_22_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reduce_scatter -> origin/JackCaoG/backport_22_reduce_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reducescatter_out -> origin/JackCaoG/backport_22_reducescatter_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_scalar -> origin/JackCaoG/backport_22_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_transfer_gpu_scalar -> origin/JackCaoG/backport_22_transfer_gpu_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_24_7327 -> origin/JackCaoG/backport_24_7327
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_24_7341 -> origin/JackCaoG/backport_24_7341
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_24_eager_1 -> origin/JackCaoG/backport_24_eager_1
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_aws_zero1 -> origin/JackCaoG/backport_aws_zero1
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_dynamo_cpu_move -> origin/JackCaoG/backport_dynamo_cpu_move
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_dynamo_doc_eval -> origin/JackCaoG/backport_dynamo_doc_eval
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_flash_attention_dynamo -> origin/JackCaoG/backport_flash_attention_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_frame_count -> origin/JackCaoG/backport_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_oerserve_rng -> origin/JackCaoG/backport_oerserve_rng
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_revert_mul -> origin/JackCaoG/backport_revert_mul
Step #0 - "git_fetch": * [new branch] JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch": * [new branch] JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch": * [new branch] JackCaoG/buffer_donor_2 -> origin/JackCaoG/buffer_donor_2
Step #0 - "git_fetch": * [new branch] JackCaoG/buffer_donor_custom_op -> origin/JackCaoG/buffer_donor_custom_op
Step #0 - "git_fetch": * [new branch] JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/check_upstream_failure -> origin/JackCaoG/check_upstream_failure
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_frame_count -> origin/JackCaoG/cherry_pick_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/clang-format-11 -> origin/JackCaoG/clang-format-11
Step #0 - "git_fetch": * [new branch] JackCaoG/clang-format-11-2 -> origin/JackCaoG/clang-format-11-2
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_gettensor -> origin/JackCaoG/clean_up_gettensor
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch": * [new branch] JackCaoG/cleanup_docs -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch": * [new branch] JackCaoG/compile_ds_detection -> origin/JackCaoG/compile_ds_detection
Step #0 - "git_fetch": * [new branch] JackCaoG/compile_full_gaph -> origin/JackCaoG/compile_full_gaph
Step #0 - "git_fetch": * [new branch] JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_fix -> origin/JackCaoG/debug_fix
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_tool -> origin/JackCaoG/debug_tool
Step #0 - "git_fetch": * [new branch] JackCaoG/decoder_only_example -> origin/JackCaoG/decoder_only_example
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_ds_dynamo_test -> origin/JackCaoG/disable_ds_dynamo_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_2 -> origin/JackCaoG/disable_flaky_test_2
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_3 -> origin/JackCaoG/disable_flaky_test_3
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_pjrt_computation_client_test -> origin/JackCaoG/disable_pjrt_computation_client_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_some_test_v2 -> origin/JackCaoG/disable_some_test_v2
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch": * [new branch] JackCaoG/doc_img_try_2 -> origin/JackCaoG/doc_img_try_2
Step #0 - "git_fetch": * [new branch] JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_aliasing_2 -> origin/JackCaoG/dynamo_aliasing_2
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_do_not_markstep_to_sync_input -> origin/JackCaoG/dynamo_do_not_markstep_to_sync_input
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_doc -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_ds_symint_return -> origin/JackCaoG/dynamo_ds_symint_return
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_dynamic_cache_optimization -> origin/JackCaoG/dynamo_dynamic_cache_optimization
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_inference_test -> origin/JackCaoG/dynamo_inference_test
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_sync_input -> origin/JackCaoG/dynamo_sync_input
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch": * [new branch] JackCaoG/eager_docs -> origin/JackCaoG/eager_docs
Step #0 - "git_fetch": * [new branch] JackCaoG/eager_faster -> origin/JackCaoG/eager_faster
Step #0 - "git_fetch": * [new branch] JackCaoG/eager_fsdpv2 -> origin/JackCaoG/eager_fsdpv2
Step #0 - "git_fetch": * [new branch] JackCaoG/eager_random_seed -> origin/JackCaoG/eager_random_seed
Step #0 - "git_fetch": * [new branch] JackCaoG/eager_spmd -> origin/JackCaoG/eager_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_cumsum_long -> origin/JackCaoG/enable_cumsum_long
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_mp_test_on_tpuci -> origin/JackCaoG/enable_mp_test_on_tpuci
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_reduce_scatter_test_gpu -> origin/JackCaoG/enable_reduce_scatter_test_gpu
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_traceable_collective_test -> origin/JackCaoG/enable_traceable_collective_test
Step #0 - "git_fetch": * [new branch] JackCaoG/error_with_tensor_id -> origin/JackCaoG/error_with_tensor_id
Step #0 - "git_fetch": * [new branch] JackCaoG/example -> origin/JackCaoG/example
Step #0 - "git_fetch": * [new branch] JackCaoG/example_pr_pin -> origin/JackCaoG/example_pr_pin
Step #0 - "git_fetch": * [new branch] JackCaoG/example_spmd_ddp -> origin/JackCaoG/example_spmd_ddp
Step #0 - "git_fetch": * [new branch] JackCaoG/example_torch_pin_pr -> origin/JackCaoG/example_torch_pin_pr
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_diagonal_scatter_dim -> origin/JackCaoG/fix_diagonal_scatter_dim
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash_profiler -> origin/JackCaoG/fix_dynamo_crash_profiler
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_input_sharding_test -> origin/JackCaoG/fix_dynamo_input_sharding_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_full_dtype -> origin/JackCaoG/fix_full_dtype
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_log_spam -> origin/JackCaoG/fix_log_spam
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_log_spam_2 -> origin/JackCaoG/fix_log_spam_2
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_multi_inplace_update_aliasing -> origin/JackCaoG/fix_multi_inplace_update_aliasing
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_spmd_doc_string -> origin/JackCaoG/fix_spmd_doc_string
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_tuple_type_annotation -> origin/JackCaoG/fix_tuple_type_annotation
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_typo -> origin/JackCaoG/fix_typo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_warning_message -> origin/JackCaoG/fix_warning_message
Step #0 - "git_fetch": * [new branch] JackCaoG/flash_attention_custom_op -> origin/JackCaoG/flash_attention_custom_op
Step #0 - "git_fetch": * [new branch] JackCaoG/flash_attention_test_precision -> origin/JackCaoG/flash_attention_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch": * [new branch] JackCaoG/full_graph_comment -> origin/JackCaoG/full_graph_comment
Step #0 - "git_fetch": * [new branch] JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch": * [new branch] JackCaoG/graph_hash_debug -> origin/JackCaoG/graph_hash_debug
Step #0 - "git_fetch": * [new branch] JackCaoG/graph_hbm_usage -> origin/JackCaoG/graph_hbm_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch": * [new branch] JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch": * [new branch] JackCaoG/in_place_update_egaer -> origin/JackCaoG/in_place_update_egaer
Step #0 - "git_fetch": * [new branch] JackCaoG/increase_max_in_flight_operation -> origin/JackCaoG/increase_max_in_flight_operation
Step #0 - "git_fetch": * [new branch] JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch": * [new branch] JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch": * [new branch] JackCaoG/make_libtpu_install_more_explict -> origin/JackCaoG/make_libtpu_install_more_explict
Step #0 - "git_fetch": * [new branch] JackCaoG/make_load_fail_more_explict -> origin/JackCaoG/make_load_fail_more_explict
Step #0 - "git_fetch": * [new branch] JackCaoG/memory_info_default_arg -> origin/JackCaoG/memory_info_default_arg
Step #0 - "git_fetch": * [new branch] JackCaoG/merge_step_and_compile -> origin/JackCaoG/merge_step_and_compile
Step #0 - "git_fetch": * [new branch] JackCaoG/move_docs_for_image -> origin/JackCaoG/move_docs_for_image
Step #0 - "git_fetch": * [new branch] JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pin_2_0 -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch": * [new branch] JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch": * [new branch] JackCaoG/move_test_grad_checkpoint_to_tpu -> origin/JackCaoG/move_test_grad_checkpoint_to_tpu
Step #0 - "git_fetch": * [new branch] JackCaoG/move_to_25 -> origin/JackCaoG/move_to_25
Step #0 - "git_fetch": * [new branch] JackCaoG/multi_process_eager -> origin/JackCaoG/multi_process_eager
Step #0 - "git_fetch": * [new branch] JackCaoG/new_alias_api -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/nightly_build_python -> origin/JackCaoG/nightly_build_python
Step #0 - "git_fetch": * [new branch] JackCaoG/no_trim_graph -> origin/JackCaoG/no_trim_graph
Step #0 - "git_fetch": * [new branch] JackCaoG/para_wrap_spmd_follow_up -> origin/JackCaoG/para_wrap_spmd_follow_up
Step #0 - "git_fetch": * [new branch] JackCaoG/parallel_spmd -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_expecttest -> origin/JackCaoG/pin_expecttest
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_5 -> origin/JackCaoG/pt_xla_debug_5
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_dynamo_execute -> origin/JackCaoG/pt_xla_debug_dynamo_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_level -> origin/JackCaoG/pt_xla_debug_level
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_max_frame -> origin/JackCaoG/pt_xla_debug_max_frame
Step #0 - "git_fetch": * [new branch] JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch": * [new branch] JackCaoG/r24_eager_doc -> origin/JackCaoG/r24_eager_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_ABI_0 -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch": * [new branch] JackCaoG/reduce_decoder_size -> origin/JackCaoG/reduce_decoder_size
Step #0 - "git_fetch": * [new branch] JackCaoG/register_trilinear_decomp -> origin/JackCaoG/register_trilinear_decomp
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_experimental_spmd -> origin/JackCaoG/remove_experimental_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_pin -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_reshape_again -> origin/JackCaoG/remove_reshape_again
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_reshape_lowering -> origin/JackCaoG/remove_reshape_lowering
Step #0 - "git_fetch": * [new branch] JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_mul -> origin/JackCaoG/revert_mul
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/run_examples_tpu_ci -> origin/JackCaoG/run_examples_tpu_ci
Step #0 - "git_fetch": * [new branch] JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_partitioner -> origin/JackCaoG/skip_partitioner
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/spamd_doc -> origin/JackCaoG/spamd_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_aliasing -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_gpu_doc -> origin/JackCaoG/spmd_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch": * [new branch] JackCaoG/support_mark_dynamic -> origin/JackCaoG/support_mark_dynamic
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/test_upstream -> origin/JackCaoG/test_upstream
Step #0 - "git_fetch": * [new branch] JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch": * [new branch] JackCaoG/torch_xla_compile -> origin/JackCaoG/torch_xla_compile
Step #0 - "git_fetch": * [new branch] JackCaoG/torch_xla_compile2 -> origin/JackCaoG/torch_xla_compile2
Step #0 - "git_fetch": * [new branch] JackCaoG/torch_xla_compile_name -> origin/JackCaoG/torch_xla_compile_name
Step #0 - "git_fetch": * [new branch] JackCaoG/tracing_timer -> origin/JackCaoG/tracing_timer
Step #0 - "git_fetch": * [new branch] JackCaoG/trouble_shooting_update -> origin/JackCaoG/trouble_shooting_update
Step #0 - "git_fetch": * [new branch] JackCaoG/truncate_python_stack -> origin/JackCaoG/truncate_python_stack
Step #0 - "git_fetch": * [new branch] JackCaoG/unify_UseEagerDebugMode -> origin/JackCaoG/unify_UseEagerDebugMode
Step #0 - "git_fetch": * [new branch] JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCaoG/updateDocker -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch": * [new branch] JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch": * [new branch] JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc1 -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc2 -> origin/JackCaoG/update_doc2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc_remove_xrt -> origin/JackCa
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/c2d2a107-ea03-4b01-8b18-99b97899d555?project=1001674285173.]
...
'/usr/share/keyrings/llvm.pgp'}) => changed=true
Step #2 - "build_xla_docker_image": ansible_loop_var: item
Step #2 - "build_xla_docker_image": checksum_dest: null
Step #2 - "build_xla_docker_image": checksum_src: 4fd4bfd415cf46818f6da618bdb6ded06d4dac61
Step #2 - "build_xla_docker_image": dest: /usr/share/keyrings/llvm.pgp
Step #2 - "build_xla_docker_image": elapsed: 0
Step #2 - "build_xla_docker_image": gid: 0
Step #2 - "build_xla_docker_image": group: root
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": attributes: null
Step #2 - "build_xla_docker_image": backup: false
Step #2 - "build_xla_docker_image": checksum: ''
Step #2 - "build_xla_docker_image": ciphers: null
Step #2 - "build_xla_docker_image": client_cert: null
Step #2 - "build_xla_docker_image": client_key: null
Step #2 - "build_xla_docker_image": decompress: true
Step #2 - "build_xla_docker_image": dest: /usr/share/keyrings/llvm.pgp
Step #2 - "build_xla_docker_image": force: false
Step #2 - "build_xla_docker_image": force_basic_auth: false
Step #2 - "build_xla_docker_image": group: null
Step #2 - "build_xla_docker_image": headers: null
Step #2 - "build_xla_docker_image": http_agent: ansible-httpget
Step #2 - "build_xla_docker_image": mode: u=rw,g=r,o=r
Step #2 - "build_xla_docker_image": owner: null
Step #2 - "build_xla_docker_image": selevel: null
Step #2 - "build_xla_docker_image": serole: null
Step #2 - "build_xla_docker_image": setype: null
Step #2 - "build_xla_docker_image": seuser: null
Step #2 - "build_xla_docker_image": timeout: 10
Step #2 - "build_xla_docker_image": tmp_dest: null
Step #2 - "build_xla_docker_image": unredirected_headers: []
Step #2 - "build_xla_docker_image": unsafe_writes: false
Step #2 - "build_xla_docker_image": url: https://apt.llvm.org/llvm-snapshot.gpg.key
Step #2 - "build_xla_docker_image": url_password: null
Step #2 - "build_xla_docker_image": url_username: null
Step #2 - "build_xla_docker_image": use_gssapi: false
Step #2 - "build_xla_docker_image": use_netrc: true
Step #2 - "build_xla_docker_image": use_proxy: true
Step #2 - "build_xla_docker_image": validate_certs: true
Step #2 - "build_xla_docker_image": item:
Step #2 - "build_xla_docker_image": keyring: /usr/share/keyrings/llvm.pgp
Step #2 - "build_xla_docker_image": url: https://apt.llvm.org/llvm-snapshot.gpg.key
Step #2 - "build_xla_docker_image": md5sum: 2d829959186730c5043c47d412c6c358
Step #2 - "build_xla_docker_image": mode: '0644'
Step #2 - "build_xla_docker_image": msg: OK (3145 bytes)
Step #2 - "build_xla_docker_image": owner: root
Step #2 - "build_xla_docker_image": size: 3145
Step #2 - "build_xla_docker_image": src: /root/.ansible/tmp/ansible-tmp-1725918267.5872774-146-119639125335619/tmpp57z6274
Step #2 - "build_xla_docker_image": state: file
Step #2 - "build_xla_docker_image": status_code: 200
Step #2 - "build_xla_docker_image": uid: 0
Step #2 - "build_xla_docker_image": url: https://apt.llvm.org/llvm-snapshot.gpg.key
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208 `" && echo ansible-tmp-1725918268.3117874-146-14182233058208="` echo /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/get_url.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmpwy4e0sg_ TO /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/AnsiballZ_get_url.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/ /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/AnsiballZ_get_url.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/AnsiballZ_get_url.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": changed: [localhost] => (item={'url': 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub', 'keyring': '/usr/share/keyrings/cuda.pgp'}) => changed=true
Step #2 - "build_xla_docker_image": ansible_loop_var: item
Step #2 - "build_xla_docker_image": checksum_dest: null
Step #2 - "build_xla_docker_image": checksum_src: 34bb9f7e66744d7b2944d0565db6687560d5d6e3
Step #2 - "build_xla_docker_image": dest: /usr/share/keyrings/cuda.pgp
Step #2 - "build_xla_docker_image": elapsed: 0
Step #2 - "build_xla_docker_image": gid: 0
Step #2 - "build_xla_docker_image": group: root
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": attributes: null
Step #2 - "build_xla_docker_image": backup: false
Step #2 - "build_xla_docker_image": checksum: ''
Step #2 - "build_xla_docker_image": ciphers: null
Step #2 - "build_xla_docker_image": client_cert: null
Step #2 - "build_xla_docker_image": client_key: null
Step #2 - "build_xla_docker_image": decompress: true
Step #2 - "build_xla_docker_image": dest: /usr/share/keyrings/cuda.pgp
Step #2 - "build_xla_docker_image": force: false
Step #2 - "build_xla_docker_image": force_basic_auth: false
Step #2 - "build_xla_docker_image": group: null
Step #2 - "build_xla_docker_image": headers: null
Step #2 - "build_xla_docker_image": http_agent: ansible-httpget
Step #2 - "build_xla_docker_image": mode: u=rw,g=r,o=r
Step #2 - "build_xla_docker_image": owner: null
Step #2 - "build_xla_docker_image": selevel: null
Step #2 - "build_xla_docker_image": serole: null
Step #2 - "build_xla_docker_image": setype: null
Step #2 - "build_xla_docker_image": seuser: null
Step #2 - "build_xla_docker_image": timeout: 10
Step #2 - "build_xla_docker_image": tmp_dest: null
Step #2 - "build_xla_docker_image": unredirected_headers: []
Step #2 - "build_xla_docker_image": unsafe_writes: false
Step #2 - "build_xla_docker_image": url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
Step #2 - "build_xla_docker_image": url_password: null
Step #2 - "build_xla_docker_image": url_username: null
Step #2 - "build_xla_docker_image": use_gssapi: false
Step #2 - "build_xla_docker_image": use_netrc: true
Step #2 - "build_xla_docker_image": use_proxy: true
Step #2 - "build_xla_docker_image": validate_certs: true
Step #2 - "build_xla_docker_image": item:
Step #2 - "build_xla_docker_image": keyring: /usr/share/keyrings/cuda.pgp
Step #2 - "build_xla_docker_image": url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
Step #2 - "build_xla_docker_image": md5sum: 527c4db3feb71dd7a69a60fba46c99db
Step #2 - "build_xla_docker_image": mode: '0644'
Step #2 - "build_xla_docker_image": msg: OK (1650 bytes)
Step #2 - "build_xla_docker_image": owner: root
Step #2 - "build_xla_docker_image": size: 1650
Step #2 - "build_xla_docker_image": src: /root/.ansible/tmp/ansible-tmp-1725918268.3117874-146-14182233058208/tmpdxehm663
Step #2 - "build_xla_docker_image": state: file
Step #2 - "build_xla_docker_image": status_code: 200
Step #2 - "build_xla_docker_image": uid: 0
Step #2 - "build_xla_docker_image": url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub
Step #2 - "build_xla_docker_image":
Step #2 - "build_xla_docker_image": TASK [install_deps : Add apt repositories into sources list] *******************
Step #2 - "build_xla_docker_image": task path: /ansible/roles/install_deps/tasks/main.yaml:11
Step #2 - "build_xla_docker_image": Monday 09 September 2024 21:44:28 +0000 (0:00:01.296) 0:00:04.501 ******
Step #2 - "build_xla_docker_image": <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096 `" && echo ansible-tmp-1725918268.8846881-196-32377105533096="` echo /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmpl9orfbu_ TO /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096/AnsiballZ_apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096/ /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918268.8846881-196-32377105533096/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": changed: [localhost] => (item=deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main) => changed=true
Step #2 - "build_xla_docker_image": ansible_loop_var: item
Step #2 - "build_xla_docker_image": diff: []
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": codename: null
Step #2 - "build_xla_docker_image": filename: null
Step #2 - "build_xla_docker_image": install_python_apt: true
Step #2 - "build_xla_docker_image": mode: null
Step #2 - "build_xla_docker_image": repo: deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": update_cache: true
Step #2 - "build_xla_docker_image": update_cache_retries: 5
Step #2 - "build_xla_docker_image": update_cache_retry_max_delay: 12
Step #2 - "build_xla_docker_image": validate_certs: true
Step #2 - "build_xla_docker_image": item: deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": repo: deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": sources_added:
Step #2 - "build_xla_docker_image": - /etc/apt/sources.list.d/apt_llvm_org_bullseye.list
Step #2 - "build_xla_docker_image": sources_removed: []
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399 `" && echo ansible-tmp-1725918276.8040047-196-34956751757399="` echo /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmpflpdvoow TO /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399/AnsiballZ_apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399/ /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918276.8040047-196-34956751757399/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": changed: [localhost] => (item=deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main) => changed=true
Step #2 - "build_xla_docker_image": ansible_loop_var: item
Step #2 - "build_xla_docker_image": diff: []
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": codename: null
Step #2 - "build_xla_docker_image": filename: null
Step #2 - "build_xla_docker_image": install_python_apt: true
Step #2 - "build_xla_docker_image": mode: null
Step #2 - "build_xla_docker_image": repo: deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": update_cache: true
Step #2 - "build_xla_docker_image": update_cache_retries: 5
Step #2 - "build_xla_docker_image": update_cache_retry_max_delay: 12
Step #2 - "build_xla_docker_image": validate_certs: true
Step #2 - "build_xla_docker_image": item: deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": repo: deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main
Step #2 - "build_xla_docker_image": sources_added: []
Step #2 - "build_xla_docker_image": sources_removed: []
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703 `" && echo ansible-tmp-1725918279.1183023-196-199521162203703="` echo /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmp7ssxckyr TO /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703/AnsiballZ_apt_repository.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703/ /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703/AnsiballZ_apt_repository.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918279.1183023-196-199521162203703/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": changed: [localhost] => (item=deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /) => changed=true
Step #2 - "build_xla_docker_image": ansible_loop_var: item
Step #2 - "build_xla_docker_image": diff: []
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": codename: null
Step #2 - "build_xla_docker_image": filename: null
Step #2 - "build_xla_docker_image": install_python_apt: true
Step #2 - "build_xla_docker_image": mode: null
Step #2 - "build_xla_docker_image": repo: deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": update_cache: true
Step #2 - "build_xla_docker_image": update_cache_retries: 5
Step #2 - "build_xla_docker_image": update_cache_retry_max_delay: 12
Step #2 - "build_xla_docker_image": validate_certs: true
Step #2 - "build_xla_docker_image": item: deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /
Step #2 - "build_xla_docker_image": repo: deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /
Step #2 - "build_xla_docker_image": sources_added:
Step #2 - "build_xla_docker_image": - /etc/apt/sources.list.d/developer_download_nvidia_com_compute_cuda_repos_debian11_x86_64.list
Step #2 - "build_xla_docker_image": sources_removed: []
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image":
Step #2 - "build_xla_docker_image": TASK [install_deps : Update apt cache] *****************************************
Step #2 - "build_xla_docker_image": task path: /ansible/roles/install_deps/tasks/main.yaml:18
Step #2 - "build_xla_docker_image": Monday 09 September 2024 21:44:41 +0000 (0:00:12.779) 0:00:17.280 ******
Step #2 - "build_xla_docker_image": <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385 `" && echo ansible-tmp-1725918281.6291528-1533-170589264463385="` echo /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/apt.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmp4sy7o1sw TO /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385/AnsiballZ_apt.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385/ /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385/AnsiballZ_apt.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385/AnsiballZ_apt.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918281.6291528-1533-170589264463385/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": ok: [localhost] => changed=false
Step #2 - "build_xla_docker_image": cache_update_time: 1725918281
Step #2 - "build_xla_docker_image": cache_updated: false
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": allow_change_held_packages: false
Step #2 - "build_xla_docker_image": allow_downgrade: false
Step #2 - "build_xla_docker_image": allow_unauthenticated: false
Step #2 - "build_xla_docker_image": autoclean: false
Step #2 - "build_xla_docker_image": autoremove: false
Step #2 - "build_xla_docker_image": cache_valid_time: 0
Step #2 - "build_xla_docker_image": clean: false
Step #2 - "build_xla_docker_image": deb: null
Step #2 - "build_xla_docker_image": default_release: null
Step #2 - "build_xla_docker_image": dpkg_options: force-confdef,force-confold
Step #2 - "build_xla_docker_image": fail_on_autoremove: false
Step #2 - "build_xla_docker_image": force: false
Step #2 - "build_xla_docker_image": force_apt_get: false
Step #2 - "build_xla_docker_image": install_recommends: null
Step #2 - "build_xla_docker_image": lock_timeout: 60
Step #2 - "build_xla_docker_image": only_upgrade: false
Step #2 - "build_xla_docker_image": package: null
Step #2 - "build_xla_docker_image": policy_rc_d: null
Step #2 - "build_xla_docker_image": purge: false
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": update_cache: true
Step #2 - "build_xla_docker_image": update_cache_retries: 5
Step #2 - "build_xla_docker_image": update_cache_retry_max_delay: 12
Step #2 - "build_xla_docker_image": upgrade: null
Step #2 - "build_xla_docker_image":
Step #2 - "build_xla_docker_image": TASK [install_deps : Install apt packages] *************************************
Step #2 - "build_xla_docker_image": task path: /ansible/roles/install_deps/tasks/main.yaml:23
Step #2 - "build_xla_docker_image": Monday 09 September 2024 21:44:45 +0000 (0:00:03.567) 0:00:20.847 ******
Step #2 - "build_xla_docker_image": <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979 `" && echo ansible-tmp-1725918285.236236-1894-262876937572979="` echo /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979 `" ) && sleep 0'
Step #2 - "build_xla_docker_image": Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/apt.py
Step #2 - "build_xla_docker_image": <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7qyky0w5h/tmpuac4ijim TO /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979/AnsiballZ_apt.py
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979/ /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979/AnsiballZ_apt.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979/AnsiballZ_apt.py && sleep 0'
Step #2 - "build_xla_docker_image": <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1725918285.236236-1894-262876937572979/ > /dev/null 2>&1 && sleep 0'
Step #2 - "build_xla_docker_image": The full traceback is:
Step #2 - "build_xla_docker_image": File "/tmp/ansible_ansible.builtin.apt_payload_rhxry8px/ansible_ansible.builtin.apt_payload.zip/ansible/modules/apt.py", line 526, in package_status
Step #2 - "build_xla_docker_image": File "/usr/lib/python3/dist-packages/apt/cache.py", line 283, in __getitem__
Step #2 - "build_xla_docker_image": raise KeyError('The cache has no package named %r' % key)
Step #2 - "build_xla_docker_image": fatal: [localhost]: FAILED! => changed=false
Step #2 - "build_xla_docker_image": invocation:
Step #2 - "build_xla_docker_image": module_args:
Step #2 - "build_xla_docker_image": allow_change_held_packages: false
Step #2 - "build_xla_docker_image": allow_downgrade: false
Step #2 - "build_xla_docker_image": allow_unauthenticated: false
Step #2 - "build_xla_docker_image": autoclean: false
Step #2 - "build_xla_docker_image": autoremove: false
Step #2 - "build_xla_docker_image": cache_valid_time: 0
Step #2 - "build_xla_docker_image": clean: false
Step #2 - "build_xla_docker_image": deb: null
Step #2 - "build_xla_docker_image": default_release: null
Step #2 - "build_xla_docker_image": dpkg_options: force-confdef,force-confold
Step #2 - "build_xla_docker_image": fail_on_autoremove: false
Step #2 - "build_xla_docker_image": force: false
Step #2 - "build_xla_docker_image": force_apt_get: false
Step #2 - "build_xla_docker_image": install_recommends: null
Step #2 - "build_xla_docker_image": lock_timeout: 60
Step #2 - "build_xla_docker_image": name:
Step #2 - "build_xla_docker_image": - ccache
Step #2 - "build_xla_docker_image": - curl
Step #2 - "build_xla_docker_image": - git
Step #2 - "build_xla_docker_image": - gnupg
Step #2 - "build_xla_docker_image": - libopenblas-dev
Step #2 - "build_xla_docker_image": - ninja-build
Step #2 - "build_xla_docker_image": - procps
Step #2 - "build_xla_docker_image": - python3-pip
Step #2 - "build_xla_docker_image": - rename
Step #2 - "build_xla_docker_image": - vim
Step #2 - "build_xla_docker_image": - wget
Step #2 - "build_xla_docker_image": - clang-format
Step #2 - "build_xla_docker_image": - gcc-10
Step #2 - "build_xla_docker_image": - g++-10
Step #2 - "build_xla_docker_image": - lcov
Step #2 - "build_xla_docker_image": - less
Step #2 - "build_xla_docker_image": - cuda-libraries-12-4
Step #2 - "build_xla_docker_image": - cuda-toolkit-12-4
Step #2 - "build_xla_docker_image": - cuda-minimal-build-12-4
Step #2 - "build_xla_docker_image": - libcudnn-cuda-12=9.1.1.17-1
Step #2 - "build_xla_docker_image": - libcudnn-dev-cuda-12=9.1.1.17-1
Step #2 - "build_xla_docker_image": only_upgrade: false
Step #2 - "build_xla_docker_image": package:
Step #2 - "build_xla_docker_image": - ccache
Step #2 - "build_xla_docker_image": - curl
Step #2 - "build_xla_docker_image": - git
Step #2 - "build_xla_docker_image": - gnupg
Step #2 - "build_xla_docker_image": - libopenblas-dev
Step #2 - "build_xla_docker_image": - ninja-build
Step #2 - "build_xla_docker_image": - procps
Step #2 - "build_xla_docker_image": - python3-pip
Step #2 - "build_xla_docker_image": - rename
Step #2 - "build_xla_docker_image": - vim
Step #2 - "build_xla_docker_image": - wget
Step #2 - "build_xla_docker_image": - clang-format
Step #2 - "build_xla_docker_image": - gcc-10
Step #2 - "build_xla_docker_image": - g++-10
Step #2 - "build_xla_docker_image": - lcov
Step #2 - "build_xla_docker_image": - less
Step #2 - "build_xla_docker_image": - cuda-libraries-12-4
Step #2 - "build_xla_docker_image": - cuda-toolkit-12-4
Step #2 - "build_xla_docker_image": - cuda-minimal-build-12-4
Step #2 - "build_xla_docker_image": - libcudnn-cuda-12=9.1.1.17-1
Step #2 - "build_xla_docker_image": - libcudnn-dev-cuda-12=9.1.1.17-1
Step #2 - "build_xla_docker_image": policy_rc_d: null
Step #2 - "build_xla_docker_image": purge: false
Step #2 - "build_xla_docker_image": state: present
Step #2 - "build_xla_docker_image": update_cache: true
Step #2 - "build_xla_docker_image": update_cache_retries: 5
Step #2 - "build_xla_docker_image": update_cache_retry_max_delay: 12
Step #2 - "build_xla_docker_image": upgrade: null
Step #2 - "build_xla_docker_image": msg: No package matching 'libcudnn-cuda-12' is available
Step #2 - "build_xla_docker_image":
Step #2 - "build_xla_docker_image": PLAY RECAP *********************************************************************
Step #2 - "build_xla_docker_image": localhost : ok=8 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Step #2 - "build_xla_docker_image":
Step #2 - "build_xla_docker_image": Monday 09 September 2024 21:44:48 +0000 (0:00:03.395) 0:00:24.242 ******
Step #2 - "build_xla_docker_image": ===============================================================================
Step #2 - "build_xla_docker_image": install_deps : Add apt repositories into sources list ------------------ 12.78s
Step #2 - "build_xla_docker_image": /ansible/roles/install_deps/tasks/main.yaml:11 --------------------------------
Step #2 - "build_xla_docker_image": install_deps : Update apt cache ----------------------------------------- 3.57s
Step #2 - "build_xla_docker_image": /ansible/roles/install_deps/tasks/main.yaml:18 --------------------------------
Step #2 - "build_xla_docker_image": install_deps : Install apt packages ------------------------------------- 3.40s
Step #2 - "build_xla_docker_image": /ansible/roles/install_deps/tasks/main.yaml:23 --------------------------------
Step #2 - "build_xla_docker_image": install_deps : Add apt keys --------------------------------------------- 1.30s
Step #2 - "build_xla_docker_image": /ansible/roles/install_deps/tasks/main.yaml:1 ---------------------------------
Step #2 - "build_xla_docker_image": Gathering Facts --------------------------------------------------------- 1.17s
Step #2 - "build_xla_docker_image": /ansible/playbook.yaml:1 ------------------------------------------------------
Step #2 - "build_xla_docker_image": bazel : Download bazelisk v1.15.0 --------------------------------------- 1.03s
Step #2 - "build_xla_docker_image": /ansible/roles/bazel/tasks/main.yaml:1 ----------------------------------------
Step #2 - "build_xla_docker_image": bazel : Bazel --version runs succesfully -------------------------------- 0.84s
Step #2 - "build_xla_docker_image": /ansible/roles/bazel/tasks/tests.yaml:1 ---------------------------------------
Step #2 - "build_xla_docker_image": Include vars from config files ------------------------------------------ 0.09s
Step #2 - "build_xla_docker_image": /ansible/playbook.yaml:25 -----------------------------------------------------
Step #2 - "build_xla_docker_image": bazel : Tests ----------------------------------------------------------- 0.04s
Step #2 - "build_xla_docker_image": /ansible/roles/bazel/tasks/main.yaml:7 ----------------------------------------
Step #2 - "build_xla_docker_image": The command '/bin/sh -c ansible-playbook -vvv playbook.yaml -e "stage=build" -e "${ansible_vars}" --tags "bazel,configure_env,install_deps"' returned a non-zero code: 2
Finished Step #2 - "build_xla_docker_image"
ERROR
ERROR: build step 2 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 2
Loading