Fori loop simple case test test #7028
Closed
Google Cloud Build / dev-3-10-cuda-12-3 (tpu-pytorch-releases)
succeeded
May 1, 2024 in 17m 16s
Summary
Build Information
Trigger | dev-3-10-cuda-12-3 |
Build | 2827e3f6-1b6b-4b1a-ac95-31ef63be687e |
Start | 2024-05-01T12:42:37-07:00 |
Duration | 16m30.369s |
Status | SUCCESS |
Steps
Step | Status | Duration |
---|---|---|
git_fetch | SUCCESS | 9.128s |
git_checkout | SUCCESS | 599ms |
build_development_docker_image | SUCCESS | 8m21.011s |
push_development_docker_image | SUCCESS | 7m56.424s |
Details
starting build "2827e3f6-1b6b-4b1a-ac95-31ef63be687e"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/pytorch/xla
* branch cbbefa2c129ae48c6e868b9eeb4985b6e562dc1c -> FETCH_HEAD
HEAD is now at cbbefa2 Support pin pr number in new .torch_pin (#6998)
BUILD
Starting Step #0 - "git_fetch"
Step #0 - "git_fetch": Already have image (with digest): gcr.io/cloud-builders/git
Step #0 - "git_fetch": From https://github.com/pytorch/xla
Step #0 - "git_fetch": * [new branch] 01172023_pin_update -> origin/01172023_pin_update
Step #0 - "git_fetch": * [new branch] 2-0-python-3-10 -> origin/2-0-python-3-10
Step #0 - "git_fetch": * [new branch] 4483-improve-resnet50-data-loading-performance -> origin/4483-improve-resnet50-data-loading-performance
Step #0 - "git_fetch": * [new branch] 5d-unit-test -> origin/5d-unit-test
Step #0 - "git_fetch": * [new branch] AddVimToReleasePackage -> origin/AddVimToReleasePackage
Step #0 - "git_fetch": * [new branch] FixTransferToServer -> origin/FixTransferToServer
Step #0 - "git_fetch": * [new branch] JackCaaoG/spmd_device_python_test -> origin/JackCaaoG/spmd_device_python_test
Step #0 - "git_fetch": * [new branch] JackCao/Disable_hlo_dump_test_on_tpu -> origin/JackCao/Disable_hlo_dump_test_on_tpu
Step #0 - "git_fetch": * [new branch] JackCao/Dynamo_sync_input_counter -> origin/JackCao/Dynamo_sync_input_counter
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_1 -> origin/JackCao/cherry_pick_dynamo_1
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_2 -> origin/JackCao/cherry_pick_dynamo_2
Step #0 - "git_fetch": * [new branch] JackCao/cherry_pick_dynamo_3 -> origin/JackCao/cherry_pick_dynamo_3
Step #0 - "git_fetch": * [new branch] JackCao/spmd_python_device -> origin/JackCao/spmd_python_device
Step #0 - "git_fetch": * [new branch] JackCao/unordered_sharding_spec -> origin/JackCao/unordered_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCao/update_to_2.0 -> origin/JackCao/update_to_2.0
Step #0 - "git_fetch": * [new branch] JackCao/update_to_223 -> origin/JackCao/update_to_223
Step #0 - "git_fetch": * [new branch] JackCao/workaround_einsum_faketensor -> origin/JackCao/workaround_einsum_faketensor
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-1 -> origin/JackCaoG-patch-1
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-2 -> origin/JackCaoG-patch-2
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-3 -> origin/JackCaoG-patch-3
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-4 -> origin/JackCaoG-patch-4
Step #0 - "git_fetch": * [new branch] JackCaoG-patch-5 -> origin/JackCaoG-patch-5
Step #0 - "git_fetch": * [new branch] JackCaoG-remove-pin -> origin/JackCaoG-remove-pin
Step #0 - "git_fetch": * [new branch] JackCaoG/1_13_revert_addcmul -> origin/JackCaoG/1_13_revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/2.0_patch -> origin/JackCaoG/2.0_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/AddDynamoSpmdMetrics -> origin/JackCaoG/AddDynamoSpmdMetrics
Step #0 - "git_fetch": * [new branch] JackCaoG/CI_build_cpp_test -> origin/JackCaoG/CI_build_cpp_test
Step #0 - "git_fetch": * [new branch] JackCaoG/Dyanmo_execute_async -> origin/JackCaoG/Dyanmo_execute_async
Step #0 - "git_fetch": * [new branch] JackCaoG/Dynamo_training_test -> origin/JackCaoG/Dynamo_training_test
Step #0 - "git_fetch": * [new branch] JackCaoG/PJRT_wait_buffer_sync -> origin/JackCaoG/PJRT_wait_buffer_sync
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_execute_metrics -> origin/JackCaoG/SPMD_execute_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/SPMD_int_to_cpu -> origin/JackCaoG/SPMD_int_to_cpu
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_short_metrics -> origin/JackCaoG/Update_short_metrics
Step #0 - "git_fetch": * [new branch] JackCaoG/Update_wheel_docker_to_table -> origin/JackCaoG/Update_wheel_docker_to_table
Step #0 - "git_fetch": * [new branch] JackCaoG/XLAShardedTensor_torch_dispatch -> origin/JackCaoG/XLAShardedTensor_torch_dispatch
Step #0 - "git_fetch": * [new branch] JackCaoG/add_2_1_py311_build -> origin/JackCaoG/add_2_1_py311_build
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_counter_check -> origin/JackCaoG/add_dynamo_test_counter_check
Step #0 - "git_fetch": * [new branch] JackCaoG/add_dynamo_test_to_tpu_ci -> origin/JackCaoG/add_dynamo_test_to_tpu_ci
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc -> origin/JackCaoG/add_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/add_gpu_doc_readme -> origin/JackCaoG/add_gpu_doc_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_dynamo_graph_dump -> origin/JackCaoG/add_test_dynamo_graph_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/add_test_print -> origin/JackCaoG/add_test_print
Step #0 - "git_fetch": * [new branch] JackCaoG/api_init_runtime -> origin/JackCaoG/api_init_runtime
Step #0 - "git_fetch": * [new branch] JackCaoG/backout_22_mesh_spmd -> origin/JackCaoG/backout_22_mesh_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_allgather_out -> origin/JackCaoG/backport_22_allgather_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_diagonal_scatter -> origin/JackCaoG/backport_22_diagonal_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_doc -> origin/JackCaoG/backport_22_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reduce_scatter -> origin/JackCaoG/backport_22_reduce_scatter
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_reducescatter_out -> origin/JackCaoG/backport_22_reducescatter_out
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_scalar -> origin/JackCaoG/backport_22_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_22_transfer_gpu_scalar -> origin/JackCaoG/backport_22_transfer_gpu_scalar
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_aws_zero1 -> origin/JackCaoG/backport_aws_zero1
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_dynamo_cpu_move -> origin/JackCaoG/backport_dynamo_cpu_move
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_dynamo_doc_eval -> origin/JackCaoG/backport_dynamo_doc_eval
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_flash_attention_dynamo -> origin/JackCaoG/backport_flash_attention_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_frame_count -> origin/JackCaoG/backport_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/backport_oerserve_rng -> origin/JackCaoG/backport_oerserve_rng
Step #0 - "git_fetch": * [new branch] JackCaoG/barrier_comment -> origin/JackCaoG/barrier_comment
Step #0 - "git_fetch": * [new branch] JackCaoG/better_dynamo_error_message -> origin/JackCaoG/better_dynamo_error_message
Step #0 - "git_fetch": * [new branch] JackCaoG/buffer_donor_custom_op -> origin/JackCaoG/buffer_donor_custom_op
Step #0 - "git_fetch": * [new branch] JackCaoG/build_audio_whl -> origin/JackCaoG/build_audio_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/bump+gpu_coverage_timeout -> origin/JackCaoG/bump+gpu_coverage_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/bump_build_timeout -> origin/JackCaoG/bump_build_timeout
Step #0 - "git_fetch": * [new branch] JackCaoG/cache_sharded_placeholder_dynamo -> origin/JackCaoG/cache_sharded_placeholder_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_dynamo_test -> origin/JackCaoG/cherry_pick_dynamo_test
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_pick_frame_count -> origin/JackCaoG/cherry_pick_frame_count
Step #0 - "git_fetch": * [new branch] JackCaoG/cherry_picl_xrt_crash_2.1 -> origin/JackCaoG/cherry_picl_xrt_crash_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/clang-format-11 -> origin/JackCaoG/clang-format-11
Step #0 - "git_fetch": * [new branch] JackCaoG/clang-format-11-2 -> origin/JackCaoG/clang-format-11-2
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_gettensor -> origin/JackCaoG/clean_up_gettensor
Step #0 - "git_fetch": * [new branch] JackCaoG/clean_up_use_virtual_device -> origin/JackCaoG/clean_up_use_virtual_device
Step #0 - "git_fetch": * [new branch] JackCaoG/cleanup_docs -> origin/JackCaoG/cleanup_docs
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi -> origin/JackCaoG/colab_whl_abi
Step #0 - "git_fetch": * [new branch] JackCaoG/colab_whl_abi_2 -> origin/JackCaoG/colab_whl_abi_2
Step #0 - "git_fetch": * [new branch] JackCaoG/data_to_string -> origin/JackCaoG/data_to_string
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_fix -> origin/JackCaoG/debug_fix
Step #0 - "git_fetch": * [new branch] JackCaoG/debug_tool -> origin/JackCaoG/debug_tool
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_coverage -> origin/JackCaoG/disable_coverage
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_dynamo_different_shape_test -> origin/JackCaoG/disable_dynamo_different_shape_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test -> origin/JackCaoG/disable_flaky_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_2 -> origin/JackCaoG/disable_flaky_test_2
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_flaky_test_3 -> origin/JackCaoG/disable_flaky_test_3
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic -> origin/JackCaoG/disable_more_dynamic
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_more_dynamic_test -> origin/JackCaoG/disable_more_dynamic_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_pjrt_computation_client_test -> origin/JackCaoG/disable_pjrt_computation_client_test
Step #0 - "git_fetch": * [new branch] JackCaoG/disable_test_xla_sharding -> origin/JackCaoG/disable_test_xla_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/disbale_docker_sandbox -> origin/JackCaoG/disbale_docker_sandbox
Step #0 - "git_fetch": * [new branch] JackCaoG/dump_output_sharding -> origin/JackCaoG/dump_output_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_2.0_cherry_pick -> origin/JackCaoG/dynamo_2.0_cherry_pick
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_aliasing_2 -> origin/JackCaoG/dynamo_aliasing_2
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_debug_message -> origin/JackCaoG/dynamo_debug_message
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_different_graph -> origin/JackCaoG/dynamo_different_graph
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_do_not_markstep_to_sync_input -> origin/JackCaoG/dynamo_do_not_markstep_to_sync_input
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_doc -> origin/JackCaoG/dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_dump_tensor_file -> origin/JackCaoG/dynamo_dump_tensor_file
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_note_backport_2.1 -> origin/JackCaoG/dynamo_note_backport_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_only_warm_up_cache -> origin/JackCaoG/dynamo_only_warm_up_cache
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_optimizer -> origin/JackCaoG/dynamo_optimizer
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_reduce_execute -> origin/JackCaoG/dynamo_reduce_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_rng_seed_without_mark_step -> origin/JackCaoG/dynamo_rng_seed_without_mark_step
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd -> origin/JackCaoG/dynamo_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_input_sharding_change -> origin/JackCaoG/dynamo_spmd_input_sharding_change
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_spmd_optimization -> origin/JackCaoG/dynamo_spmd_optimization
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_sync_input -> origin/JackCaoG/dynamo_sync_input
Step #0 - "git_fetch": * [new branch] JackCaoG/dynamo_warning -> origin/JackCaoG/dynamo_warning
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_cumsum_long -> origin/JackCaoG/enable_cumsum_long
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_dynamo_upstream_test -> origin/JackCaoG/enable_dynamo_upstream_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_fsdp_gpu_test -> origin/JackCaoG/enable_fsdp_gpu_test
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_mp_test_on_tpuci -> origin/JackCaoG/enable_mp_test_on_tpuci
Step #0 - "git_fetch": * [new branch] JackCaoG/enable_reduce_scatter_test_gpu -> origin/JackCaoG/enable_reduce_scatter_test_gpu
Step #0 - "git_fetch": * [new branch] JackCaoG/example_torch_pin_pr -> origin/JackCaoG/example_torch_pin_pr
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_cc_linter_once -> origin/JackCaoG/fix_cc_linter_once
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_deprecated_test_oepration_api -> origin/JackCaoG/fix_deprecated_test_oepration_api
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_diagonal_scatter_dim -> origin/JackCaoG/fix_diagonal_scatter_dim
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dyanmo_additional_tracing -> origin/JackCaoG/fix_dyanmo_additional_tracing
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo -> origin/JackCaoG/fix_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash -> origin/JackCaoG/fix_dynamo_crash
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_crash_profiler -> origin/JackCaoG/fix_dynamo_crash_profiler
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_inplace -> origin/JackCaoG/fix_dynamo_inplace
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_input_sharding_test -> origin/JackCaoG/fix_dynamo_input_sharding_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_dynamo_text_allclose -> origin/JackCaoG/fix_dynamo_text_allclose
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_full_dtype -> origin/JackCaoG/fix_full_dtype
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump -> origin/JackCaoG/fix_hlo_dump
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_hlo_dump_spmd -> origin/JackCaoG/fix_hlo_dump_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_log_spam -> origin/JackCaoG/fix_log_spam
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mark_step_on_spmd_device -> origin/JackCaoG/fix_mark_step_on_spmd_device
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_mp_save_test -> origin/JackCaoG/fix_mp_save_test
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_send_cpu_data -> origin/JackCaoG/fix_send_cpu_data
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_spmd_buff_is_null -> origin/JackCaoG/fix_spmd_buff_is_null
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_tpu_ci_accuracy -> origin/JackCaoG/fix_tpu_ci_accuracy
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_typo -> origin/JackCaoG/fix_typo
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_vision_build -> origin/JackCaoG/fix_vision_build
Step #0 - "git_fetch": * [new branch] JackCaoG/fix_warning_message -> origin/JackCaoG/fix_warning_message
Step #0 - "git_fetch": * [new branch] JackCaoG/flash_attention_custom_op -> origin/JackCaoG/flash_attention_custom_op
Step #0 - "git_fetch": * [new branch] JackCaoG/flash_attention_test_precision -> origin/JackCaoG/flash_attention_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/fsdp_autowrap_test -> origin/JackCaoG/fsdp_autowrap_test
Step #0 - "git_fetch": * [new branch] JackCaoG/gpu_build_pytorch_cuda -> origin/JackCaoG/gpu_build_pytorch_cuda
Step #0 - "git_fetch": * [new branch] JackCaoG/graph_hash_debug -> origin/JackCaoG/graph_hash_debug
Step #0 - "git_fetch": * [new branch] JackCaoG/handle_dynamo_fn_without_arg -> origin/JackCaoG/handle_dynamo_fn_without_arg
Step #0 - "git_fetch": * [new branch] JackCaoG/improve_data_tostring -> origin/JackCaoG/improve_data_tostring
Step #0 - "git_fetch": * [new branch] JackCaoG/increase_timeout_docker -> origin/JackCaoG/increase_timeout_docker
Step #0 - "git_fetch": * [new branch] JackCaoG/ir_value_sharding -> origin/JackCaoG/ir_value_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/lower_dropout_backward -> origin/JackCaoG/lower_dropout_backward
Step #0 - "git_fetch": * [new branch] JackCaoG/move_op_lowering_guide -> origin/JackCaoG/move_op_lowering_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pin_2_0 -> origin/JackCaoG/move_pin_2_0
Step #0 - "git_fetch": * [new branch] JackCaoG/move_pjrt_lock_location -> origin/JackCaoG/move_pjrt_lock_location
Step #0 - "git_fetch": * [new branch] JackCaoG/move_return_future -> origin/JackCaoG/move_return_future
Step #0 - "git_fetch": * [new branch] JackCaoG/move_test_grad_checkpoint_to_tpu -> origin/JackCaoG/move_test_grad_checkpoint_to_tpu
Step #0 - "git_fetch": * [new branch] JackCaoG/new_alias_api -> origin/JackCaoG/new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/nightly_build_python -> origin/JackCaoG/nightly_build_python
Step #0 - "git_fetch": * [new branch] JackCaoG/parallel_spmd -> origin/JackCaoG/parallel_spmd
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_expecttest -> origin/JackCaoG/pin_expecttest
Step #0 - "git_fetch": * [new branch] JackCaoG/pin_update_20230808 -> origin/JackCaoG/pin_update_20230808
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_5 -> origin/JackCaoG/pt_xla_debug_5
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_dynamo_execute -> origin/JackCaoG/pt_xla_debug_dynamo_execute
Step #0 - "git_fetch": * [new branch] JackCaoG/pt_xla_debug_max_frame -> origin/JackCaoG/pt_xla_debug_max_frame
Step #0 - "git_fetch": * [new branch] JackCaoG/r20_backportdoc -> origin/JackCaoG/r20_backportdoc
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_ABI_0 -> origin/JackCaoG/r2_0_ABI_0
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_back_port_profile_flag -> origin/JackCaoG/r2_0_back_port_profile_flag
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread -> origin/JackCaoG/r2_0_backport_loader_thread
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_backport_loader_thread_2 -> origin/JackCaoG/r2_0_backport_loader_thread_2
Step #0 - "git_fetch": * [new branch] JackCaoG/r2_0_cherry_pick_defer_init -> origin/JackCaoG/r2_0_cherry_pick_defer_init
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_pin -> origin/JackCaoG/remove_pin
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_reshape_again -> origin/JackCaoG/remove_reshape_again
Step #0 - "git_fetch": * [new branch] JackCaoG/remove_reshape_lowering -> origin/JackCaoG/remove_reshape_lowering
Step #0 - "git_fetch": * [new branch] JackCaoG/restrict_mark_sharding -> origin/JackCaoG/restrict_mark_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_addcmul -> origin/JackCaoG/revert_addcmul
Step #0 - "git_fetch": * [new branch] JackCaoG/revert_protobuf_tf_patch -> origin/JackCaoG/revert_protobuf_tf_patch
Step #0 - "git_fetch": * [new branch] JackCaoG/shard_device_data_ir -> origin/JackCaoG/shard_device_data_ir
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_partitioner -> origin/JackCaoG/skip_partitioner
Step #0 - "git_fetch": * [new branch] JackCaoG/skip_tpu_test_precision -> origin/JackCaoG/skip_tpu_test_precision
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_aliasing -> origin/JackCaoG/spmd_aliasing
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_dynamo_sharding_changed -> origin/JackCaoG/spmd_dynamo_sharding_changed
Step #0 - "git_fetch": * [new branch] JackCaoG/spmd_incorrect_result -> origin/JackCaoG/spmd_incorrect_result
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_debugging_api -> origin/JackCaoG/tensor_debugging_api
Step #0 - "git_fetch": * [new branch] JackCaoG/tensor_with_xla_data_sharding -> origin/JackCaoG/tensor_with_xla_data_sharding
Step #0 - "git_fetch": * [new branch] JackCaoG/test_torch_compile_usage -> origin/JackCaoG/test_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/tf_update_20230201 -> origin/JackCaoG/tf_update_20230201
Step #0 - "git_fetch": * [new branch] JackCaoG/tracing_timer -> origin/JackCaoG/tracing_timer
Step #0 - "git_fetch": * [new branch] JackCaoG/trouble_shooting_update -> origin/JackCaoG/trouble_shooting_update
Step #0 - "git_fetch": * [new branch] JackCaoG/truncate_python_stack -> origin/JackCaoG/truncate_python_stack
Step #0 - "git_fetch": * [new branch] JackCaoG/unordered_partial_sharding_spec -> origin/JackCaoG/unordered_partial_sharding_spec
Step #0 - "git_fetch": * [new branch] JackCaoG/updateDocker -> origin/JackCaoG/updateDocker
Step #0 - "git_fetch": * [new branch] JackCaoG/update_2_0_ABI -> origin/JackCaoG/update_2_0_ABI
Step #0 - "git_fetch": * [new branch] JackCaoG/update_API_GUIDE -> origin/JackCaoG/update_API_GUIDE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE -> origin/JackCaoG/update_XLA_IR_SHAPE_CACHE_SIZE
Step #0 - "git_fetch": * [new branch] JackCaoG/update_build_guide -> origin/JackCaoG/update_build_guide
Step #0 - "git_fetch": * [new branch] JackCaoG/update_colab_2_10_whl -> origin/JackCaoG/update_colab_2_10_whl
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc1 -> origin/JackCaoG/update_doc1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_doc_remove_xrt -> origin/JackCaoG/update_doc_remove_xrt
Step #0 - "git_fetch": * [new branch] JackCaoG/update_docker_instruction -> origin/JackCaoG/update_docker_instruction
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_name -> origin/JackCaoG/update_dynamo_backend_name
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_backend_nograd -> origin/JackCaoG/update_dynamo_backend_nograd
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_counter_check -> origin/JackCaoG/update_dynamo_counter_check
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_doc -> origin/JackCaoG/update_dynamo_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_doc_to_remove_eval -> origin/JackCaoG/update_dynamo_doc_to_remove_eval
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_number_2.1 -> origin/JackCaoG/update_dynamo_number_2.1
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_readme -> origin/JackCaoG/update_dynamo_readme
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_save_graph_path -> origin/JackCaoG/update_dynamo_save_graph_path
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_torch_equal -> origin/JackCaoG/update_dynamo_torch_equal
Step #0 - "git_fetch": * [new branch] JackCaoG/update_dynamo_training_compile_count -> origin/JackCaoG/update_dynamo_training_compile_count
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_dev -> origin/JackCaoG/update_gpu_dev
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_doc -> origin/JackCaoG/update_gpu_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheel_link -> origin/JackCaoG/update_gpu_wheel_link
Step #0 - "git_fetch": * [new branch] JackCaoG/update_gpu_wheellink -> origin/JackCaoG/update_gpu_wheellink
Step #0 - "git_fetch": * [new branch] JackCaoG/update_pjrt_future -> origin/JackCaoG/update_pjrt_future
Step #0 - "git_fetch": * [new branch] JackCaoG/update_readme_2 -> origin/JackCaoG/update_readme_2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_set_default_tensor_type -> origin/JackCaoG/update_set_default_tensor_type
Step #0 - "git_fetch": * [new branch] JackCaoG/update_tf_2023018 -> origin/JackCaoG/update_tf_2023018
Step #0 - "git_fetch": * [new branch] JackCaoG/update_to_2.2 -> origin/JackCaoG/update_to_2.2
Step #0 - "git_fetch": * [new branch] JackCaoG/update_to_223 -> origin/JackCaoG/update_to_223
Step #0 - "git_fetch": * [new branch] JackCaoG/update_torch_compile_usage -> origin/JackCaoG/update_torch_compile_usage
Step #0 - "git_fetch": * [new branch] JackCaoG/update_trouble_shotting_doc -> origin/JackCaoG/update_trouble_shotting_doc
Step #0 - "git_fetch": * [new branch] JackCaoG/update_troubleshotting_with_sanitycheck -> origin/JackCaoG/update_troubleshotting_with_sanitycheck
Step #0 - "git_fetch": * [new branch] JackCaoG/update_wheel_docker_format -> origin/JackCaoG/update_wheel_docker_format
Step #0 - "git_fetch": * [new branch] JackCaoG/update_xla_pin_040922024 -> origin/JackCaoG/update_xla_pin_040922024
Step #0 - "git_fetch": * [new branch] JackCaoG/use_new_alias_api -> origin/JackCaoG/use_new_alias_api
Step #0 - "git_fetch": * [new branch] JackCaoG/weak_ref_dynamo -> origin/JackCaoG/weak_ref_dynamo
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherry_pick_07_10 -> origin/JackCaoG/xrt_cherry_pick_07_10
Step #0 - "git_fetch": * [new branch] JackCaoG/xrt_cherrypick_0721 -> origin/JackCaoG/xrt_cherrypick_0721
Step #0 - "git_fetch": * [new branch] JackCaog/dynamo_input_sharding -> origin/JackCaog/dynamo_input_sharding
Step #0 - "git_fetch": * [new branch] JackCaog/quick_spmd_fix -> origin/JackCaog/quick_spmd_fix
Step #0 - "git_fetch": * [new branch] JackCaog/spmd_device_default -> origin/JackCaog/spmd_device_default
Step #0 - "git_fetch": * [new branch] Liyang90-patch-1 -> origin/Liyang90-patch-1
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-1 -> origin/ManfeiBai-patch-1
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-10 -> origin/ManfeiBai-patch-10
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-11 -> origin/ManfeiBai-patch-11
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-12 -> origin/ManfeiBai-patch-12
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-13 -> origin/ManfeiBai-patch-13
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-14 -> origin/ManfeiBai-patch-14
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-15 -> origin/ManfeiBai-patch-15
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-16 -> origin/ManfeiBai-patch-16
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-17 -> origin/ManfeiBai-patch-17
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-18 -> origin/ManfeiBai-patch-18
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-19 -> origin/ManfeiBai-patch-19
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-2 -> origin/ManfeiBai-patch-2
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-20 -> origin/ManfeiBai-patch-20
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-21 -> origin/ManfeiBai-patch-21
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-22 -> origin/ManfeiBai-patch-22
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-23 -> origin/ManfeiBai-patch-23
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-24 -> origin/ManfeiBai-patch-24
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-25 -> origin/ManfeiBai-patch-25
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-26 -> origin/ManfeiBai-patch-26
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-27 -> origin/ManfeiBai-patch-27
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-28 -> origin/ManfeiBai-patch-28
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-29 -> origin/ManfeiBai-patch-29
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-3 -> origin/ManfeiBai-patch-3
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-30 -> origin/ManfeiBai-patch-30
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-31 -> origin/ManfeiBai-patch-31
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-32 -> origin/ManfeiBai-patch-32
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-33 -> origin/ManfeiBai-patch-33
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-34 -> origin/ManfeiBai-patch-34
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-35 -> origin/ManfeiBai-patch-35
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-36 -> origin/ManfeiBai-patch-36
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-37 -> origin/ManfeiBai-patch-37
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-38 -> origin/ManfeiBai-patch-38
Step #0 - "git_fetch": * [new branch] ManfeiBai-patch-39 -> origin/ManfeiBai-patch-39
Step #0 - "gi
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/2827e3f6-1b6b-4b1a-ac95-31ef63be687e?project=1001674285173.]
...
bi3-manylinux_2_28_x86_64.whl (4.6 MB)
Step #2 - "build_development_docker_image": ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 74.9 MB/s eta 0:00:00
Step #2 - "build_development_docker_image": Collecting resolvelib<1.1.0,>=0.5.3
Step #2 - "build_development_docker_image": Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB)
Step #2 - "build_development_docker_image": Collecting packaging
Step #2 - "build_development_docker_image": Downloading packaging-24.0-py3-none-any.whl (53 kB)
Step #2 - "build_development_docker_image": ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 10.0 MB/s eta 0:00:00
Step #2 - "build_development_docker_image": Collecting PyYAML>=5.1
Step #2 - "build_development_docker_image": Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
Step #2 - "build_development_docker_image": ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 kB 60.9 MB/s eta 0:00:00
Step #2 - "build_development_docker_image": Collecting MarkupSafe>=2.0
Step #2 - "build_development_docker_image": Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Step #2 - "build_development_docker_image": Collecting cffi>=1.12
Step #2 - "build_development_docker_image": Downloading cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (443 kB)
Step #2 - "build_development_docker_image": ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 443.9/443.9 kB 50.3 MB/s eta 0:00:00
Step #2 - "build_development_docker_image": Collecting pycparser
Step #2 - "build_development_docker_image": Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Step #2 - "build_development_docker_image": ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 21.7 MB/s eta 0:00:00
Step #2 - "build_development_docker_image": Installing collected packages: resolvelib, PyYAML, pycparser, packaging, MarkupSafe, jinja2, cffi, cryptography, ansible-core, ansible
Step #2 - "build_development_docker_image": Successfully installed MarkupSafe-2.1.5 PyYAML-6.0.1 ansible-9.5.1 ansible-core-2.16.6 cffi-1.16.0 cryptography-42.0.5 jinja2-3.1.3 packaging-24.0 pycparser-2.22 resolvelib-1.0.1
Step #2 - "build_development_docker_image": �[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Step #2 - "build_development_docker_image": �[0m�[91m
Step #2 - "build_development_docker_image": [notice] A new release of pip is available: 23.0.1 -> 24.0
Step #2 - "build_development_docker_image": [notice] To update, run: pip install --upgrade pip
Step #2 - "build_development_docker_image": �[0mRemoving intermediate container ae378b76f877
Step #2 - "build_development_docker_image": ---> 84a10cb21346
Step #2 - "build_development_docker_image": Step 5/10 : COPY . /ansible
Step #2 - "build_development_docker_image": ---> 193c7290ae08
Step #2 - "build_development_docker_image": Step 6/10 : WORKDIR /ansible
Step #2 - "build_development_docker_image": ---> Running in cbeb774e3e27
Step #2 - "build_development_docker_image": Removing intermediate container cbeb774e3e27
Step #2 - "build_development_docker_image": ---> 5fcfc99d541b
Step #2 - "build_development_docker_image": Step 7/10 : ENV TAGS="bazel,configure_env,install_deps"
Step #2 - "build_development_docker_image": ---> Running in 1c8bc05a3e2a
Step #2 - "build_development_docker_image": Removing intermediate container 1c8bc05a3e2a
Step #2 - "build_development_docker_image": ---> 175e28ade48b
Step #2 - "build_development_docker_image": Step 8/10 : ARG ansible_vars
Step #2 - "build_development_docker_image": ---> Running in c60b7e291925
Step #2 - "build_development_docker_image": Removing intermediate container c60b7e291925
Step #2 - "build_development_docker_image": ---> e766f5d76d88
Step #2 - "build_development_docker_image": Step 9/10 : RUN ansible-playbook playbook.yaml -e "stage=build" -e "${ansible_vars}" --tags "${TAGS}"
Step #2 - "build_development_docker_image": ---> Running in d5ce950de2c8
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": PLAY [Install build dependencies] **********************************************
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [Gathering Facts] *********************************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:45 +0000 (0:00:00.027) 0:00:00.027 *********
Step #2 - "build_development_docker_image": ok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [Include vars from config files] ******************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:46 +0000 (0:00:01.027) 0:00:01.055 *********
Step #2 - "build_development_docker_image": ok: [localhost] => (item=vars.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=cuda_deps.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=apt.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=pip.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=env.yaml)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Download bazelisk v1.15.0] ***************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:46 +0000 (0:00:00.072) 0:00:01.127 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Tests] ***********************************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:47 +0000 (0:00:00.938) 0:00:02.065 *********
Step #2 - "build_development_docker_image": included: /ansible/roles/bazel/tasks/tests.yaml for localhost
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Bazel --version runs succesfully] ********************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:47 +0000 (0:00:00.029) 0:00:02.094 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Add apt keys] *********************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:48 +0000 (0:00:00.692) 0:00:02.787 *********
Step #2 - "build_development_docker_image": changed: [localhost] => (item={'url': 'https://apt.llvm.org/llvm-snapshot.gpg.key', 'keyring': '/usr/share/keyrings/llvm.pgp'})
Step #2 - "build_development_docker_image": changed: [localhost] => (item={'url': 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub', 'keyring': '/usr/share/keyrings/cuda.pgp'})
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Add apt repositories into sources list] *******************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:43:49 +0000 (0:00:00.920) 0:00:03.708 *********
Step #2 - "build_development_docker_image": changed: [localhost] => (item=deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Update apt cache] *****************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:44:00 +0000 (0:00:11.783) 0:00:15.491 *********
Step #2 - "build_development_docker_image": ok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install apt packages] *************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:44:03 +0000 (0:00:03.106) 0:00:18.598 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install pip packages] *************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:48:59 +0000 (0:04:55.635) 0:05:14.233 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install pip packages without deps (--no-deps)] ************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:49:39 +0000 (0:00:40.166) 0:05:54.400 *********
Step #2 - "build_development_docker_image": �[91m[WARNING]: No valid name or requirements file found.
Step #2 - "build_development_docker_image": �[0mok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [configure_env : Append environment variables required during runtime to ~/.bashrc] ***
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:49:40 +0000 (0:00:00.319) 0:05:54.720 *********
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=CC)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=CXX)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=LD_LIBRARY_PATH)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=TF_CUDA_COMPUTE_CAPABILITIES)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=XLA_CUDA)
Step #2 - "build_development_docker_image": skipping: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [configure_env : Append environment variables required during runtime to ~/.zshrc] ***
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:49:40 +0000 (0:00:00.064) 0:05:54.784 *********
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=CC)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=CXX)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=LD_LIBRARY_PATH)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=TF_CUDA_COMPUTE_CAPABILITIES)
Step #2 - "build_development_docker_image": skipping: [localhost] => (item=XLA_CUDA)
Step #2 - "build_development_docker_image": skipping: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": PLAY RECAP *********************************************************************
Step #2 - "build_development_docker_image": localhost : ok=11 changed=6 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:49:40 +0000 (0:00:00.085) 0:05:54.870 *********
Step #2 - "build_development_docker_image": ===============================================================================
Step #2 - "build_development_docker_image": install_deps : Install apt packages ----------------------------------- 295.64s
Step #2 - "build_development_docker_image": install_deps : Install pip packages ------------------------------------ 40.17s
Step #2 - "build_development_docker_image": install_deps : Add apt repositories into sources list ------------------ 11.78s
Step #2 - "build_development_docker_image": install_deps : Update apt cache ----------------------------------------- 3.11s
Step #2 - "build_development_docker_image": Gathering Facts --------------------------------------------------------- 1.03s
Step #2 - "build_development_docker_image": bazel : Download bazelisk v1.15.0 --------------------------------------- 0.94s
Step #2 - "build_development_docker_image": install_deps : Add apt keys --------------------------------------------- 0.92s
Step #2 - "build_development_docker_image": bazel : Bazel --version runs succesfully -------------------------------- 0.69s
Step #2 - "build_development_docker_image": install_deps : Install pip packages without deps (--no-deps) ------------ 0.32s
Step #2 - "build_development_docker_image": configure_env : Append environment variables required during runtime to ~/.zshrc --- 0.09s
Step #2 - "build_development_docker_image": Include vars from config files ------------------------------------------ 0.07s
Step #2 - "build_development_docker_image": configure_env : Append environment variables required during runtime to ~/.bashrc --- 0.06s
Step #2 - "build_development_docker_image": bazel : Tests ----------------------------------------------------------- 0.03s
Step #2 - "build_development_docker_image": Removing intermediate container d5ce950de2c8
Step #2 - "build_development_docker_image": ---> 75d8733f33ab
Step #2 - "build_development_docker_image": Step 10/10 : RUN ansible-playbook playbook.yaml -e "stage=release" -e "${ansible_vars}" --tags "${TAGS}"
Step #2 - "build_development_docker_image": ---> Running in 8666ef6ebe69
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": PLAY [Install build dependencies] **********************************************
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [Gathering Facts] *********************************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:54 +0000 (0:00:00.024) 0:00:00.024 *********
Step #2 - "build_development_docker_image": ok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [Include vars from config files] ******************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:55 +0000 (0:00:01.121) 0:00:01.146 *********
Step #2 - "build_development_docker_image": ok: [localhost] => (item=vars.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=cuda_deps.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=apt.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=pip.yaml)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=env.yaml)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Download bazelisk v1.15.0] ***************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:55 +0000 (0:00:00.074) 0:00:01.220 *********
Step #2 - "build_development_docker_image": ok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Tests] ***********************************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:56 +0000 (0:00:00.846) 0:00:02.066 *********
Step #2 - "build_development_docker_image": included: /ansible/roles/bazel/tasks/tests.yaml for localhost
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [bazel : Bazel --version runs succesfully] ********************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:56 +0000 (0:00:00.029) 0:00:02.096 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Add apt keys] *********************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:57 +0000 (0:00:00.582) 0:00:02.679 *********
Step #2 - "build_development_docker_image": ok: [localhost] => (item={'url': 'https://apt.llvm.org/llvm-snapshot.gpg.key', 'keyring': '/usr/share/keyrings/llvm.pgp'})
Step #2 - "build_development_docker_image": ok: [localhost] => (item={'url': 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub', 'keyring': '/usr/share/keyrings/cuda.pgp'})
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Add apt repositories into sources list] *******************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:50:58 +0000 (0:00:00.905) 0:00:03.584 *********
Step #2 - "build_development_docker_image": ok: [localhost] => (item=deb [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=deb-src [signed-by=/usr/share/keyrings/llvm.pgp] http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye-17 main)
Step #2 - "build_development_docker_image": ok: [localhost] => (item=deb [signed-by=/usr/share/keyrings/cuda.pgp] https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/ /)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Update apt cache] *****************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:00 +0000 (0:00:02.605) 0:00:06.190 *********
Step #2 - "build_development_docker_image": skipping: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install apt packages] *************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:00 +0000 (0:00:00.024) 0:00:06.214 *********
Step #2 - "build_development_docker_image": changed: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install pip packages] *************************************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:06 +0000 (0:00:05.576) 0:00:11.791 *********
Step #2 - "build_development_docker_image": ok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [install_deps : Install pip packages without deps (--no-deps)] ************
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:08 +0000 (0:00:01.496) 0:00:13.287 *********
Step #2 - "build_development_docker_image": �[91m[WARNING]: No valid name or requirements file found.
Step #2 - "build_development_docker_image": �[0mok: [localhost]
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [configure_env : Append environment variables required during runtime to ~/.bashrc] ***
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:08 +0000 (0:00:00.363) 0:00:13.650 *********
Step #2 - "build_development_docker_image": changed: [localhost] => (item=CC)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=CXX)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=LD_LIBRARY_PATH)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=TF_CUDA_COMPUTE_CAPABILITIES)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=XLA_CUDA)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": TASK [configure_env : Append environment variables required during runtime to ~/.zshrc] ***
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:09 +0000 (0:00:01.249) 0:00:14.899 *********
Step #2 - "build_development_docker_image": changed: [localhost] => (item=CC)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=CXX)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=LD_LIBRARY_PATH)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=TF_CUDA_COMPUTE_CAPABILITIES)
Step #2 - "build_development_docker_image": changed: [localhost] => (item=XLA_CUDA)
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": PLAY RECAP *********************************************************************
Step #2 - "build_development_docker_image": localhost : ok=12 changed=4 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Step #2 - "build_development_docker_image":
Step #2 - "build_development_docker_image": Wednesday 01 May 2024 19:51:10 +0000 (0:00:01.157) 0:00:16.057 *********
Step #2 - "build_development_docker_image": ===============================================================================
Step #2 - "build_development_docker_image": install_deps : Install apt packages ------------------------------------- 5.58s
Step #2 - "build_development_docker_image": install_deps : Add apt repositories into sources list ------------------- 2.61s
Step #2 - "build_development_docker_image": install_deps : Install pip packages ------------------------------------- 1.50s
Step #2 - "build_development_docker_image": configure_env : Append environment variables required during runtime to ~/.bashrc --- 1.25s
Step #2 - "build_development_docker_image": configure_env : Append environment variables required during runtime to ~/.zshrc --- 1.16s
Step #2 - "build_development_docker_image": Gathering Facts --------------------------------------------------------- 1.12s
Step #2 - "build_development_docker_image": install_deps : Add apt keys --------------------------------------------- 0.91s
Step #2 - "build_development_docker_image": bazel : Download bazelisk v1.15.0 --------------------------------------- 0.85s
Step #2 - "build_development_docker_image": bazel : Bazel --version runs succesfully -------------------------------- 0.58s
Step #2 - "build_development_docker_image": install_deps : Install pip packages without deps (--no-deps) ------------ 0.36s
Step #2 - "build_development_docker_image": Include vars from config files ------------------------------------------ 0.07s
Step #2 - "build_development_docker_image": bazel : Tests ----------------------------------------------------------- 0.03s
Step #2 - "build_development_docker_image": install_deps : Update apt cache ----------------------------------------- 0.02s
Step #2 - "build_development_docker_image": Removing intermediate container 8666ef6ebe69
Step #2 - "build_development_docker_image": ---> 74b80b3ac315
Step #2 - "build_development_docker_image": Successfully built 74b80b3ac315
Step #2 - "build_development_docker_image": Successfully tagged us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:cuda
Step #2 - "build_development_docker_image": Successfully tagged us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:3.10_cuda_12.3
Step #2 - "build_development_docker_image": Successfully tagged us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:3.10_cuda_12.3_20240501
Step #2 - "build_development_docker_image": Successfully tagged local_image:latest
Finished Step #2 - "build_development_docker_image"
Starting Step #3 - "push_development_docker_image"
Step #3 - "push_development_docker_image": Already have image (with digest): gcr.io/cloud-builders/docker
Step #3 - "push_development_docker_image": The push refers to repository [us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development]
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Preparing
Step #3 - "push_development_docker_image": b9109473f507: Preparing
Step #3 - "push_development_docker_image": 1060896a6be8: Preparing
Step #3 - "push_development_docker_image": c4f44703eb27: Preparing
Step #3 - "push_development_docker_image": a105bc96d154: Preparing
Step #3 - "push_development_docker_image": 1e15f07af706: Preparing
Step #3 - "push_development_docker_image": 392f916c39cc: Preparing
Step #3 - "push_development_docker_image": 8aec4a886a70: Preparing
Step #3 - "push_development_docker_image": c4c533a462b4: Preparing
Step #3 - "push_development_docker_image": f50eca7ee9ab: Preparing
Step #3 - "push_development_docker_image": 04a325c21dbd: Preparing
Step #3 - "push_development_docker_image": 9dcbabc40b87: Preparing
Step #3 - "push_development_docker_image": 1e15f07af706: Waiting
Step #3 - "push_development_docker_image": 392f916c39cc: Waiting
Step #3 - "push_development_docker_image": 8aec4a886a70: Waiting
Step #3 - "push_development_docker_image": c4c533a462b4: Waiting
Step #3 - "push_development_docker_image": f50eca7ee9ab: Waiting
Step #3 - "push_development_docker_image": 04a325c21dbd: Waiting
Step #3 - "push_development_docker_image": 9dcbabc40b87: Waiting
Step #3 - "push_development_docker_image": a105bc96d154: Layer already exists
Step #3 - "push_development_docker_image": 1e15f07af706: Layer already exists
Step #3 - "push_development_docker_image": 392f916c39cc: Layer already exists
Step #3 - "push_development_docker_image": 1060896a6be8: Pushed
Step #3 - "push_development_docker_image": 8aec4a886a70: Layer already exists
Step #3 - "push_development_docker_image": c4c533a462b4: Layer already exists
Step #3 - "push_development_docker_image": f50eca7ee9ab: Layer already exists
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Pushed
Step #3 - "push_development_docker_image": 04a325c21dbd: Layer already exists
Step #3 - "push_development_docker_image": 9dcbabc40b87: Layer already exists
Step #3 - "push_development_docker_image": c4f44703eb27: Pushed
Step #3 - "push_development_docker_image": b9109473f507: Pushed
Step #3 - "push_development_docker_image": 3.10_cuda_12.3: digest: sha256:f2ebe3f72678f78c9b07b0209365f49ee6ac8a284b2dca917a0806129a525514 size: 2853
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Preparing
Step #3 - "push_development_docker_image": b9109473f507: Preparing
Step #3 - "push_development_docker_image": 1060896a6be8: Preparing
Step #3 - "push_development_docker_image": c4f44703eb27: Preparing
Step #3 - "push_development_docker_image": a105bc96d154: Preparing
Step #3 - "push_development_docker_image": 1e15f07af706: Preparing
Step #3 - "push_development_docker_image": 392f916c39cc: Preparing
Step #3 - "push_development_docker_image": 8aec4a886a70: Preparing
Step #3 - "push_development_docker_image": c4c533a462b4: Preparing
Step #3 - "push_development_docker_image": f50eca7ee9ab: Preparing
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Layer already exists
Step #3 - "push_development_docker_image": 04a325c21dbd: Preparing
Step #3 - "push_development_docker_image": 9dcbabc40b87: Preparing
Step #3 - "push_development_docker_image": b9109473f507: Layer already exists
Step #3 - "push_development_docker_image": f50eca7ee9ab: Waiting
Step #3 - "push_development_docker_image": 1e15f07af706: Waiting
Step #3 - "push_development_docker_image": 1060896a6be8: Layer already exists
Step #3 - "push_development_docker_image": 392f916c39cc: Waiting
Step #3 - "push_development_docker_image": c4f44703eb27: Layer already exists
Step #3 - "push_development_docker_image": 9dcbabc40b87: Waiting
Step #3 - "push_development_docker_image": 8aec4a886a70: Waiting
Step #3 - "push_development_docker_image": c4c533a462b4: Waiting
Step #3 - "push_development_docker_image": 04a325c21dbd: Waiting
Step #3 - "push_development_docker_image": a105bc96d154: Layer already exists
Step #3 - "push_development_docker_image": 1e15f07af706: Layer already exists
Step #3 - "push_development_docker_image": 392f916c39cc: Layer already exists
Step #3 - "push_development_docker_image": 8aec4a886a70: Layer already exists
Step #3 - "push_development_docker_image": f50eca7ee9ab: Layer already exists
Step #3 - "push_development_docker_image": 04a325c21dbd: Layer already exists
Step #3 - "push_development_docker_image": 9dcbabc40b87: Layer already exists
Step #3 - "push_development_docker_image": c4c533a462b4: Layer already exists
Step #3 - "push_development_docker_image": 3.10_cuda_12.3_20240501: digest: sha256:f2ebe3f72678f78c9b07b0209365f49ee6ac8a284b2dca917a0806129a525514 size: 2853
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Preparing
Step #3 - "push_development_docker_image": b9109473f507: Preparing
Step #3 - "push_development_docker_image": 1060896a6be8: Preparing
Step #3 - "push_development_docker_image": c4f44703eb27: Preparing
Step #3 - "push_development_docker_image": a105bc96d154: Preparing
Step #3 - "push_development_docker_image": 1e15f07af706: Preparing
Step #3 - "push_development_docker_image": 392f916c39cc: Preparing
Step #3 - "push_development_docker_image": 8aec4a886a70: Preparing
Step #3 - "push_development_docker_image": c4c533a462b4: Preparing
Step #3 - "push_development_docker_image": f50eca7ee9ab: Preparing
Step #3 - "push_development_docker_image": 04a325c21dbd: Preparing
Step #3 - "push_development_docker_image": 9dcbabc40b87: Preparing
Step #3 - "push_development_docker_image": 4b9d4b7a4e5f: Layer already exists
Step #3 - "push_development_docker_image": 1060896a6be8: Layer already exists
Step #3 - "push_development_docker_image": c4f44703eb27: Layer already exists
Step #3 - "push_development_docker_image": c4c533a462b4: Waiting
Step #3 - "push_development_docker_image": 392f916c39cc: Layer already exists
Step #3 - "push_development_docker_image": f50eca7ee9ab: Waiting
Step #3 - "push_development_docker_image": 8aec4a886a70: Waiting
Step #3 - "push_development_docker_image": 8aec4a886a70: Layer already exists
Step #3 - "push_development_docker_image": a105bc96d154: Layer already exists
Step #3 - "push_development_docker_image": 04a325c21dbd: Waiting
Step #3 - "push_development_docker_image": 9dcbabc40b87: Waiting
Step #3 - "push_development_docker_image": 9dcbabc40b87: Layer already exists
Step #3 - "push_development_docker_image": f50eca7ee9ab: Layer already exists
Step #3 - "push_development_docker_image": c4c533a462b4: Layer already exists
Step #3 - "push_development_docker_image": 04a325c21dbd: Layer already exists
Step #3 - "push_development_docker_image": b9109473f507: Layer already exists
Step #3 - "push_development_docker_image": 1e15f07af706: Layer already exists
Step #3 - "push_development_docker_image": cuda: digest: sha256:f2ebe3f72678f78c9b07b0209365f49ee6ac8a284b2dca917a0806129a525514 size: 2853
Finished Step #3 - "push_development_docker_image"
PUSH
DONE
Loading