feat(.dockerignore): revise `.dockerignore` to correctly ignore copying unrelated files #5009

youtalk · 2024-07-18T11:03:38Z

Description

This is the first countermeasure raised in the dicussion https://github.com/orgs/autowarefoundation/discussions/5007.

I found that the current .dockerignore does not properly ignore the copying of unrelated files at all. Therefore, this PR revises the .dockerignore to prevent unnecessary files under src from being copied into the container image.

As a result, the compressed image size of devel has been reduced by 1.5GB. The most effective one is .git.

In addition, since .dockerignore has been properly configured, I believe that executing the COPY command will allow the layer cache to function correctly.

Tests performed

Before revising the .dockerignore:

After revising the .dockerignore:

Effects on system behavior

Not applicable.

Interface changes

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: Yutaka Kondo <[email protected]>

youtalk · 2024-07-18T11:21:41Z

The list of extensions to ignore was selected for larger sizes using the following one-liner.

$ find src/universe -type d -name ".git" -prune -o -type f -exec du -hs {} + | sort -rh | head -n 100
27M     src/universe/autoware.universe/common/autoware_test_utils/test_map/pointcloud_map.pcd
19M     src/universe/external/llh_converter/data/gsigeo2011_ver2_1.asc
19M     src/universe/external/eagleye/eagleye_core/coordinate/data/gsigeo2011_ver2.asc
17M     src/universe/autoware.universe/localization/pose_instability_detector/media/pose_instabilty_detector_procedure.svg
17M     src/universe/autoware.universe/control/autoware_smart_mpc_trajectory_follower/autoware_smart_mpc_trajectory_follower/python_simulator/slalom_course_data.csv
8.6M    src/universe/autoware.universe/perception/detected_object_validation/image/obstacle_pointcloud_based_validator/debug_image.gif
5.1M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_static_obstacle_avoidance_module/images/how_to_decide_path_shape_one_object.drawio.svg
3.9M    src/universe/autoware.universe/perception/probabilistic_occupancy_grid_map/image/pointcloud_based_occupancy_grid_map_sample_image.gif
3.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_static_obstacle_avoidance_module/images/shift_length_parameters.drawio.svg
3.1M    src/universe/autoware.universe/planning/behavior_velocity_planner/autoware_behavior_velocity_intersection_module/docs/static-occlusion-timeout.png
2.7M    src/universe/autoware.universe/planning/autoware_static_centerline_generator/test/data/bag_ego_trajectory_turn-left-right.db3
2.6M    src/universe/autoware.universe/planning/autoware_static_centerline_generator/test/data/bag_ego_trajectory_turn-right.db3
2.4M    src/universe/autoware.universe/planning/behavior_velocity_planner/autoware_behavior_velocity_intersection_module/docs/ttc.gif
2.2M    src/universe/external/muSSP/input_MOT_seq07_followme.txt
2.2M    src/universe/autoware.universe/planning/behavior_velocity_planner/autoware_behavior_velocity_run_out_module/docs/collision_detection_for_shape.svg
2.2M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_lane_change_module/images/lane_change-enable_collision_check_at_prepare_phase.png
2.1M    src/universe/autoware.universe/perception/traffic_light_visualization/images/roi-visualization.png
2.1M    src/universe/autoware.universe/docs/assets/images/autoware_universe_front.png
2.0M    src/universe/autoware.universe/perception/occupancy_grid_map_outlier_filter/image/occupancy_grid_map_outlier_filter.drawio.svg
2.0M    src/universe/autoware.universe/perception/ground_segmentation/test/data/test.pcd
1.7M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_static_obstacle_avoidance_module/images/how_to_decide_path_shape_multi_object_one_direction.drawio.svg
1.7M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_static_obstacle_avoidance_module/images/how_to_decide_path_shape_multi_object_both_direction.drawio.svg
1.6M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-threads.drawio.svg
1.6M    src/universe/autoware.universe/perception/traffic_light_map_based_detector/docs/traffic_light_map_based_detector_result.svg
1.5M    src/universe/autoware.universe/localization/autoware_landmark_based_localizer/doc_image/principle.png
1.5M    src/universe/autoware.universe/localization/autoware_landmark_based_localizer/autoware_ar_tag_based_localizer/doc_image/sample_result_in_awsim.png
1.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_planner/image/limitations/limitation-chattering_shifts.png
1.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-refined_goal.drawio.svg
1.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-goal_distance.drawio.svg
1.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-goal_candidates.drawio.svg
1.4M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-collision_check_margin.drawio.svg
1.3M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-safety_check.drawio.svg
1.3M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-object_to_avoid.drawio.svg
1.3M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-longitudinal_margin.drawio.svg
1.3M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-is_safe.drawio.svg
1.3M    src/universe/autoware.universe/planning/behavior_path_planner/autoware_behavior_path_goal_planner_module/images/goal_planner-deciding_path.drawio.svg
1.3M    src/universe/autoware.universe/control/autoware_smart_mpc_trajectory_follower/images/autoware_smart_mpc.png
1.2M    src/universe/autoware.universe/tools/reaction_analyzer/media/sc2-awsim.png
1.2M    src/universe/autoware.universe/tools/reaction_analyzer/media/sc1-awsim.png
1.1M    src/universe/external/ndt_omp/data/251371071.pcd
1.1M    src/universe/external/ndt_omp/data/251370668.pcd
1016K   src/universe/external/muSSP/muSSP/cmake-build-debug/muSSP
1004K   src/universe/autoware.universe/planning/behavior_velocity_planner/autoware_behavior_velocity_run_out_module/docs/create_dynamic_obstacle.svg
...

Signed-off-by: Yutaka Kondo <[email protected]>

oguzkaganozt · 2024-07-18T12:36:16Z

Looking good @youtalk, but the caching needs to be tested after this merged. Thanks !

youtalk · 2024-07-18T21:17:35Z

The first time of the docker-build-and-push was finally success. https://github.com/autowarefoundation/autoware/actions/runs/9991232729
The second one is also being executed to make sure.
https://github.com/autowarefoundation/autoware/actions/runs/9998728294

The autoware-universe and devel image sizes are significantly slimming up by more than 1.5GB!
https://autowarefoundation.github.io/autoware-ci-metrics/

@oguzkaganozt It's time to review please.

youtalk · 2024-07-18T21:29:27Z

@xmfcx Please review this too. The heal-check was success in this case.

xmfcx · 2024-07-18T21:31:39Z

https://github.com/autowarefoundation/autoware/actions/runs/9998715168/job/27638240438?pr=5009#step:5:779

colcon build command on cuda version was cached and not executed. Is this normal?

Shouldn't it actually build it?

And non-cuda version is still building https://github.com/autowarefoundation/autoware/actions/runs/9998715168/job/27638240136?pr=5009

xmfcx · 2024-07-18T21:32:16Z

feels suspicious 🤔

youtalk · 2024-07-18T21:45:52Z

The layer cache of the docker build is now functioning correctly. Previously, the .git directory of all repositories under src was unintentionally being copied, causing the layer hash to change and triggering a rebuild whenever src was copied. I think this was due to some files under the .git directory having their hashes changed just by running git clone.

However, since this PR, .git is now ignored, preventing the unintentional copying of such files, and thus the layer cache works properly. The health-check's cuda job finished instantly because all layers utilized the build cache. On the other hand, the no-cuda job likely reran colcon build because the hash of some copied files changed.

This behavior is intentional for me, as stated in the PR description.

In addition, since .dockerignore has been properly configured, I believe that executing the COPY command will allow the layer cache to function correctly.

xmfcx · 2024-07-18T21:53:38Z

On the other hand, the no-cuda job likely reran colcon build because the hash of some copied files changed.

But how can no-cuda job run again while cuda job is cached?

Same hashes should affect both of them.

Maybe it was cached here: https://github.com/autowarefoundation/autoware/actions/runs/9998728294/job/27638271926

Anyway, let's merge and see, thank you!

youtalk · 2024-07-18T21:57:59Z

@xmfcx The build cache is different between cuda and no-cuda jobs. That's why their behaviors can be different.
https://github.com/autowarefoundation/autoware/pkgs/container/autoware-buildcache

youtalk added 2 commits July 17, 2024 21:18

fix image organization

0aed5bc

Signed-off-by: Yutaka Kondo <[email protected]>

disable cache (#77)

e600c5e

Signed-off-by: Yutaka Kondo <[email protected]>

youtalk self-assigned this Jul 18, 2024

youtalk added the type:containers Docker containers, containerization of components, or container orchestration. label Jul 18, 2024

youtalk changed the title ~~feat(.dockerignore): Revise .dockerignore to correctly ignore copying unrelated src files~~ feat(.dockerignore): Revise .dockerignore to correctly ignore copying unrelated files Jul 18, 2024

youtalk added 4 commits July 18, 2024 21:09

fix .dockerignore

922e379

Signed-off-by: Yutaka Kondo <[email protected]>

not ignore etc

c1713b0

Signed-off-by: Yutaka Kondo <[email protected]>

Update .dockerignore

357ce43

Update .dockerignore

a4bca27

youtalk force-pushed the upstream-update-dockerignore branch from efb731c to a4bca27 Compare July 18, 2024 12:09

xmfcx added the tag:run-health-check Run health-check label Jul 18, 2024

youtalk changed the title ~~feat(.dockerignore): Revise .dockerignore to correctly ignore copying unrelated files~~ feat(.dockerignore): revise .dockerignore to correctly ignore copying unrelated files Jul 18, 2024

Merge branch 'main' into upstream-update-dockerignore

ceccfb3

youtalk marked this pull request as ready for review July 18, 2024 21:12

youtalk requested review from oguzkaganozt, mitsudome-r and xmfcx as code owners July 18, 2024 21:12

youtalk mentioned this pull request Jul 18, 2024

feat(ci,docker): disable var-lib-apt cache #5004

Merged

5 tasks

xmfcx approved these changes Jul 18, 2024

View reviewed changes

xmfcx merged commit eac84a4 into main Jul 18, 2024
20 of 21 checks passed

xmfcx deleted the upstream-update-dockerignore branch July 18, 2024 21:54

tier4-autoware-public-bot bot mentioned this pull request Jul 19, 2024

chore: sync upstream tier4/autoware#5

Open

youtalk mentioned this pull request Jul 24, 2024

feat(docker): copy src by --mount=type=bind instead of COPY #5027

Merged

4 tasks

youtalk mentioned this pull request Sep 6, 2024

fix(.dockerignore): not ignore specific csv and png files #5187

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(.dockerignore): revise `.dockerignore` to correctly ignore copying unrelated files #5009

feat(.dockerignore): revise `.dockerignore` to correctly ignore copying unrelated files #5009

youtalk commented Jul 18, 2024 •

edited by xmfcx

Loading

youtalk commented Jul 18, 2024

oguzkaganozt commented Jul 18, 2024

youtalk commented Jul 18, 2024

youtalk commented Jul 18, 2024

xmfcx commented Jul 18, 2024

xmfcx commented Jul 18, 2024

youtalk commented Jul 18, 2024

xmfcx commented Jul 18, 2024

youtalk commented Jul 18, 2024

feat(.dockerignore): revise .dockerignore to correctly ignore copying unrelated files #5009

feat(.dockerignore): revise .dockerignore to correctly ignore copying unrelated files #5009

Conversation

youtalk commented Jul 18, 2024 • edited by xmfcx Loading

Description

Tests performed

Effects on system behavior

Interface changes

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

youtalk commented Jul 18, 2024

oguzkaganozt commented Jul 18, 2024

youtalk commented Jul 18, 2024

youtalk commented Jul 18, 2024

xmfcx commented Jul 18, 2024

xmfcx commented Jul 18, 2024

youtalk commented Jul 18, 2024

xmfcx commented Jul 18, 2024

youtalk commented Jul 18, 2024

feat(.dockerignore): revise `.dockerignore` to correctly ignore copying unrelated files #5009

feat(.dockerignore): revise `.dockerignore` to correctly ignore copying unrelated files #5009

youtalk commented Jul 18, 2024 •

edited by xmfcx

Loading