Dockerfile: update ALPINE_VERSION to 3.18 #4354

thaJeztah · 2023-06-14T18:48:21Z

relates to update go to go1.20.5, alpine 3.17 #4350
depends on e2e: update to use compose v2, and don't depend on distro-packages #4576
depends on Dockerfile: build gotestsum and goversioninfo without cgo #4577

This also moves musl-dev to the alpine-base stage, due to changes in
Alpine 3.18 causing gotestsum build to fail because stdlib.h was missing;

#17 5.065 # runtime/cgo
#17 5.065 In file included from _cgo_export.c:3:
#17 5.065 /usr/include/fortify/stdlib.h:23:15: fatal error: stdlib.h: No such file or directory
#17 5.065    23 | #include_next <stdlib.h>
#17 5.065       |               ^~~~~~~~~~

alpine 3.17:

/ # find / | grep stdlib.h
/usr/include/c++/12.2.1/tr1/stdlib.h
/usr/include/c++/12.2.1/stdlib.h

alpine 3.18

/ # find / | grep stdlib.h
/usr/lib/llvm16/lib/clang/16/include/__clang_hip_stdlib.h
/usr/include/fortify/stdlib.h
/usr/include/c++/12.2.1/tr1/stdlib.h
/usr/include/c++/12.2.1/stdlib.h

- What I did

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

thaJeztah · 2023-06-14T18:51:02Z

Ah, found the culprit; it's not Go, but the alpine upgrade;

#17 5.065 # runtime/cgo
#17 5.065 In file included from _cgo_export.c:3:
#17 5.065 /usr/include/fortify/stdlib.h:23:15: fatal error: stdlib.h: No such file or directory
#17 5.065    23 | #include_next <stdlib.h>
#17 5.065       |               ^~~~~~~~~~

the _cgo_export.c is from https://github.com/golang/go/blob/c5463218a228b082661df3f5f1ba0492a4d3df18/src/cmd/cgo/doc.go#L676-L689

Given the input Go files x.go and y.go, cgo generates these source
files:

	x.cgo1.go       # for gc (cmd/compile)
	y.cgo1.go       # for gc
	_cgo_gotypes.go # for gc
	_cgo_import.go  # for gc (if -dynout _cgo_import.go)
	x.cgo2.c        # for gcc
	y.cgo2.c        # for gcc
	_cgo_defun.c    # for gcc (if -gccgo)
	_cgo_export.c   # for gcc
	_cgo_export.h   # for gcc
	_cgo_main.c     # for gcc
	_cgo_flags      # for alternative build tools

codecov-commenter · 2023-06-14T18:53:32Z

Codecov Report

Merging #4354 (6a74a63) into master (49f7de7) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4354   +/-   ##
=======================================
  Coverage   59.75%   59.75%           
=======================================
  Files         287      287           
  Lines       24821    24821           
=======================================
  Hits        14832    14832           
  Misses       9103     9103           
  Partials      886      886

thaJeztah · 2023-06-15T07:48:17Z

It looks like there's a couple of options;

remove `clang` for building `gotestsum`

it looks like gotestsum doesn't need it, and the error only happens if clang is installed (we should try to understand why that changed in alpine 3.18 / newer clang?). I think we can move clang from build-base-alpine to build-alpine;

cli/Dockerfile

Lines 13 to 21 in 7d723e2

    
           FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine${ALPINE_VERSION} AS build-base-alpine 
        
           COPY --from=xx / / 
        
           RUN apk add --no-cache bash clang lld llvm file git 
        
           WORKDIR /go/src/github.com/docker/cli 
        
           FROM build-base-alpine AS build-alpine 
        
           ARG TARGETPLATFORM 
        
           # gcc is installed for libgcc only 
        
           RUN xx-apk add --no-cache musl-dev gcc

add `musl-dev` to `build-base-alpine`

Adding musl-dev brings the header

Reproduce

Here's a minimal dockerfile and build-args to test the variants;

# syntax=docker/dockerfile:1

ARG ALPINE_VERSION=3.17
ARG BASE=default

FROM golang:1.20-alpine${ALPINE_VERSION} AS base-no-clang

FROM base-no-clang AS base-default
RUN apk add --no-cache clang

FROM base-default AS base-clang-musl-dev
RUN apk add --no-cache musl-dev

FROM base-${BASE}
ARG GOTESTSUM_VERSION=v1.8.2
RUN --mount=type=cache,target=/go/pkg/mod \
    go install "gotest.tools/gotestsum@${GOTESTSUM_VERSION}" && gotestsum --version

x	ALPINE_VERSION	clang?	musl-dev?	cmd
✅	3.17	-	-	`docker build --build-arg BASE=no-clang .`
✅	3.17	✅	-	`docker build .`
✅	3.17	✅	✅	`docker build --build-arg BASE=clang-musl-dev .`
✅	3.18	-	-	`docker build --build-arg ALPINE_VERSION=3.18 --build-arg BASE=no-clang .`
❌	3.18	✅	-	`docker build --build-arg ALPINE_VERSION=3.18 .`
✅	3.18	✅	✅	`docker build --build-arg ALPINE_VERSION=3.18 --build-arg BASE=clang-musl-dev .`

thaJeztah · 2023-06-15T08:24:34Z

From a discussion on Slack;

after installing clang

alpine 3.17:

/ # find / | grep stdlib.h
/usr/include/c++/12.2.1/tr1/stdlib.h
/usr/include/c++/12.2.1/stdlib.h

alpine 3.18

/ # find / | grep stdlib.h
/usr/lib/llvm16/lib/clang/16/include/__clang_hip_stdlib.h
/usr/include/fortify/stdlib.h
/usr/include/c++/12.2.1/tr1/stdlib.h
/usr/include/c++/12.2.1/stdlib.h

thaJeztah · 2023-09-26T09:51:42Z

LOL, I was wondering why the e2e tests were failing, and was looking through logs, but couldn't find an actual failure there;

make -f docker.Makefile test-e2e-non-experimental

#6 [2/2] COPY ./notary/ /fixtures/
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:caf0815eaa7da5cd65244aa32318f904951ed74e2ee2c3c5bad2af2ed299d85e done
#7 naming to docker.io/library/cliendtoendsuite-notary-server done
#7 DONE 0.0s
 Container cliendtoendsuite-notary-server-1  Running
 Container cliendtoendsuite-evil-notary-server-1  Running
 Container cliendtoendsuite-registry-1  Running
 Container cliendtoendsuite-engine-1  Running
Error: No such object: cliendtoendsuite_engine_1
make: *** [docker.Makefile:142: test-e2e-non-experimental] Error 1

And it was staring me in the face:

 Container cliendtoendsuite-engine-1  Running
Error: No such object: cliendtoendsuite_engine_1

Looks like something in the e2e tests is expecting compose to be creating containers using underscores, and we're installing docker-compose from alpine's package repositories; and it looks like alpine 3.18 switched over to use compose v2:

On alpine 3.17:

make -f docker.Makefile build-e2e-image
docker run --rm docker-cli-e2e docker-compose --version
docker-compose version 1.29.2, build unknown

On alpine 3.18:

make -f docker.Makefile build-e2e-image
docker run --rm docker-cli-e2e docker-compose --version
Docker Compose version v2.17.3

thaJeztah · 2023-09-26T11:55:24Z

Dockerfile

-RUN apk add --no-cache bash clang lld llvm file git
+RUN apk add --no-cache bash clang lld llvm musl-dev file git
 WORKDIR /go/src/github.com/docker/cli

 FROM build-base-alpine AS build-alpine
 ARG TARGETPLATFORM
 # gcc is installed for libgcc only
-RUN xx-apk add --no-cache musl-dev gcc
+RUN xx-apk add --no-cache gcc


I worked around this by not using cgo for gotestsum, but wondering if we still need this, or something similar, but not sure if musl-dev must be installed through xx here, or if it's safe to install it using a plain apk add

/cc @crazy-max perhaps you have ideas here

This also moves `musl-dev` to the alpine-base stage, due to changes in Alpine 3.18 causing gotestsum build to fail because stdlib.h was missing; docker#17 5.065 # runtime/cgo docker#17 5.065 In file included from _cgo_export.c:3: docker#17 5.065 /usr/include/fortify/stdlib.h:23:15: fatal error: stdlib.h: No such file or directory docker#17 5.065 23 | #include_next <stdlib.h> docker#17 5.065 | ^~~~~~~~~~ alpine 3.17: / # find / | grep stdlib.h /usr/include/c++/12.2.1/tr1/stdlib.h /usr/include/c++/12.2.1/stdlib.h alpine 3.18 / # find / | grep stdlib.h /usr/lib/llvm16/lib/clang/16/include/__clang_hip_stdlib.h /usr/include/fortify/stdlib.h /usr/include/c++/12.2.1/tr1/stdlib.h /usr/include/c++/12.2.1/stdlib.h Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2023-11-23T22:43:09Z

Alright, this one is happy now as well; no longer had to change the musl-dev parts

/cc @crazy-max ptal 🤗

thaJeztah added status/2-code-review process/cherry-picked area/packaging labels Jun 14, 2023

thaJeztah added this to the 25.0.0 milestone Jun 14, 2023

This was referenced Jun 14, 2023

update go to go1.20.5, alpine 3.17 #4350

Merged

[24.0 backport] update go to go1.20.5, alpine 3.17 #4351

Merged

thaJeztah mentioned this pull request Jun 29, 2023

dockerfile: update Alpine to 3.18 moby/buildkit#3976

Merged

thaJeztah force-pushed the update_alpine_3.18 branch 3 times, most recently from 3f2b5c7 to e972730 Compare September 26, 2023 08:44

thaJeztah mentioned this pull request Sep 26, 2023

e2e: update to use compose v2, and don't depend on distro-packages #4576

Merged

thaJeztah force-pushed the update_alpine_3.18 branch 3 times, most recently from 6d8bb42 to 273c3ef Compare September 26, 2023 10:24

thaJeztah mentioned this pull request Sep 26, 2023

Dockerfile: build gotestsum and goversioninfo without cgo #4577

Merged

thaJeztah force-pushed the update_alpine_3.18 branch from 273c3ef to f945b5b Compare September 26, 2023 11:53

thaJeztah marked this pull request as draft September 26, 2023 11:53

thaJeztah commented Sep 26, 2023

View reviewed changes

thaJeztah force-pushed the update_alpine_3.18 branch from f945b5b to d2d4be9 Compare November 23, 2023 22:29

thaJeztah force-pushed the update_alpine_3.18 branch from d2d4be9 to 6a74a63 Compare November 23, 2023 22:37

thaJeztah marked this pull request as ready for review November 23, 2023 22:41

crazy-max approved these changes Nov 23, 2023

View reviewed changes

thaJeztah merged commit 6d820e8 into docker:master Nov 23, 2023
76 checks passed

thaJeztah deleted the update_alpine_3.18 branch November 23, 2023 22:51

This was referenced Jan 10, 2024

[24.0 backport] Dockerfile: update ALPINE_VERSION to 3.18 #4768

Merged

[24.0] update to go1.20.13 moby/moby#47054

Merged

thaJeztah mentioned this pull request Jan 24, 2024

[23.0] update to go1.20.13 #4825

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dockerfile: update ALPINE_VERSION to 3.18 #4354

Dockerfile: update ALPINE_VERSION to 3.18 #4354

thaJeztah commented Jun 14, 2023 •

edited

Loading

thaJeztah commented Jun 14, 2023 •

edited

Loading

codecov-commenter commented Jun 14, 2023 •

edited

Loading

thaJeztah commented Jun 15, 2023

thaJeztah commented Jun 15, 2023

thaJeztah commented Sep 26, 2023

thaJeztah Sep 26, 2023 •

edited

Loading

thaJeztah commented Nov 23, 2023

Dockerfile: update ALPINE_VERSION to 3.18 #4354

Dockerfile: update ALPINE_VERSION to 3.18 #4354

Conversation

thaJeztah commented Jun 14, 2023 • edited Loading

thaJeztah commented Jun 14, 2023 • edited Loading

codecov-commenter commented Jun 14, 2023 • edited Loading

Codecov Report

thaJeztah commented Jun 15, 2023

remove clang for building gotestsum

add musl-dev to build-base-alpine

Reproduce

thaJeztah commented Jun 15, 2023

thaJeztah commented Sep 26, 2023

thaJeztah Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

thaJeztah commented Nov 23, 2023

thaJeztah commented Jun 14, 2023 •

edited

Loading

thaJeztah commented Jun 14, 2023 •

edited

Loading

codecov-commenter commented Jun 14, 2023 •

edited

Loading

remove `clang` for building `gotestsum`

add `musl-dev` to `build-base-alpine`

thaJeztah Sep 26, 2023 •

edited

Loading