Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate docker build issue on linux/arm64 platform #846

Closed
masih opened this issue Jan 24, 2025 · 6 comments · Fixed by #853 or #857
Closed

Investigate docker build issue on linux/arm64 platform #846

masih opened this issue Jan 24, 2025 · 6 comments · Fixed by #853 or #857
Assignees

Comments

@masih
Copy link
Member

masih commented Jan 24, 2025

Initially observed in #834 but it is unrelated to the changes in that PR: confirmed by re-running the docker build on latest commit of main here. The issue:

> [linux/arm64 build 6/6] RUN go build -o /go/bin/f3 ./cmd/f3:
87.02 runtime/cgo: gcc: signal: segmentation fault (core dumped)
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:9
--------------------
   7 |     
   8 |     COPY . .
   9 | >>> RUN go build -o /go/bin/f3 ./cmd/f3
  10 |     
  11 |     FROM gcr.io/distroless/cc
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c go build -o /go/bin/f3 ./cmd/f3" did not complete successfully: exit code: 1

Why has it started failing now?

Because go-f3 container build is not reproducible:

  • the dockerfile uses latest tags for the build and base images.
  • the GitHub actions use a soft version for the build setup.

Change at any of the above layers could have caused the issue, e.g. gcc linker upgrade.
Also, yesterday GitHub Actions was having a bad day with a flurry of errors specifically in docker build jobs which may be related.

Building multi-platform containers locally (using podman) passes for me with no issues.

Unblocking the merge of that PR by capturing this issue to investigate the problem separately.

@github-project-automation github-project-automation bot moved this to Todo in F3 Jan 24, 2025
@masih masih added this to the M2: Mainnet Passive Testing milestone Jan 24, 2025
@masih
Copy link
Member Author

masih commented Jan 27, 2025

Builds now started passing on main without any updates to docker build process. Also see: docker/build-push-action#1309

@masih masih self-assigned this Jan 27, 2025
@masih masih moved this from Todo to In progress in F3 Jan 27, 2025
masih added a commit that referenced this issue Jan 27, 2025
There is no reason not to use a more recent version of go to build the
binary in docker images.

This should also reduce the chances of random segmentation fault when
building for arm64 platform.

Fixes #846
@FacuM
Copy link

FacuM commented Jan 27, 2025

IMO it looks like the VM is running out of RAM and causing some weird behavior. It looks like I've found a workaround, as I've commented in my issue:

jobs:

  build:
    runs-on: ubuntu-22.04

Please try like that and let me know if it helps. :)

github-merge-queue bot pushed a commit that referenced this issue Jan 28, 2025
There is no reason not to use a more recent version of go to build the
binary in docker images.

This should also reduce the chances of random segmentation fault when
building for arm64 platform.

Fixes #846
@github-project-automation github-project-automation bot moved this from In progress to Done in F3 Jan 28, 2025
@masih masih reopened this Jan 28, 2025
@github-project-automation github-project-automation bot moved this from Done to Todo in F3 Jan 28, 2025
@masih
Copy link
Member Author

masih commented Jan 28, 2025

Thank you @FacuM, I'll give that a shot 👍

@masih
Copy link
Member Author

masih commented Jan 28, 2025

@FacuM In our case using the ubuntu-22.04 runner image made no difference. See: https://github.com/filecoin-project/go-f3/actions/runs/13007480659/job/36277473006?pr=855

@masih
Copy link
Member Author

masih commented Jan 28, 2025

@masih masih moved this from Todo to In progress in F3 Jan 28, 2025
@masih
Copy link
Member Author

masih commented Jan 28, 2025

Update using qemu v8 works.

@masih masih moved this from In progress to In review in F3 Jan 28, 2025
@github-project-automation github-project-automation bot moved this from In review to Done in F3 Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
2 participants