Support ARM64

[androidacy-user]

Self-Hosted Version

n/a

CPU Architecture

arm64

Docker Version

20.10.12

Docker Compose Version

1.29.2

Steps to Reproduce

1. Try to run ./install.sh on a relatively fresh install of ubuntu 22.04
2. Install fails

Expected Result

Install succeeds

Actual Result

Hits an error:

Step 4/5 : RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 2ae646b00261
exec /bin/sh: exec format error

Note we don't have an enhance-image.sh anywhere in the directory

[aminvakil]

@ethanhs Can getsentry/sentry docker image run on a host with an arm64 cpu?

It may also help if to you try running ./install.sh again like this DEBUG=1 ./install.sh and paste the output in a gist.

[emmatyping]

Can getsentry/sentry docker image run on a host with an arm64 cpu?

No I think on macOS it is getting run by Rosetta maybe? I'm a bit uncertain. Theoretically it is unlikely that any of its dependencies won't work on ARM, I will ask around to see how feasible it is to build native ARM images.

[androidacy-user]

We're not on MacOS but Ubuntu 22.04 running on GCP.

@aminvakil would that even help? I think it's pretty obvious, the image used to run that script isn't respecting the system architecture and as such failing to run as it defaults to amd64

[aminvakil]

@androidacy-user What's the output of this command: docker info --format '{{.Architecture}}'?

[androidacy-user]

aarch64 as expected

[emmatyping]

I'll try to reproduce and fix this later.

[emmatyping]

Hm, unfortunately I think we will be stalled on this until we upgrade zookeeper, ref #1292 as confluence's zookeeper image seems not to support ARM until the 7.x series.

[licstar]

Hm, unfortunately I think we will be stalled on this until we upgrade zookeeper, ref #1292 as confluence's zookeeper image seems not to support ARM until the 7.x series.

Can I replace zookeeper to some arm64 version? such as watershine

[emmatyping]

We use a rather old version of zookeeper, I'm not sure if it would be safe to upgrade but I'm looking into it.

[licstar]

I successfully run sentry on an ARM server, using this third-party version: https://github.com/Sentry-ARM/onpremise

[emmatyping]

Note that we don't support third-party repos for deployment. I do want to add aarch64 for Linux at some point, but there are several changes we'd need to make to see that happen.

[nickjmv]

We have also run into ARM issues while running the ./install.sh script. Apparently the getsentry/relay docker image is not built for ARM, only for AMD. We opened a separate issue for this: #1642

[armenzg]

I think I have code to build an arm64 build of Sentry in getsentry/sentry#38180

[checkolala]

Hello,

we are facing the same issue on aws graviton (aarch64) while running ./install.sh
seems, that the current sentry-image doesn't provide the platform aarch64

#7 [3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi
#7 0.490 exec /bin/sh: exec format error
#7 ERROR: executor failed running [/bin/sh -c if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi]: exit code: 1
------
 > [3/4] RUN if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi:
#7 0.490 exec /bin/sh: exec format error
------
failed to solve: executor failed running [/bin/sh -c if [ -s /usr/src/sentry/enhance-image.sh ]; then     /usr/src/sentry/enhance-image.sh; fi]: exit code: 1
Error in build-docker-images.sh:6.
'$dc build --build-arg "http_proxy=${http_proxy:-}" --build-arg "https_proxy=${https_proxy:-}" --build-arg "no_proxy=${no_proxy:-}" --force-rm web' exited with status 17
-> ./install.sh:main:30
--> build-docker-images.sh:source:6

[johnsturgeon]

Getting the following error trying to build on my M1 Mac

container for service "zookeeper" is unhealthy
Error in bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:33
--> bootstrap-snuba.sh:source:3

Cleaning up...

[emmatyping]

@johnsturgeon I believe that is an inconsistent issue that only happens some of the time, if you try again does it happen?

[johnsturgeon]

I tried again, and it did happen again.

I got around it by just doing a docker compose up that fired up quite a few containers...
then docker compose down
then redid the install.sh

[johnsturgeon]

The only problem I have now is whenever I start the stack the nginx container usually does not start, so I have to stop it / and the web container and restart them. That usually gets it going

[johnsturgeon]

This is the normal state after I start the stack

[hubertdeng123]

note: Maybe M1 Max vs M1 Pro is causing issues with compatibility

I have to comment out this for self-hosted to work on M1 Max in docker-compose.yml while Chad and Ethan do not.

platform: ${DOCKER_PLATFORM:-} 

[aldy505]

An update:

• Relay have arm64 version already from a while ago
• Symbolicator now have arm64 version on the GitHub Container Registry. DockerHub soon. https://github.com/getsentry/symbolicator/pkgs/container/symbolicator/340834879?tag=latest
• Vroom PR is here ci: provide arm64 image build vroom#552
• Snuba PR is here ci: provide arm64 image snuba#6803, but closed due to waiting for the GCR migration (see ref(ci): Refactor self-hosted e2e and dockerhub steps to not use GCR image snuba#6800). New PR is here ci: provide arm64 image build snuba#6825
• Sentry PR is here ci: provide arm64 image build sentry#83907

More things to do (I hope people subscribed to this issue can help):

• Find an alternative for tianon/exim4 that supports multiarch

  self-hosted/docker-compose.yml

  Lines 88 to 91 in d637ed1

  	smtp:
  	<<: *restart_policy
  	platform: linux/amd64
  	image: tianon/exim4

• Make sure all the dependencies supports multiarch (linux/amd64 and linux/arm64)

  self-hosted/docker-compose.yml

  Line 98 in d637ed1

  image: "memcached:1.6.26-alpine"

  self-hosted/docker-compose.yml

  Line 106 in d637ed1

  image: "redis:6.2.14-alpine"

  self-hosted/docker-compose.yml

  Line 124 in d637ed1

  image: "postgres:14.11"

  self-hosted/docker-compose.yml

  Line 141 in d637ed1

  image: "confluentinc/cp-kafka:7.6.1"

  self-hosted/docker-compose.yml

  Line 451 in d637ed1

  image: "nginx:1.25.4-alpine"

  self-hosted/docker-compose.yml

  Line 182 in d637ed1

  BASE_IMAGE: "altinity/clickhouse-server:23.8.11.29.altinitystable"

[ezhevita]

@aldy505

Make sure all the dependencies supports multiarch (linux/amd64 and linux/arm64)

• memcached:1.6.26-alpine: supported
• redis:6.2.14-alpine: supported
• postgres:14.11: supported
• confluentinc/cp-kafka:7.6.1: supported
• nginx:1.25.4-alpine: supported
• altinity/clickhouse-server:23.8.11.29.altinitystable: supported (arm64/v8 only)

[ezhevita]

Users report having hardware issues with arm64 GHA runners, so we might have to wait?
rust-lang/rust#135867
https://github.com/orgs/community/discussions/149715
actions/partner-runner-images#36
rust-lang/rust#135939

[aldy505]

Users report having hardware issues with arm64 GHA runners, so we might have to wait? rust-lang/rust#135867 https://github.com/orgs/community/discussions/149715 actions/partner-runner-images#36 rust-lang/rust#135939

Not sure, since we're using GHA runners with Docker instead of Rust directly. So far it's working well for symbolicator and vroom

[aldy505]

Also bumping this again:

People, we need to find an alternative for tianon/exim4 that supports multiarch. Any suggestions?

self-hosted/docker-compose.yml

Lines 88 to 91 in d637ed1

	smtp:
	<<: *restart_policy
	platform: linux/amd64
	image: tianon/exim4

[androidacy-user]

Also bumping this again:

People, we need to find an alternative for tianon/exim4 that supports multiarch. Any suggestions?

self-hosted/docker-compose.yml

Lines 88 to 91 in d637ed1

	smtp:
	<<: *restart_policy
	platform: linux/amd64
	image: tianon/exim4

Their recommendation for a replacement is https://wiki.archlinux.org/title/Msmtp. Could try and install that in the relevant container I suppose.

[peterdk]

Also bumping this again:

People, we need to find an alternative for tianon/exim4 that supports multiarch. Any suggestions?

self-hosted/docker-compose.yml

Lines 88 to 91 in d637ed1
smtp:
<<: *restart_policy
platform: linux/amd64
image: tianon/exim4

Should be fairly simple to build it for arm64 and upload it to docker? Until sentry itself also deprecates it? Or is it arm64 incompatible?

[aldy505]

Update: There will be delays.

Quoting from @bc-sentry (engineering manager of devinfra):

Thanks for reaching out regarding this effort. Our concern is that we are already bumping up against the limits of the Github runners free pool. Since we would need to build arm64 images for a number of repos, this could exacerbate that issue. We are looking into a project to add capacity by introducing self-hosted runners, but we have to build a proof-of-concept and test that before committing to it. As a result, this would not be available in the near term. I would recommend that we wait to introduce the arm64 images until the runner issue is resolved. I understand the community will be disappointed with this delay, but I'm not certain of a viable alternative.

The issue is it that this will double our image builds and double our test runs. The free pool of runners is shared across all of our repos. We are seeing a sizable percentage of GHA jobs getting queued due to runner exhaustion. We plan on working on this runner capacity issue in the first half of 2025, so it would be best if you waited for that to happen.

One other thing to mention is that while arm64 is in beta the global pool is severely constrained and will result in queuing even if we haven't exhausted ours. So let's circle back when arm64 runners are out of beta.

[halettedurand]

Our concern is that we are already bumping up against the limits of the Github runners free pool.

LOL at running a VC backed SaaS and not being willing to pay for someone else's services.

[clarity99]

this is indeed disappointing :( How about at least getting it ready so we can build it ourselves? the above mentioned https://github.com/Sentry-ARM/self-hosted seems to work well, although it is now a bit out of date. Didn't seem to be so many changes to get it working.