Container exits on MacOS / ARM64

[jryanearl]

Describe the bug

I am trying to start the latest-arm64 container on an M1 MacBook Pro that has Rancher Desktop installed and configured. I'm using compose to try and bring it up, and I get an error.

To Reproduce

Steps to reproduce the behavior:

1. On an Apple Silicon Mac, setup Rancher Desktop https://rancherdesktop.io
2. Choose the nerdctl replacement for docker.
3. 'nerdctl compose up' with the docker-compose.yaml provided below.

Expected behavior

Expect the container to run and PalWorld Server to update and run.

Actual behavior

It appears to call box86 instead of box64 when running steamcmd, I think. I was looking at this late and I need help, I think there's something broken in the underlying steamcmd-arm64 image.

$ nerdctl compose up
INFO[0000] Creating network jre_default                 
INFO[0000] Ensuring image thijsvanloef/palworld-server-docker:latest-arm64 
INFO[0000] Creating container Pal1                      
INFO[0000] Attaching to logs                            
Pal1 |*****EXECUTING USERMOD*****
INFO[0000] Container "Pal1" exited                      
INFO[0000] All the containers have exited               
INFO[0000] Stopping containers (forcibly)               
INFO[0000] Stopping container Pal1                      

Screenshots

OS information

• OS: macOS Sonoma 14.3.1
• Palworld Version: 1.4.1.0
• Docker Image Version:

Client:
 Version:	v1.7.3
 OS/Arch:	linux/arm64
 Git commit:	0a464409d0178e16d3d2bed36222937ec3fc9c77
 buildctl:
  Version:	v0.12.5
  GitCommit:	bac3f2b673f3f9d33e79046008e7a38e856b3dc6

Server:
 containerd:
  Version:	v1.7.2
  GitCommit:	0cae528dd6cb557f7201036e9f43420650207b58
 runc:
  Version:	1.1.12
  GitCommit:	51d5e94601ceffbbd85688df1c928ecccbfa4685

Hardware information

docker-compose.yml contents

services:
   palworld:
      image: thijsvanloef/palworld-server-docker:latest-arm64
      restart: always
      container_name: Pal1
      ports:
         - 25575:25575/tcp
         - 8215:8215/udp
         - 27015:27015/udp
      environment:
         - PUID=501
         - PGID=20
         - PUBLIC_IP=1.2.3.4
         - PUBLIC_PORT=8215
         - PORT=8215 # Optional but recommended
         - PLAYERS=16 # Optional but recommended
         - SERVER_PASSWORD="MacOS arm64 test" # Optional but recommended
         - MULTITHREADING=true
         - RCON_ENABLED=true
         - RCON_PORT=25575
         - TZ=UTC
         - ADMIN_PASSWORD="arm64-test"
         - COMMUNITY=true  # Enable this if you want your server to show up in the community servers tab, USE WITH SERVER_PASSWORD!
         - SERVER_NAME="ARM64 on Mac PalWorld"
         - REGION="NA"
         - SERVER_DESCRIPTION="MacOS Container Test - M1 Pro 32GB"
         - BACKUP_ENABLED=true
         - IS_MULTIPLAYER=true
         - BACKUP_CRON_EXPRESION="0 0 * * *"
         - AUTO_UPDATE_ENABLED=true
         - AUTO_UPDATE_CRON_EXPRESSION="0 * * * *"
         - AUTO_UPDATE_WARN_MINUTES=30
         - AUTO_REBOOT_ENABLED=true
         - AUTO_REBOOT_WARN_MINUTES=15
         - AUTO_REBOOT_CRON_EXPRESSION="0 * * * *"
         - UPDATE_ON_BOOT=true
      volumes:
         - /Users/jre/palworld:/palworld/

Container/Host Logs

*****EXECUTING USERMOD*****
usermod: no changes
*****GENERATING CONFIGS*****
Compiling PalWorldSettings.ini...
Compiling PalWorldSettings.ini done!
*****STARTING INSTALL/UPDATE*****
/home/steam/steamcmd/steamcmd.sh: line 38: /usr/local/bin/box86: cannot execute binary file: Exec format error
cp: cannot stat './PalServer.sh': No such file or directory
sed: can't read ./PalServer-arm64.sh: No such file or directory
chmod: cannot access './PalServer-arm64.sh': No such file or directory
./PalServer-arm64.sh does not exist.
Try restarting with UPDATE_ON_BOOT=true

Note UPDATE_ON_BOOT=true is in the docker-compose.yaml

Additional context

Rancher Desktop is a license unencumbered alternative to Docker Desktop and has better performance on MacOS supposedly. Anyway, it's free to use.

[sonroyaalmerol]

Box86 being called when running steamcmd is expected. The steamcmd executable is a 32-bit application, which is incompatible with Box64.

cannot execute binary file: Exec format error is an error you usually see when running an executable built on one architecture but executed on another. It might also have something to do with the platform being unable to run 32-bit executables.

Sadly, I don't have an M1 or Rancher Desktop to help you test things but I've seen people successfully using the latest-arm64 tag with Apple Silicon (#278). One thing I could suggest is try using the latest tag and let Rosetta do the emulation instead by adding platform: linux/amd64 to your compose file like so:

services:
   palworld:
      image: thijsvanloef/palworld-server-docker:latest
      platform: linux/amd64
      restart: always

[Phelms215]

Does it create a folder /palworld/backups ? Or /palworld/PalServer.sh for that matter?
This also might be fixed by #356

[jryanearl]

So this is honestly kind of crazy, I guess docker desktop is using Rosetta under the hood or something. Rancher Desktop is making a HVM for the container. box86 won't run on Apple Silicon... Could really use a 64-bit build of steamcmd. And MacOS has a native steamcmd is the other thing if that could somehow be leveraged.

Some captures:

root@palworld:/home/steam/server# file /usr/local/bin/box86
/usr/local/bin/box86: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=33966ad429a30e97eaf2a8a68db3bc098b9a2c95, stripped

root@palworld:/home/steam/server# file /usr/local/bin/box64
/usr/local/bin/box64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=1894a657c81ac7bac52469f02b3b005561c81e2a, with debug_info, not stripped```

root@palworld:/home/steam# box86 --help
bash: /usr/local/bin/box86: cannot execute binary file: Exec format error

root@palworld:/home/steam# box64 --help
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS SHA1 SHA2 PageSize:4096 Running on unknown aarch64 cpu with 5 Cores
Params database has 57 entries
This is Box64, The Linux x86_64 emulator with a twist

Usage is 'box64 [options] path/to/software [args]' to launch x86_64 software.
 options are:
    '-v'|'--version' to print box64 version and quit
    '-h'|'--help' to print this and quit
    '-f'|'--flags' to print box64 flags and quit

root@palworld:/home/steam/steamcmd/linux32# box86 steamcmd -h
bash: /usr/local/bin/box86: cannot execute binary file: Exec format error

root@palworld:/home/steam/steamcmd/linux32# box64 steamcmd -h
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS SHA1 SHA2 PageSize:4096 Running on unknown aarch64 cpu with 5 Cores
Params database has 57 entries
Box64 with Dynarec v0.2.7 fed47bea built on Feb 11 2024 06:20:00
BOX64: Detected 48bits at least of address space
Counted 51 Env var
BOX64 LIB PATH: ./:lib/:lib64/:x86_64/:bin64/:libs64/:/usr/lib/x86_64-linux-gnu/
BOX64 BIN PATH: ./:bin/:/usr/local/sbin/:/usr/local/bin/:/usr/sbin/:/usr/bin/:/sbin/:/bin/
Looking for steamcmd
argv[1]="-h"
This is a 32bits ELF! box64 can only run 64bits ELF (/home/steam/steamcmd/linux32/steamcmd)!
Error: Reading elf header of /home/steam/steamcmd/linux32/steamcmd, Try to launch using box86 instead
/usr/local/bin/box86: 1: Syntax error: word unexpected (expecting ")")

2. MacOS with M1/M2 is on AWS

You can get MacOS on Apple hardware in AWS very cheaply on spot instances for testing, like $1 for 5-hours just FYI: https://aws.amazon.com/ec2/instance-types/mac/

I've not really used Rosetta, my brain hurts now LOL. Only box64 runs, but it doesn't run the 32-bit steamcmd in Linux.

[jryanearl]

So I managed to update with the MacOS version of steamcmd from outside the container into the bind volume. I modified start.sh to skip updates and such, and was able to get it to run actually.

However, performance is horrible. I don't think there is much value to run this with emulation. If they release a native ARM64 build, and Unreal Engine 5.2 supports Apple Silicon natively now: https://www.unrealengine.com/en-US/tech-blog/unreal-engine-5-2-brings-native-support-for-apple-silicon-and-other-developments-for-macos However, PalWorld is on UnrealEngine 5.1, so maybe they will update the backend engine at some point. Here's the output of me finally getting it running:

steam@palworld:~/server$ ./start-mac.sh 
*****STARTING SERVER*****
./PalServer-arm64.sh
Dynarec for ARM64, with extension: ASIMD AES CRC32 PMULL ATOMICS SHA1 SHA2 PageSize:4096 Running on unknown aarch64 cpu with 5 Cores
Params database has 57 entries
Box64 with Dynarec v0.2.7 fed47bea built on Feb 11 2024 06:20:00
BOX64: Detected 48bits at least of address space
Counted 15 Env var
BOX64 LIB PATH: /home/steam/steamcmd/linux64/:./:lib/:lib64/:x86_64/:bin64/:libs64/:/usr/lib/x86_64-linux-gnu/
BOX64 BIN PATH: ./:bin/:/usr/local/bin/:/usr/bin/:/bin/:/usr/local/games/:/usr/games/
Looking for /palworld/Pal/Binaries/Linux/PalServer-Linux-Test
argv[1]="Pal"
Rename process to "PalServer-Linux-Test"
Using native(wrapped) libpthread.so.0
Using native(wrapped) libdl.so.2
Using emulated /palworld/Pal/Binaries/Linux/../../../Engine/Binaries/ThirdParty/Steamworks/Steamv153/x86_64-unknown-linux-gnu/libsteam_api.so
Redirecting overridden malloc from symtab function for /palworld/Pal/Binaries/Linux/../../../Engine/Binaries/ThirdParty/Steamworks/Steamv153/x86_64-unknown-linux-gnu/libsteam_api.so
Using emulated /palworld/Pal/Binaries/Linux/../../../Engine/Binaries/Linux/libEOSSDK-Linux-Shipping.so
Using native(wrapped) librt.so.1
Using native(wrapped) libm.so.6
Using native(wrapped) libc.so.6
Using native(wrapped) ld-linux-x86-64.so.2
Using native(wrapped) libutil.so.1
Using native(wrapped) libbsd.so.0
Using emulated /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
Warning: Global Symbol _ZTHN2UE5Trace7Private15GTlsWriteBufferE not found, cannot apply R_X86_64_GLOB_DAT @0xb6ee120 ((nil)) in /palworld/Pal/Binaries/Linux/PalServer-Linux-Test
Shutdown handler: initalize.
Increasing per-process limit of core file size to infinity.
- Existing per-process limit (soft=18446744073709551615, hard=18446744073709551615) is enough for us (need only 18446744073709551615)

Using emulated /home/steam/steamcmd/linux64/steamclient.so
Redirecting overridden malloc from symtab function for /home/steam/steamcmd/linux64/steamclient.so
Warning: Weak Symbol _ITM_RU1 not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0614dad8 (0xac9f10)
Warning: Weak Symbol _ZGTtnam not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0614dae0 (0xac9f10)
Warning: Weak Symbol _ITM_memcpyRtWn not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0614dae8 (0xac9f10)
Warning: Weak Symbol _ITM_RU8 not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0614daf0 (0xac9f10)
[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
Using native(wrapped) crashhandler.so
CAppInfoCacheReadFromDiskThread took 3 milliseconds to initialize
Using emulated /home/steam/steamcmd/linux64/steamservice.so
Redirecting overridden malloc from symtab function for /home/steam/steamcmd/linux64/steamservice.so
Warning: Weak Symbol _ITM_RU1 not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0914dad8 (0xac9f10)
Warning: Weak Symbol _ZGTtnam not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0914dae0 (0xac9f10)
Warning: Weak Symbol _ITM_memcpyRtWn not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0914dae8 (0xac9f10)
Warning: Weak Symbol _ITM_RU8 not found, cannot apply R_X86_64_JUMP_SLOT @0x7fff0914daf0 (0xac9f10)
Steam Service Error: Failed to get Steam Service Start function
Setting breakpad minidump AppID = 2394010
Error loading needed lib libsteam.so
Warning: Cannot dlopen("libsteam.so"/0x7fff0003f7d9, 2)
[S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
[S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
[S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
[S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
Error loading needed lib libSDL3.so.0
Warning: Cannot dlopen("libSDL3.so.0"/0x7fff04a65a5b, 2)

Anyway, I saw what I wanted to test. I was hoping it would perform well, but it's much, much worse than on slower x86-64 CPUs under emulation.

[sonroyaalmerol]

So this is honestly kind of crazy, I guess docker desktop is using Rosetta under the hood or something. Rancher Desktop is making a HVM for the container. box86 won't run on Apple Silicon... Could really use a 64-bit build of steamcmd.

That is an interesting find. I would've thought that Rancher Desktop would be using similar Docker APIs as Docker Desktop. I have no experience with Rancher Desktop so I wouldn't really know. As far as I know, Docker Desktop on the Apple Silicon automatically uses Rosetta on incompatible images as soon as you specify the platform.

root@palworld:/home/steam/steamcmd/linux32# box86 steamcmd -h
bash: /usr/local/bin/box86: cannot execute binary file: Exec format error

I guess Docker Desktop emulates AArch32 (32-bit ARM) with Rosetta and Rancher probably does not do it automatically? I would assume so since people using Docker Desktop were successful with running Box86 within the container. I guess you learn something new everyday.

However, performance is horrible. I don't think there is much value to run this with emulation. If they release a native ARM64 build, and Unreal Engine 5.2 supports Apple Silicon natively now: https://www.unrealengine.com/en-US/tech-blog/unreal-engine-5-2-brings-native-support-for-apple-silicon-and-other-developments-for-macos However, PalWorld is on UnrealEngine 5.1, so maybe they will update the backend engine at some point.

Yup, emulation in addition to the memory leaks and (probably) unoptimized code, it is a struggle. Heck, even some amd64 hosts are struggling to serve this game. Hopefully they do release an arm64 dedicated server somehow. But I don't know how they would distribute it as steamcmd is also not built for arm64.

[pyaehein]

Same issue happened on ARM64 "Orange pi 5b".

[sonroyaalmerol]

@jryanearl I know you've already figured it out on your own but do you mind testing #464 with the same setup?

Feel free to test it as well @pyaehein