Problems with TOPCAT v4.9 MacOS DMG file

[mbtaylor]

Background

Alongside the platform-independent topcat-full.jar file which I generally recommend for running TOPCAT, I also provide as a convenience to MacOS users a topcat-full.dmg file. This is built using https://github.com/TheInfiniteKind/appbundler/.

In topcat versions before v4.9, the topcat-full.dmg file only worked for Intel architectures. Chris Carroll kindly supplied a version for topcat v4.8-8 that worked for ARM (Apple Silicon) but not Intel, still available at https://www.star.bristol.ac.uk/mbt/releases/topcat/v4.8-8/topcat-full-b.dmg This one is built differently than the previously supplied one (using jpackage) and it bundled the Java Runtime Environment with it, so it's much larger.

Current status

At TOPCAT v4.9 (1 Nov 2023), I changed the way that the DMG file is built and it is now supposed to work for both Intel and ARM (an updated version of appbundler contains a fat version of the JavaAppLauncher binary). It does indeed work for some users on both Intel and ARM, but others have had difficulties, namely java crashes at startup on ARM.

I don't know why this is, but it looks like a bug in the MacOS ARM java. Whether it happens or not may depend on the java installation. I so far have reports of:

• OpenJDK Java 21: works on Intel and ARM
• Oracle Java 8u391: crashes on ARM
• Corretto Java 8: works on Intel, crashes on ARM

But maybe it's not the java installation after all and there's something else going wrong. So I'd appreciate user reports of what does and doesn't work for you on ARM.

Note that as far as I know running java on the topcat-full.jar file works in all cases, so if you can live without clicking on a yellow cat's head, just do:

   java -jar topcat-full.jar

Note also this is not about whether MacOS wants to prevent you from running this application, that's a separate issue discussed in the FAQ.

Next steps

This issue serves as a place for:

1. users to report success/failure in using the DMG file(s)
2. me to provide updates about conclusions or fixes if I make some

So if you've tried running the TOPCAT v4.9 topcat-full.dmg file on ARM and it either has or hasn't worked for you, and a setup like yours has not already been noted here, please consider adding a comment to report your experience.

[augustfly]

I was literally just looking for how to report something that sounds like this problem when you posted this to the discussion board!

Outcomes:

• New Version 4.9 App fails to run. I cannot not find any console log to see the actual errors reported during the attempt. All I have is that the UI reports in a popup:

This application requires that Java 8 or later be installed on your computer. Please download and install the latest version of Java from www.java.com and try again.

This doesn't make any sense bc

❯ java -version
java version "1.8.0_391"
Java(TM) SE Runtime Environment (build 1.8.0_391-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.391-b13, mixed mode)

and

❯ java -jar $HOME/Applications/bin/topcat-full.jar

works completely fine with the version 4.9 jar file.

finally things you might need:

❯ system_profiler -detailLevel mini SPSoftwareDataType
Software:

System Software Overview:

  System Version: macOS 13.6.1 (22G313)
  Kernel Version: Darwin 22.6.0
  Time since boot: 4 days, 4 hours, 8 minutes

❯ system_profiler SPHardwareDataType
Hardware:

Hardware Overview:

  Model Name: MacBook Pro
  Model Identifier: MacBookPro18,3
  Model Number: Z15H00105LL/A
  Chip: Apple M1 Pro

[mbtaylor]

Hi Gus. This is actually a somewhat different problem than the one I mentioned at the top, though it's related so here's a good place to discuss it. It seems that MacOS (or something) is picky about what/where the JRE is that it's looking for. Upgrading/re-installing Java may fix this, or at least turn it into a different problem... Any java version >=8 ought in principle to be OK, but based on the reports I've had so far, OpenJDK 21 from https://jdk.java.net/ might be a good bet. That comes as a tar.gz archive which may or may not be to your taste... can anybody recommend an ARM JRE that works with the topcat 4.9 DMG?

[stevemaser]

So I'm the one with this report: "Corretto Java 8: works on Intel, crashes on ARM". I think we'll just end up seeing if those that need it can get used to using the .jar file instead in the interim as we have a pressing need to get the arm version of Corretto out.

[monodera]

I've tried a few different OpenJDK builds for ARM 64 on M1 Pro MacBook Pro (macOS 14.1).

• Azul Zulu JRE 8 ARM 64 (8u392b08): failed on startup
• Azul Zulu JRE 21 ARM 64 (21.0.1+12): succeeded to launch
• Adoptium Eclipse Temurin JRE 21 aarch64 (based on jdk-21.0.1+12): succeeded to launch

Maybe version 8 does not work nicely with Apple Silicon? I'll use one of the two version 21 JRE builds.

[mbtaylor]

I am still looking into this, but haven't reached a conclusion yet ...

It seems that running java -jar /path/to/topcat-full.jar works for everybody, given a working java installation. Does anybody more Mac-savvy than me know if/how I can build a DMG file which just contains that jar file and a launcher that's a shell script with the content java -jar topcat-full.jar?
Or is there some fundamental flaw in that approach?

[nxg]

A fragment of an answer: an excutable shell script named foo.command, when double-clicked, will run the script. That might be something to investigate.

A different fragment: I think there are no longer any good macOS technical mailing lists, but Apple.SE is generally quite reliable for questions of this sort.

[ricardo-rizzo]

I'm running successfully TOPCAT 4.9 in two MacBook Air which differ only in the RAM available.
With 8GB ram it's working with openJDK

openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment Zulu16.32+15-CA (build 16.0.2+7)
OpenJDK 64-Bit Server VM Zulu16.32+15-CA (build 16.0.2+7, mixed mode) 

The other one is running the Oracle's version, with 16 GB RAM:

java version "21.0.1" 2023-10-17 LTS
Java(TM) SE Runtime Environment (build 21.0.1+12-LTS-29)
Java HotSpot(TM) 64-Bit Server VM (build 21.0.1+12-LTS-29, mixed mode, sharing)

Thank you!

[mbtaylor]

I'm clutching at straws a little bit here, but I've had a go at building a different version of the DMG file that I'm hoping (a) will work on both ARM and Intel and (b) won't crash on startup. If anybody here has experienced a crash at startup (not just "can't find java 8") and is willing to have a try with a different version, I'd be grateful. The new one is at https://www.star.bristol.ac.uk/mbt/releases/topcat/pre/topcat-full-d.dmg

[mbtaylor]

Apologies, the link I posted in the previous comment was initially wrong - I've edited it and it should be correct now.

[augustfly]

I've successfully switched sides from "can't find java 8" to crashing on startup! Here is my current install (8 JRE not SDK and not 21)

❯ java -version  
java version "1.8.0_391"
Java(TM) SE Runtime Environment (build 1.8.0_391-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.391-b13, mixed mode)

I'd have to spend more minutes than I have figuring out how to install 21 right now. Here is my current report:

Using new 11/29 Topcat version -- Running it says Java not installed (same as before)

Gus's next steps:

• Removed Java (using Oracle installer); reinstalled using same installer
• Now the new Topcat version throws a Kernel exception error!
• The November 1 Topcat also now throws a Kernel exception error!
• I show the first few lines of the Kernel Exception from my Mac below.
• So that was a bad idea? At least now my issue is pertinent to the GitHub issue!
• With the reinstalled Java JRE 8, the command line version still runs. Whew!

Of note on Macs: the java command, /usr/bin/java
is installed by Apple. This is NOT an Oracle-installed command.

---

Translated Report (Full Report Below)

Process: JavaAppLauncher [85998]
Path: /Applications/TOPCAT.app/Contents/MacOS/JavaAppLauncher
Identifier: uk.ac.starlink.topcat
Version: 4.9 (1.0)
Code Type: ARM-64 (Native)
Parent Process: launchd [1]
User ID:

Date/Time: 2023-11-29 09:06:27.9321 -0500
OS Version: macOS 13.6.2 (22G320)
Report Version: 12
Anonymous UUID:

Sleep/Wake UUID:

Time Awake Since Boot: 670000 seconds
Time Since Wake: 38878 seconds

System Integrity Protection: enabled

Crashed Thread: 4

Exception Type: EXC_BAD_ACCESS (SIGABRT)
Exception Codes: KERN_PROTECTION_FAILURE at 0x000000080061002e
Exception Codes: 0x0000000000000002, 0x000000080061002e

Termination Reason: Namespace SIGNAL, Code 6 Abort trap: 6
Terminating Process: JavaAppLauncher [85998]

[mbtaylor]

Thanks Gus, that's very useful. Doesn't get me much closer to a fix, but it does narrow things down a bit.

[mbtaylor]

OK here we go again. I've taken a completely different approach this time, so I'd be surprised if we see exactly the same problems as before... This time there are two alternatives, one which includes a bundled Java Runtime Environment (it's Corretto 8) and one which does not.

• http://www.star.bristol.ac.uk/mbt/releases/topcat/pre/topcat-full-e.dmg (93Mb, includes JRE)
• http://www.star.bristol.ac.uk/mbt/releases/topcat/pre/topcat-lite-e.dmg (36Mb)

I believe:

• both should run on both ARM and Intel - hopefully no startup crash, and no failure for having the wrong architecture
• topcat-full should avoid any problems about not being able to find Java 8 (topcat-lite might do as well as long as you have Java installed, I'm not sure)
• neither is signed or notarized, so you will still have the same issues as before about having to sidestep MacOS's security concerns

Again, I'd be grateful if somebody could have a go running these. If they do work, I'd be interested in anything weird/suboptimal/different from the previous arrangement that you notice.

[mjcowley]

Hi Mark,

I decided to update TOPCAT today and encountered the Java problem described above. I tested your JRE bundle (topcat-full-e.dmg) and, despite a security warning (resolved by removing the quarantine attribute using "xattr -d com.apple.quarantine /Applications/TOPCAT.app"), it is running smoothly on my M1 Pro with Sonoma 14.1.2.

Thanks!

[mbtaylor]

That is good news, thank you @mjcowley. Given the variability of experience I've seen so far, I'd still welcome reports of other people's attempts.

[augustfly]

Here's my report:

• The "lite" e version does not run. It issues a popup as shown in attached screenshot. This popup is entirely new (i.e., is not the same popup I got with the Nov 1 version -- before reinstalling Java caused the Nov 1 to also simply crash), and I've never heard of "adoptium.net".

• the bundled "e" version runs after I use the xattr cleanup.

Gus

[mbtaylor]

Thanks Gus. That corresponds to the message you were getting before, i.e. not being able to find a suitable Java 8 on your machine (even though apparently you have one). So at the expense of downloading quite a few more bytes (the -full version), it seems that it's possible to get around that, and the crash seems to have gone away. This is looking fairly positive.

[augustfly]

Contextually, yes the same popup/warning. Precisely, the popup is completely different, which I assume is a TOPCAT not a Gus's computer change.

[stevemaser]

So I tried the “full” — with the xattr modification — and it launches on my Sonoma M1 box here. I tried the “light” (same xattr) with Corretto 8 (arm version) and it launched — it also launched with Corretto 21 So promising from my end! - Steve

…

On Dec 1, 2023, at 10:35 AM, Mark Taylor ***@***.***> wrote: Thanks Gus. That corresponds to the message you were getting before, i.e. not being able to find a suitable Java 8 on your machine (even though apparently you have one). So at the expense of downloading quite a few more bytes (the -full version), it seems that it's possible to get around that, and the crash seems to have gone away. This is looking fairly positive. — Reply to this email directly, view it on GitHub <#65 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ART5QHVBH6LGT4ZI7U4FF23YHH2LHAVCNFSM6AAAAAA64RKKQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZWGMYTONJWHE>. You are receiving this because you commented.

[davidwcraig]

I can confirm that the topcat-full-e.dmg version works with the xattr fix on a MacBook Air M1 (Model A2337) using Sonoma 14.1.2 as well. I got the same popup with the lite version.
As a long-time TOPCAT user, this is great!

[mbtaylor]

@augustfly and @davidwcraig if I understand your reports correctly, you have java (>=1.8) installed on your machine, but the topcat-lite-e.dmg image doesn't find it, i.e. if you try to use topcat-lite-e.dmg a popup tells you to install Java 1.8. Can you (and anybody else that this applies to) tell me where the java you have is installed? I guess "which java" ought to tell you this.

[augustfly]

I reported this before, but this time I dug a little deeper. This may be random, unhelpful information but:

❯ which java
/usr/bin/java

❯ /usr/libexec/java_home
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home

❯ ls -l /usr/bin/java
... Nov  2 18:23 ...

which is the same timestamp as everything else in /usr/bin and corresponded to the timestamp of the last Apple OS update I did. Noting that nothing is ever supposed to write to /usr/bin on a Mac and that this java command had never been removed no matter how many times I ran the Oracle Java uninstaller, I wondered if this was an Apple command.

❯ codesign -dv /usr/bin/java 
Executable=/usr/bin/java
Identifier=com.apple.wsimport
Format=Mach-O universal (x86_64 arm64e)
CodeDirectory v=20400 size=587 flags=0x0(none) hashes=13+2 location=embedded
Platform identifier=14
Signature size=4442
Signed Time=Jun 25, 2023 at 11:24:31 PM
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=1 size=68

digging around in the binary's signature:

❯ codesign -dv /usr/bin/java --extract-certificates
❯ openssl x509 -inform DER -in codesign0 -text 
...

...

            X509v3 Certificate Policies: 
                Policy: 1.2.840.113635.100.5.1
                  User Notice:
                    Explicit Text: This certificate is to be used exclusively for functions internal to Apple Products and/or Apple processes.


...

hmmm, unsure if that is helpful, but that looks like java is an explicitly Apple binary. Maybe this is more helpful?

❯ java -verbose topcat-full.jar
[Opened /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/rt.jar]
[Loaded java.lang.Object from /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/rt.jar]
[Loaded java.io.Serializable from /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/rt.jar]
... <pages of verbosity> ...

[davidwcraig]

Right now I don't have any jre installed, sorry!

The latest version of Oracle's Java would not install on my machine at all (u391) I got an 'error 1' from the installer that was not very informative. I also tried to use OpenJDK installed with homebrew, but I could not get that to work earlier. I was trying several options and realized just now that I had removed OpenJDK. If I get a chance soon I will try it again.

[davidwcraig]

Addition: I'm not sure I tried topcat-lite-e with openjdk, so I am not very informative.

[mbtaylor]

Thanks @davidwcraig - I think I have enough to go on from @augustfly so don't worry too much about this.

[mbtaylor]

I've given up on the -lite DMG file, since it fails for reasons that I don't understand on some Macs, so I will be offering only one DMG file for download in future, topcat-all.dmg, which contains a bundled (Corretto 8) JRE. This is built using Install4j, a rather nice commercial multi-platform installer builder application which is kind enough to offer free licences to open source projects.

This version is now available for v4.9 on the TOPCAT home page at topcat-all.dmg. It's not identical to the one that various people here have tested, but it should be close enough that I expect it to work on all reasonable ARM or Intel Macs. But if anybody here is willing to try it out and check, I'd be grateful for reports of problems. If I don't hear anything in the next few days I'll announce the new DMG file on the mailing lists.

Hopefully this is the end of this story for now. Many thanks to all those who have helped me to investigate it and sort it out.

[mjcowley]

I've given up on the -lite DMG file, since it fails for reasons that I don't understand on some Macs, so I will be offering only one DMG file for download in future, topcat-all.dmg, which contains a bundled (Corretto 8) JRE. This is built using Install4j, a rather nice commercial multi-platform installer builder application which is kind enough to offer free licences to open source projects.

This version is now available for v4.9 on the TOPCAT home page at topcat-all.dmg. It's not identical to the one that various people here have tested, but it should be close enough that I expect it to work on all reasonable ARM or Intel Macs. But if anybody here is willing to try it out and check, I'd be grateful for reports of problems. If I don't hear anything in the next few days I'll announce the new DMG file on the mailing lists.

Hopefully this is the end of this story for now. Many thanks to all those who have helped me to investigate it and sort it out.

I've tested out v4.9 for a few hours today and didn't encounter any problems, Mark. The only thing was the need to remove the quarantine attribute, but other than that, it installed without a hitch on macOS Sonoma 14.1.2.