Move to 17 using DS-Memory 4.0.0 with FFM/Panama

[leerho]

This must wait until DS-Memory 4.0.0 has been released.

[frankgrimes97]

@leerho This is very interesting and promising work!
FWIW, I started looking into this as means to get myself familiar with the new FFM APIs, however I noticed that the APIs have changed significantly since the first preview.
Notably it's moved from jdk.incubator.foreign to java.lang.foreign and the segment management has been folded into an Arena interface since Java 22 (JEP-454).
Are there any plans to move this work to use the updated FFM APIs?
Would a PR on datasketches-memory be welcome should I find time to contribute?

[leerho]

@frankgrimes97,
Hello Frank, I think I have seen your name before. Nonetheless, thank you for your observations and I welcome your participation and contributions to our library!

As you recognize, the ds-java repo depends on ds-memory for all of the serialization, deserialization, state-machine and off-heap handling. And I have been working on moving the older version of ds-memory, which handled Java 8 and 11, forward to be able to handle the next LTS, Java 17. This is the ds-memory 4.X releases that are fairly recent. And, as you also recognize, the FFM API changed considerably from Java 17 to Java 21. As a result, the ds-memory 4.X releases will compile only with Java 17, but should run with later versions of Java if Java's promise of backward compatibility applies to their incubation code, but I haven't had the chance to confirm this.

Nonetheless, I am planning a ds-memory 5.X release that will be based on the Java 21 FFM API, and there will have to be a concomitant ds-java major release to depend on it. From what I can tell, the FFM API doesn't change much after Java 21, so I'm assuming that it might also compile (or run) under Java 25, but Java 25 isn't due out until September 2025 so we will have to wait and see.

FFM became formally part of Java as of Java 22, so I'm thinking that the ds-memory 5.X release may be the last. At that point, I may as well update the ds-java to use FFM directly, eliminating the need for the ds-memory entirely.

The ds-memory was originally developed to provide efficient off-heap state-machine handling for Java 7 and later Java 8 & 11. But to do that required extensive use of JVM internals (unsafe and several others), which over time with newer Java versions, have been progressively locked out for Java users. And a useful implementation of Panama (for us) didn't appear until Java 17.

Many of our customers for our DataSketches library had not moved beyond Java 8 until recently thus the current need to move our library to Java 17 and soon Java 21. Unfortunately, we do not have the resources to support the non-LTS Java releases.

If you are interested in contributing, you might try modify the ds-memory 4.1.0 to take advantage of the FFM in Java 21. This would become the ds-memory 5.X release. It should be straightforward. Although it might be tempting to move it to Java 22, I would not recommend that. Most major customers of our project ignore the non-LTS releases. It must at least compile on Java 21.

Cheers,
Lee.