Support for multiple plattforms at the same time

[kernle32dll]

Hi,

currently development on our local machines is a bit of a hassle, since we have developers use all kind of OSses, e.g Linux, macOSX and even Windows. So looking at the latest build refinements in J2V8 - would it be possible to provide a maven artifact, which contains all the lib files for all supported platforms? From what I can tell the lib loader should be able to handle that.

In hindsight it would also be nice to have a "core" image, which does not contain any lib whatsoever, and instead relies on the platform to provide one (#252).

[irbull]

See #261. I'm sorry, but I just don't have the resources to support all the different versions of J2V8 that everyone seems to want. I'm hoping we can find a workable solution where others can help with the maven builds and we can share the artifacts.

[kernle32dll]

@drywolf : Do you have some thoughts on the matter? If the naming of the library files is distinct (that's why I made that change in the Alpine PR), it should be possible to just create an maven artifact which contains all library files.

As I said - I think such an artifact would be very nice for development, as a team with different dev OSes (such as our team) could use the same maven artifact for development (atm it is very nasty for us, since the J2V8 dependency is currently hardwired deeply in an application SDK, which needs to be rebuild locally on non-linux machines. Duh...)

[drywolf]

I have done some research on this (focusing on #188 and #289)

IMO there are two problems that need to be solved well:

1. a distinctive file naming scheme is needed for the native library, that uniquely identifies the different libraries across linux distributions
2. retrieving system triplet from the Java runtime & loading the compatible & available native lib

To get a consistent and reliable library-loading scheme going for the linux distributions and all their subtle differences, I would propose:

1) File naming scheme

To settle on a uniform naming scheme for the native libraries, gcc seems to have this covered well already (see gcc -dumpmachine).

GCC Target triplet structure: {arch}-[vendor]-{operating_system}-[cstdlib/abi_env]
* Parts in angular brackets are optional
** For a full listing of possible values for each, I think LLVM has the most complete list

gcc -dumpmachine examples:

• Alpine: x86_64-alpine-linux-musl
• Centos: x86_64-redhat-linux
• Ubuntu: x86_64-linux-gnu
• Debian: x86_64-linux-gnu

The structure of a system-triple for the filename of the native library should be less ambiguous, and include all the relevant parts:

{arch}-{vendor}-{operating_system}-{clib_env/abi_env}

examples:

• Alpine: x86_64-alpine-linux-musl
• Centos: x86_64-centos-linux-gnu
• Ubuntu: x86_64-ubuntu-linux-gnu
• Debian: x86_64-debian-linux-gnu

2) LibraryLoader.java implementation:

The current implementation of the library loader heavily relies on some Java system properties (os.name, java.specification.vendor, os.arch) to try and figure out which OS is the JVM host.
The vendor part being the tricky one that needs solving, I looked into an approach that would work more consistently and match with the parts outlined in point 1) above.
Every linux distribution that I tested (see above) has the /etc/os-release meta-data file implemented at this point, and in this file you reliably can get the OS "ID" attribute which delivers the vendor string that was uncertain/inconsistent before.

For the fourth part (🤷‍♂️) of the filename system-triplet ... the clib_env/abi_env .... I think we can just keep an internal table of the most relevant linux distributions and which clib they use, and just return it from there based on which distribution was detected via /etc/os-release.

PPS: The LibraryLoader code should also be able to handle a fallback case, for example when there is a native lib located in the JAR that does not specify the vendor-part, then the LibraryLoader should pick this library if no other fitting vendor-specific library is located in the JAR. This should cover the case if we want to continue to deliver a default linux JAR via maven, but which is mostly only compatible with debian-based derivates.

The only thing that I couldn't confirm myself was the file permissions that a "regular" user might have for the /etc/os-release file (since I'm only testing the Linux distributions via Docker on Windows, and docker by default always uses root permissions)

@kernle32dll could you please help me out and check what the file permissions on that file are on a usual Linux machine with a user that you would usually use to run your J2V8 software ?

PS: is just found this code in a maven plugin that pretty much does exactly what I outlined above, so the general approach seems to be feasable 😄

[matiwinnetou]

@drywolf Awesome investigation...

[kernle32dll]

@drywolf : Awesome investigation indeed!

I'm all for the suggested naming scheme. At least on my dev machine at work was no /etc/os-version to be found, but /etc/os-release. The linked maven plugin also seems to indicate that "os-release" is the right name. The only remark I have is that I don't know if it is necessary to enforce the vendor part. Because if we do so, it would require to build J2V8 for every operating system out there :(

As for permissions, it is indeed root owned. But I did not encounter problems reading (cat) from it as a regular user. So I think we should be good with that.

Also, as you might have seen in #289 (see this line in particular), I did use a similar trick (/etc/alpine-release) to detect alpine in the LibraryLoader. And this code was tested briefly with our regular production Docker images (with non-root users). So I suspect if Alpine does have a /etc/os-release file, it should work just the same.

[drywolf]

Thanks for testing the file permissions @kernle32dll

The only remark I have is that I don't know if it is necessary to enforce the vendor part. Because if we do so, it would require to build J2V8 for every operating system out there

That's what I also thought, and mentioned in the PPS part above.
I am currently changing the LibraryLoader code to behave that way and use the new naming scheme.

PS: I meant to say /etc/os-release in the above summary, I corrected it now 😉

[AlexTrotsenko]

Is there any updates on this topic?
We are building Android app, but would like to run pure JUnit tests on different environments: Jenkins (Linux), MacOS, Window.
Is there already a way to accomplish this somehow?

[caer]

Hey @AlexTrotsenko,

While developing http://alicorn.io and https://github.com/alicorn-systems/v8-adapter, I had a similar issue with supporting multiple build environments.

On the desktop, my solution was to use Maven profiles; for each platform, I attempt to have Maven auto-detect the platform and use the correct V8 dependency (which you can see in action in the V8-Adapter POM: https://github.com/alicorn-systems/v8-adapter/blob/master/pom.xml#L140).

On Android (with Gradle), I exclude the V8 dependency from the normal POM and re-add the .AAR file from J2V8:

compile ('io.alicorn:my-project:0.1-SNAPSHOT') {
        exclude group: 'com.eclipsesource.j2v8'
}
compile (name: 'j2v8', ext: 'aar')

Hope this helps!

[AlexTrotsenko]

@crahda thanks for suggestions. I will check and try it!

[irbull]

The J2V8 project will only be providing binaries for Android. Others are free to build and distribute binaries for other platforms. This has been the case for a while now because I haven't had the time to support all the different configurations. Please see #441 for more information.

I'm currently in the process of building an aar with support for x86, x86_64, arm and arm64.

[JessHolle]

@irbull So how does one build both 32 and 64-bit ARM binaries for Android into 1 .aar? It's not clear from the existing build system?

[irbull]

I'll add this to the docs. The way I do it:
First I build for one platform and perform all the steps
python build.py -t android -a arm --docker all

Then I build for another platform with the --keep-native-libs flag:
python build.py -t android -a x86 --keep-native-libs --docker nodejs j2v8cmake j2v8jni j2v8cpp j2v8optimize j2v8java

I don't run the tets steps here because 1) we've already run all the emulator tests, and 2) I can't seem to get x86 emulator tests running inside a docker container while running on an x86 host.

I repeat this for the other platforms:
python build.py -t android -a x86_64 --keep-native-libs --docker nodejs j2v8cmake j2v8jni j2v8cpp j2v8optimize j2v8java
python build.py -t android -a arm64 --keep-native-libs --docker nodejs j2v8cmake j2v8jni j2v8cpp j2v8optimize j2v8java

This produces an aar with all the platform binaries.

[irbull]

I've re-opened this issue as a TODO to update the docs.