NCDFE in Liberty startup #20444
Comments
|
Issue Number: 20444 |
|
From the tracepoint: There are 2 threads loading classes: However, both threads got The tracepoint is: openj9/runtime/vm/classsupport.c Line 778 in a08b536 and openj9/runtime/vm/classsupport.c Line 742 in a08b536 Could you look at why thread |
|
We have also observed this strange java.lang.NoSuchMethodError: java/util/concurrent/locks/ReentrantReadWriteLock$WriteLock.getPrivateLibraries(Lcom/ibm/wsspi/classloading/ClassLoaderConfiguration;)Ljava/util/List; (loaded from jrt:/java.base by <Bootstrap Loader>) called from class com.ibm.ws.classloading.internal.AppClassLoader (loaded from file:/C:/Users/jazz_build/Build/jazz-build-engines/wasrtc/EBCPROD/build/dev/image/output/wlp/lib/com.ibm.ws.classloading_1.1.95.jar by org.eclipse.osgi.internal.loader.EquinoxClassLoader@b19957cc[com.ibm.ws.classloading:1.1.95.202410071314(id=48)]).
at com.ibm.ws.classloading.internal.AppClassLoader.<init>(AppClassLoader.java:172)
at com.ibm.ws.classloading.internal.ClassLoaderFactory.createClassLoader(ClassLoaderFactory.java:138)
at com.ibm.ws.classloading.internal.ClassLoaderFactory.create(ClassLoaderFactory.java:116)
at com.ibm.ws.classloading.internal.ClassLoadingServiceImpl.createAppClassLoader(ClassLoadingServiceImpl.java:349)
at com.ibm.ws.classloading.internal.ClassLoadingServiceImpl.createTopLevelClassLoader(ClassLoadingServiceImpl.java:321)
at com.ibm.ws.classloading.internal.ClassLoadingServiceImpl.createTopLevelClassLoader(ClassLoadingServiceImpl.java:103)
at com.ibm.ws.classloading.ClassLoaderConfigHelper.createTopLevelClassLoader(ClassLoaderConfigHelper.java:314)
at com.ibm.ws.app.manager.module.internal.DeployedAppInfoBase.createTopLevelClassLoader(DeployedAppInfoBase.java:635)
at com.ibm.ws.app.manager.war.internal.WARDeployedAppInfo.createModuleClassLoader(WARDeployedAppInfo.java:105)
at com.ibm.ws.app.manager.module.internal.ExtendedModuleInfoImpl.getClassLoader(ExtendedModuleInfoImpl.java:124)
at com.ibm.ws.container.service.annocache.internal.ModuleAnnotationsImpl.<init>(ModuleAnnotationsImpl.java:198)
at com.ibm.ws.container.service.annocache.internal.ModuleAnnotationsImpl.<init>(ModuleAnnotationsImpl.java:161)
at com.ibm.ws.container.service.annocache.internal.WebAnnotationsImpl.<init>(WebAnnotationsImpl.java:91)
at com.ibm.ws.container.service.annocache.internal.WebAnnotationsAdapterImpl.adapt(WebAnnotationsAdapterImpl.java:63)
at com.ibm.ws.container.service.annocache.internal.WebAnnotationsAdapterImpl.adapt(WebAnnotationsAdapterImpl.java:35)
at com.ibm.ws.adaptable.module.internal.AdapterFactoryServiceImpl.adapt(AdapterFactoryServiceImpl.java:202)
at com.ibm.ws.adaptable.module.internal.AdaptableContainerImpl.adapt(AdaptableContainerImpl.java:176)
at com.ibm.ws.container.service.annocache.AnnotationsBetaHelper.getWebAnnotations(AnnotationsBetaHelper.java:270)
at com.ibm.ws.webcontainer.osgi.container.config.WebAppConfigurator.setWebAnnotations(WebAppConfigurator.java:88)
at com.ibm.ws.webcontainer.osgi.container.config.WebAppConfigurator.configure(WebAppConfigurator.java:424)
at com.ibm.ws.webcontainer.osgi.container.config.WebAppConfigurationAdapter.adapt(WebAppConfigurationAdapter.java:121)
at com.ibm.ws.webcontainer.osgi.container.config.WebAppConfigurationAdapter.adapt(WebAppConfigurationAdapter.java:45)
at com.ibm.ws.adaptable.module.internal.AdapterFactoryServiceImpl.adapt(AdapterFactoryServiceImpl.java:202)
at com.ibm.ws.adaptable.module.internal.AdaptableContainerImpl.adapt(AdaptableContainerImpl.java:176)
at com.ibm.ws.webcontainer.osgi.WebContainer.createModuleMetaData(WebContainer.java:872)
at com.ibm.ws.app.manager.module.internal.ModuleHandlerBase.createModuleMetaData(ModuleHandlerBase.java:65)
at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase$ModuleContainerInfoBase.createModuleMetaData(SimpleDeployedAppInfoBase.java:222)
at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.preDeployApp(SimpleDeployedAppInfoBase.java:608)
at com.ibm.ws.app.manager.module.internal.SimpleDeployedAppInfoBase.installApp(SimpleDeployedAppInfoBase.java:584)
at com.ibm.ws.app.manager.module.internal.DeployedAppInfoBase.deployApp(DeployedAppInfoBase.java:625)
at com.ibm.ws.app.manager.war.internal.WARApplicationHandlerImpl.install(WARApplicationHandlerImpl.java:67)
at com.ibm.ws.app.manager.internal.statemachine.StartAction.execute(StartAction.java:199)
at com.ibm.ws.app.manager.internal.statemachine.ApplicationStateMachineImpl.enterState(ApplicationStateMachineImpl.java:1369)
at com.ibm.ws.app.manager.internal.statemachine.ApplicationStateMachineImpl.run(ApplicationStateMachineImpl.java:912)
at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:298)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:857) |
|
@TobiAjila @gacholio Does this ring any bell to you ? 2 threads loading 2 different classes simultaneously. Thread1 loaded the correct class, but thread2 got the class loaded by thread1. |
|
For Liberty we have three levels of class loaders
All three levels are storing classes into the Open J9 shared class cache. So far I only recall seeing these strange errors for the classes loaded by the Equinox bundle class loaders. This makes me suspect there is something specific to how Equinox uses the The call to Similarly the method Again, this call has no locks held by the class loader while calling the |
|
re comment There is only one copy of Also there is only one copy of I believe this JVM called |
Actually The normal sequence should be the class loader calls |
Because Equinox class loaders are a graph that may contain cycles it does not hold locks for an initial call to
Could there be some inconsistency with allowing another thread to get a non-null result from the call to |
I realized the wrong class returned is
It is possible that a thread in step 1 can get a not null class while another thread is defining the class in step 4 (the thread should already be near the end of step 4 after the class is already added to the hashtable keyed on the class name, the class name in step 1 and 4 should be the same). |
I'm out of ideas on the Equinox side how that could happen. I assume that must be going wrong down in the JVM? |
This is a really strange issue. By looking at the code, I don't see how the JVM could be wrong about the class name either. Not sure if things go wrong inside JVM or before calling into JVM. But we can add assertions in the JVM code that the class being searched for and the class returned has the same name. |
|
@thallium is looking at adding the assertions. |
|
@tjwatson Hi, just want to confirm that you're looking for a build of JDK17 on windows? |
We have a very large set of configurations we run our tests on ranging from Java 8 to Java 21. If we got a test build we would have to figure out a way to update some set of them to run with the debugging build. I also have a separate build setup just for InstantOn where I have tested out development builds of OpenJ9, but that setup only runs instantOn tests and I am unsure we have seen this type of failure using Semeru InstantOn with any regularity. Regardless, if you have a developer build you want us to try let me know and I can see if there is something we can do. |
|
@tjwatson Hi, I've built JDK 17 and JDK 21 on both Linux and Windows: |
What do we expect to see if the issue happens when running these builds? |
It should trigger an assertion and I also added a trace point containing relevant information. You can find my changes here: #20501 |
@thallium It will be more helpful if you can add the message about assertion trace id and line number etc which would be printed out in the console so that Thomas can easily identify. |
|
The assertion is just The trace point added is: |
|
Will any additional -Xdump options be necessary or helpful along with this build? Or are there other options I need to set to make the trace enabled? |
The new trace are enabled by default. The trace assertion failure will automatically trigger diagnostic files by default. No more option needed. |
|
We have observed this strange verify error that I think can be explained by the wrong class being returned: Note that The stack frame |
|
The error in comment is more related to verifier that does not allow uninitialized this to be assigned to the super class. |
This is related to Open J9 issue eclipse-openj9/openj9#20444 There is a concern that the initial call to findLoadedClass by Equinox is not protected by the same loader-classname lock as the later calls to findLoadedClass/defineClass calls. There are concerns that the unprotected findLoadedClass call could catch the JVM in an invalid state if there is another thread in the middle of defining the class. This changes replaces the clunky way of doing loader-classname locks with a more efficient strategy. This way we can use this lock around the initial call to findLoadedClass without too much concern over the performance impact. This change also removes some of the old code that still supproted Java 6. It is always assumed we now can register the class loaders as parallel capable.
Similar to the exception, I cannot make any sense of that. Nothing in the code is trying to assign unninialized this to the super class type. It seems more likely we got the wrong class back and trying to assign to the wrong class fails. |
This is related to Open J9 issue eclipse-openj9/openj9#20444 There is a concern that the initial call to findLoadedClass by Equinox is not protected by the same loader-classname lock as the later calls to findLoadedClass/defineClass are. There are concerns that the unprotected findLoadedClass call could catch the JVM in an invalid state if there is another thread in the middle of defining the class. This change replaces the clunky way of doing loader-classname locks with a more efficient strategy. This way we can use this lock around the initial call to findLoadedClass without too much concern over the performance impact. This change also removes some of the old code that still supported Java 6. It is always assumed we now can register the class loaders as parallel capable.
This is related to Open J9 issue eclipse-openj9/openj9#20444 There is a concern that the initial call to findLoadedClass by Equinox is not protected by the same loader-classname lock as the later calls to findLoadedClass/defineClass are. There are concerns that the unprotected findLoadedClass call could catch the JVM in an invalid state if there is another thread in the middle of defining the class. This change replaces the clunky way of doing loader-classname locks with a more efficient strategy. This way we can use this lock around the initial call to findLoadedClass without too much concern over the performance impact. This change also removes some of the old code that still supported Java 6. It is always assumed we now can register the class loaders as parallel capable.
This is related to Open J9 issue eclipse-openj9/openj9#20444 There is a concern that the initial call to findLoadedClass by Equinox is not protected by the same loader-classname lock as the later calls to findLoadedClass/defineClass are. There are concerns that the unprotected findLoadedClass call could catch the JVM in an invalid state if there is another thread in the middle of defining the class. This change replaces the clunky way of doing loader-classname locks with a more efficient strategy. This way we can use this lock around the initial call to findLoadedClass without too much concern over the performance impact. This change also removes some of the old code that still supported Java 6. It is always assumed we now can register the class loaders as parallel capable.
This is related to Open J9 issue eclipse-openj9/openj9#20444 There is a concern that the initial call to findLoadedClass by Equinox is not protected by the same loader-classname lock as the later calls to findLoadedClass/defineClass are. There are concerns that the unprotected findLoadedClass call could catch the JVM in an invalid state if there is another thread in the middle of defining the class. This change replaces the clunky way of doing loader-classname locks with a more efficient strategy. This way we can use this lock around the initial call to findLoadedClass without too much concern over the performance impact. This change also removes some of the old code that still supported Java 6. It is always assumed we now can register the class loaders as parallel capable.
Java -version output
Summary of problem
java.lang.NoClassDefFoundError thrown when loading class com.ibm.ws.classloading.internal.providers.Providers
Diagnostic files
Diagnostic files here captured by option:
-Xdump:system+java+snap:events=throw+systhrow,filter=java/lang/NoClassDefFoundError,msg_filter=com.ibm.ws.classloading.internal.providers.Providers,request=exclusive+prepwalk+serial+preemptThe text was updated successfully, but these errors were encountered: