Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

openJcePlusTests_0_FAILED Segmentation error vmState=0x00020019 #19643

Closed
JasonFengJ9 opened this issue Jun 5, 2024 · 18 comments
Closed

openJcePlusTests_0_FAILED Segmentation error vmState=0x00020019 #19643

JasonFengJ9 opened this issue Jun 5, 2024 · 18 comments
Labels
comp:crypto test failure
Milestone
Release 0.46 (Jav...

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Jun 5, 2024 •
edited
Loading

Failure link

From an internal build(ubu20le-rtp-rt12-1):

openjdk version "22.0.1-beta" 2024-04-16
IBM Semeru Runtime Open Edition 22.0.1+8-202406050155 (build 22.0.1-beta+8-202406050155)
Eclipse OpenJ9 VM 22.0.1+8-202406050155 (build master-228602c60d, JRE 22 Linux ppc64le-64-Bit Compressed References 20240605_83 (JIT enabled, AOT enabled)
OpenJ9   - 228602c60d
OMR      - a3bc75a88
JCL      - 0e12d35c8 based on jdk-22.0.1+8)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-06-05T03:03:33.046Z] variation: NoOptions
[2024-06-05T03:03:33.046Z] JVM_OPTIONS:  

[2024-06-05T03:04:46.703Z]      [test]     [junit] Test finished: ibm.jceplus.junit.openjceplus.multithread.TestDSAKey
[2024-06-05T03:04:46.703Z]      [test]     [junit] Test calling: ibm.jceplus.junit.openjceplus.multithread.TestAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:04:46.703Z]      [test]     [junit] executing testAES_128
[2024-06-05T03:05:05.528Z]      [test]     [junit] Unhandled exception
[2024-06-05T03:05:05.528Z]      [test]     [junit] Type=Segmentation error vmState=0x00020019
[2024-06-05T03:05:05.528Z]      [test]     [junit] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2024-06-05T03:05:05.528Z]      [test]     [junit] Handler1=0000706424632130 Handler2=00007064245597C0
[2024-06-05T03:05:05.528Z]      [test]     [junit] R0=000070641E68FAD0 R1=00007063E056B350 R2=000070641E7F5500 R3=0000706420061A90
[2024-06-05T03:05:05.528Z]      [test]     [junit] R4=0000000000000000 R5=00000000E1FCA338 R6=00007063E056B5B0 R7=00000000000004F8
[2024-06-05T03:05:05.528Z]      [test]     [junit] R8=000070641E4A52A0 R9=0000000000000000 R10=0000000000000000 R11=0000000000000000
[2024-06-05T03:05:05.528Z]      [test]     [junit] R12=000070641E6DBEB0 R13=00007063E05768F0 R14=00000000005AF620 R15=000000000051C400
[2024-06-05T03:05:05.528Z]      [test]     [junit] R16=00007063E3A30038 R17=0000000081327990 R18=00000000810E2E68 R19=0000000000010000
[2024-06-05T03:05:05.528Z]      [test]     [junit] R20=000000000000003C R21=0000000000000010 R22=000000000000004A R23=00007063E056B5B0
[2024-06-05T03:05:05.528Z]      [test]     [junit] R24=0000000000000001 R25=0000000000000000 R26=00000000E1FCA338 R27=0000706420054480
[2024-06-05T03:05:05.528Z]      [test]     [junit] R28=0000706420300DE8 R29=00007063E056B5B0 R30=0000706420054480 R31=00000000E1FCA338
[2024-06-05T03:05:05.528Z]      [test]     [junit] NIP=000070641E6E9194 MSR=800000000280F033 ORIG_GPR3=C000000000F2ECAC CTR=000070641E6DBEB0
[2024-06-05T03:05:05.528Z]      [test]     [junit] LINK=000070641E6E92D0 XER=0000000020000000 CCR=0000000024884824 SOFTE=0000000000000001
[2024-06-05T03:05:05.528Z]      [test]     [junit] TRAP=0000000000000300 DAR=0000000000000018 dsisr=0000000040000000 RESULT=0000000000000000
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR0=0000000000010008 (f: 65544.000000, d: 3.238304e-319)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR1=0000000000010821 (f: 67617.000000, d: 3.340724e-319)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR2=3f292696d10db7dd (f: 3507337216.000000, d: 1.918849e-04)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR3=20746f6e20646944 (f: 543451456.000000, d: 2.438612e-152)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR4=3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR5=bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR6=3fd27616c9496e0b (f: 3377032704.000000, d: 2.884576e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR7=bfd71969a075c67a (f: 2692072960.000000, d: -3.609261e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR8=3ff71547652ab82b (f: 1697298432.000000, d: 1.442695e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR9=3fdec70a6ca7badd (f: 1822931712.000000, d: 4.808985e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR10=3fcebfce50fac4f3 (f: 1358611712.000000, d: 2.402285e-01)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR11=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR12=47efffffe0000000 (f: 3758096384.000000, d: 3.402823e+38)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR13=4008000000000000 (f: 0.000000, d: 3.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-06-05T03:05:05.528Z]      [test]     [junit] Module=/home/jenkins/workspace/Test_openjdk22_j9_extended.functional_ppc64le_linux_testList_0/jdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2024-06-05T03:05:05.528Z]      [test]     [junit] Module_base_address=000070641E470000
[2024-06-05T03:05:05.528Z]      [test]     [junit] Target=2_90_20240605_83 (Linux 5.4.0-182-generic)
[2024-06-05T03:05:05.528Z]      [test]     [junit] CPU=ppc64le (4 logical CPUs) (0x1fcc00000 RAM)
[2024-06-05T03:05:05.528Z]      [test]     [junit] ----------- Stack Backtrace -----------
[2024-06-05T03:05:05.528Z]      [test]     [junit] _ZN33MM_IndexableObjectAllocationModel25initializeIndexableObjectEP18MM_EnvironmentBasePv+0x1b4 (0x000070641E6E9194 [libj9gc29.so+0x279194])
[2024-06-05T03:05:05.528Z]      [test]     [junit] _ZN22GC_ObjectModelDelegate20initializeAllocationEP18MM_EnvironmentBasePvP25MM_AllocateInitialization+0x30 (0x000070641E68FAD0 [libj9gc29.so+0x21fad0])
[2024-06-05T03:05:05.528Z]      [test]     [junit] _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x16c (0x000070641E60767C [libj9gc29.so+0x19767c])
[2024-06-05T03:05:05.528Z]      [test]     [junit] J9AllocateIndexableObjectNoGC+0x384 (0x000070641E4D1284 [libj9gc29.so+0x61284])
[2024-06-05T03:05:05.528Z]      [test]     [junit] fast_jitNewArray+0x58 (0x000070641F4594E8 [libj9jit29.so+0xc594e8])
[2024-06-05T03:05:05.528Z]      [test]     [junit]  (0x000070641F46FB84 [libj9jit29.so+0xc6fb84])
[2024-06-05T03:05:05.528Z]      [test]     [junit] runJavaThread+0x250 (0x00007064246089E0 [libj9vm29.so+0x189e0])
[2024-06-05T03:05:05.528Z]      [test]     [junit] javaProtectedThreadProc+0xc8 (0x00007064246A8908 [libj9vm29.so+0xb8908])
[2024-06-05T03:05:05.528Z]      [test]     [junit] omrsig_protect+0x3e4 (0x000070642455AC74 [libj9prt29.so+0x3ac74])
[2024-06-05T03:05:05.528Z]      [test]     [junit] javaThreadProc+0x60 (0x00007064246A3DE0 [libj9vm29.so+0xb3de0])
[2024-06-05T03:05:05.528Z]      [test]     [junit] thread_wrapper+0x190 (0x00007064244ED820 [libj9thr29.so+0xd820])
[2024-06-05T03:05:05.528Z]      [test]     [junit] start_thread+0xe8 (0x00007064250E8838 [libpthread.so.0+0x8838])
[2024-06-05T03:05:05.528Z]      [test]     [junit] clone+0x74 (0x0000706424FBBA44 [libc.so.6+0x14ba44])
[2024-06-05T03:05:05.528Z]      [test]     [junit] ---------------------------------------
[2024-06-05T03:05:05.528Z]      [test]     [junit] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/06/04 20:05:04 - please wait.

.....

[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400: Object neither in heap nor stack-allocated in thread pool-16-thread-3
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	O-Slot=00000000005AF598
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	O-Slot value=0000000000010000
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	PC=00007064023B9F20
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	framesWalked=1
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	arg0EA=00000000005AF598
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	walkSP=00000000005AF3F0
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	literals=0000000000000008
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	jitInfo=00007063E3BEA778
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	method=000000000052E230 (com/ibm/crypto/plus/provider/ock/SymmetricCipher.initCipher(Z[B[B)V) (JIT)
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	stack=00000000005AADF0-00000000005B0770
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400: Object neither in heap nor stack-allocated in thread pool-16-thread-3
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	O-Slot=00000000005AF5D0
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	O-Slot value=000000000000003A
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	PC=0000706402386128
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	framesWalked=2
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	arg0EA=00000000005AF5F0
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	walkSP=00000000005AF3F0
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	literals=0000000000000008
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	jitInfo=00007063E3BF3A78
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	method=000000000052E210 (com/ibm/crypto/plus/provider/ock/SymmetricCipher.initCipherDecrypt([B[B)V) (JIT)
[2024-06-05T03:05:21.820Z]      [test]     [junit] 000000000051C400:	stack=00000000005AADF0-00000000005B0770
[2024-06-05T03:05:21.820Z]      [test]     [junit] 
[2024-06-05T03:05:21.820Z]      [test]     [junit] 
[2024-06-05T03:05:21.820Z]      [test]     [junit] *** Invalid JIT return address 00000000810EF1A8 in 00007063E33BB968
[2024-06-05T03:05:21.820Z]      [test]     [junit] 
[2024-06-05T03:05:23.183Z]      [test]     [junit] JVMDUMP053I JIT dump is recompiling java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
[2024-06-05T03:05:23.183Z]      [test]     [junit] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk22_j9_extended.functional_ppc64le_linux_testList_0/jvmtest/functional/OpenJcePlusTests/jitdump.20240604.200504.64446.0004.dmp
[2024-06-05T03:05:23.183Z]      [test]     [junit] JVMDUMP013I Processed dump event "gpf", detail "".
[2024-06-05T03:05:23.183Z]      [test]     [junit] Running ibm.jceplus.junit.TestMultithread
[2024-06-05T03:05:23.183Z]      [test]     [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
[2024-06-05T03:05:23.183Z]      [test] 
[2024-06-05T03:05:23.183Z]      [test] BUILD FAILED
[2024-06-05T03:05:23.183Z]      [test] /home/jenkins/workspace/Test_openjdk22_j9_extended.functional_ppc64le_linux_testList_0/jvmtest/functional/OpenJcePlusTests/test.xml:61: Test ibm.jceplus.junit.TestMultithread failed (crashed)
[2024-06-05T03:05:23.183Z]      [test] 
[2024-06-05T03:05:23.183Z]      [test] Total time: 1 minute 46 seconds
[2024-06-05T03:05:23.183Z] 
[2024-06-05T03:05:23.183Z] BUILD FAILED
[2024-06-05T03:05:23.183Z] /home/jenkins/workspace/Test_openjdk22_j9_extended.functional_ppc64le_linux_testList_0/jvmtest/functional/OpenJcePlusTests/test.xml:35: Java returned: 1
[2024-06-05T03:05:23.183Z] 
[2024-06-05T03:05:23.183Z] Total time: 1 minute 49 seconds
[2024-06-05T03:05:23.183Z] -----------------------------------
[2024-06-05T03:05:23.183Z] openJcePlusTests_0_FAILED

50x internal grinder - 2 original failures were reproduced and 2 other similar ones seem the same issues.

FYI @jasonkatonica @KostasTsiounis
@dmitripivkine
Copy link
Contributor

There is no stored results for openJcePlusTests_0 test, folder is empty. Also, there are couple of reproductions in the grinder, but there are no stored results also.
This crash occur at the attempt to initialize header of the newly allocated indexable object, most likely due heap corruption. We have seen failures like this before.

@llxia
Copy link
Contributor

llxia commented Jun 6, 2024

The dmp is generated under /home/jenkins/workspace/Test_openjdk22_j9_extended.functional_ppc64le_linux_testList_0/jvmtest/functional/OpenJcePlusTests/jitdump.20240604.200504.64446.0004.dmp, not testoutput dir. This should be fixed.

For now, I am running Grinder with KEEP_WORKSPACE = true and I will upload the core later.

@dmitripivkine
Copy link
Contributor

core from this job should be good enough, this is what are looking for: https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_2/3655/tapResults/

     [test]     [junit] Indexable object allocation sanity failure: object 00000000E58F2218, requested 16 bytes, but read 1048584, MM_IndexableObjectAllocationModel 000071498647B5B0
     [test]     [junit] 15:48:35.449 0x4c9100    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk22/jdk22-linux-ppc64le-openj9/workspace/build/src/openj9/runtime/gc_modron_startup/mgcalloc.cpp:367: ((false))
     [test]     [junit] JVMDUMP039I Processing dump event "traceassert", detail "" at 2024/06/06 08:48:35 - please wait.
     [test]     [junit] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_2/jvmtest/functional/OpenJcePlusTests/core.20240606.084835.104694.0001.dmp' in response to an event
     [test]     [junit] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_2/jvmtest/functional/OpenJcePlusTests/core.20240606.084835.104694.0001.dmp

@dmitripivkine
Copy link
Contributor

...but I don't see it on machine. May be I am looking in the wrong place?

@dmitripivkine
Copy link
Contributor

There is another one https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_3/2992/tapResults/:

     [test]     [junit] Unhandled exception
     [test]     [junit] Type=Segmentation error vmState=0x00000000
     [test]     [junit] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
     [test]     [junit] Handler1=00007FFFBDB72130 Handler2=00007FFFBDA997C0
     [test]     [junit] R0=00007FFFBC061284 R1=00007FFF60F6B530 R2=00007FFFBC385500 R3=00000000ECFF11A8
     [test]     [junit] R4=0000000000000000 R5=00007FFF60F6B5A0 R6=00007FFFB80AA450 R7=0000000000000000
     [test]     [junit] R8=00000000007D17E0 R9=0000000000000000 R10=00007FFFB802F148 R11=0000000000000010
     [test]     [junit] R12=00007FFFBC15FAB0 R13=00007FFF60F768E0 R14=0000000000584430 R15=0000000000495D00
     [test]     [junit] R16=00007FFF996C0038 R17=00000000E234CE30 R18=00000000EC64D148 R19=00000000FD063330
     [test]     [junit] R20=00000000FDF8CF50 R21=00007FFFB80572B0 R22=00000000E6C6C0A8 R23=00007FFF60F6B590
     [test]     [junit] R24=0000000000000001 R25=0000000000000000 R26=00000000000B7C00 R27=00007FFFB83A6C08
     [test]     [junit] R28=0000000000495D00 R29=00000000ECFF11A8 R30=0000000000080000 R31=00007FFFB80573C0
     [test]     [junit] NIP=00007FFFBC0612AC MSR=800000000280F033 ORIG_GPR3=00007FFF611AAA90 CTR=00007FFFBC15FAB0
     [test]     [junit] LINK=00007FFFBC061284 XER=0000000020000000 CCR=0000000042884842 SOFTE=0000000000000001
     [test]     [junit] TRAP=0000000000000300 DAR=0000000000000018 dsisr=0000000040000000 RESULT=0000000000000000
     [test]     [junit] FPR0=3fcbec0940000000 (f: 1073741824.000000, d: 2.181408e-01)
     [test]     [junit] FPR1=0000000000093301 (f: 602881.000000, d: 2.978628e-318)
     [test]     [junit] FPR2=3ff027caefc0401f (f: 4022353920.000000, d: 1.009715e+00)
     [test]     [junit] FPR3=20746f6e20646944 (f: 543451456.000000, d: 2.438612e-152)
     [test]     [junit] FPR4=3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
     [test]     [junit] FPR5=bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
     [test]     [junit] FPR6=3fd27616c9496e0b (f: 3377032704.000000, d: 2.884576e-01)
     [test]     [junit] FPR7=3ff71547652ab82b (f: 1697298432.000000, d: 1.442695e+00)
     [test]     [junit] FPR8=3fdec70a6ca7badd (f: 1822931712.000000, d: 4.808985e-01)
     [test]     [junit] FPR9=3e7798867c25010b (f: 2082799872.000000, d: 8.790121e-08)
     [test]     [junit] FPR10=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
     [test]     [junit] FPR11=0000000000042102 (f: 270594.000000, d: 1.336912e-318)
     [test]     [junit] FPR12=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
     [test]     [junit] FPR13=4008000000000000 (f: 0.000000, d: 3.000000e+00)
     [test]     [junit] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] Module=/home/jenkins/workspace/Grinder_iteration_3/jdkbinary/j2sdk-image/lib/default/libj9gc29.so
     [test]     [junit] Module_base_address=00007FFFBC000000
     [test]     [junit] Target=2_90_20240605_83 (Linux 4.18.0-513.24.1.el8_9.ppc64le)
     [test]     [junit] CPU=ppc64le (4 logical CPUs) (0x1dcc60000 RAM)
     [test]     [junit] ----------- Stack Backtrace -----------
     [test]     [junit] J9AllocateIndexableObjectNoGC+0x3ac (0x00007FFFBC0612AC [libj9gc29.so+0x612ac])
     [test]     [junit] fast_jitNewArray+0x58 (0x00007FFFBCFE94E8 [libj9jit29.so+0xc594e8])
     [test]     [junit]  (0x00007FFFBCFFFB84 [libj9jit29.so+0xc6fb84])
     [test]     [junit] runJavaThread+0x250 (0x00007FFFBDB489E0 [libj9vm29.so+0x189e0])
     [test]     [junit] javaProtectedThreadProc+0xc8 (0x00007FFFBDBE8908 [libj9vm29.so+0xb8908])
     [test]     [junit] omrsig_protect+0x3e4 (0x00007FFFBDA9AC74 [libj9prt29.so+0x3ac74])
     [test]     [junit] javaThreadProc+0x60 (0x00007FFFBDBE3DE0 [libj9vm29.so+0xb3de0])
     [test]     [junit] thread_wrapper+0x190 (0x00007FFFBDA2D820 [libj9thr29.so+0xd820])
     [test]     [junit] start_thread+0xf8 (0x00007FFFBE5A9718 [libpthread.so.0+0x9718])
     [test]     [junit] clone+0x74 (0x00007FFFBE48B798 [libc.so.6+0x13b798])
     [test]     [junit] ---------------------------------------
     [test]     [junit] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/06/06 09:35:17 - please wait.
     [test]     [junit] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_3/jvmtest/functional/OpenJcePlusTests/core.20240606.093517.85515.0001.dmp' in response to an event
     [test]     [junit] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_3/jvmtest/functional/OpenJcePlusTests/core.20240606.093517.85515.0001.dmp

but I have no access to machine

@dmitripivkine
Copy link
Contributor

and another one https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_4/2848/tapResults/

     [test]     [junit] Type=Segmentation error vmState=0x00020019
     [test]     [junit] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
     [test]     [junit] Handler1=000070F47E4E2130 Handler2=000070F47E4097C0
     [test]     [junit] R0=000070F47C6EFAD0 R1=000070F45B58B350 R2=000070F47C855500 R3=000070F478060870
     [test]     [junit] R4=0000000000000000 R5=00000000F8814568 R6=000070F45B58B5B0 R7=0000000000000001
     [test]     [junit] R8=000070F47C5052A0 R9=0000000000000000 R10=0000000000000000 R11=0000000000000010
     [test]     [junit] R12=000070F47C73BEB0 R13=000070F45B5968F0 R14=00000000005E2470 R15=0000000000492E00
     [test]     [junit] R16=000070F4598F0038 R17=FFFFFFFFFFFFFFFF R18=0000000000000000 R19=0000000000000010
     [test]     [junit] R20=00000000FCD20D48 R21=00000000F8774550 R22=00000000F86C4530 R23=000070F45B58B5B0
     [test]     [junit] R24=0000000000000001 R25=0000000000000000 R26=00000000F8814568 R27=000070F478053230
     [test]     [junit] R28=000070F4783AA828 R29=000070F45B58B5B0 R30=000070F478053230 R31=00000000F8814568
     [test]     [junit] NIP=000070F47C749194 MSR=800000000280F033 ORIG_GPR3=000070F45BA9AA90 CTR=000070F47C73BEB0
     [test]     [junit] LINK=000070F47C7492D0 XER=0000000020000000 CCR=0000000024884840 SOFTE=0000000000000001
     [test]     [junit] TRAP=0000000000000300 DAR=0000000000000018 dsisr=0000000040000000 RESULT=0000000000000000
     [test]     [junit] FPR0=0000000000080008 (f: 524296.000000, d: 2.590366e-318)
     [test]     [junit] FPR1=0000000000093301 (f: 602881.000000, d: 2.978628e-318)
     [test]     [junit] FPR2=3f29bf881b96de5f (f: 462872160.000000, d: 1.964430e-04)
     [test]     [junit] FPR3=6574707972636544 (f: 1919116672.000000, d: 5.300838e+180)
     [test]     [junit] FPR4=3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
     [test]     [junit] FPR5=bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
     [test]     [junit] FPR6=3fd27616c9496e0b (f: 3377032704.000000, d: 2.884576e-01)
     [test]     [junit] FPR7=bfd71969a075c67a (f: 2692072960.000000, d: -3.609261e-01)
     [test]     [junit] FPR8=3ff71547652ab82b (f: 1697298432.000000, d: 1.442695e+00)
     [test]     [junit] FPR9=3fdec70a6ca7badd (f: 1822931712.000000, d: 4.808985e-01)
     [test]     [junit] FPR10=3fcebfce50fac4f3 (f: 1358611712.000000, d: 2.402285e-01)
     [test]     [junit] FPR11=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
     [test]     [junit] FPR12=47efffffe0000000 (f: 3758096384.000000, d: 3.402823e+38)
     [test]     [junit] FPR13=4008000000000000 (f: 0.000000, d: 3.000000e+00)
     [test]     [junit] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [test]     [junit] Module=/home/jenkins/workspace/Grinder_iteration_4/jdkbinary/j2sdk-image/lib/default/libj9gc29.so
     [test]     [junit] Module_base_address=000070F47C4D0000
     [test]     [junit] Target=2_90_20240605_83 (Linux 5.4.0-182-generic)
     [test]     [junit] CPU=ppc64le (4 logical CPUs) (0x1fcc00000 RAM)
     [test]     [junit] ----------- Stack Backtrace -----------
     [test]     [junit] _ZN33MM_IndexableObjectAllocationModel25initializeIndexableObjectEP18MM_EnvironmentBasePv+0x1b4 (0x000070F47C749194 [libj9gc29.so+0x279194])
     [test]     [junit] _ZN22GC_ObjectModelDelegate20initializeAllocationEP18MM_EnvironmentBasePvP25MM_AllocateInitialization+0x30 (0x000070F47C6EFAD0 [libj9gc29.so+0x21fad0])
     [test]     [junit] _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x16c (0x000070F47C66767C [libj9gc29.so+0x19767c])
     [test]     [junit] J9AllocateIndexableObjectNoGC+0x384 (0x000070F47C531284 [libj9gc29.so+0x61284])
     [test]     [junit] fast_jitNewArray+0x58 (0x000070F47D4B94E8 [libj9jit29.so+0xc594e8])
     [test]     [junit]  (0x000070F47D4CFB84 [libj9jit29.so+0xc6fb84])
     [test]     [junit] runJavaThread+0x250 (0x000070F47E4B89E0 [libj9vm29.so+0x189e0])
     [test]     [junit] javaProtectedThreadProc+0xc8 (0x000070F47E558908 [libj9vm29.so+0xb8908])
     [test]     [junit] omrsig_protect+0x3e4 (0x000070F47E40AC74 [libj9prt29.so+0x3ac74])
     [test]     [junit] javaThreadProc+0x60 (0x000070F47E553DE0 [libj9vm29.so+0xb3de0])
     [test]     [junit] thread_wrapper+0x190 (0x000070F47E39D820 [libj9thr29.so+0xd820])
     [test]     [junit] start_thread+0xe8 (0x000070F47EF98838 [libpthread.so.0+0x8838])
     [test]     [junit] clone+0x74 (0x000070F47EE6BA44 [libc.so.6+0x14ba44])
     [test]     [junit] ---------------------------------------
     [test]     [junit] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/06/06 17:41:16 - please wait.
     [test]     [junit] 0000000000492E00: Object neither in heap nor stack-allocated in thread pool-18-thread-10
     [test]     [junit] 0000000000492E00:	O-Slot=00000000005E2530
     [test]     [junit] 0000000000492E00:	O-Slot value=0000000000080000
     [test]     [junit] 0000000000492E00:	PC=000070F45C5E5518
     [test]     [junit] 0000000000492E00:	framesWalked=2
     [test]     [junit] 0000000000492E00:	arg0EA=00000000005E2598
     [test]     [junit] 0000000000492E00:	walkSP=00000000005E22E8
     [test]     [junit] 0000000000492E00:	literals=0000000000000000
     [test]     [junit] 0000000000492E00:	jitInfo=000070F459A98778
     [test]     [junit] 0000000000492E00:	method=0000000000508840 (com/ibm/crypto/plus/provider/AESCipher.internalInit(ILjava/security/Key;[B)V) (JIT)
     [test]     [junit] 0000000000492E00:	stack=00000000005DD9D8-00000000005E32E0
     [test]     [junit] 
     [test]     [junit] 
     [test]     [junit] *** Invalid JIT return address 00000000F3B02490 in 000070F45914B968
     [test]     [junit] 
     [test]     [junit] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_4/jvmtest/functional/OpenJcePlusTests/core.20240606.174116.837929.0001.dmp' in response to an event

but there is nothing left on machine also

@llxia
Copy link
Contributor

llxia commented Jun 7, 2024

The workspace does not exist anymore. I think the clean up job (runs twice a day) wiped out everything.
I will modify the test to move the dmp into the expected dir.

@llxia
Copy link
Contributor

llxia commented Jun 7, 2024

just for the record, the output is in link.

@llxia llxia mentioned this issue Jun 7, 2024
Closed
@llxia
Copy link
Contributor

llxia commented Jun 7, 2024

Test issue is opened at adoptium/aqa-tests#5373 for core files generated outside of REPORTDIR.

@dmitripivkine
Copy link
Contributor

There is reproduction in internal grinder, core is available:

10:13:52       [test]     [junit] Type=Segmentation error vmState=0x00020019
10:13:52       [test]     [junit] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
10:13:52       [test]     [junit] Handler1=00007FFFBDBC2130 Handler2=00007FFFBDAE97C0
10:13:52       [test]     [junit] R0=00007FFFBC1EBD7C R1=00007FFECF60AF80 R2=00007FFFBC3D5500 R3=00007FFFBDE76C00
10:13:52       [test]     [junit] R4=00007FFECF60AFE8 R5=0000000000000100 R6=00007FFFBC32AAE8 R7=00007FFECF60B148
10:13:52       [test]     [junit] R8=00007FFFBDFDF7F0 R9=0000000000000000 R10=000000003F34F581 R11=0000000000000000
10:13:52       [test]     [junit] R12=0000000000000000 R13=00007FFECF6168E0 R14=0000000000BA10C0 R15=00000000004D5F00
10:13:52       [test]     [junit] R16=0000000000000000 R17=00000001C032CD18 R18=0000000000000000 R19=0000000000002208
10:13:52       [test]     [junit] R20=000000000000000D R21=00000006F0276C88 R22=00000000000029F8 R23=5A7AC0BF092F3F9E
10:13:52       [test]     [junit] R24=00007FFFB8108560 R25=00007FFFB81083A8 R26=00007FFECF60B5A0 R27=0000000000000000
10:13:52       [test]     [junit] R28=0000000000000001 R29=00007FFECF60AFE8 R30=00007FFFBDDABD10 R31=00007FFFBDE76C00
10:13:52       [test]     [junit] NIP=0000000000000000 MSR=800000004280F033 ORIG_GPR3=00007FFFBC1D1A48 CTR=0000000000000000
10:13:52       [test]     [junit] LINK=00007FFFBC1D1A4C XER=0000000000000000 CCR=0000000084882A28 SOFTE=0000000000000001
10:13:52       [test]     [junit] TRAP=0000000000000400 DAR=0000000000000000 dsisr=0000000040000000 RESULT=0000000000000000
10:13:52       [test]     [junit] FPR0=00000000000000ed (f: 237.000000, d: 1.170936e-321)
10:13:52       [test]     [junit] FPR1=406db38200000000 (f: 0.000000, d: 2.376096e+02)
10:13:52       [test]     [junit] FPR2=3ff018277cec2ac4 (f: 2095852288.000000, d: 1.005897e+00)
10:13:52       [test]     [junit] FPR3=3fee666660000000 (f: 1610612736.000000, d: 9.500000e-01)
10:13:52       [test]     [junit] FPR4=3fce840b4ac4e4d2 (f: 1254417664.000000, d: 2.384047e-01)
10:13:52       [test]     [junit] FPR5=bfe7154748bef6c8 (f: 1220474624.000000, d: -7.213475e-01)
10:13:52       [test]     [junit] FPR6=3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
10:13:52       [test]     [junit] FPR7=bfd57bf7808caade (f: 2156702464.000000, d: -3.356913e-01)
10:13:52       [test]     [junit] FPR8=bfdffffef20a4123 (f: 4060758272.000000, d: -4.999997e-01)
10:13:52       [test]     [junit] FPR9=bfd00ea348b88334 (f: 1220051712.000000, d: -2.508934e-01)
10:13:52       [test]     [junit] FPR10=bfe01e76d9261ee1 (f: 3643154176.000000, d: -5.037188e-01)
10:13:52       [test]     [junit] FPR11=3f204dca6b289cb9 (f: 1797823616.000000, d: 1.243887e-04)
10:13:52       [test]     [junit] FPR12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR13=00007ffecf84a7c0 (f: 3481577472.000000, d: 6.953103e-310)
10:13:52       [test]     [junit] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
10:13:52       [test]     [junit] FPR31=5a7ac0bf092f3f9e (f: 154091424.000000, d: 7.243847e+127)
10:13:52       [test]     [junit] Target=2_90_20240605_83 (Linux 4.18.0-513.24.1.el8_9.ppc64le)
10:13:52       [test]     [junit] CPU=ppc64le (24 logical CPUs) (0x1ef26a0000 RAM)
10:13:52       [test]     [junit] ----------- Stack Backtrace -----------
10:13:52       [test]     [junit]  (0x0000000000000000 [<unknown>+0x0])
10:13:52       [test]     [junit] _ZN27MM_LargeObjectAllocateStats17getSizeClassIndexEm+0xac (0x00007FFFBC1EBD7C [libj9gc29.so+0x19bd7c])
10:13:52       [test]     [junit] _ZN27MM_LargeObjectAllocateStats32decrementFreeEntrySizeClassStatsEmP26MM_FreeEntrySizeClassStatsm+0x38 (0x00007FFFBC1EFDF8 [libj9gc29.so+0x19fdf8])
10:13:52       [test]     [junit] _ZN27MM_LargeObjectAllocateStats32decrementFreeEntrySizeClassStatsEm+0x20 (0x00007FFFBC1F00E0 [libj9gc29.so+0x1a00e0])
10:13:52       [test]     [junit] _ZN31MM_MemoryPoolAddressOrderedList14allocateObjectEP18MM_EnvironmentBaseP22MM_AllocateDescription+0x308 (0x00007FFFBC1BEA68 [libj9gc29.so+0x16ea68])
10:13:52       [test]     [junit] _ZN24MM_MemorySubSpaceGeneric14allocateObjectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP17MM_MemorySubSpaceS5_b+0x32c (0x00007FFFBC28E2AC [libj9gc29.so+0x23e2ac])
10:13:52       [test]     [junit] _ZN25MM_TLHAllocationInterface21allocateArrayletSpineEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x2a8 (0x00007FFFBC1DECD8 [libj9gc29.so+0x18ecd8])
10:13:52       [test]     [junit] _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0xf0 (0x00007FFFBC1E7600 [libj9gc29.so+0x197600])
10:13:52       [test]     [junit] J9AllocateIndexableObjectNoGC+0x384 (0x00007FFFBC0B1284 [libj9gc29.so+0x61284])
10:13:52       [test]     [junit] fast_jitNewArray+0x58 (0x00007FFFBD0394E8 [libj9jit29.so+0xc594e8])
10:13:52       [test]     [junit]  (0x00007FFFBD04FB84 [libj9jit29.so+0xc6fb84])
10:13:52       [test]     [junit] runJavaThread+0x250 (0x00007FFFBDB989E0 [libj9vm29.so+0x189e0])
10:13:52       [test]     [junit] javaProtectedThreadProc+0xc8 (0x00007FFFBDC38908 [libj9vm29.so+0xb8908])
10:13:52       [test]     [junit] omrsig_protect+0x3e4 (0x00007FFFBDAEAC74 [libj9prt29.so+0x3ac74])
10:13:52       [test]     [junit] javaThreadProc+0x60 (0x00007FFFBDC33DE0 [libj9vm29.so+0xb3de0])
10:13:52       [test]     [junit] thread_wrapper+0x190 (0x00007FFFBDA7D820 [libj9thr29.so+0xd820])
10:13:52       [test]     [junit] start_thread+0xf8 (0x00007FFFBE5D9678 [libpthread-2.28.so+0x9678])
10:13:52       [test]     [junit] clone+0x74 (0x00007FFFBE4C8968 [libc-2.28.so+0x138968])
10:13:52       [test]     [junit] ---------------------------------------
10:13:52       [test]     [junit] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/06/07 10:13:52 - please wait.
10:13:52       [test]     [junit] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_5/jvmtest/functional/OpenJcePlusTests/core.20240607.101352.57094.0001.dmp' in response to an event
10:13:55       [test]     [junit] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_5/jvmtest/functional/OpenJcePlusTests/core.20240607.101352.57094.0001.dmp

Just to summarize uo front - there is heap corruption at 0x6F0285478: GC expects Linked Free header but there is !j9indexableobject 0x6F0285478 0 10816

  • An allocation request !MM_IndexableObjectAllocationModel 0x7FFECF60B590 for byte array allocation 0x29F0 bytes.
  • current TLH for !j9vmthread 0x4d5f00 is not large enough to allocate object:
	0x60: U8* heapAlloc = !j9x 0x00000006F0241D50 // ""
	0x68: U8* heapTop = !j9x 0x00000006F0243AA0 // "" <---- 0x1D50 left
  • allocation request goes to !mm_memorypooladdressorderedlist 0x00007FFFB8108050 to get larger piece of memory
  • the free list header in the MP _heapFreeList = !mm_heaplinkedfreeheader 0x00000006F024DE00
  • iterating list for search proper memory chunk:
0x6F024DE00 :  00000006f02527f9 0000000000001dd0 <-- too small
0x6F02527F8 :  00000006f0253f79 0000000000001098 <-- too small
..... until reach
0x6F0276C88 :  00000006f0285479 0000000000001220 <-- too small
0x6F0285478 :  00002a40000ca000 5a7ac0bf092f3f9e <---- corrupted

So, GC expects free memory starts with Linked Free Header, but there is a byte array !j9indexableobject 0x6F0285478 0 10816written after last GC. GC interpreted 0x5a7ac0bf092f3f9e as a free memory chunk size and crashed in the MM_LargeObjectAllocateStats1 code eventually.

The only mentioning of address 0x6F0285478 is in !j9vmthread 0x004daa00 (pool-17-thread-10) in the

	0x120: UDATA returnValue = 0x00000006F0285478 (29798978680)

@KostasTsiounis
Copy link
Contributor

I don't know if that helps, but OpenJCEPlus uses ByteBuffer instances in some points and allocates space outside the heap. Maybe that's what the GC runs into.

@dmitripivkine
Copy link
Contributor

there is one more failure in internal grinder, results are available:

     [test]     [junit] Indexable object allocation sanity failure: object 00000000F86AF1D8, requested 16 bytes, but read 1048584, MM_IndexableObjectAllocationModel 00007A289812B5B0
     [test]     [junit] 13:40:37.968 0x52fa00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk22/jdk22-linux-ppc64le-openj9/workspace/build/src/openj9/runtime/gc_modron_startup/mgcalloc.cpp:367: ((false))

     [test]     [junit] JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_4/jvmtest/functional/OpenJcePlusTests/core.20240607.064037.1915170.0001.dmp
     [test]     [junit] 000000000052FA00: Object neither in heap nor stack-allocated in thread pool-18-thread-7
     [test]     [junit] 000000000052FA00:	O-Slot=00000000005587B8
     [test]     [junit] 000000000052FA00:	O-Slot value=0000000000000010
     [test]     [junit] 000000000052FA00:	PC=00007A28D2FDD920
     [test]     [junit] 000000000052FA00:	framesWalked=1
     [test]     [junit] 000000000052FA00:	arg0EA=00000000005587B8
     [test]     [junit] 000000000052FA00:	walkSP=0000000000558618
     [test]     [junit] 000000000052FA00:	literals=0000000000000000
     [test]     [junit] 000000000052FA00:	jitInfo=00007A28D0A5C938
     [test]     [junit] 000000000052FA00:	method=00000000005743E0 (com/ibm/crypto/plus/provider/ock/SymmetricCipher.initCipher(Z[B[B)V) (JIT)
     [test]     [junit] 000000000052FA00:	stack=0000000000553E90-0000000000559810
     [test]     [junit] 000000000052FA00: Object neither in heap nor stack-allocated in thread pool-18-thread-7
     [test]     [junit] 000000000052FA00:	O-Slot=00000000005588B8
     [test]     [junit] 000000000052FA00:	O-Slot value=000000000000004C
     [test]     [junit] 000000000052FA00:	PC=00007A28D358F4A0
     [test]     [junit] 000000000052FA00:	framesWalked=2
     [test]     [junit] 000000000052FA00:	arg0EA=00000000005588C8
     [test]     [junit] 000000000052FA00:	walkSP=0000000000558618
     [test]     [junit] 000000000052FA00:	literals=0000000000000000
     [test]     [junit] 000000000052FA00:	jitInfo=00007A28D0A57D38
     [test]     [junit] 000000000052FA00:	method=000000000050FC80 (com/ibm/crypto/plus/provider/AESCipher.internalInit(ILjava/security/Key;[B)V) (JIT)
     [test]     [junit] 000000000052FA00:	stack=0000000000553E90-0000000000559810
     [test]     [junit] 000000000052FA00: Object neither in heap nor stack-allocated in thread pool-18-thread-7
     [test]     [junit] 000000000052FA00:	O-Slot=0000000000558880
     [test]     [junit] 000000000052FA00:	O-Slot value=0000000000000010
     [test]     [junit] 000000000052FA00:	PC=00007A28D358F4A0
     [test]     [junit] 000000000052FA00:	framesWalked=2
     [test]     [junit] 000000000052FA00:	arg0EA=00000000005588C8
     [test]     [junit] 000000000052FA00:	walkSP=0000000000558618
     [test]     [junit] 000000000052FA00:	literals=0000000000000000
     [test]     [junit] 000000000052FA00:	jitInfo=00007A28D0A57D38
     [test]     [junit] 000000000052FA00:	method=000000000050FC80 (com/ibm/crypto/plus/provider/AESCipher.internalInit(ILjava/security/Key;[B)V) (JIT)
     [test]     [junit] 000000000052FA00:	stack=0000000000553E90-0000000000559810
     [test]     [junit] 
     [test]     [junit] 
     [test]     [junit] *** Invalid JIT return address 0000000000000000 in 00007A28D019B968
     [test]     [junit]

triaging it.

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Jun 7, 2024 •
edited
Loading 

[test]     [junit] Indexable object allocation sanity failure: object 00000000F86AF1D8, requested 16 bytes, but read 1048584, MM_IndexableObjectAllocationModel 00007A289812B5B0
     [test]     [junit] 13:40:37.968 0x52fa00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk22/jdk22-linux-ppc64le-openj9/workspace/build/src/openj9/runtime/gc_modron_startup/mgcalloc.cpp:367: ((false))

> !MM_IndexableObjectAllocationModel 0x00007A289812B5B0
MM_IndexableObjectAllocationModel at 0x7a289812b5b0 {
  Fields for MM_Base:
  Fields for MM_AllocateInitialization:
	0x0: const U64 _allocationCategory = 0x0000000000000001 (1)
	0x8: bool _isAllocatable = true
	0x10: class MM_AllocateDescription _allocateDescription = !mm_allocatedescription 0x00007A289812B5C0
  Fields for MM_JavaObjectAllocationModel:
	0x88: struct J9Class* _class = !j9class 0x00000000000B7C00 // [B
  Fields for MM_IndexableObjectAllocationModel:
	0x90: const U32 _numberOfIndexedFields = 0x00100000 (1048576)
	0x98: const U64 _dataSize = 0x0000000000100000 (1048576)
	0xa0: const enum GC_ArrayletObjectModelBase::ArrayLayout _layout = 0x1 (1) //InlineContiguous
	0xa4: const bool _alignSpineDataSection = true
	0xa8: const U64 _numberOfArraylets = 0x0000000000000001 (1)
}
> !mm_allocatedescription 0x00007A289812B5C0
MM_AllocateDescription at 0x7a289812b5c0 {
  Fields for MM_Base:
  Fields for MM_AllocateDescription:
	0x0: U64 _bytesRequested = 0x0000000000100008 (1048584) <-------
	0x8: U64 _allocateFlags = 0x0000000000000061 (97)
	0x10: U32 _objectFlags = 0x00000000 (0)
	0x14: bool _allocationSucceeded = true
	0x18: class MM_MemorySpace* _memorySpace = !mm_memoryspace 0x00007A28F00AC090
	0x20: class MM_MemorySubSpace* _memorySubSpace = !mm_memorysubspacegeneric 0x00007A28F00A63D0
	0x28: U64 _allocationTaxSize = 0x0000000000100008 (1048584)
	0x30: bool _tlhAllocation = false
	0x31: bool _nurseryAllocation = true
	0x32: bool _loaAllocation = false
	0x38: U64 _spineBytes = 0x0000000000100008 (1048584)
	0x40: U64 _numArraylets = 0x0000000000000001 (1)
	0x48: bool _chunkedArray = false
	0x49: bool _dataAdjacentToHeader = false
	0x50: struct J9IndexableObject* _spine = !j9indexableobject 0x00000000F86AF1D8 <-----
	0x58: bool _threadAtSafePoint = false
	0x60: class MM_MemoryPool* _memoryPool = !mm_memorypooladdressorderedlist 0x00007A28F00A0BF0
	0x68: bool _collectorAllocateExpandOnFailure = false
	0x69: bool _collectorAllocateSatisfyAnywhere = false
	0x6c: MM_MemorySubSpace$AllocationType _allocationType = 0x0 (0) //ALLOCATION_TYPE_INVALID
	0x70: bool _collectAndClimb = false
	0x71: bool _climb = false
	0x72: bool _completedFromTlh = false
}
>

0xF86AF1A0 :  cfa0c19d7050d739 0b31dd94e893b16c [ 9.Pp....l.....1. ]
0xF86AF1B0 :  198550b7aa0ada98 0dd1a877217ab19f [ .....P....z!w... ]
0xF86AF1C0 :  34ad6b8cd70f29af 8b24f13925ece2b4 [ .)...k.4...%9.$. ]
0xF86AF1D0 :  40d88899d62c1d40 00000000000b7c00 [ @.,....@.|...... ] <----- class but not size
0xF86AF1E0 :  0000000000000000 0000000000000000 [ ................ ]
0xF86AF1F0 :  0000000000000000 0000000000000000 [ ................ ]

GC is allocating array at 0x00000000F86AF1D8. Class (0xB7C00) and size (0x100000) fields have been written to the object header successfully. However on the way out GC double check them: class field is correct but size is reset to zero. This discovery triggers the assertion. Also there is byte array located right before. Operations with this array might be reason for out-of-range write.

Bad O-slots: for `!j9vmthread 0x0052fa00 (pool-18-thread-7):

<52fa00> JIT frame: bp = 0x0000000000558798, pc = 0x00007A28D2FDD920, unwindSP = 0x0000000000558640, cp = 0x0000000000573960, arg0EA = 0x00000000005587B8, jitInfo = 0x00007A28D0A5C938
<52fa00> 	Method: com/ibm/crypto/plus/provider/ock/SymmetricCipher.initCipher(Z[B[B)V !j9method 0x00000000005743E0
<52fa00> 	Bytecode index = 116, inlineDepth = 0, PC offset = 0x00000000000003B0
<52fa00> 	stackMap=0x00007A28D0A5CBBE, slots=I16(0x0004) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0004), localBaseOffset=I16(0xFF98)
<52fa00> 	Described JIT args starting at 0x00000000005587A0 for U16(0x0004) slots
<52fa00> 		O-Slot: : a3[0x00000000005587A0] = 0x00000000E498F5B8
<52fa00> 		O-Slot: : a2[0x00000000005587A8] = 0x000000008139E158
<52fa00> 		I-Slot: : a1[0x00000000005587B0] = 0x00000000E0340850
<52fa00> 		O-Slot: : a0[0x00000000005587B8] = 0x0000000000000010 <--- BAD
<52fa00> 	Described JIT temps starting at 0x0000000000558730 for IDATA(0x000000000000000D) slots
<52fa00> 		I-Slot: : t12[0x0000000000558730] = 0x00000000811B2D00
... 
<52fa00> JIT frame: bp = 0x00000000005588A8, pc = 0x00007A28D358F4A0, unwindSP = 0x00000000005587A0, cp = 0x0000000000572C40, arg0EA = 0x00000000005588C8, jitInfo = 0x00007A28D0A57D38
<52fa00> 	Method: com/ibm/crypto/plus/provider/AESCipher.internalInit(ILjava/security/Key;[B)V !j9method 0x000000000050FC80
<52fa00> 	Bytecode index = 217, inlineDepth = 0, PC offset = 0x0000000000000630
<52fa00> 	stackMap=0x00007A28D0A5826C, slots=I16(0x0004) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0004), localBaseOffset=I16(0xFFA0)
<52fa00> 	Described JIT args starting at 0x00000000005588B0 for U16(0x0004) slots
<52fa00> 		O-Slot: : a3[0x00000000005588B0] = 0x00000000FCD3D698
<52fa00> 		O-Slot: : a2[0x00000000005588B8] = 0x000000000000004C <--- BAD
<52fa00> 		I-Slot: : a1[0x00000000005588C0] = 0x0000000080E33928
<52fa00> 		O-Slot: : a0[0x00000000005588C8] = 0x0000000080E33DB8
<52fa00> 	Described JIT temps starting at 0x0000000000558848 for IDATA(0x000000000000000C) slots
<52fa00> 		I-Slot: : t11[0x0000000000558848] = 0x00000000F43AEFC0
<52fa00> 		I-Slot: : t10[0x0000000000558850] = 0x0000000000100000
...
<52fa00> JIT frame: bp = 0x00000000005588A8, pc = 0x00007A28D358F4A0, unwindSP = 0x00000000005587A0, cp = 0x0000000000572C40, arg0EA = 0x00000000005588C8, jitInfo = 0x00007A28D0A57D38
<52fa00> 	Method: com/ibm/crypto/plus/provider/AESCipher.internalInit(ILjava/security/Key;[B)V !j9method 0x000000000050FC80
<52fa00> 	Bytecode index = 217, inlineDepth = 0, PC offset = 0x0000000000000630
<52fa00> 	stackMap=0x00007A28D0A5826C, slots=I16(0x0004) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0004), localBaseOffset=I16(0xFFA0)
<52fa00> 	Described JIT args starting at 0x00000000005588B0 for U16(0x0004) slots
<52fa00> 		O-Slot: : a3[0x00000000005588B0] = 0x00000000FCD3D698
<52fa00> 		O-Slot: : a2[0x00000000005588B8] = 0x000000000000004C
<52fa00> 		I-Slot: : a1[0x00000000005588C0] = 0x0000000080E33928
<52fa00> 		O-Slot: : a0[0x00000000005588C8] = 0x0000000080E33DB8
<52fa00> 	Described JIT temps starting at 0x0000000000558848 for IDATA(0x000000000000000C) slots
<52fa00> 		I-Slot: : t11[0x0000000000558848] = 0x00000000F43AEFC0
<52fa00> 		I-Slot: : t10[0x0000000000558850] = 0x0000000000100000
<52fa00> 		I-Slot: : t9[0x0000000000558858] = 0x0000000000000000
<52fa00> 		O-Slot: : t8[0x0000000000558860] = 0x00000000E49A2A40
<52fa00> 		I-Slot: : t7[0x0000000000558868] = 0x00000000E49A2C70
<52fa00> 		I-Slot: : t6[0x0000000000558870] = 0x000000008139E158
<52fa00> 		I-Slot: : t5[0x0000000000558878] = 0x00000000E0340850
<52fa00> 		O-Slot: : t4[0x0000000000558880] = 0x0000000000000010 <--- BAD
<52fa00> 		I-Slot: : t3[0x0000000000558888] = 0x00000000F3BAEF80

@dmitripivkine
Copy link
Contributor

I don't know if that helps, but OpenJCEPlus uses ByteBuffer instances in some points and allocates space outside the heap. Maybe that's what the GC runs into.

@KostasTsiounis I think the problem is out-of bound array heap corruption caused by wrong size of data (16+ bytes more, than necessary) used to copy data from heap to native memory and back to the heap by tested code. As a result, it is invisible if memory (16 bytes or more) followed byte array was not changed between copy from and copy to (copy the same extra reminder somewhere and back). In the case these few bytes were updated it caused heap corruption.
The same effect might be observed if size is correct but start address is off. I guess this case might be ruled out - it should impact test correctness(?).
Would you please check details how copied data size is calculated? I presume object header is not the part of copied area. If it is a case, and code calculates copied size based on total object size without header, how array header size is obtained? Currently the type of Indexable object header is selected dynamically at JVM startup based on GC policy and it has different size for Gencon and Balanced. So static value like constant can not represent object header size properly any more.
Please ping me on Slack if you need more details or just want to discuss this issue.

@GioChkhaidze GioChkhaidze mentioned this issue Jun 11, 2024
Closed
@LongyuZhang
Copy link
Contributor

Test issue is opened at adoptium/aqa-tests#5373 for core files generated outside of REPORTDIR.

FYI, the test has been updated to generate core inside the REPORTDIR and will be automatically uploaded into the jenkins build artifactory via adoptium/aqa-tests#5389.

@dmitripivkine
Copy link
Contributor

Remove comp:gc due heap corruption is not caused by GC code.

@pshipton
Copy link
Member

pshipton commented Jun 28, 2024 •
edited
Loading

IBM/OpenJCEPlus#107 through IBM/OpenJCEPlus#112 were opened.

@pshipton
Copy link
Member

pshipton commented Jul 3, 2024 •
edited
Loading

The PRs were merged and https://github.com/ibmruntimes/OpenJCEPlus is updated, as well as the 22.0.2, 21.0.4, 17.0.12, 11.0.24 branches.

@pshipton pshipton closed this as completed Jul 3, 2024
@jasonkatonica jasonkatonica mentioned this issue Jul 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:crypto test failure
Projects
None yet
Milestone
Release 0.46 (Java 8, 11, 17, 21, 22)...
Development

No branches or pull requests
6 participants
@JasonFengJ9 @pshipton @dmitripivkine @llxia @LongyuZhang @KostasTsiounis