Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK21 serviceability_jvmti_j9_0_FAILED serviceability/jvmti/vthread/BoundVThreadTest/BoundVThreadTest.java Type=Segmentation error vmState=0x00040000 #17868

Closed
JasonFengJ9 opened this issue Jul 27, 2023 · 3 comments
Closed

JDK21 serviceability_jvmti_j9_0_FAILED serviceability/jvmti/vthread/BoundVThreadTest/BoundVThreadTest.java Type=Segmentation error vmState=0x00040000 #17868

JasonFengJ9 opened this issue Jul 27, 2023 · 3 comments
Assignees
@babsingh
Labels
comp:jvmti comp:vm jdk21 test failure
Milestone
Java 21 (0.42)

Comments

@JasonFengJ9
Copy link
Member

Failure link

From an internal grinder build(paix906) - launched at #17865

08:08:05  openjdk version "21-internal" 2023-09-19
08:08:05  OpenJDK Runtime Environment (build 21-internal-adhoc.jenkins.BuildJDK21ppc64aixPersonal)
08:08:05  Eclipse OpenJ9 VM (build master-c9a98669642, JRE 21 AIX ppc64-64-Bit Compressed References 20230726_28 (JIT enabled, AOT enabled)
08:08:05  OpenJ9   - c9a98669642
08:08:05  OMR      - 436bad3c438
08:08:05  JCL      - 78015256cbd based on jdk-21+31)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

11:54:29  variation: Mode150
11:54:29  JVM_OPTIONS:  -XX:+UseCompressedOops 

12:03:07  TEST: serviceability/jvmti/vthread/BoundVThreadTest/BoundVThreadTest.java

12:03:07  STDERR:
12:03:07  Unhandled exception
12:03:07  Type=Segmentation error vmState=0x00040000
12:03:07  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000032
12:03:07  Handler1=09001000A12F32D0 Handler2=09001000A10704A8
12:03:07  R0=0000000000000020 R1=00000100230677C0 R2=09001000A12F70E0 R3=000000003020E400
12:03:07  R4=B7654321254A7A05 R5=09001000A12F2FA0 R6=0000000000000003 R7=F70001001406C286
12:03:07  R8=000000000000007D R9=0000000000000000 R10=0000010022D77272 R11=0000000000000000
12:03:07  R12=000000002200022B R13=0000010023073800 R14=00000000301FE630 R15=0000000000000002
12:03:07  R16=00000000301FE508 R17=00000000302146E8 R18=00000000302146D0 R19=000001001012DB10
12:03:07  R20=0000000000000001 R21=09001000A0FE11F8 R22=0000000000000002 R23=0000010022EE4DA6
12:03:07  R24=09001000A0004B50 R25=0000000000000100 R26=0000000000000020 R27=00000000302146D8
12:03:07  R28=0000000000000000 R29=09001000A12AFF90 R30=B7654321254A7A05 R31=000000003020E400
12:03:07  IAR=090000000EAD1E58 LR=0900000011782308 MSR=A00000000200D032 CTR=090000000EAD1E00
12:03:07  CR=2200022020000000 FPSCR=8202000000000000 XER=2000000082020000
12:03:07  FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
12:03:07  FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
12:03:07  FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
12:03:07  FPR6 4114e35800000000 (f: 0.000000, d: 3.422300e+05)
12:03:07  FPR7 412e848000000000 (f: 0.000000, d: 1.000000e+06)
12:03:07  FPR8 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
12:03:07  FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
12:03:07  FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
12:03:07  FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
12:03:07  FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
12:03:07  FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
12:03:07  Module=/home/jenkins/workspace/Grinder_iteration_2/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
12:03:07  Module_base_address=090000000EA67000
12:03:07  Target=2_90_20230726_28 (AIX 7.2)
12:03:07  CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
12:03:07  ----------- Stack Backtrace -----------
12:03:07  IPRA.$_ZL32test_unsupported_jvmti_functionsP9_jvmtiEnvP7JNIEnv_P8_jobjectS4_+0x424 (0x0900000011782308 [libBoundVThreadTest.so+0x2308])
12:03:07  Java_BoundVThreadTest_testJvmtiFunctions+0x68 (0x0900000011781D8C [libBoundVThreadTest.so+0x1d8c])
12:03:07  (0x090000000EB4783C [libj9vm29.so+0xe083c])
12:03:07  ffi_call+0x110 (0x090000000EB47174 [libj9vm29.so+0xe0174])
12:03:07  (0x090000000EC1B080 [libj9vm29.so+0x1b4080])
12:03:07  (0x090000000EAF4368 [libj9vm29.so+0x8d368])
12:03:07  runJavaThread+0x28c (0x090000000EADBA50 [libj9vm29.so+0x74a50])
12:03:07  _ZL23javaProtectedThreadProcP13J9PortLibraryPv+0xf0 (0x090000000EA69FF4 [libj9vm29.so+0x2ff4])
12:03:07  omrsig_protect+0x4fc (0x090000000EE581E0 [libj9prt29.so+0x5f1e0])
12:03:07  javaThreadProc+0x70 (0x090000000EA69E94 [libj9vm29.so+0x2e94])
12:03:07  thread_wrapper+0x14c (0x090000000C08E590 [libj9thr29.so+0x5590])
12:03:07  _pthread_body+0xe8 (0x09000000005B404C [libpthreads.a+0x404c])
12:03:07  ---------------------------------------
12:03:07  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/07/27 12:00:06 - please wait.
12:04:30  TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
12:04:30  --------------------------------------------------
12:04:32  Test results: passed: 121; failed: 2
12:04:56  Report written to /home/jenkins/workspace/Grinder_iteration_2/jvmtest/openjdk/report/html/report.html
12:04:56  Results written to /home/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_16904731027876/serviceability_jvmti_j9_0/work
12:04:56  Error: Some tests failed or other problems occurred.
12:04:56  -----------------------------------
12:04:56  serviceability_jvmti_j9_0_FAILED

50x internal grinder - 3/50 failing rate

FYI @babsingh
@JasonFengJ9 JasonFengJ9 added this to the Java 21 milestone Jul 27, 2023
@babsingh
Copy link
Contributor

babsingh commented Aug 3, 2023

gdb stack trace

#11 0x00007ff3ff9d5645 in mainSynchSignalHandler (signal=11, sigInfo=0x7ff3d874ed70, contextInfo=0x7ff3d874ec40) at /root/openj9-openjdk-jdk21/omr/port/unix/omrsignal.c:1066
#12 <signal handler called>
#13 isVirtualThread (env=0x1d1300, obj=0xda929232b7654321) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/jnicsup.cpp:2508
#14 0x00007ff3ffb768bb in test_unsupported_jvmti_functions (group=0x7ff3d874f260, vthread=0x7ff3d874f258, jni=0x7ff3f80a4678, jvmti=0x229cf0)
    at test/hotspot/jtreg/serviceability/jvmti/vthread/BoundVThreadTest/libBoundVThreadTest.cpp:145
#15 Java_BoundVThreadTest_testJvmtiFunctions (jni=0x7ff3f80a4678, cls=<optimized out>, vthread=0x7ff3d874f258, group=0x7ff3d874f260)

Test source code

https://github.com/ibmruntimes/openj9-openjdk-jdk21/blob/eaf8e5f386d40a6194646173e68cc2f5c9904e3d/test/hotspot/jtreg/serviceability/jvmti/vthread/BoundVThreadTest/libBoundVThreadTest.cpp#L126-L144

  LOG("Testing GetAllThreads\n");
  err = jvmti->GetAllThreads(&thread_cnt, &threads_ptr);
  check_jvmti_status(jni, err, "test_unsupported_jvmti_functions: error in JVMTI GetAllThreads");
  for (int idx = 0; idx < thread_cnt; idx++) {
    jthread thread = threads_ptr[idx];
    if (jni->IsVirtualThread(thread)) {
      fatal(jni, "GetAllThreads should not include virtual threads");
    }
  }

  LOG("Testing GetAllStackTraces\n");
  err = jvmti->GetAllStackTraces(MAX_FRAMES, &stack_info, &thread_cnt);
  check_jvmti_status(jni, err, "test_unsupported_jvmti_functions: error in JVMTI GetAllStackTraces");
  for (int idx = 0; idx < thread_cnt; idx++) {
    jthread thread = threads_ptr[idx];
    if (jni->IsVirtualThread(thread)) { // [CRASH] A bad object ("thread") is passed
      fatal(jni, "GetAllStackTraces should not include virtual threads");
    }
  }

Summary

  • GetAllThreads and GetAllStackTraces both track live platform threads.
  • threads_ptr and thread_cnt are populated during the call to GetAllThreads.
  • After the call to GetAllThreads, GetAllStackTraces is invoked, which re-evaluates thread_cnt, but threads_ptr is not updated and live platform threads might have been created or destroyed between the call to GetAllThreads and GetAllStackTraces.
  • We are using a stale threads_ptr array with a newer value of thread_cnt, and a bad thread object is accessed in the stale threads_ptr array.

Solution

Derive the thread object through the jvmtiStackInfo array, which is returned by GetAllStackTraces, instead of the stale threads_ptr array.

  ...
  LOG("Testing GetAllStackTraces\n");
  err = jvmti->GetAllStackTraces(MAX_FRAMES, &stack_info, &thread_cnt);
  check_jvmti_status(jni, err, "test_unsupported_jvmti_functions: error in JVMTI GetAllStackTraces");
  for (int idx = 0; idx < thread_cnt; idx++) {
    jvmtiStackInfo *stackInfo = &stack_info[idx];
    if (jni->IsVirtualThread(stackInfo->thread)) { // CORRECTION
      fatal(jni, "GetAllStackTraces should not include virtual threads");
    }
  }

References

@babsingh babsingh self-assigned this Aug 3, 2023
babsingh added a commit to babsingh/openj9 that referenced this issue Aug 3, 2023
@babsingh
[JDK19+] Fix JVMTI GetAllStackTraces
1359006 
JVMTI GetAllStackTraces only tracks live platform threads.

This change correctly and consistently derives the threadObject
from J9VMThread->carrierThreadObject in GetAllStackTraces for JDK19+.

Currently, J9VMThread->threadObject is used in some places, which is
incorrect since it can also point to a virtual thread.

Related: eclipse-openj9#17868

Signed-off-by: Babneet Singh <[email protected]>
This was referenced Aug 3, 2023
Merged
Merged
babsingh added a commit to babsingh/openj9-openjdk-jdk21 that referenced this issue Aug 3, 2023
@babsingh
Fix BoundVThreadTest - Bad Thread Object
5cc2491 
After GetAllStackTraces, derive the thread object from the
stack_info array, which is filled by GetAllStackTraces.

Currently, the thread object is retrieved from the threads_ptr
array, which is filled by GetAllThreads before the call to
GetAllStackTraces. Live platform threads might have been created
or destroyed between the call to GetAllThreads and GetAllStackTraces.
This makes the threads_ptr array stale after GetAllStackTraces. A bad
thread object is accessed from the stale threads_ptr which causes a
segfault.

Related: eclipse-openj9/openj9#17868

Signed-off-by: Babneet Singh <[email protected]>
@babsingh babsingh mentioned this issue Aug 3, 2023
Merged
@babsingh
Copy link
Contributor

babsingh commented Aug 3, 2023

PRs which will FIX this issue:

tajila pushed a commit to ibmruntimes/openj9-openjdk-jdk that referenced this issue Aug 12, 2023
@babsingh
Fix BoundVThreadTest - Bad Thread Object
d867a25 
After GetAllStackTraces, derive the thread object from the
stack_info array, which is filled by GetAllStackTraces.

Currently, the thread object is retrieved from the threads_ptr
array, which is filled by GetAllThreads before the call to
GetAllStackTraces. Live platform threads might have been created
or destroyed between the call to GetAllThreads and GetAllStackTraces.
This makes the threads_ptr array stale after GetAllStackTraces. A bad
thread object is accessed from the stale threads_ptr which causes a
segfault.

Related: eclipse-openj9/openj9#17868

Signed-off-by: Babneet Singh <[email protected]>
@babsingh
Copy link
Contributor

Closing, all PRs in #17868 (comment) have been merged.

jasonkatonica pushed a commit to jasonkatonica/openj9-openjdk-jdk that referenced this issue Aug 22, 2023
@babsingh @jasonkatonica
Fix BoundVThreadTest - Bad Thread Object
35d3dc1 
After GetAllStackTraces, derive the thread object from the
stack_info array, which is filled by GetAllStackTraces.

Currently, the thread object is retrieved from the threads_ptr
array, which is filled by GetAllThreads before the call to
GetAllStackTraces. Live platform threads might have been created
or destroyed between the call to GetAllThreads and GetAllStackTraces.
This makes the threads_ptr array stale after GetAllStackTraces. A bad
thread object is accessed from the stale threads_ptr which causes a
segfault.

Related: eclipse-openj9/openj9#17868

Signed-off-by: Babneet Singh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@babsingh babsingh

Labels
comp:jvmti comp:vm jdk21 test failure
Projects
None yet
Milestone
Java 21 (0.42)
Development

No branches or pull requests
2 participants
@babsingh @JasonFengJ9