Configure runtime augument for java native image

[zhoufenqin]

I use kpack 0.9.2 and java-native-image.

From GraalVM Debugging and Diagnostics doc, I want to configure heapdump, see the doc: https://www.graalvm.org/22.3/reference-manual/native-image/guides/create-heap-dump/

In build phase, add BP_NATIVE_IMAGE_BUILD_ARGUMENTS="--enable-monitoring=heapdump"
In runtime phase, I hope to run {command} -XX:+DumpHeapAndExit, but there is no place to pass -XX:+DumpHeapAndExit. I directly run the command in a container, the error shows:

Expected Behavior

Current Behavior

should pass some argument to runtime phase

Possible Solution

Steps to Reproduce

Motivations

[dmikusa]

1. I think you want this -> https://buildpacks.io/docs/app-developer-guide/run-an-app/#default-process-type-with-additional-arguments. That will let you add arguments to the default process which should be your application.

2. You could include a Procfile & specify the command that way but it's not really necessary because of 1.).

I haven't tried it, but you might also try sending the SIGUSR1 signal. That might let you trigger a heap dump without modifying the start command.

[zhoufenqin]

1. I think you want this -> https://buildpacks.io/docs/app-developer-guide/run-an-app/#default-process-type-with-additional-arguments. That will let you add arguments to the default process which should be your application.
2. You could include a Procfile & specify the command that way but it's not really necessary because of 1.).

I haven't tried it, but you might also try sending the SIGUSR1 signal. That might let you trigger a heap dump without modifying the start command.

Hi @dmikusa set the default process with arguments using https://buildpacks.io/docs/app-developer-guide/run-an-app/#default-process-type-with-additional-arguments do not work well because Java native image layer is not put into the runtime image.

the /layers show only the ca-certificate buildpack layer is put into the final container image.

So it may need the buildpack to do some changes to support the heapdump/jfr shoubleshooting, like the default path for the dump file needs to be changed.

Besides, I didn't use the SIGUSR1 signal because I don't want to send a signal from an application, hope to use a heap dump like in JVM way (e.g. set BPL_HEAP_DUMP_PATH when using JDK)

[dmikusa]

do not work well because Java native image layer is not put into the runtime image.

Yes, the application binary gets moved into /workspace. It's strange that it's not writing the heap dump to the current working directory. Do you know if there is a way to tell GraalVM to write the heap dump to a specific location?

Besides, I didn't use the SIGUSR1 signal because I don't want to send a signal from an application, hope to use a heap dump like in JVM way (e.g. set BPL_HEAP_DUMP_PATH when using JDK)

I don't think that -XX:+DumpHeapAndExit works the same as the settings we use with the JDK and BPL_HEAP_DUMP_PATH. In the JDK we use a setting that says to dump the heap to this location when the heap is exhausted. It allows you to get a picture of the heap when it runs out of space.

My understanding of -XX:+DumpHeapAndExit is that it starts your app and immediately dumps the heap. That its purpose is different, and it is used to see what gets put onto the heap in your native app.

At any rate, if GraalVM does not give you a way to change where the heap dump is written, there's not a lot you can do, besides the custom route.

[zhoufenqin]

Thanks @dmikusa

Do you know if there is a way to tell GraalVM to write the heap dump to a specific location?

From GraalVM dev build, I see that the -XX:HeapDumpPath option can be used to specify an alternative filename or directory, but in the latest version, it seems that the option is removed. And I use -XX:PrintFlags= to show all supported command, it can't support specify heap dump path too.

> k logs -f test2-default-5-594c7c4b7d-xt2k2
Defaulted container "test2" out of: test2, m
  -XX:ActiveProcessorCount=-1                  Overwrites the available number of processors provided by the OS. Any value <= 0 means using the processor count from
                                               the OS.
  -XX:?AutomaticReferenceHandling              Determines if the reference handling is executed automatically or manually. Default: + (enabled).
  -XX:?CollectYoungGenerationSeparately        Determines if a full GC collects the young generation separately or together with the old generation. Serial GC only.
  -XX:DiagnosticDetails=""                     Specifies how many details are printed for certain diagnostic thunks, e.g.: 'DumpThreads:1,DumpRegisters:2'. A value of
                                               1 will result in the maximum amount of information, higher values will print less information. By default, the most
                                               detailed output is enabled for all diagnostic thunks. Wildcards (*) are supported in the name of the diagnostic thunk.
  -XX:?DisableExplicitGC                       Ignore calls to System.gc(). Default: - (disabled).
  -XX:?DumpHeapAndExit                         Create a heap dump and exit. Default: - (disabled).
  -XX:?EagerSnippets                           Eagerly construct extra snippet info. Default: - (disabled).
  -XX:?EnableSignalHandling                    Enables signal handling.
  -XX:?ExitOnOutOfMemoryError                  Exit on the first occurrence of an out-of-memory error that is thrown because the Java heap is out of memory. Default: -
                                               (disabled).
  -XX:?InstallSegfaultHandler                  Install segfault handler that prints register contents and full Java stacktrace. Default: enabled for an executable,
                                               disabled for a shared library.
  -XX:?LoopOnFatalError                        Execute an endless loop before printing diagnostics for a fatal error. Default: - (disabled).
  -XX:MaxDirectMemorySize=0                    Maximum total size of NIO direct-buffer allocations.
  -XX:MaxHeapFree=0                            The maximum free bytes reserved for allocations, in bytes (0 for automatic according to GC policy). Serial GC only.
  -XX:MaxHeapSize=0                            The maximum heap size at run-time, in bytes.
  -XX:MaxJavaStackTraceDepth=1024              The maximum number of lines in the stack trace for Java exceptions (0 means all).
  -XX:MaxNewSize=0                             The maximum size of the young generation at run-time, in bytes.
  -XX:MaximumHeapSizePercent=80                The maximum heap size as percent of physical memory. Serial and epsilon GC only.
  -XX:MaximumYoungGenerationSizePercent=10     The maximum size of the young generation as a percentage of the maximum heap size. Serial and epsilon GC only.
  -XX:MinHeapSize=0                            The minimum heap size at run-time, in bytes.
  -XX:?PreferContainerQuotaForCPUCount         Calculate the container CPU availability based on the value of quotas (if set), when true. Otherwise, use the CPU shares
                                               value, provided it is less than quota. Default: + (enabled).
  -XX:PrintFlags=...                           Show available options based on comma-separated option-types (allowed categories: User, Expert, Debug). Default: None
  -XX:PrintFlagsWithExtraHelp=...              Print extra help, if available, based on comma-separated option names. Pass * to show all options that contain extra
                                               help. Default: None
  -XX:?PrintGC                                 Print summary GC information after each collection. Default: - (disabled).
  -XX:?PrintGCSummary                          Print summary GC information after application main method returns. Serial GC only. Default: - (disabled).
  -XX:?PrintGCTimeStamps                       Print a time stamp at each collection, if +PrintGC or +VerboseGC. Serial GC only. Default: - (disabled).
  -XX:?PrintGCTimes                            Print the time for each of the phases of each collection, if +VerboseGC. Serial GC only. Default: - (disabled).
  -XX:?PrintHeapShape                          Print the shape of the heap before and after each collection, if +VerboseGC. Serial GC only. Default: - (disabled).
  -XX:SafepointPromptnessFailureNanos=0        Exit the VM if I can not come to a safepoint in this many nanoseconds. 0 implies forever.
  -XX:SafepointPromptnessWarningNanos=0        Print a warning if I can not come to a safepoint in this many nanoseconds. 0 implies forever.
  -XX:StackSize=0                              The size of each thread stack at run-time, in bytes.
  -XX:TearDownFailureNanos=0                   The number of nanoseconds before tearing down an isolate gives a failure message.  0 implies no message.
  -XX:TearDownWarningNanos=0                   The number of nanoseconds before and between which tearing down an isolate gives a warning message.  0 implies no
                                               warning.
  -XX:?TraceHeapChunks                         Trace heap chunks during collections, if +VerboseGC and +PrintHeapShape. Serial GC only. Default: - (disabled).
  -XX:?VerboseGC                               Print more information about the heap before and after each collection. Default: - (disabled).

My understanding of -XX:+DumpHeapAndExit is that it starts your app and immediately dumps the heap. That its purpose is different, and it is used to see what gets put onto the heap in your native app.

Yes, I think so, it's different with BPL_HEAP_DUMP_PATH in JVM.

So I think maybe the heapdump/JFR throubleshooting is not support well in native image

[dmikusa]

I'm not sure why they would remove the ability to set the heap dump path. Maybe ask the GraalVM team? We can't really do anything without that option because even if we published the native-image layer, that would be a read-only layer so it couldn't be written to. I'm sure we're not the only ones creating images where the JVM is on a read-only layer. It has to have some way to write the heap dump elsewhere.

You might also ask if they plan to add an option like -XX:+HeapDumpOnOutOfMemoryError. We'd need both of those to enable BPL_HEAP_DUMP_PATH to work the same for a native-image binary.

[zhoufenqin]

The default path for heap dump file is /layers/paketo-buildpacks_native-image/native-image rather than /workspace， I think it's because the native-image command has params like this:

-H:Name=/layers/paketo-buildpacks_native-image/native-image/io.paketo.demo.DemoApplication

The executable file io.paketo.demo.DemoApplication is finally put into /workspace/io.paketo.demo.DemoApplication

Is it a bug in native-image buildpack? see https://github.com/paketo-buildpacks/native-image/blob/main/native/arguments.go#L159

[dmikusa]

No, that's intentional. The native-image file is put in the layer so that it can be cached. If you rebuild and your application hasn't changed, then it'll reuse that layer and skip the expensive build.

Could it be done differently? Yes. There are probably 5 different ways you could technically handle this, but presently it's working the way it was designed.