Garbage collection JVM options

Imply recommends the following JVM Garbage Collection (GC) related options to be used in the jvm.config of every service:

-XX:+UseG1GC
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:+PrintGC
-XX:HeapDumpPath=/tmp/druid-heap.hprof

For these options to apply to indexing tasks as well, they must be added to druid.indexer.runner.javaOpts in middleManager/runtime.properties. For example:

druid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/druid-heap.hprof

You should customize the HeapDumpPath to specify a path on your server that has enough storage space to store a heap dump, in the event that the server has an out-of-memory event. This heap dump will potentially be multiple gigabytes.

The rationale behind these settings is:

• G1GC is an effective, general purpose garbage collector that does not require much tuning.
• The ExitOnOutOfMemoryError option allows the process to exit and be automatically restarted, in the event that it runs out of memory.
• The HeapDumpOnOutOfMemoryError option allows you and Imply support to debug out-of-memory conditions if they occur.
• The PrintGC option allows you to see statistics about garbage collection timings in service logs, which will help you and Imply support tune settings if needed.

If you see any of the following symptoms, you have a garbage collection problem on your hands:

• Long (multi-second) "Full GC" pauses in the PrintGC messages; or long pauses in the "Total GC time" metric in Clarity's Server view.
• OutOfMemoryErrors with the messages "GC overhead limit exceeded" or "Java heap space"
• Heap dumps generated in the HeapDumpPath you specified.