There are times when a historical will not start even though plenty of heap has been given to the JVM and ulimit has been set appropriately. The historical startup sequence will look normal loading the segments from the segmentCache and will fail with the following type of error:
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007ee213a7f000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 262144 bytes for committing reserved memory.
# An error report file with more information is saved as:
This error will not be resolved via the typical means of increasing heap. This is a direct reference to a setting on the node called max_map_count. This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.
Historical processes and to a lesser extent, MiddleManager and Indexer processes memory map segment files, so depending on the number of segments per server, /proc/sys/vm/max_map_count might need to be increased. This should be done with consultation of your systems administrator and can vary depending on your variant of Linux.