If you're having a "too many open files" problem.. I feel bad for you son, but I got 99 problems and a ulimit setting ain't one.

Permanently deleted user

Updated April 10, 2020 19:10

Symptoms:

"Too many open files" messages showing up in your logs? Tasks failing?

First step - check your ulimit

ulimit -a

If the value is < 10,000 you should probably increase it.

Root Cause:

The default open file settings for linux or often too low for an Apache Druid system, which can be interacting with thousands of files simultaneously.

Solution:

Per session solution (will be reverted on reboot):

NOTE: Ensure you have root access

Execute the following

ulimit -n 65536
ulimit -u 65536

Permanent solution:

vi /etc/security/limits.conf

Ensure the following values are set:

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

Wait! are you on RHEL 7 or using systemd to boot the systems?

Above wouldn't work! To set the values, review the following RedHat Subscriber Article instead:

https://access.redhat.com/solutions/1257953

Reference:

https://access.redhat.com/solutions/61334

https://www.youtube.com/watch?v=6uikJTnmtgw