OBJECTIVE:
This article explains a configuration that can prevent index_parallel deadlocks in a cluster with high middle manager task slot utilization.
ENVIRONMENT:
This applies to clusters with multiple index_parallel tasks that have many sub-tasks where worker slot utilization is high and the slots may be taken up by all index_parallel tasks which can never complete their sub-tasks. This can create a deadlock situation where all worker slots may be running index_parallel tasks with each waiting for their sub-tasks to run and complete but there are no worker slots are available to do so.
PROCEDURE:
There is an available configuration change that can be applied to the overlord properties. This configuration is druid.indexer.runner.parallelIndexTaskSlotRatio.Configure the value to be less than 1, (for example 0.5). This will prevent more than half the slots from being utilized by index_parallel tasks leaving some open to run the resulting subtasks preventing a deadlock from occurring.