How to avoid index_parallel deadlocks in a busy cluster

OBJECTIVE:

This article explains a configuration that can prevent index_parallel deadlocks in a cluster with high middle manager task slot utilization.

ENVIRONMENT:

This applies to clusters with multiple index_parallel tasks that have many sub-tasks where worker slot utilization is high and the slots may be taken up by all index_parallel tasks which can never complete their sub-tasks.  This can create a deadlock situation where all worker slots may be running index_parallel tasks with each waiting for their sub-tasks to run and complete but there are no worker slots are available to do so.

PROCEDURE:

There is an available configuration change that can be applied to the overlord properties. This configuration is druid.indexer.runner.parallelIndexTaskSlotRatio.Configure the value to be less than 1, (for example 0.5).  This will prevent more than half the slots from being utilized by index_parallel tasks leaving some open to run the resulting subtasks preventing a deadlock from occurring.