Upgrading Imply (running in kubernetes) to any higher version using Imply-Manager is getting stuck at "waiting for data nodes to be available"

ISSUE:

Upgrading Imply to any higher version using Imply-Manager is getting stuck at "waiting for data nodes to be available".  There are multiple possible causes - this article discusses an issue with istio-mesh versions applicable to some kubernetes environments.

ENVIRONMENT:

Imply-3.x and above in kubernetes, with Imply-Manager-3.x and above.

ADDITIONAL INFORMATION:

These are the additional symptoms if Istio-mesh was upgraded from 1.7 to 1.8:

Even though Imply-Manager says the upgrade is stuck, in the background, Imply was successfully upgraded to the desired higher version. Pivot is up and running, the druid unified console is up and running, and ingestions are succeeding in the new higher version of Imply.

All the kubernetes pods are up and running.

All the druid services inside those kubernetes pods are up and running.

But Imply Manager thinks that the data nodes are down and the cluster is unhealthy, so it is stuck during the upgrade process even though the upgrade successfully completed and the underlying druid cluster is very healthy.

CAUSE:

Istio-Mesh-1.8 is preventing the health checks to data nodes, which is making data nodes appear to not come up. For more info on Istio-Mesh, refer to this link https://istio.io/latest/docs/concepts/what-is-istio/

WorkAround:

Downgrade Istio-Mesh from 1.8 back to it's original version 1.7 till the issue in Istio-Mesh 1.8 is fixed.