ISSUE
Historical server fails to come up and the following error is observed in the log file:
2019-12-16T10:11:06,965 WARN [main-SendThread(10.13.44.53:2181)] org.apache.zookeeper.ClientCnxn - Session 0x105e3e8f3a5002f for server 10.13.44.53/10.13.44.53:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Packet len10233413 is out of range!
at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) ~[zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) [zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
ROOT CAUSE
By default zookeeper communication packet size is 1Mb. If a client process tries to send a packet size bigger than 1Mb, the above error is returned.
WORKAROUND
In general Druid doesn't create packet size of greater than 1Mb. Options would be to either to reduce the packet size generated by Druid or increase the zookeeper packet size.
Option 1
Try altering the following parameters in IMPLY_LOC/conf/druid/_common/common.runtime.properties and bringing up historicals:
| Property | Description | Default |
|---|---|---|
druid.announcer.segmentsPerNode |
Each Znode contains info for up to this many segments. | 50 |
druid.announcer.maxBytesPerNode |
Max byte size for Znode. | 524288 |
druid.announcer.skipDimensionsAndMetrics |
Skip Dimensions and Metrics list from segment announcements. NOTE: Enabling this will also remove the dimensions and metrics list from Coordinator and Broker endpoints. | false |
druid.announcer.skipLoadSpec |
Skip segment LoadSpec from segment announcements. NOTE: Enabling this will also remove the loadspec from Coordinator and Broker endpoints. | false |
If the datasources have high number of metrics and dimensions, following could be set:
druid.announcer.skipDimensionsAndMetrics=true
Option 2
Based on the Packet Len errors observed in historical log files, add `jute.maxbuffers` as follows in the Zookeeper configs in ZOOKEEPER_PATH/conf
- In java.env,
export JVMFLAGS="-Dmax.jutebuffer=10485760" - In zookeeper.properties, export max.jutebuffer=10485760
Also add -Dmax.jutebuffer=10485760 to the JVMFLAGS in all the Druid services runtime.properties file.
REFERENCES