ISSUE:
In Imply 3.x, running a compaction ask with single-dim in partionsSpec fails with the following error:
2020-06-29T05:14:40,409 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while running task[AbstractTask{id='partial_dimension_distribution_TestDs_eghdjnoh_2020-06-29T05:13:07.913Z', groupId='compact_TestDs_onnbilcc_2020-06-29T05:09:49.094Z', taskResource=TaskResource{availabilityGroup='partial_dimension_distribution_TestDs_eghdjnoh_2020-06-29T05:13:07.913Z', requiredCapacity=1}, dataSource='TestDs', context={appenderatorTrackingTaskId=compact_TestDs_onnbilcc_2020-06-29T05:09:49.094Z, forceTimeChunkLock=true, priority=25}}]
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Cannot': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"Cannot find any task with id: [compact_TestDs_onnbilcc_2020-06-29T05:09:49.094Z_0]"; line: 1, column: 7]
at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTaskStatus(HttpIndexingServiceClient.java:268) ~[druid-server-0.18.0-iap8.jar:0.18.0-iap8]
...
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Cannot':
was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"Cannot find any task with id: [compact_TestDs_onnbilcc_2020-06-29T05:09:49.094Z_0]"; line: 1, column: 7]
ENVIRONMENT:
Imply Version 3.x and above. (This is fixed in Imply 4.0 and higher.)
CAUSE:
In compaction and native ingestion tasks, in 3.x, single_dim is supported only for parallel tasks, i.e.,maxNumConcurrentSubTasks should be set to a value higher than 1.
In this version, default value for maxNumConcurrentSubTasks is 1 and it was not defined in the compaction task spec. Hence the task failed.
RESOLUTION:
Add maxNumConcurrentSubTasks to the tuningConfig and set it to a value > 1. For example,
{
"type": "compact",
"dataSource": "TestDs",
"ioConfig": {
"type": "compact",
"inputSpec": {
"type": "interval",
"interval": "2020-04-27/2020-04-28"
}
},
"tuningConfig": {
"type": "index_parallel",
"maxNumConcurrentSubTasks": 5,
"forceGuaranteedRollup": true,
"partitionsSpec": {
"type": "single_dim",
"targetRowsPerSegment": 5000000,
"partitionDimension": "test"
}
}
}
REFERENCES:
Apache Druid Documentation: Single-dimension range partitioning
Comments
0 comments
Article is closed for comments.