ISSUE:
During kinesis ingestion, the kinesis supervisor displays "Lost Connectivity to Stream" error and finally goes to "Unhealthy_state". The task log shows the following error :
Starting sequenceNumber [1234] is no longer available for partition [shardId-xxxx]
(earliest: [null]) and resetOffsetAutomatically is not enabled
ENVIRONMENT:
Imply-3.x and Imply-4.x in AWS as this is Kinesis issue.
ADDITIONAL INFORMATION:
CAUSE:
Either the shard in kinesis stream is not getting data at all or not getting data continuously.
RESOLUTION:
Set the following in Kinesis ingestion spec and submit the supervisor. It is important to set resetOffsetAutomatically=false when we set skipSequenceNumberAvailabilityCheck=true. Otherwise, we will never know when we start losing the data. When skipSequenceNumberAvailabilityCheck=true, `IteratorAgeMilliseconds` will not show in the AWS Cloudwatch metrics, hence we should not keep skipSequenceNumberAvailabilityCheck=true indefinitely.
skipSequenceNumberAvailabilityCheck=true
resetOffsetAutomatically=false