null ShardIterator Error when using closed kinesis shards

[Ramshackle-Jamathon]

null ShardIterator Error when using closed kinesis shards

[Ramshackle-Jamathon]

Sorry for not writing this up better, I'm trying to find some good aws docs on closed shards. So far the last section of https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-after-resharding.html is the best I've found

Therefore, assuming that the order of the data is important, you should, after a reshard, always continue to read data from the parent shards until it is exhausted, and only then begin reading data from the child shards. When getRecordsResult.getNextShardIterator returns null, it indicates that you have read all the data in the parent shard. If you are reading data using the Kinesis Client Library, the library ensures that you receive the data in order even if a reshard occurs.

I think describeStream will also return null shardId values. I'll do some testing? 🕵️ 👨‍🔬

[Ramshackle-Jamathon]

I made a little harness to test this:

const AWS = require('aws-sdk')
const kread = require('./kinesis-reader')
const through2 = require('through2')

const kinesis = new AWS.Kinesis({ region: 'us-east-1' });

const reader = kread(kinesis, 'jvds-rad-stream', {readpause: 5000})

reader
    .pipe(
      through2.obj(function (records, enc, next) {
        console.log(records);
        next(null, '')
      })
    )

setInterval(() => {
    kinesis.putRecords({
        Records: [ /* required */
            {
              Data: 'STRING_VALUE' /* Strings will be Base-64 encoded on your behalf */, /* required */
              PartitionKey: 'woooo', /* required */
            }
          ],
          StreamName: 'jvds-rad-stream'
    }, () => {})
}, 4000)

steps to reproduce

1. run the above script
2. change the shard count of jvds-rad-stream in kinesis
3. the following error will be produced

MissingRequiredParameter: Missing required key 'ShardIterator' in params
    at ParamValidator.fail (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/param_validator.js:50:37)
    at ParamValidator.validateStructure (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/param_validator.js:61:14)
    at ParamValidator.validateMember (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/param_validator.js:88:21)
    at ParamValidator.validate (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/param_validator.js:34:10)
    at Request.VALIDATE_PARAMETERS (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/event_listeners.js:125:42)
    at Request.callListeners (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at callNextListener (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
    at /Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/event_listeners.js:85:9
    at finish (/Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/config.js:320:7)
    at /Users/joevandrunen/Documents/workspace/kinesis-cycling-readable/node_modules/aws-sdk/lib/config.js:338:9

[tcql]

Merged #4, and published as v1.1.0