crccheck/kinesis-console-consumer

AWS Kinesis stream consumer for the console

kinesis-console-consumer's Issues

Output can't be grepped

kinesis-console-consumer stream-name | grep . won't output anything.

workaround is:

kinesis-console-consumer stream-name | jq -cC . | grep .

Keep track of shard iterators used

maintain a Set of shard iterators to keep track of state. Part of #27

Make list streams support pagination

list streams is done now, just need to wait for enough streams to need pagination

add support to filter data by shard-ids

created a PR to implement this :

#59

Use kinesis-streams instead of our own kinesis stream reader

Add test coverage to cli

Some problems:

• need to mock process.exit
• But if you stub out process.exit, then --help doesn't exit, which causes a block to run that normally wouldn't.
• argv

Add "update-notifier" so end users know if they're using an old version

https://github.com/yeoman/update-notifier

Drop Node 4 support?

It's getting real annoying using older syntax

Make this work on node 4

Without transpilation. AFAIK just needs 'use strict'

make sure to update .nvmrc,package.json

allow user to set the poll duration

currently, it's in a setTimeout(..., 2000)

Remove update-notifier?

it's a heavy weight requirement and if this is used as a library, we don't need it.

it doesn't really make sense to make it an optional peer requirement because no one is going to do that.

The other option is to split this up into two libraries.

Add keywords to the package.json

aws, kinesis, cli

Add basic tests

It's very hard to justify right now because after stubbing aws-sdk, there's very little code. But I really really should do it.

Disable stream reading when paused/ending Streamable

to follow best practices

rename --regex-filter to --grep?

I just noticed/remembed Mocha a very similar option https://mochajs.org/#-g---grep-pattern
Since that's already a thing, shouldn't we follow?

It's also nice because it's a bigger hint to just use grep if you got it.

Read from all shards if more than one shard is open

Refactor to make this a plain Kinesis consumer

It turns out that the library code is pretty good just for consuming a Kinesis stream. Might as well refactor this so others can benefit from it.

Now that this is 1.0, I should probably add a CHANGELOG

Pass in AWS client in constructor

It's sorta like nasty dependency injection, but it's also quite practical because right now I'm making assumptions about how the client should be instantiated. And this will make writing tests so much easier!

inspired by https://github.com/rclark/kinesis-readable