theburningmonk/darkseid

Actor-based library to help you more effectively and easily push data into an Amazon Kinesis stream by taking care of the heavy lifting for you.

Hi there 👋

My name is Yan. I'm an AWS Serverless Hero and an independent consultant. I help companies go faster for less by adopting Serverless technologies successfully. If you're interested in working with me, then check out my consulting services.

How to reach me:

• Blog: theburningmonk.com
• Twitter: @theburningmonk
• LinkedIn
• Real-World Serverless podcast
• YouTube channel

Workshops:

• Production-Ready Serverless

Video courses & books:

• Testing Serverless Architectures
• AppSync Masterclass
• Learn you some Lambda best practice for great good!
• Complete guide to AWS Step Functions
• Serverless Architectures on AWS, 2nd Edition (Manning)

darkseid's Issues

Allow glorious godfrey to be disposed and wait for all records to be flushed

Support synchronous mode so that there's no worry of loss of uncommitted data when node dies

Auto merge when utilization level drops below threshold

Wrap functionalities into interfaces

Remove unnecessary Limit when getting shards

Allow configuration to be supplied via app.config

Report errors pushing records to Kinesis as events

Write documentation

Create/initialize the stream in the put record request if it doesn't exist already

Allow user to handle the FailEvent from AeroTrooper

So they can do something about these records that keep failing

Allow user to configure throttle call thresholds

mscorlib.dll exception:

Hi, I'm seeing an exception in Darkseid in: Utils.fs line 59>
`/// Applies memoization to the supplied function f
let memoize (f : 'a -> 'b) =
let cache = new Dictionary<'a, 'b>()

    let memoizedFunc (input : 'a) =
        // check if there is a cached result for this input
        match cache.TryGetValue(input) with
        | true, x   -> x
        | false, _  ->
            // evaluate and add result to cache
            let result = f input
            cache.Add(input, result)
            result`

System.ArgumentException was unhandled by user code
Message: An exception of type 'System.ArgumentException' occurred in mscorlib.dll but was not handled in user code
Additional information: An item with the same key has already been added.

cache is initialized (ie not null) and input, result are also not null.

I could have a dll conflict, but seems unlikely in mscorlib.dll

stack:

Darkseid.dll!Darkseid.Utils.Utils.memoizedFunc@53<string,Amazon.CloudWatch.Model.Dimension[]>.Invoke(string input) Line 53 F#
Darkseid.dll!Darkseid.VirmanVundabar.TrackSuccessfulSend(string shardId, int payloadSize) Line 155 F#
Darkseid.dll!<StartupCode$Darkseid>.$[email protected](Darkseid.Model.InternalModel.Result<Amazon.Kinesis.Model.PutRecordResponse,System.Exception> _arg1) Line 197 F#
[External Code]

This was originally happening with the current nuget package (v0.2.0). I got the source and included it into my project to get a better handle on where it's going wrong.

Any ideas?

Deal with stream with large number of shards

the KinesisUtils.getShards method doesn't deal with cases where there're more shards not returnd in the response.

getAvgStreamMetric excepts when there's no datapoints

Allow dynamic scaling up and down of AeroTroopers

One solution is to allow AeroTroopers to 'promoted' into a commander and have its own fleet of AeroTroopers sending records for him using the same approach of hashing the partition key. So that even though the base number of AeroTroopers we start with does not change, we are still able to increase/decrease the overall throughput of the system as a whole.

Godfrey/Virman will be responsible for dictating when to promote AeroTroopers and monitor the throughput of the system based on the action to see if promotion/demotion makes an observable difference and choose the best approach best on empirical data.

Swap out the batch helper function with a more efficient version

such as this one : http://fssnip.net/1o