doitintl/banias

Opinionated serverless event analytics pipeline

banias's Issues

From DavidB

went over it, looks ok but I miss comments so don't really understand what many parts of code do without looking at the import packages first. For example, g.Add that uses oklog which i don't know what it is and so on. So from the main func code i could hardly understand what it does, especially with things like 'logger.Info("exit", zap.Error(g.Run()))'. I think g.Run() should be more explicit because it's your main loop and my eye automatically skips log lines. Also pls highlight somewhere what the endpoints are (/track and /metrics), unless you have a dedicated client in which case it's not that important.
collector looks fine, needs more comments though. Things like 'if counter != 7' need to be more clear
Also I'd do some wrapper around tag, for example instead of tag.NewKey("banias/keys/code") it would be mytag.Key("code") and instead of "banias/measures/request_latency" it would be mytag.Measure("request_latency") because 'banias' is constant and keys/measures/etc are categories

Consider exporting monitoring metrics to opencensus.io

Consider exporting monitoring metrics to opencensus.io instead of just Prometheus

Invalid argument in backend/Makefile

There are a few occurrences of --GCSSchemasBucketName=(SCHEMAS_BUCKET) in the Makefile, which cause errors. I presume it is meant to read --GCSSchemasBucketName=$(SCHEMAS_BUCKET).

Is the current Makefile working for you with this error?

Resource requests and limits

Need to add Resource requests and limits to the banias/frontend/deploy/frontend-deployment.yaml

Add support for AWS Kinesis Streams / Firehose

https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html#putrecordbatch

Ingress

banias/frontend/deploy/frontend-service.yaml has to be changed to Ingress to support global ip and federation