Comments (7)
Hi @sam701, this source is a little more tricky since:
- Vector must implement continuous shard discovery.
- Vector must handle various shard states (ex: a closed stream).
- Vector must coordinate with other Vector instances to ensure only one Vector instance is reading a shard. (distributed locking).
- Vector must maintain checkpoint state.
Kafka, for example, handles a lot of this bookkeeping, making the integration much easier. That's why this source is not done yet. It's questionable if all of the above fits within the scope of Vector. Especially stream exclusivity, which would require distributed locking. That would obviously need to be delegated to a system designed for that.
The best solution, imo, is the wrap Vector in a system that handles the above. Which is easier said than done. But, for example, if Vector was integrated into an AWS Lambda function you could leverage AWS' kinesis -> lambda integration, which handles all of this for you.
from vector.
There is a client for Rust, already being used by the kinesis sink, but the issue is in the complexity of synchronization and shard discovery. The Logstash input uses the AWS KCL that needs a DynamoDB table (e.g. similar to how Kafka uses zookeeper for state).
@binarylogic's comment explains well the complexity behind this.
I think however another option could be to run the AWS Java KCL MultiLangDaemon and pipe the output to a Vector stdin
source.
from vector.
@binarylogic How would you recommend to handle this scenario in spring 2020? Having a lambda + vector with HTTP source? I saw, the HTTP source is not production ready yet.
from vector.
Any news on this?
from vector.
I'd also be appreciative of this feature being added. In terms of the complexity, logstash already supports this so perhaps its solutions to the locking problem/etc can be reused?
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kinesis.html
from vector.
I'd also be appreciative of this feature being added. In terms of the complexity, logstash already supports this so perhaps its solutions to the locking problem/etc can be reused? https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kinesis.html
Looks like logstash is able to support cause is using the aws client, but there is no rust one at this moment: https://github.com/awslabs?q=kinesis-client&type=all&language=&sort=
I guess that we can call some java or python code from rust but I seems bad to me.
from vector.
There is https://crates.io/crates/aws-sdk-kinesis linked to from https://awslabs.github.io/aws-sdk-rust/ so perhaps that have most of what's needed?
from vector.
Related Issues (20)
- Enhance Azure Identity Integration in Vector: Ability to Utilize Passwordless Connections
- Kafka integration test `drains_acknowledgements_during_rebalance_sticky_assignments` is flaky
- oauth client credentials authentication in http sink HOT 5
- Vector high memory utilization HOT 4
- Documentation missing for decrypt VRL function reference HOT 1
- Why JSON logs cannot be directly parsed? HOT 7
- Vector Disk Buffer Error: Last written record was unable to be deserialized. Corruption likely HOT 1
- high latency of log collect
- [AWS] support AWS Pod Identity authentication - AWS SDK bump
- Vector v0.39.0 release
- Log error=Missing environment variable in config for comment in config HOT 1
- compact fail or not HOT 6
- Unable to suppress pod metadata with log_namespace: true HOT 3
- SERVICE AVAILABILITY: vector validate hangs indefinitely when using kafka source on v0.34.0 and beyond - Denial of Service for systemd managed instances HOT 7
- Kafka sink metrics incorrect after reload Kafka config.
- clickhouse sink doesn't support encoding.codec = raw_message HOT 10
- Loki sink: out-of-order logs might cause loss of other in-order logs? HOT 3
- The Vector release binary is not stripped HOT 1
- Path exists in the event, but VRL doesn't see it HOT 4
- `demo_logs` fails with GELF encoder HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vector.