Comments (12)
From @mbogoevici on February 24, 2016 18:2
We need to discuss backwards compatibility options for 0.8 too I believe.
from spring-cloud-stream-binder-kafka.
Yeah, maybe we would be fine with existing tests, but chances are @garyrussell would want to add new tests, too.
The backward compatibility is a whole different story though. Gary's thoughts below.
Gary Russell [9:47 AM]
But I think Marius is planning to create the binder as a new project so there could be two binders; but I don’t know how long he would want to support that
Gary Russell [9:47 AM]
The 0.9 client is sooooo much simpler, it makes no sense (from an SI perspective) to support both.
from spring-cloud-stream-binder-kafka.
From @mbogoevici on February 24, 2016 18:13
Just to clarify: I do not feel we should support both in the same release. But there should be an option for users that are on 0.8 - even if that means moving out the 0.8 version post 1.1 (when it is the most likely we'll upgrade to 0.9).
from spring-cloud-stream-binder-kafka.
From @mbogoevici on February 24, 2016 18:17
And for 1.0 we can provide a separate binder 0.9 implementation that could be developed between the RC and GA timeline for 1.0 (also more lenient for dependencies, e.g using Mx releases of Spring Integration Kafka, which is typically a non-starter for RC releases).
from spring-cloud-stream-binder-kafka.
From @cdupuis on May 24, 2016 10:2
@mbogoevici and @garyrussell what are the plans for this? Do you guys have any idea on timing on this?
Thanks, cd
from spring-cloud-stream-binder-kafka.
From @mbogoevici on May 24, 2016 10:39
@cdupuis: Based on the current progress of Spring Kafka and Spring Integration Kafka 0.9, we will be able to provide a milestone that supports it in the next weeks with the goal of aligning Kafka 0.9 support in SCSt with the Spring Cloud Data Flow 1.0 release.
Note that the current implementation still supports interacting with 0.9 brokers, but it uses the simple consumer API of 0.8.x (which is still compatible). The major goal of this is to get advantage of the new 0.9 client features, and especially of rebalancing and dynamic scaling of clients.
from spring-cloud-stream-binder-kafka.
From @mbogoevici on May 24, 2016 10:51
To wit, the only blocker of some sorts is spring-projects/spring-kafka#84 and we are positive that it will be solved soon.
from spring-cloud-stream-binder-kafka.
From @cdupuis on May 25, 2016 15:38
Thanks @mbogoevici.
from spring-cloud-stream-binder-kafka.
Closed via #13
from spring-cloud-stream-binder-kafka.
@sabbyanandan
Does it mean that we can dynamically add Kafka consumers in v1.1.0.RELEASE?
According to spring data flow latest documentation:
"Currently, this is not supported with the Kafka binder (based on the 0.8 simple consumer at the time of the release), as well as partitioned streams, for which the suggested workaround is redeploying the stream with an updated number of instances. Both cases require a static consumer set up based on information about the total instance count and current instance index, a limitation intended to be addressed in future releases. For example, Kafka 0.9 and higher provides good infrastructure for scaling applications dynamically and will be available as an alternative to the current Kafka 0.8 based binder in the near future. One specific concern regarding scaling partitioned streams is the handling of local state, which is typically reshuffled as the number of instances is changed. This is also intended to be addressed in the future versions, by providing first class support for local state management."
Reference:
http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#arch-runtime-scaling
Great work it's an awesome integration!
from spring-cloud-stream-binder-kafka.
Hi, @Kenvelo90:
Does it mean that we can dynamically add Kafka consumers in v1.1.0.RELEASE?
If you're building standalone Spring Cloud Stream (SCSt) applications, yes, you could and that is supported.
In Spring Cloud Data Flow, there's no tight coupling of SCSt except of course the applications built with SCSt are registered and used in the stream definitions. That said, we are in the process of upgrading OOTB applications with the latest SCSt 1.1.0.RELEASE and when that is done, we will update the docs. In the meantime, you can build and register custom SCSt applications with the latest Kafka binder to take advantage of the auto-balancing capabilities.
from spring-cloud-stream-binder-kafka.
@sabbyanandan Great, Thank You!
from spring-cloud-stream-binder-kafka.
Related Issues (20)
- Live producer threads and dynamic-destination-cache-size property HOT 2
- How to wait for future inside Kafka Stream map()? HOT 2
- DefaultErrorHandler not exists HOT 4
- ClassCastException in pipeline: Kafka Streams Function -> Kafka Function -> Kafka Streams Consumer HOT 5
- NOT_ENOUGH_REPLICAS Errors After Upgrading The Spring Boot And Spring Cloud Version HOT 5
- KafkaBinderMetrics prevent JVM shutdown due to not shutting down scheduler thread pool! HOT 8
- Kafka bindings from previous test class remains active in the next test class HOT 2
- Kafka Streams Serdes having nested generic doesn't work HOT 3
- Support for DLQ in batch mode HOT 1
- spring.cloud.stream.bindings.process-in-0.consumer.concurrency not working in case of UpperCase letter in process name HOT 1
- ListenerContainerWithDlqAndRetryCustomizer template points to localhost:9092 HOT 5
- Looking for a sample with AWS MSK IAM
- [Feature Request] Allow to accept a list of customizers instead of a unique one for KafkaMessageChannelBinder HOT 1
- spring-kafka-test is compile (not test) dependency of spring-cloud-stream-binder-kafka-streams:3.2.3 and 3.2.4
- Batch mode is not working HOT 1
- Unknown Autoconfigured KafkaAvroSerializer and KafkaAvroDeSerializers HOT 2
- Spring Cloud stream binder kafka DLQ Handling for Batching doesn't work HOT 2
- Enable DLQ with destinations as pattern
- Fetch message timestamp
- Health Check for Binders Always Shows 'UNKNOWN' Status Due to Empty binderHealthContributors Map (Missing Bean Configuration for ReactorKafkaBinderHealthIndicator?) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spring-cloud-stream-binder-kafka.