Comments (5)
Hi yanglei99,
thank you for posting your problem and providing a fix.
Right now the workaround is to use broker options like --options=host.name=,advertised.host.name= (options param in REST) during broker add|update.
Regarding your fix. I need to make a decision about applicability of it. The policy is following:
- for any specific use case any option could be overriden by --options (options param in REST);
- defaults should be provided in code only for a generic (70%+) usecase;
So the question is: is your usecase generic enough to support it in the code?
To clarify this I need to know more about your environment:
- how cluster is ran: in vagrant, in AWS, in VMS, etc.
- how many and what public interfaces do the master|slaves have? What is chosen by Kafka Broker by default?
- how does Mesos master knows where the correct (public accessible) interface is? What are the guarantees that offer.getHostname would return the correct interface? i.e is it safe to rely on offer.getHostname in environments with multiple interfaces?
Sorry, for a pedantry, but this info is really important in order to make a decision about integrating your fix.
from kafka.
Hello dmitrypekar,
Thank you for your recommendation on using --options!
However I still think using mesos-slave's announced hostname is a better approach. As you will leave the decision on which network interface (or hostname ) to use to the people setting up mesos cluster. And it will also make it consistent on Kafka side, as you do show the mesos-slave offer's hostname which is in the form of ip address for my environment.
To answer your questions:
- how cluster is ran: in vagrant, in AWS, in VMS, etc.
Softlayer
- how many and what public interfaces do the master|slaves have? What is chosen by Kafka Broker by default?
Have one public ip and one private ip on each VM (master and slave). I define the one I use in mesos-slave /etc/mesos-slave/ip and /etc/mesos-slave/hostname. Kafka Broker does show the right ip when using "kafka-mesos.sh status" as you get it from mesos's offer. It is the Kafka (shown in Broker's log) that used the VM's hostname not mesos slave ip for communication.
- how does Mesos master knows where the correct (public accessible) interface is? What are the guarantees that offer.getHostname would return the correct interface? i.e is it safe to rely on offer.getHostname in environments with multiple interfaces?
It is a good question. I think it is down to what the contract is between mesos(slave) and framework executor. First, mesos master and slave do have contract in announcing what hostname(or ip) to use for their communication, configurations defined under /etc/ mesos-slave. Now, about the contract between executor and slave, I have assumed by default it is what the salve uses and passed by offer. But then I may have missed the reason we choose not to do so.
Really appreciate your thoughts and help!
Yang Lei
from kafka.
Hi yanglei99,
Thank you very much for a clarification.
I've done several tests and now I see that your changes absolutely makes sense as a defaults except for 'advertised.host.name', which by default in Kafka equals to 'host.name'. So I've added host.name=offer.getHostname as default option which could be overridden if required.
The changes are available in branch kafka-default-bind-addr, which will be merged by Joe Stein in soon (in day or two). You can still checkout a branch and use it directly if required.
Please also comment, if setting 'advertised.host.name' as a default is required in your env. I think, that everything should work wo those option. If not - let's discuss that. Maybe it is correct to add those option to the defaults too.
from kafka.
Thanks dmitrypekar!
Yes, I know the two settings are kind of redundant. :-) As advertised.host.name should derive from host.name. I will give it a try again and let you know if it does not work.
Thanks again.
Yang
from kafka.
The patch lgtm, I merged this to master.
The host.name is how the broker ultimately binds to that address.
The advertised.host.name is what the broker says to connect to.
Same with host.port and advertised.host.port respectively.
So, the broker can bind on 0.0.0.0 but say connect to me on 10.10.5.2 or whatever.
Binding on the slave offer's hostname does makes sense. This however, may not be routable to other brokers, producer and consumers and that value will just have to be updated through the CLI which can be updated through options in cli or rest api.
from kafka.
Related Issues (20)
- Host advertisement doesn't seem to work on Kafka 0.9.0+
- Unable to Start Brokers Executor - java.lang.reflect.InvocationTargetException HOT 1
- Can't modify bind-address of broker HOT 2
- Security support for mesos-kafka HOT 7
- very quick offerRescinded, brokers not starting HOT 11
- Not able to start broker with Vagrant example HOT 3
- Topic rebalance REST API: NPE for get status
- How to add kafka connectors
- Multiple version of kafka same schedular HOT 2
- Does adding topic have to be through the framework HOT 1
- Exception when creating topics HOT 1
- Question: Differerence between mesos-kafka & DCOS-kafka-service? HOT 1
- Framework stuck at broker reconciling state HOT 5
- More config support for scheduler from Apache Kafka? HOT 6
- Error: java.io.IOException: 400 - java.lang.NullPointerException, when broker start
- Missing broker task attribute 'endpoint' on broker startup / reconciliation HOT 3
- "listeners" manipulation when bind-address set breaks ability to use SSL
- Build Failing for openJdk7 & KafkaVer:0.10.2.0
- Will Kafka 0.11.x be supported? HOT 10
- broker start timeout when Mesos agent start with gpu/nvidia isolation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka.