Kafka Broker should use Mesos Slave's hostname as host.name and advertised.host.name?

[mesos]

Kafka Broker should use Mesos Slave's hostname as host.name and advertised.host.name?

[dmitrypekar]

Hi yanglei99,
thank you for posting your problem and providing a fix.

Right now the workaround is to use broker options like --options=host.name=,advertised.host.name= (options param in REST) during broker add|update.

Regarding your fix. I need to make a decision about applicability of it. The policy is following:

• for any specific use case any option could be overriden by --options (options param in REST);
• defaults should be provided in code only for a generic (70%+) usecase;

So the question is: is your usecase generic enough to support it in the code?

To clarify this I need to know more about your environment:

1. how cluster is ran: in vagrant, in AWS, in VMS, etc.
2. how many and what public interfaces do the master|slaves have? What is chosen by Kafka Broker by default?
3. how does Mesos master knows where the correct (public accessible) interface is? What are the guarantees that offer.getHostname would return the correct interface? i.e is it safe to rely on offer.getHostname in environments with multiple interfaces?

Sorry, for a pedantry, but this info is really important in order to make a decision about integrating your fix.

[yanglei99]

Hello dmitrypekar,

Thank you for your recommendation on using --options!

However I still think using mesos-slave's announced hostname is a better approach. As you will leave the decision on which network interface (or hostname ) to use to the people setting up mesos cluster. And it will also make it consistent on Kafka side, as you do show the mesos-slave offer's hostname which is in the form of ip address for my environment.

To answer your questions:

1. how cluster is ran: in vagrant, in AWS, in VMS, etc.

Softlayer

1. how many and what public interfaces do the master|slaves have? What is chosen by Kafka Broker by default?

Have one public ip and one private ip on each VM (master and slave). I define the one I use in mesos-slave /etc/mesos-slave/ip and /etc/mesos-slave/hostname. Kafka Broker does show the right ip when using "kafka-mesos.sh status" as you get it from mesos's offer. It is the Kafka (shown in Broker's log) that used the VM's hostname not mesos slave ip for communication.

1. how does Mesos master knows where the correct (public accessible) interface is? What are the guarantees that offer.getHostname would return the correct interface? i.e is it safe to rely on offer.getHostname in environments with multiple interfaces?

It is a good question. I think it is down to what the contract is between mesos(slave) and framework executor. First, mesos master and slave do have contract in announcing what hostname(or ip) to use for their communication, configurations defined under /etc/ mesos-slave. Now, about the contract between executor and slave, I have assumed by default it is what the salve uses and passed by offer. But then I may have missed the reason we choose not to do so.

Really appreciate your thoughts and help!

Yang Lei

[dmitrypekar]

Hi yanglei99,

Thank you very much for a clarification.

I've done several tests and now I see that your changes absolutely makes sense as a defaults except for 'advertised.host.name', which by default in Kafka equals to 'host.name'. So I've added host.name=offer.getHostname as default option which could be overridden if required.

The changes are available in branch kafka-default-bind-addr, which will be merged by Joe Stein in soon (in day or two). You can still checkout a branch and use it directly if required.

Please also comment, if setting 'advertised.host.name' as a default is required in your env. I think, that everything should work wo those option. If not - let's discuss that. Maybe it is correct to add those option to the defaults too.

[yanglei99]

Thanks dmitrypekar!

Yes, I know the two settings are kind of redundant. :-) As advertised.host.name should derive from host.name. I will give it a try again and let you know if it does not work.

Thanks again.

Yang

[joestein]

The patch lgtm, I merged this to master.

The host.name is how the broker ultimately binds to that address.

The advertised.host.name is what the broker says to connect to.

Same with host.port and advertised.host.port respectively.

So, the broker can bind on 0.0.0.0 but say connect to me on 10.10.5.2 or whatever.

Binding on the slave offer's hostname does makes sense. This however, may not be routable to other brokers, producer and consumers and that value will just have to be updated through the CLI which can be updated through options in cli or rest api.