Segmentation fault using bridging

[jpmens]

Segmentation fault using bridging

[jpmens]

Excellent error report; thanks and sorry!

I need to know whether the plugin on mosq(a) and (b) were compuled with just one BE_ backend or more than one.

I think this may be an off by one error (again: sigh). I'll try and look into this asap.

[oshoemaker]

Thanks for the quick reply! Yes mosqa(a) was compiled with a single backend (redis). Mosqa(b) has multiple. Mosqa(a) is the one crashing.

• Owen

On Mar 27, 2014, at 10:11 AM, JP Mens [email protected] wrote:

Excellent error report; thanks and sorry!

I need to know whether the plugin on mosq(a) and (b) were compuled with just one BE_ backend or more than one.

I think this may be an off by one error (again: sigh). I'll try and look into this asap.
—
Reply to this email directly or view it on GitHub.

[jpmens]

May I ask you to do what you'll dread me asking? Can you add a backend to the one with just one? CDB is probably easiest... (Source of cdb is included. )

[jpmens]

@oshoemaker Don't bother, please. I see it now in your report:

1395917713: |-- user bridge was authenticated in back-end 16 (<nil>)

[jpmens]

According to your log above, the client bridge never authenticated to the target broker, which is why the back-end hadn't been selected. (This is very strange -- I didn't think that was possible... maybe we've uncovered a bug in Mosquitto, but let me assume for the moment, that that isn't the case.)

Do me a favor please: pull the ldap branch in which I've (hopefully) fixed this bug. (In any case I've added a bounds-check for the index, and the ACL check should quietly fail with NOT AUTHORIZED and log accordingly).

Before pulling the ldap branch, save your Makefile and then copy it over. That ought to be all that's required.

[oshoemaker]

1395945845: Received SUBACK from monolith.fromCloud
1395945851: |-- nord is -1: unpossible!
1395945851: Denied PUBLISH from monolith.toCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395945851: |-- nord is -1: unpossible!
1395945851: Denied PUBLISH from monolith.fromCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))

It seem that this is the cause…

On Mar 27, 2014, at 11:19 AM, JP Mens [email protected] wrote:

According to your log above, the client bridge never authenticated to the target broker, which is why the back-end hadn't been selected. (This is very strange -- I didn't think that was possible... maybe we've uncovered a bug in Mosquitto, but let me assume for the moment, that that isn't the case.)

Do me a favor please: pull the ldap branch in which I've (hopefully) fixed this bug. (In any case I've added a bounds-check for the index, and the ACL check should quietly fail with NOT AUTHORIZED and log accordingly).

Before pulling the ldap branch, save your Makefile and then copy it over. That ought to be all that's required.

—
Reply to this email directly or view it on GitHub.

[jpmens]

I'm not able to reproduce this... You're quite sure you don't see a line like

|-- mosquitto_auth_unpwd_check(.........)

on stderr? Could you please configure log_type debug and show me what you get? (You can send me the output privately if you prefer)

[oshoemaker]

Mosq(a):

root@monolith:/usr/local/src/mosquitto-auth-plug# vi /etc/mosquitto/mosquitto.conf
root@monolith:/usr/local/src/mosquitto-auth-plug# /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf -v
1395953856: mosquitto version 1.3 (build date 2014-03-27 00:13:27+0000) starting
1395953856: Config loaded from /etc/mosquitto/mosquitto.conf.
*** auth-plug: backend=BACKEND
1395953856: |-- AuthOptions: key=backends, val=redis
1395953856: |-- AuthOptions: key=host, val=localhost
1395953856: |-- AuthOptions: key=port, val=6379
** Configured order: redis
1395953856: |-- }}}} Redis
1395953856: Opening ipv4 listen socket on port 1883.
1395953856: Opening ipv6 listen socket on port 1883.
1395953856: Opening ipv4 listen socket on port 1884.
1395953856: Opening ipv6 listen socket on port 1884.
1395953856: Bridge monolith.toCloud doing local SUBSCRIBE on topic /remote/#
1395953856: Connecting bridge toCloud (10.0.0.7:1883)
1395953856: Bridge monolith.toCloud sending CONNECT
1395953856: Connecting bridge fromCloud (10.0.0.7:1883)
1395953856: Bridge monolith.fromCloud sending CONNECT
1395953856: Received CONNACK on connection monolith.toCloud.
1395953856: Bridge monolith.toCloud sending UNSUBSCRIBE (Mid: 34, Topic: /remote/#)
1395953856: Received CONNACK on connection monolith.fromCloud.
1395953856: Bridge monolith.fromCloud sending SUBSCRIBE (Mid: 4, Topic: /remote/test/control, QoS: 0)
1395953856: Received PUBACK from monolith.toCloud (Mid: 33)
1395953856: Received UNSUBACK from monolith.toCloud
1395953856: Received PUBACK from monolith.fromCloud (Mid: 3)
1395953856: Received SUBACK from monolith.fromCloud
1395953859: |-- nord is -1: unpossible!
1395953859: Denied PUBLISH from monolith.toCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395953859: |-- nord is -1: unpossible!
1395953859: Denied PUBLISH from monolith.fromCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))

Mosq(b)

1395953859: New bridge connected from 10.0.0.150 as monolith.fromCloud (c0, k60, ubridge).
1395953859: Sending CONNACK to monolith.fromCloud (0)
1395953859: |-- aclcheck(bridge, $SYS/broker/connection/monolith.toCloud/state, 2) SUPERUSER=Y by mysql
1395953859: Received PUBLISH from monolith.toCloud (d0, q1, r1, m33, '$SYS/broker/connection/monolith.toCloud/state', ... (1 bytes))
1395953859: Sending PUBACK to monolith.toCloud (Mid: 33)
1395953859: Received UNSUBSCRIBE from monolith.toCloud
1395953859: /remote/#
1395953859: monolith.toCloud /remote/#
1395953859: |-- aclcheck(bridge, $SYS/broker/connection/monolith.fromCloud/state, 2) SUPERUSER=Y by mysql
1395953859: Received PUBLISH from monolith.fromCloud (d0, q1, r1, m3, '$SYS/broker/connection/monolith.fromCloud/state', ... (1 bytes))
1395953859: Sending PUBACK to monolith.fromCloud (Mid: 3)
1395953859: Received SUBSCRIBE from monolith.fromCloud
1395953859: /remote/test/control (QoS 0)
1395953859: monolith.fromCloud 0 /remote/test/control
1395953859: Sending SUBACK to monolith.fromCloud
1395953861: New connection from 10.0.0.156 on port 1883.
1395953861: |-- mosquitto_auth_unpwd_check(bridge)
1395953861: |-- ** checking backend mysql
1395953861: |-- getuser(bridge) AUTHENTICATED=1 by mysql
1395953861: New client connected from 10.0.0.156 as mosqpub/21962-ow3nd.loc (c1, k60, ubridge).
1395953861: Sending CONNACK to mosqpub/21962-ow3nd.loc (0)
1395953861: |-- aclcheck(bridge, /remote/test/control, 2) SUPERUSER=Y by mysql
1395953861: Received PUBLISH from mosqpub/21962-ow3nd.loc (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395953861: |-- aclcheck(bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1395953861: |-- aclcheck(bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1395953861: |-- aclcheck(bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1395953861: |-- aclcheck(bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1395953861: Sending PUBLISH to monolith.toCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395953861: Sending PUBLISH to monolith.fromCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395953861: Sending PUBLISH to bridge (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1395953861: |-- aclcheck(bridge, /bridge/remote/test/control, 2) SUPERUSER=Y by mysql
1395953861: Received PUBLISH from bridge (d0, q0, r0, m0, ‘/bridge/remote/test/control', ... (18 bytes))
1395953861: Received DISCONNECT from mosqpub/21962-ow3nd.loc

The main difference between Mosq(A) and Mosq(B) is A is Debian Wheezy on ARM (32-bit) and the B is Ubuntu Precise on Intel (64-bit).

On Mar 27, 2014, at 12:17 PM, JP Mens [email protected] wrote:

I'm not able to reproduce this... You're quite sure you don't see a line like

|-- mosquitto_auth_unpwd_check(.........)
on stderr? Could you please configure log_type debug and show me what you get? (You can send me the output privately if you prefer)

—
Reply to this email directly or view it on GitHub.

[jpmens]

We're getting closer, but this is a tough one. Thanks to a hint from Roger, it would seem as though local incoming bridge configurations don't authenticate locally.

Do me a favor please on mosq(a):

pull the code again and run your test, pasting the log once again (and hopefully for the last time)

I want to see it all, but I'm particularly interested in a line which starts like

|-- mosquitto_auth_acl_check

[oshoemaker]

Here is the log from Mosq(a):

1396001364: Received UNSUBACK from monolith.toCloud
1396001364: Received PUBACK from monolith.fromCloud (Mid: 1)
1396001364: Received SUBACK from monolith.fromCloud
1396001366: |-- mosquitto_auth_acl_check(..., monolith.toCloud, elarm-bridge, /remote/test/control, 2)
1396001366: |-- nord is -1: unpossible!
1396001366: Denied PUBLISH from monolith.toCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1396001366: |-- mosquitto_auth_acl_check(..., monolith.fromCloud, elarm-bridge, /remote/test/control, 2)
1396001366: |-- nord is -1: unpossible!
1396001366: Denied PUBLISH from monolith.fromCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))

And Mosq(b):

1396001366: New connection from 10.0.0.150 on port 1883.
1396001366: |-- mosquitto_auth_unpwd_check(elarm-bridge)
1396001366: |-- ** checking backend mysql
1396001366: |-- getuser(elarm-bridge) AUTHENTICATED=1 by mysql
1396001366: |-- aclcheck(elarm-bridge, $SYS/broker/connection/monolith.toCloud/state, 2) SUPERUSER=Y by mysql
1396001366: New bridge connected from 10.0.0.150 as monolith.toCloud (c0, k60, uelarm-bridge).
1396001366: Sending CONNACK to monolith.toCloud (0)
1396001366: New connection from 10.0.0.150 on port 1883.
1396001366: |-- mosquitto_auth_unpwd_check(elarm-bridge)
1396001366: |-- ** checking backend mysql
1396001366: |-- getuser(elarm-bridge) AUTHENTICATED=1 by mysql
1396001366: |-- aclcheck(elarm-bridge, $SYS/broker/connection/monolith.fromCloud/state, 2) SUPERUSER=Y by mysql
1396001366: New bridge connected from 10.0.0.150 as monolith.fromCloud (c0, k60, uelarm-bridge).
1396001366: Sending CONNACK to monolith.fromCloud (0)
1396001366: |-- aclcheck(elarm-bridge, $SYS/broker/connection/monolith.toCloud/state, 2) SUPERUSER=Y by mysql
1396001366: Received PUBLISH from monolith.toCloud (d0, q1, r1, m1, '$SYS/broker/connection/monolith.toCloud/state', ... (1 bytes))
1396001366: Sending PUBACK to monolith.toCloud (Mid: 1)
1396001366: Received UNSUBSCRIBE from monolith.toCloud
1396001366: /remote/#
1396001366: monolith.toCloud /remote/#
1396001366: |-- aclcheck(elarm-bridge, $SYS/broker/connection/monolith.fromCloud/state, 2) SUPERUSER=Y by mysql
1396001366: Received PUBLISH from monolith.fromCloud (d0, q1, r1, m1, '$SYS/broker/connection/monolith.fromCloud/state', ... (1 bytes))
1396001366: Sending PUBACK to monolith.fromCloud (Mid: 1)
1396001366: Received SUBSCRIBE from monolith.fromCloud
1396001366: /remote/test/control (QoS 0)
1396001366: monolith.fromCloud 0 /remote/test/control
1396001366: Sending SUBACK to monolith.fromCloud
1396001368: New connection from 10.0.0.156 on port 1883.
1396001368: |-- mosquitto_auth_unpwd_check(elarm-bridge)
1396001368: |-- ** checking backend mysql
1396001368: |-- getuser(elarm-bridge) AUTHENTICATED=1 by mysql
1396001368: New client connected from 10.0.0.156 as mosqpub/23941-ow3nd.loc (c1, k60, uelarm-bridge).
1396001368: Sending CONNACK to mosqpub/23941-ow3nd.loc (0)
1396001368: |-- aclcheck(elarm-bridge, /remote/test/control, 2) SUPERUSER=Y by mysql
1396001368: Received PUBLISH from mosqpub/23941-ow3nd.loc (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1396001368: |-- aclcheck(elarm-bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1396001368: |-- aclcheck(elarm-bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1396001368: |-- aclcheck(elarm-bridge, /remote/test/control, 1) SUPERUSER=Y by mysql
1396001368: Sending PUBLISH to monolith.toCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1396001368: Sending PUBLISH to monolith.fromCloud (d0, q0, r0, m0, '/remote/test/control', ... (18 bytes))
1396001368: Received DISCONNECT from mosqpub/23941-ow3nd.loc

On Mar 28, 2014, at 2:49 AM, JP Mens [email protected] wrote:

We're getting closer, but this is a tough one. Thanks to a hint from Roger, it would seem as though local incoming bridge configurations don't authenticate locally.

Do me a favor please on mosq(a):

pull the code again and run your test, pasting the log once again (and hopefully for the last time)

I want to see it all, but I'm particularly interested in a line which starts like

|-- mosquitto_auth_acl_check
—
Reply to this email directly or view it on GitHub.

[jpmens]

Perfect @oshoemaker, thank you very much. This is what I was looking for, and I can reproduce the error as you described it:

|-- mosquitto_auth_acl_check(..., monolith.toCloud, elarm-bridge, /remote/test/control, 2)

The good news is we've found why it happens, and it would in theory be easy to fix.

The bad news is, that this is going to mean a change in the Mosquitto broker code. Your mosq(a) broker has a username/password configured for mosq(b), and it (a) bridges topics IN. That username/password is used by (a) to authenticate to (b).

When a receives a PUB on that INcoming topic, it invokes the authentication plugin to determine whether it may publish to the particular topic (ACL check). It has, however, not yet authenticated to the plugin (because it simply doesn't do that; recall the configured username/password is used to authenticate to b!) Since I (well, auth-plugin) haven't seen the (local broker) authenticate, I don't know which of the auth-plugin back-ends is responsible for that particular ACL check, and we give up. (Giving up meant crashing, now it means logging and returning DENIED by default).

I'm talking to @ralight at the moment, and this is probably going to mean adding something like local_username to Mosquitto's configuration.

To cut a long story short: we cannot currently use auth-plug for bridges configured as you have your mosq(a). Bridges that operate as mosq(b) does are fine (as you have already confirmed.

[oshoemaker]

Thanks for all the help. I apologize that I can’t be of more help but my C skills are pretty weak.

I am guessing I don’t need to file a bug with @ralight as you are working with him. If you need anything else from me let me know. It would be really nice to have this feature working but I can get by for the time being with a small workaround.

On Mar 28, 2014, at 3:25 AM, JP Mens [email protected] wrote:

Perfect @oshoemaker, thank you very much. This is what I was looking for, and I can reproduce the error as you described it:

|-- mosquitto_auth_acl_check(..., monolith.toCloud, elarm-bridge, /remote/test/control, 2)
The good news is we've found why it happens, and it would in theory be easy to fix.

The bad news is, that this is going to mean a change in the Mosquitto broker code. Your mosq(a) broker has a username/password configured for mosq(b), and it (a) bridges topics IN. That username/password is used by (a) to authenticate to (b).

When a receives a PUB on that INcoming topic, it invokes the authentication plugin to determine whether it may publish to the particular topic (ACL check). It has, however, not yet authenticated to the plugin (because it simply doesn't do that; recall the configured username/password is used to authenticate to b!) Since I (well, auth-plugin) haven't seen the (local broker) authenticate, I don't know which of the auth-plugin back-ends is responsible for that particular ACL check, and we give up. (Giving up meant crashing, now it means logging and returning DENIED by default).

I'm talking to @ralight at the moment, and this is probably going to mean adding something like local_username to Mosquitto's configuration.

To cut a long story short: we cannot currently use auth-plug for bridges configured as you have your mosq(a). Bridges that operate as mosq(b) does are fine (as you have already confirmed.

—
Reply to this email directly or view it on GitHub.

[jpmens]

What I can do at short notice, but I'd like you to think about whether this would be useful, is to add a few parameters for auth-plug to work around this issue, at least until Roger has considered how to solve the issue on the Mosquitto side.

Something along the lines of:

auth_opt_local_client monolith.toCloud
auth_opt_local_username elarm-bridge
auth_opt_local_backend redis

It's a bit of a hack, but it could solve the issue. I haven't yet thought about this carefully, so please do so for me.... ;-)

[jpmens]

Forgot to say the important part: the plugin would use those values as defaults if, and only if, an ACL check is requested without prior authentication request, which is the case we're concerned with here.

[oshoemaker]

So let me understand this correctly….

Mosq(a) establishes a connection to Mosq(b).
Subscribes to a topic using user/password (configured in the bridge config)
I send a message to Mosq(b) which is authenticated by the ACL
Mosq(b) bridges that to the topic that Mosq(a) is subscribed to.
*** Here is where the issue is ***
Mosq(a) receives the MQTT message and then goes to check it’s ACL as there is no previous auth backend for it.
Mosq(a) denies the message.

So what you are proposing is to add password for the bridge into the backend (in my case redis) as well as in the bridge config so it would work in both scenarios.

This sounds like a plausible workaround and would definitely get us rolling with full authentication in both directions if I am understanding this correctly.

On Mar 28, 2014, at 4:24 AM, JP Mens [email protected] wrote:

Forgot to say the important part: the plugin would use those values as defaults if, and only if, an ACL check is requested without prior authentication request, which is the case we're concerned with here.

—
Reply to this email directly or view it on GitHub.

[jpmens]

I haven't forgotten you, but I am having second thoughts about this ...

Yes, you understood correctly, and yes, this is a horribly workaround. The reason for my second thoughts is that it would force a particular auth backend (on mosq(a)) for all bridges configured, and I don't like that.

I'm going to hang on a bit and see if @ralight has thought about implementing something like local_user and local_password for solving this issue more generically. (He has been very busy I hear, so this could take a bit, but I have spoken to him about it.)

[ralight]

I think this patch to mosquitto should allow the problem to be fixed by specifying a local username and/or password for the bridge.

http://h.ral.me/mqtt/bridge.patch

[oshoemaker]

Awesome. I will give it a test tomorrow.

• Owen

On Apr 4, 2014, at 1:51 PM, Roger Light [email protected] wrote:

I think this patch to mosquitto should allow the problem to be fixed by specifying a local username and/or password for the bridge.

http://h.ral.me/mqtt/bridge.patch

—
Reply to this email directly or view it on GitHub.

[jpmens]

I'm testing this right now, and it looks very good, thank you Roger. For the record, I'm mentioning here what I just said on IRC, so that @oshoemaker sees it:

connection fromCloud
address 127.0.0.1:9993
username FOOX
local_username CRASHME
local_password OOPSIE
cleansession true
clientid LDAP-BRIDGE
start_type automatic
topic # in 0 a/ a/

the local_username works to authenticate. When the acl_check is run, a NULL username is passed in unless I specify username as well.

[jpmens]

@oshoemaker Please apply this 2nd patch on top of the first; you can then drop username from the above config.

It would be nice if you could report results.

[oshoemaker]

I will have time to test this tomorrow. I have been a bit swamped this weekend.

• Owen

On Apr 7, 2014, at 12:25 AM, JP Mens [email protected] wrote:

@oshoemaker Please apply this 2nd patch on top of the first; you can then drop username from the above config.

It would be nice if you could report results.

—
Reply to this email directly or view it on GitHub.

[jpmens]

I know the feeling ... ;-)

[oshoemaker]

No luck for me. I used the following config parameters:

local_username
local_password

Also tried with just local_username and jpmens patch. I get the log line:

1396916100: Bridge monolith.toCloud failed authentication on local broker.
1396916100: Bridge monolith.fromCloud failed authentication on local broker.

• Owen

On Apr 4, 2014, at 1:51 PM, Roger Light [email protected] wrote:

I think this patch to mosquitto should allow the problem to be fixed by specifying a local username and/or password for the bridge.

http://h.ral.me/mqtt/bridge.patch

—
Reply to this email directly or view it on GitHub.

[jpmens]

So you applied both patches, and you created the local_name user
with its local_name password in your authentication backend? And you
verified that a connect to your Mosquitto was possible (mosquitto_pub)
with that local_username and local_password? And it didn't work?

Can you please post the debug output of the first PUB to the broker from
which the IN bridge then subscribes?

[oshoemaker]

Ok, that may have been the part that I missed. I created local_username & local_password but I did not configure them in the “local” backend. I only had them configured on the remote side.

So first I tried bridge.patch. It worked with the local_username & local_password set.

Next I tried the bridge-2.patch. It worked with the local_username & local_password set.

However, bridge-2.patch did not work if I did not set local_password. Is this the expected behavior?

BTW, this is with the latest tarball of mosquitto (1.3) with only the patches applied and using this commit from MASTER of the git-auth-log (not ldap):

commit 79af802
Author: Jan-Piet Mens [email protected]
Date: Wed Nov 20 07:52:21 2013 +0100

• Owen

On Apr 7, 2014, at 10:42 PM, JP Mens [email protected] wrote:

So you applied both patches, and you created the local_name user
with its local_name password in your authentication backend? And you
verified that a connect to your Mosquitto was possible (mosquitto_pub)
with that local_username and local_password? And it didn't work?

Can you please post the debug output of the first PUB to the broker from
which the IN bridge then subscribes?
—
Reply to this email directly or view it on GitHub.

[jpmens]

Well, I'd prefer if you use the 'ldap' branch (more debugging), but you've basically confirmed that the two patches work! (FWIW, I don't think you were lucky with just bridge.patch' patch applied (unless you also hadusername' configured...)

Anyway, to cut a long story short: will you ACK that this is now working as you expected it to? If so, maybe we can convince @ralight to include those patches in the next release. :-)

[oshoemaker]

Yes, everything looks great now. The only problem now is I have to get the rest of my application written ;) Thanks for all of the help.

• Owen

On Apr 7, 2014, at 11:39 PM, JP Mens [email protected] wrote:

Well, I'd prefer if you use the 'ldap' branch (more debugging), but you've basically confirmed that the two patches work! (FWIW, I don't think you were lucky with just bridge.patch' patch applied (unless you also hadusername' configured...)

Anyway, to cut a long story short: will you ACK that this is now working as you expected it to? If so, maybe we can convince @ralight to include those patches in the next release. :-)
—
Reply to this email directly or view it on GitHub.

[jpmens]

@oshoemaker FYI, I hear from Roger, that the commits are already in the 1.4 branch.

[jpmens]

@oshoemaker FYI, the local_username/password feature is in 1.3.2 (currently on Bitbucket), and is due for release very soon, I hear.

[oshoemaker]

Awesome. Thanks for the update. I will check it out and give it a shot.

• Owen

On May 8, 2014, at 11:19 PM, JP Mens [email protected] wrote:

@oshoemaker FYI, the local_username/password feature is in 1.3.2 (currently on Bitbucket), and is due for release very soon, I hear.

—
Reply to this email directly or view it on GitHub.