monitor-events terminates with no error message

[azure]

monitor-events terminates with no error message

[atecce]

hey there. I'm dealing with the same problems. for the life of me I can't read telemetry at all from my IoT Hub. I can dump it in Time Series Insights with an Event Source and see it, but I can't use any API to get programmatic access, including several client libraries in Go and C# over the Event Hub connection string and now with az, and --debug doesn't seem to show any smoking gun, just stuff like this:

I know there's telemetry coming in, both from the time series graph that I can see in Azure Portal and from, again, TSI

[digimaun]

Can you please provide any more details? Are there multiple consumers on the $Default consumer group?

You can run the command appended with --debug for more information.

[vanti]

Hi digimaun,

How do I verify how many consumers there are in $Default consumer group?
The trace stops here while waiting for ctrl-c:

uamqp.c_uamqp : b'-> [ATTACH]* {receiver-link-d19d555e-beca-4b97-93af-c874c3ae46bb,5,true,0,1,* {amqps://iothub-ns-hub-173603-8d18738641.servicebus.windows.net/hub/ConsumerGroups/$Default/Partitions/2,NULL,NULL,NULL,NULL,NULL,NULL,{[apache.org:selector-filter:string:* amqp.annotation.x-opt-enqueuedtimeutc > 1542069893156]}},* {f7be7f0f-aef0-4beb-8447-1151319c71ea},NULL,NULL,NULL,262144}'

After 10 seconds, these are the first few messages that show up:

uamqp.c_uamqp : b'<- [ATTACH]* {receiver-link-ce4febce-9d48-49fa-84f8-d87dbbb12851,2,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL}'
uamqp.c_uamqp : b'Cannot get initial delivery count' (b'/data/src/vendor/azure-uamqp-c/src/link.c':b'link_frame_received':333)
uamqp.receiver : Receiver link failed to open - expecting to receive DETACH frame.

This only seems to happen when we run this behind our company firewall, which has a proxy plug setup for AMQPS to 'iothub-ns-hub-173603-8d18738641.servicebus.windows.net'.

[onionhammer]

It is doing this for me as well. I attach it with -t 0 and it simply stops after 30 seconds or so

[digimaun]

@onionhammer Are you also experiencing this issue behind a firewall?

[digimaun]

+ @annatisch

[onionhammer]

No, it works but then it just stops

[digimaun]

Ok, then it seems like a distinct issue from what vanti is facing. Are there any errors or more information to share from --debug logs?

[onionhammer]

Ah it looks like a consumer group conflict?

LinkDetach("ErrorCodes.LinkStolen: At least one receiver for the endpoint is created with epoch of '55', and so non-epoch receiver is not allowed. Either reconnect with a higher epoch, or make sure all epoch receivers are closed or disconnected.

So the only bug here would be the lack of any kind of error message without pouring over the logs.

[digimaun]

Yea looks like a consumer group conflict. And agreed the error should be bubbling up.

[k8iechen]

I also have the same issue: monitor just terminates without outputting anything.

[digimaun]

@k8iechen You should run the command with --debug to see if any more details show up. You could be experiencing a consumer group conflict.

[atecce]

lol. need to enable a fallback route or a custom message route for anyone else who lands here

[androidand]

I have the same issue as vanti, well, actually it doesn't look like it terminates, it just shows no event messages at all after a while.

We are using custom routes in the IoT-hub but I've also defined the endpoint "events" for TwinChangeEvents, DeviceMessages and DeviceLifecycleEvents in order to run the az iot hub monitor-events.

Running
az iot hub monitor-events -n our-iothub-name -d mydeviceid123
Shows all events for a while, then (randomly?) the events stop showing until az is stopped and started again. I can see that events are coming in on another route.

Running debug gives similar output as described by atecce, not very useful.
az iot hub monitor-events -n our-iothub-name -d mydeviceid123 --debug

azure-cli (2.0.57)
iot (0.3.5)
iotcentral (0.1.5)

[digimaun]

@androidand, please make sure you are using a dedicated consumer group when using monitor-events. The best practice is to use one consumer group per consumer.

Based on your comment looks like you have already set default/fallback routes to the default eventhub when provisioning an IoT Hub. Current monitor-events only monitors the default events endpoint.

I am not sure what version of the IoT extension you are running, but its not the latest as the latest requires azure-cli 2.0.70 or greater. So one quick thing to try is upgrading your cli core and extension version to the latest (currently 0.8.2).

Finally make sure there are no firewall related issues at play. We do not yet have proxy over websocket support.

[digimaun]

Closing this long running issue. There have been a number of improvements to monitor-events since this thread - including for example bubbling amqp errors to the user rather than having to use --debug. So please try the latest version (taking into account the tips described in the thread) and if there are issues or feedback lets create a new issue.