Periodically keep Pubnub connection alive

[jimsmith]

Periodically keep Pubnub connection alive

[jschamburger]

You're totally right!
I noticed the same thing.
For 2.3.1 I changed the push notification service to avoid battery drain caused by waking the device up every couple of minutes, but apparently it is not working as it's supposed to be yet.
I'll take care of it!

[jimsmith]

cool - I'm only using pubnub as that's what this app uses - there are a few other alternatives out there if it's easier to offer more than just this.

just a thought.

cool :)

[jschamburger]

I'm optimistic that the upcoming version 2.3.3 fixes the problem. However, I want to monitor the app over a couple of hours to be safe.
Also, I'd appreciate input on this! :-)

[jimsmith]

Just updated to 2.3.3 - more than happy to give input - going to test throughout the morning so be around 4 hours to allow battery usage and see how testing goes.

:)

[jschamburger]

Thanks for that!

Unfortunately, my own tests have already shown that the service dies after a while unless it is woken up. So we will have to wake it up periodically to make sure the push messages come through.
The more battery-efficient solution I imagined will not work unfortunately.

[jimsmith]

found this https://help.pubnub.com/entries/21182136-Can-I-Use-PubNub-for-Long-Running-Subscription-to-Push-Notifications-

interesting though I also found a 'cached' google page for pubnub mobile but this is removed from the live page 'Minimize battery drain'

[Pimmetje]

Intresting. I can image that a android service would save battery (specially when u have more programs using push). They talk about a native application in android. But i searched for it a few days ago and could not find one. If i am not mistaken google placed this service into there market application.

[jimsmith]

maybe this is it ?

https://developers.geoloqi.com/android/push-notifications
http://developer.android.com/google/gcm/index.html

[jjoe64]

https://help.pubnub.com/entries/21720011-Can-my-Android-App-Receive-Messages-While-Inactive-

[jimsmith]

makes sense - as various other andriod apps require to start at andriod boot-up
(which reminds me - skype is one of these apps which doesn't need to start at boot-up wasting valuable resource and battery lol)

[jimsmith]

confirmed battery drained (was 36% now at 9%)

[jimsmith]

using new version 2.3.4 - I'm seeing across various devices that push notifications over wifi has stopped working since upgraded to 2.3.4.

[jjoe64]

ok we changed same things about the push service in 2.3.5. Please update to it and let us know if your problems get solved. If not, we will switch the push-service from Pubnub to GCM

[geremyCohen]

@jjoe64 if you are experiencing any issues implementing PubNub, please contact us at [email protected], we're happy to work with you to get it resolved.
geremy

[Nowaker]

We have today missed very important push notifications. Both people didn't get them to their phones. Therefore, I believe push notifications in ZAX are still broken (unreliable).

[geremyCohen]

@Nowaker please email me at [email protected] so we can dig deeper... PubNub is very reliable, so I want to help get to the bottom of this as quickly as possible...

[Nowaker]

@geremyCohen I never meant it's PubNub to blame! The problem lays on ZAX app side.

[geremyCohen]

Ah, ok. :) If you need help on the PN side, don't hesitate to contact us for support on this.

[Nowaker]

@geremyCohen Thanks. No need to contact PN. Have a good day, Mr Murdock. ;)

[geremyCohen]

I love it when a plan comes together!

On Wed, Sep 10, 2014 at 8:37 PM, Damian Nowak [email protected]
wrote:

@geremyCohen https://github.com/geremyCohen Thanks. No need to contact
PN. Have a good day, Mr Murdock. ;)

Reply to this email directly or view it on GitHub
#30 (comment).

[jschamburger]

@Nowaker I am very sorry for the problems you've been having!
I just reviewed the code responsible for the Pubnub registration and the background service handling the Pubnub messages and showing the notifications looks ok so far. We have tried the app ourselves without any problems for a while. It's tough to reproduce the issue which makes it hard to solve.

@geremyCohen Maybe you could help us with the Pubnub integration as we use it in Zax.
I just looked at the Pubnub examples here: https://github.com/pubnub/java/tree/master/android/examples/SubscribeAtBoot
Apparently you are using partial wake locks for the Pubnub services. Is that the recommended way of using Pubnub? Or is it possible to use wakeful intent services?

Maybe you could take a look at the Zax source code related to Pubnub?
This is how it currently works: In the Manifest, we register the push service in a separate process: class: .push.PushService.
If Pubnub is enabled, the service is started using PushService.startOrStopPushService() if the settings are changed, the device is rebooted (.push.BootCompletedIntentReceiver) or the app is updated (OnUpgradeReceiver). This method also sets a repeating alarm to refresh the service if it has been canceled.

Some help using your expertise and experience with Pubnub would be highly appreciated!

[geremyCohen]

Happy to help! For starts, you can use a verbose log debug version of the
client, to see whats happening in greater detail on the PN side: ant clean
then ant debug build will turn that on.

Next, as you suggest, a partial wake lock (level 4) is required to keep the
necessary radio/network operations functioning when Android attempts to
sleep.

If you enable logging, and partial wake lock, and you still experience
issues, if you can send me the log, we can move forward (code review, etc).
Just shoot me an email at [email protected] so it doesn't get lost in the
mix...

geremy

On Thu, Sep 11, 2014 at 9:04 AM, Johannes Schamburger <
[email protected]> wrote:

@Nowaker https://github.com/Nowaker I am very sorry for the problems
you've been having!
I just reviewed the code responsible for the Pubnub registration and the
background service handling the Pubnub messages and showing the
notifications looks ok so far. We have tried the app ourselves without any
problems for a while. It's tough to reproduce the issue which makes it hard
to solve.

@geremyCohen https://github.com/geremyCohen Maybe you could help us
with the Pubnub integration as we use it in Zax.
I just looked at the Pubnub examples here:
https://github.com/pubnub/java/tree/master/android/examples/SubscribeAtBoot
Apparently you are using partial wake locks for the Pubnub services. Is
that the recommended way of using Pubnub? Or is it possible to use wakeful
intent services?

Maybe you could take a look at the Zax source code related to Pubnub?
This is how it currently works: In the Manifest, we register the push
service in a separate process: class: .push.PushService.
If Pubnub is enabled, the service is started using
PushService.startOrStopPushService() if the settings are changed, the
device is rebooted (.push.BootCompletedIntentReceiver) or the app is
updated (OnUpgradeReceiver). This method also sets a repeating alarm to
refresh the service if it has been canceled.

Some help using your expertise and experience with Pubnub would be highly
appreciated!

Reply to this email directly or view it on GitHub
#30 (comment).

[jjoe64]

partial wake lock is no option for us because this will cost much battery - we have tried this earlier.
The users who want this, can download app to activate the partial wake lock manually. This may fix the problem, but I'm sure their won't be happy with the battery life time.
currently the app wakes app in a interval via AlarmManager and refreshes the Pubnub connection. This solution works for us but as we can see others have problems with it.

[Nowaker]

I can just say that battery is not an indication for me. We want ZAX so notify us of every downtime no matter what. And wouldn't have turned off email notifications from Zabbix if we knew we can miss some. In fact, my phone can work no more than 5 hours without a cable, as I use it for many important things related to our business: e-mail, customer support, company chat and, obviously, monitoring. :-)

[jjoe64]

@Nowaker
install that app and try it
https://play.google.com/store/apps/details?id=eu.thedarken.wl&hl=de

[Nowaker]

I also don't think that depending on some third party app is a good idea. I even use Wake Lock app! I need it because otherwise my phone sometimes doesn't ring when someone's calling me. ;) However, it's not running at the moment, so probably it wasn't running a few days ago either when the problem occurred. I don't know why it got turned off, but it's a good tip in our case too: do not depend on some third party app running because it may stop running...

[jimsmith]

Hi, I've noticed that I've been getting missed notifications but I have common sense to have zabbix email as well !

Looks like there is dialogue between pubnub and Zax 👍 👍 this is good :)

[geremyCohen]

Looking at the code, I notice you only wake the device up every 60m to check for new messages ?

https://github.com/inovex/zax/blob/master/src/com/inovex/zabbixmobile/push/PushService.java#L459

If this is the case, you will certainly lose messages, as a message on the PN subscribe queue (Server-side) has a default life of up to ~ 10m (depending on publish rate, current environmental factors of the cluster, GC cycle specifics, etc).

If you choose not to use a wake lock, and you wake the app every 60m instead, I'd suggest using history upon wakeup (aka 'storage') to pull messages. Storage volatility is in the days (user-selected via the web admin), not in minutes like subscribe.

geremy

[geremyCohen]

And if running with history incurs a charge, let me know, I'll try to get you setup on our evangelist/open-source tier: http://www.pubnub.com/free-evangelism-program/

[Nowaker]

My suggestion is not to use "storage" as it wouldn't be cool to get a missing push after, say, 50 minutes. Sure, still better than never (current state), but ideally I'd expect ZAX to be real-time (wake lock) or one minute delay (with current wake approach). If the latter, why not use the same settings as for the widget, "Refresh interval (min)"? Then anyone can specify the value according to their business needs or battery needs.

[geremyCohen]

An option of history is to give it a time interval (via the start and end
parameters) .. so a user could define nothing more than x minutes old...
http://www.pubnub.com/docs/java/android/api/reference.html#_history

On Fri, Sep 12, 2014 at 2:19 PM, Damian Nowak [email protected]
wrote:

My suggestion is not to use "storage" as it wouldn't be cool to get a
missing push after, say, 50 minutes. Sure, still better than never (current
state), but ideally I'd expect ZAX to be real-time (wake lock) or one
minute delay (with current wake approach). If the latter, why not use the
same settings as for the widget, "Refresh interval (min)"?

Reply to this email directly or view it on GitHub
#30 (comment).

[jschamburger]

@geremyCohen We do not wake the service every 60 minutes to check for new messages. We just refresh the service (restart it if it has been killed by the system). The push messages come through when the service is running via Pubnub.
Thanks for the suggestion to use history!
We'll look into it and think about a good solution for this problem.

[Nowaker]

@jschamburger What happens if the system kills the service? Does it mean 60 minutes of notifications hole in a pessimistic situation?

[jjoe64]

@jschamburger let's check for a running pubnub connection every 8 minutes instead of 60 and see how it's going.

[Nowaker]

@jschamburger @jjoe64 Why not use the same settings as for the widget, "Refresh interval (min)"? ZAX pushes are critical to me and I really want them reach my phone as soon as possible.

[jjoe64]

1. we are not speaking about a refresh interval, because we don't poll for new messages (this is what we do on the widget)
2. this option will confuse the users. If push notifications are enabled, it just should work.

[Nowaker]

If push notifications are enabled, it just should work.

I agree, it should just work! Push notifications are meant to be instant. It's not a case when OS kills the service and it's resurrected after 60 minutes, or 8 minutes. 1 minute delay to recover the connection is needed here.

You argue that battery life may be a problem for some people. I personally don't care about the battery, but about the reliability of these notifications. Therefore, I'm suggesting to make it customizable. Those who find ZAX notifications critical will just set one minute keepalive interval to make sure pushes work all the time...

[jjoe64]

in your case I just recommend to prevent the phone from sleeping via sleep lock app, then you won't have the problem with the recover of the connection

[Nowaker]

It's far from "should just work", don't you think?

I do use Wake Lock application but the problem is the OS sometimes kills it too... See #30 (comment).

[jimsmith]

I'm thrilled for this FOSS app - think you have to bare that in mind, the guys at ZAX are doing a grand job!

I'm looking forward to testing out the updated app as soon as it can become available, for me battery life is vital as I'm 'out and about - [sorry but I don't seem to be able to carry a mains power supply with me, when I'm out :)] it's a trade off battery life versus getting 'every' notification - I suggest having a fallback SMS/EMAIL via Zabbix Server until the app matures to a point where it balances this trade off.

[Nowaker]

@jimsmith Well, if I'm to receive e-mails then I don't need ZAX pushes. Getting both at the same time is very annoying and we disabled e-mails about a year ago. Pushes are just awesome (and unreliable).

As I said, I expect ZAX pushes to be reliable at all cost - but I'm OK to make it a per-user setting. I really don't know what's the problem here as users are already able to define refresh interval for the widget.

[jimsmith]

Perhaps this might help you for time being.

My zabbix server is setup and fine tuned to alert on critical thresholds to reduce the need for the mundane day to day alerts such as cpu spikes, ping ms latency as the servers I maintain are quite extensive so having notified of everything is not the right strategy.

I also have rss notifications via rss feed plus email which my devops engineer android phone displays the gmail widget showing me real time emails and i have a live wallpaper displaying the zabbix dashboard again at a glance i get to see straight away. Then i have zax and any soa service such as third party notifications ie pubnub emails sms they do go down that's the reality.

Hope this might help towards until this app has the balance or options available. I'm more than happy to test to.

[Nowaker]

@jimsmith Thanks for your input.

having notified of everything is not the right strategy.

Agreed it's not the right strategy - that's why we get only critical triggers via PUSH notifications.

Hope this might help towards until this app has the balance or options available. I'm more than happy to test to.

Yea, I'm too. I'm just looking forward to being allowed to set my own wake interval for push service healt check. ;-)

[Nowaker]

in your case I just recommend to prevent the phone from sleeping via sleep lock app, then you won't have the problem with the recover of the connection

@jjoe64 @jschamburger Wake Lock app was running all the time on my phone, yet ~50 pushes have gone missing half an hour ago.

[jjoe64]

@Nowaker thanks for that report! We will discuss what we will do....

[jjoe64]

@Nowaker the perl script has a log file. Do you see there that your missing messages were sent to the pubnub server?

[Nowaker]

@jjoe64 You probably mean some Perl script installed on Zabbix side. I passed the question to our Zabbix guru, I'm just a user.

[geremyCohen]

To best troubleshoot this, I'd enable history on the PubNub side via
admin.pubnub.com.

Also, be sure to log the server responses for all published messages (be
sure you are seeing the entire URL in the log message, Server responses,
etc). IOW, the app performing the publishes should have debug logging
enabled.

When you know you have missed a message via the android app, note the time
range(s) of the missing message(s). Also note the server response from the
client when it wakes up to get the [missed] messages.

If a message is missed:

1. At this point, we should have a 'success' audit of the publish. In the
   form of something like [1, "Sent", TT], where TT is the sent time in 10ths
   of nanoseconds-precision UNIXTIME. (returned from the REST publish service)
2. If #1 is a go, then in history (via Android SDK, or even via REST call),
   you should see this message archived in the system at the same TT offset as
   what was returned in the TT value of the response from #1
3. Given we have data for #1 and #2, noting the logged request/response of
   the client when it goes to pull the data (and its not there), there will be
   limited scenarios for the missing data, and we'll drill down on
   lowest-hanging-fruit first.

I just ask when you get this data and wish to proceed, open a new issue and
@geremyCohen so I can work with you on it.

geremy

On Sun, Sep 21, 2014 at 8:26 AM, Damian Nowak [email protected]
wrote:

@jjoe64 https://github.com/jjoe64 You probably mean some Perl script
installed on Zabbix side. I passed the question to our Zabbix guru, I'm
just a user.

Reply to this email directly or view it on GitHub
#30 (comment).

[Nowaker]

To best troubleshoot this, I'd enable history on the PubNub side via
admin.pubnub.com.

Please give me a free helicopter ride @geremyCohen and tell me where to enable it. I logged in to the admin panel but don't see anything about history (my user ID is 211357).

I just ask when you get this data and wish to proceed, open a new issue and
@geremyCohen so I can work with you on it.

I'll get the server-side log tomorrow. Will do.

[geremyCohen]

Ok, its called "Storage and Playback" in the admin console (APIs call it
history, got my wires crossed :) Set it for the minimal number of days
you'd need to keep it before you could pull it and save it (if it would
take a few days to determine you missed a message, set it for a few days --
if you know it will take within a day, set it for 48 hours, etc)

geremy

On Sun, Sep 21, 2014 at 6:15 PM, Damian Nowak [email protected]
wrote:

To best troubleshoot this, I'd enable history on the PubNub side via
admin.pubnub.com.

Please give me a free helicopter ride @geremyCohen
https://github.com/geremyCohen and tell me where to enable it. I logged
in to the admin panel but don't see anything about history (my user ID is
211357).

I just ask when you get this data and wish to proceed, open a new issue and
@geremyCohen https://github.com/geremyCohen so I can work with you on
it.

I'll get the server-side log tomorrow. Will do.

Reply to this email directly or view it on GitHub
#30 (comment).

[Nowaker]

@jjoe64 Well, this log doesn't have timestamps. :( I'll have to follow the triggers in Zabbix to gain a foothold in the log.

[NiciDieNase]

I did a long-term test (over a few weeks) with the current version and notifications seem to arrive without a problem.