WebsocketFeed stops receiving messages

[irufus]

WebsocketFeed stops receiving messages

[kyle-blair]

I was running on a laptop at least some of the time and was wondering if it sleeping or power saving could be causing an issue. But I experience the same issues with disconnections and messages randomly stopping (while the websocket remains connected) on my dedicated desktop PC as well. The watchdog thread I tried works perfectly to close and reconnect the websocket if messages aren't being received. I've had the application running on my PC since March 11th. During that time there have been 153 reconnections, 128 of which were triggered by the watchdog thread when a message hasn't been received for 60 seconds. So that's still the main cause and I'm still not sure why it happens, but it's easily handled. The remaining 25 reconnections were triggered by the websocket actually closing and the closure codes include the following: "null", "java.io.IOException: The specified network name is no longer available.", "Unexpected end of stream", and "java.io.IOException: An existing connection was forcibly closed by the remote host". That being said, I've achieved the "stable"/resilient connection I wanted through a reconnect-on-close method and the watchdog thread so I can close this issue.

[robevansuk]

Not sure about this - how long are you running for before it ends?

[robevansuk]

Think I found and fixed what may have been an implementation error on my part. It will be fixed in the next update made by me if no one else does it before then. There is a great article detailing how to implement Swing GUIs that can update in real time here: https://dzone.com/articles/multi-threading-java-swing

Long story short. I've fixed it and hopefully when it's complete we can build something far better in time to come

[kyle-blair]

Hi @robevansuk, sorry for the super-late response. It would stop fairly randomly, sometimes within a couple hours sometimes not for a couple days. It also occurred even with the GUI completely "turned off".

I want to say it was due to me adding too much processing to the same thread that is handling the websocket messages. The problem seemed to be alleviated when I moved some of the processing to other threads and increased the websocket buffers.

I've since been able to implement a reconnect feature when the socket closes so an occasional closure isn't that significant anymore. I think that code is too reliant on other code that I've added/changed otherwise I'd submit a pull request. Thanks for looking into it!

[robevansuk]

ok. The websocket buffers will probably disappear as that approach of storing messages, only to replay them if an out of sequence message comes through appears to be an incorrect approach. I'm working on another implementation now that should remove the necessity for any significant amount of buffering. Only downside is that I'll need to add timeout handling instead.

As for timing out after a couple of hours. That'll need looking at. If you're right, I suspect its down to a memory leak or a build up of too many messages?

Also a better/more responsive GUI is on the way since I've figured out how the SwingWorkers finally work. MUCH better.

[robevansuk]

So the above is all done! And should be available shortly. Theres a preview over at my own repo www.github.com/robevansuk/gdax-java which I'm just tweaking. The GUI is incredibly responsive now and utilises a multithreaded approach that has no problem keeping up. It has a few usability issues which I'll address in a later commit but for now it demonstrates the live orderbook is working.
Two outstanding order types not handled as I've not seen them out 'in the wild'/real world are change and activate order messages. Other than that though the hard part is now done and ready for use any day now.

[robevansuk]

Hmm - keep testing it overnight and come down in the morning and find its stopped. Will hold off creating a pull request since it fails.

[robevansuk]

Found a bug in my implementation - think I've fixed it with a commit in my repo. Will run overnight - hopefully no discrepancies in the morning... 🤞

[robevansuk]

Also in relation to this - I've found a way to log the cause of the dropped connection from the message that comes back from the closed connection callback operation. So I'm going to be monitoring why it happens in the next few days. There are a standard set of reasons in the message that comes back for closing the connection...

    NORMAL_CLOSURE(1000),
    GOING_AWAY(1001),
    PROTOCOL_ERROR(1002),
    CANNOT_ACCEPT(1003),
    RESERVED(1004),
    NO_STATUS_CODE(1005),
    CLOSED_ABNORMALLY(1006),
    NOT_CONSISTENT(1007),
    VIOLATED_POLICY(1008),
    TOO_BIG(1009),
    NO_EXTENSION(1010),
    UNEXPECTED_CONDITION(1011),
    SERVICE_RESTART(1012),
    TRY_AGAIN_LATER(1013),
    TLS_HANDSHAKE_FAILURE(1015);

Worst comes to worst, I'll just add some logic to reconnect on closed connection.

[robevansuk]

have run on and off for a while now. Got one of the above messages this morning:

closing websocket: CloseReason: code [1006], reason [Unexpected end of stream]

Aside from that still haven't seen an "in the wild" activate message and related effect

[kyle-blair]

The 1006 close reason is the one I was initially experiencing "Unexpected end of stream" doesn't give a lot to go on; I'm not even 100% sure if it was closed by gdax or because of the websocket in this application. I was able to work around that by just reconnecting when the socket gets closed. But now I'm experiencing a new issue I haven't been able to find a root cause for. My program stops receiving/printing messages from the websocket but the connection doesn't close or trigger a reconnect. Looking at the network tab in Windows resource monitor I can see that java's network traffic drops to very little/zero when this happens compared to ~100kB/sec when running normally. I added a "watchdog" in the form of a scheduled thread that runs periodically and checks when the last message was received. If it's been longer than a minute I'll close the session and reconnect. If that works then I've solved both of my issues with connection stability and I can try to put them in a pull request. It's frustrating though because I haven't been able to diagnose what's causing these issues, only create workarounds to handle them.

[robevansuk]

My thinking for a workaround was going more down the road of subscribing to the heartbeat channel and use that as the indicator for when to reconnect - miss more than 1 heartbeat - reconnect. I currently have a 15s timeout function so that if a sequence ID we're expecting doesn't show up within 15s, which seems more than enough time for an out of sequence message to come through (typically no more than 5s), then a reconnect should happen - unfortunately I haven't written a test for this but it seems to work if the application doesn't fail for some other reason.

[kyle-blair]

I was planning on looking into the heartbeat eventually but had to make changes quickly to run it before I left for work. But I suppose if the heartbeat channel also stops sending messages the only way to know would be to check elapsed time with a scheduled thread anyways? Or if you keep receiving heartbeat messages but no trade messages you'd still want to reconnect. Well, I think I have the info I need to make the feed bulletproof, just a matter of testing some implementations. Have you been able to get a stable feed that lasted more than a couple of days?

[robevansuk]

Ive not actually tried to keep it running for more than a 12 hrs but perhaps I'll give it a go on a spare laptop and let you know how it goes. My implementation is available on my repo for now. I have a few changes on my local machine to commit still as they're incomplete. Id hope to get something before the end of the month Im happy with but it depends on time.

[robevansuk]

I set this running last night with a mouse jiggler (www.sticksoftware.com) on my mac to prevent the screen from shutting down. Everything seems to have held up so far for >12 hours with no issues. Can you tell us whether your computer goes to sleep when you are running the websocket feed as I thought perhaps that (or something similar) may explain why the connection was being dropped.

[robevansuk]

Saying that!!!.. I did just put things my desktop app into the background whilst updating this issue and the connection was closed! :( Something about putting the application into the background seems to cause the whole thing to fail. Odd. Otherwise, assuming you're running a command line application on a dedicated machine, presumably it would be fine for some time.

[robevansuk]

So I checked with customer support on this issue as my thinking was if they did a release you would lose your connection. Their solution: use the heartbeat channel to ensure connectivity is maintained.

I'm pretty sure this is not what was desired but it seems about the only real option. The connection cannot be guaranteed. You won't always get 100% connectivity. Best you can hope for is probably 98-99%. I think this ticket can be closed now. Its not a problem with the code, its a lack of documentation on GDAX's side really.