Java api BUG: binance and coinbase download different times (timezones?)

[RobustCoder]

Java api BUG: binance and coinbase download different times (timezones?)

[RobustCoder]

I ran my market data download code several minutes ago, on 2018-07-29 at ~04:30:00 GMT.

This time, the downloads for both binance BTC/USDT as well as coinbase BTC/USD both had their final interval have the same start time of 2018-07-29T08:00:00Z.

So, they are now both consistent, which is good.

But they are both consistently wrong: both end with a time that is several hours into the future.

Did you make an incomplete bug fix, coinapi?

[Svisstack]

@RobustCoder We are handling timestamps correctly and also correctly distribute data via the API, if there is a bug it needs to be inside Java code with the timezone handling (not assuming UTC and converting your local time to the UTC which is creating offset).

This could be the case?

[RobustCoder]

Artur, Thanks for getting back to me. *I spent time debugging this just now, and now and I now know exactly what is going on.* *The essence of the BUG is that your java_rest_coin_api is using your local time zone instead of UTC because it uses that bad old java.sql.TimeStamp class.* Its precise_time_to_string method: --is used to send times to your website json API --it takes in a TimeStamp instance--which represents an absolute moment in time--but it wrongly uses that class's date field methods to construct the time String that gets sent --the problem is that TimeStamp's date field methods use your local time zone, not UTC! Its precise_time_from_string method: --is used to parse times received from your website json API --it parses the date fields from the String received --then it wrongly uses those fields to construct a TimeStamp instance --again, the problem is that TimeStamp's date field methods assume your local time zone, not UTC! --so the TimeStamp instance now has wrongly parsed the absolute time Attached is a Java class JavaApiTimezoneBug which proves all these points. At the end of this email is the console output that it produces. Note: this output is valid for me in the EDT time zone, if you are in another time zone, you will get different results. In particular, if you are in the UTC time zone, like London, you actually will be blind to the BUG! That could be why your developer missed it... *Concerning fixes, I will respond to another of your emails, stay tuned.* Regards, Brent Boyer

…

-------------------------------------------------- timeSentIsWrong -------------------------------------------------- t_coinapi: 2018-07-30T10:02:44 t_Instant: 2018-07-30T10:02:44Z you should see that this perfectly reproduces t_coinapi followed by the explicit time zone for UTC t_Timestamp: 2018-07-30 06:02:44.0 you should see that Timestamp has converted the time into your local time zone, this is dangerous! t_apiString: 2018-07-30T06:02:44 HERE IS THE BUG: java_rest_coin_api.precise_time_to_string sends that Timestamp as the local time zone String--it failed to convert it to UTC

-------------------------------------------------- timeReceivedIsWrong -------------------------------------------------- t_coinapi: 2018-07-30T10:02:44 t_Instant.toEpochMilli(): 1532944964000 t_apiTimestamp.getTime(): 1532959364000 t_apiTimestamp.getTime() - t_Instant.toEpochMilli(): 14400000 You should see that t_Instant and t_apiTimestamp represent different absolute moments in time (units here: number of millseconds since the epoch). REASON FOR THE BUG: java_rest_coin_api.precise_time_from_string parsed t_coinapi into a Timestamp with date fields that assume the local time zone String--it failed to recognize that its fields are for UTC

On Mon, Jul 30, 2018 at 6:56 AM, Artur Pietrzyk ***@***.***> wrote: @RobustCoder <https://github.com/RobustCoder> We are handling timestamps correctly and also correctly distribute data via the API, if there is a bug it needs to be inside Java code with the timezone handling (not assuming UTC and converting your local time to the UTC which is creating offset). This could be the case? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKjnxIBHm8iNXImx5djI03Nx_Qg7_CdHks5uLuZrgaJpZM4Vizks> .

[RobustCoder]

p.s. I forgot to mention: to get my attached class JavaApiTimezoneBug to run, you need to modify the access privilege of those 2 methods in java_rest_coin_api.java. In particular, remove the "private" modifier from the definition of the precise_time_to_string and precise_time_from_string methods.

…

On Mon, Jul 30, 2018 at 11:09 AM, Brent Boyer ***@***.***> wrote: Artur, Thanks for getting back to me. *I spent time debugging this just now, and now and I now know exactly what is going on.* *The essence of the BUG is that your java_rest_coin_api is using your local time zone instead of UTC because it uses that bad old java.sql.TimeStamp class.* Its precise_time_to_string method: --is used to send times to your website json API --it takes in a TimeStamp instance--which represents an absolute moment in time--but it wrongly uses that class's date field methods to construct the time String that gets sent --the problem is that TimeStamp's date field methods use your local time zone, not UTC! Its precise_time_from_string method: --is used to parse times received from your website json API --it parses the date fields from the String received --then it wrongly uses those fields to construct a TimeStamp instance --again, the problem is that TimeStamp's date field methods assume your local time zone, not UTC! --so the TimeStamp instance now has wrongly parsed the absolute time Attached is a Java class JavaApiTimezoneBug which proves all these points. At the end of this email is the console output that it produces. Note: this output is valid for me in the EDT time zone, if you are in another time zone, you will get different results. In particular, if you are in the UTC time zone, like London, you actually will be blind to the BUG! That could be why your developer missed it... *Concerning fixes, I will respond to another of your emails, stay tuned.* Regards, Brent Boyer -------------------------------------------------- timeSentIsWrong -------------------------------------------------- t_coinapi: 2018-07-30T10:02:44 t_Instant: 2018-07-30T10:02:44Z you should see that this perfectly reproduces t_coinapi followed by the explicit time zone for UTC t_Timestamp: 2018-07-30 06:02:44.0 you should see that Timestamp has converted the time into your local time zone, this is dangerous! t_apiString: 2018-07-30T06:02:44 HERE IS THE BUG: java_rest_coin_api.precise_time_to_string sends that Timestamp as the local time zone String--it failed to convert it to UTC -------------------------------------------------- timeReceivedIsWrong -------------------------------------------------- t_coinapi: 2018-07-30T10:02:44 t_Instant.toEpochMilli(): 1532944964000 t_apiTimestamp.getTime(): 1532959364000 t_apiTimestamp.getTime() - t_Instant.toEpochMilli(): 14400000 You should see that t_Instant and t_apiTimestamp represent different absolute moments in time (units here: number of millseconds since the epoch). REASON FOR THE BUG: java_rest_coin_api.precise_time_from_string parsed t_coinapi into a Timestamp with date fields that assume the local time zone String--it failed to recognize that its fields are for UTC On Mon, Jul 30, 2018 at 6:56 AM, Artur Pietrzyk ***@***.***> wrote: > @RobustCoder <https://github.com/RobustCoder> We are handling timestamps > correctly and also correctly distribute data via the API, if there is a bug > it needs to be inside Java code with the timezone handling (not assuming > UTC and converting your local time to the UTC which is creating offset). > > This could be the case? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#36 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AKjnxIBHm8iNXImx5djI03Nx_Qg7_CdHks5uLuZrgaJpZM4Vizks> > . >

[RobustCoder]

This bug was fixed in the in the recent refactoring of coinapi's Java SDK.