Comments (16)
I also tested with loading data with timezone=UTC set and save the data as parquet format. Then load the parquet file without setting timezone. The date didn't change back. Hope this would help those who worry about setting timezone=UTC may mess up timestamp in other data sources.
from spark-sas7bdat.
if in the same timezone, get correct answer without more configure.
if different timezone, need setup timezone of spark to get the correct answer.
from spark-sas7bdat.
I think this should be added to the README because it's very easy to get tripped up by this and not realize.
from spark-sas7bdat.
There is a very active timestamp incompatibilities discussion on Spark dev list.
Here's a good summary, details and suggestions by the Spark development community
https://goo.gl/VV88c5
In short, the following is being proposed:
- The TIMESTAMP WITHOUT TIME ZONE type should have LocalDateTime
semantics. - The TIMESTAMP WITH LOCAL TIME ZONE type should have Instant semantics.
- The TIMESTAMP WITH TIME ZONE type should have OffsetDateTime semantics.
This proposal is in accordance with the SQL standard and many major DB
engines.
If stars will be aligned properly, this will be part of Spark 3.0 release..
from spark-sas7bdat.
Putting this on hold while I test more
from spark-sas7bdat.
@nelson2005 I just noticed this issue too.
xy.sas7bdat.zip
right date is 2018-03-03
but got 2018-03-02
from spark-sas7bdat.
Okay, this does appear to be really happening as described in the top post. To get the correct dates I've added a spark UDF like
spark.udf.register(Tags.date2timestamp_p1day, (date: java.sql.Date) => Try(java.sql.Timestamp.valueOf(date.toLocalDate.atStartOfDay().plusDays(1))).toOption
from spark-sas7bdat.
after double check, I am afraid it is not a bug, but because of the timezone issue of you server.
from spark-sas7bdat.
@witwall can you elaborate? Both the SAS server where the sas7bdat was created and the spark server are in the same timezone
from spark-sas7bdat.
@nelson2005 With hint from witwall, I've figured out the following working configs:
- For spark shell:
spark-shell --master=local[*] --driver-java-options="-Duser.timezone=UTC"
I believe spark-submit would work in this way too. (didn't test) - For inside a program like pyspark:
.set('spark.driver.extraJavaOptions','-Duser.timezone=UTC').set('spark.executor.extraJavaOptions','-Duser.timezone=UTC')
Hope this can help you.
from spark-sas7bdat.
@PhoenixDai thanks, that seems to work.
from spark-sas7bdat.
It is easy to get tripped up on this (and might be worth a mention README), but I think it's fundamentally a Spark ecosystem issue, nothing specific to this project. People (including me) get tripped up all the time with this, for example here because Hive applies the server timezone to timestamps.
from spark-sas7bdat.
Interesting. I'm a relative newcomer to Spark, so this is the first time I'd gotten tripped up by it.
from spark-sas7bdat.
Hi,
Can someone please help me, Im facing a similar issue. I tried setting the timezone to UTC but it is not working in spark yarn mode. It is only working in local mode.
Can someone please provide a snippet for how to get the correct date by setting timezone in pyspark yarn mode.
Thanks
Subhankar
from spark-sas7bdat.
Do these options work for you?
--driver-java-options="-Droot.logger=ERROR,console -Duser.timezone=UTC" --conf "spark.yarn.extraJavaOptions=-Duser.timezone=UTC" --conf "spark.executor.extraJavaOptions=-Duser.timezone=UTC"
from spark-sas7bdat.
I am having similar issue for date conversion where date is off by 2 days for the date year > 4000. I think that issue is tackled in parso latest version 2.14 but the Saurfang latest version is using parso 2.11. Does anyone know how to make parso 2.14 version work with saurfang 3.0 version or 2.10 version?
from spark-sas7bdat.
Related Issues (20)
- Spark 3 / Scala 2.12 compatibility HOT 24
- spark-sas7bdat wins WiseWithData ELEVATE Award HOT 1
- undoumented changes in NEWS.md HOT 3
- Issue : The tab space after the string is removed after reading in by the jar HOT 3
- Change in Parso Breaks spark-sas7bdat HOT 6
- Bintray Returning 403 Forbidden Request for Entire Spark Packages HOT 4
- UNRESOLVED DEPENDENCIES HOT 1
- Unable to read properly grouped columns with Delimiter
- java.lang.BootstrapMethodError
- All numeric columns getting converted to Double HOT 2
- Unable to read sas7bdat file in pyspark HOT 2
- Unable to read sas file without using Mounts
- Can we read SAS files with .spds9 file extension HOT 1
- Can't load directory with sas7bdat files HOT 2
- Parso Version Update
- InvalidInputException
- Don't find the latest JAR as per latest upgrade for parse 2.0.14 HOT 4
- Wrong number of arguments
- IllegalArgumentException: Method SASFilestream not found
- unable to read file - java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: com/epam/parso/impl/SasFileReaderImpl HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-sas7bdat.