julianpeeters / case-class-generator Goto Github PK
View Code? Open in Web Editor NEWDynamically defines and loads Scala classes at runtime. Useful for turning JSON schemas into Scala case classes on the fly.
License: Other
Dynamically defines and loads Scala classes at runtime. Useful for turning JSON schemas into Scala case classes on the fly.
License: Other
Hi, @julianpeeters
I used this project to generate some case class according to the csv content loaded by spark DataFrame. And I'd like to convert the dataframe to Dataset, which required an Encoder(since the Encoder support basic type and case class ).
However, it always threw some exception like following? Do you have any comments or thoughts about this? Appreciated very much!
java.lang.UnsupportedOperationException: Cannot infer type for class models.Station because it is not bean-compliant
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$serializerFor(JavaTypeInference.scala:416)
at org.apache.spark.sql.catalyst.JavaTypeInference$.serializerFor(JavaTypeInference.scala:327)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:82)
at org.apache.spark.sql.Encoders$.bean(Encoders.scala:141)
Attach my code sample as below:
_val hydrologyStationDF = spark.sqlContext.read.format("csv").option("header", true).csv(filePath)
val valueMembers : List[FieldData] = List(FieldData("STCD", typeOf[String]), FieldData("STNM", typeOf[String]),
FieldData("LGTD", typeOf[Double]), FieldData("LTTD", typeOf[Double]))
val classData = new ClassData(ClassNamespace(Some("models")), ClassName("Station"), ClassFieldData(valueMembers))
val dcc = new DynamicCaseClass(classData)
val record1 = dcc.runtimeInstance
type MyRecord = record1.type
import spark.implicits._
val points2 = hydrologyStationPropertyDF.as(Encoders.bean(record1.getClass()))_
{"result":[[{"last24hrs":{"fiat":9050,"vol":500000,"difffiat":"-182835","diffvol":"-10000000"}}],[{"max24Hr":1810000,"diff":"-20000"}],[{"min24Hr":1810000,"diff":"70000"}],[{"lasttrade":[{"ask":[[{"rate":1810000,"volume":500000,"time":1437856586316}]]}]}],{"highestBid":1810000},{"lowestAsk":1850000},{"allaskCoinSum":319800000,"allbidFiatSum":6229412}]}
Please help to figureout
After upgrading build.properties file from sbt 0.12.2 to 0.13.2, there is an error. sbt 0.13.7-M3 fixes the issue in sbt run, but now each dynamic class is loaded twice in sbt test.
This issues exists for devs, but not for users (there is no issue when this project is used as a dependency).
Hi,
I would like to generate case classes and then Map it to Bean. But The fields definition is stored in a database. So the Field names and types as strings:
So I have, for instance, {"name": "String", "age": "Int"}
How can I use your case class generator to create a case class and the map it to bean.
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.