Expando Objects

[jtgooding42]

Expando Objects

[aloneguid]

Hi. Your best bet is to use the low level API as you can build schema and data in runtime. The issue with System.Data is that it doesn't support complex types so there is no way to serialise even something simple like structs.

[jtgooding42]

Thanks, I realize that Parquet has an absurd amount of different ways to store data but from my experience, well so far on about 200 systems not one of those has ever been used. With the popularity of Snowflake, Databricks, Fabric a popular real world usage today is much more traditional types you might use than structures, embedded arrays etc.

It's just like SQL Server yeah you can create custom types, CLR types etc. but they tend to perform poorly and cause issues somewhere through some tool. I'm not saying drop that kind of support by any means but not supporting many mainstream usages because it doesn't work with one feature just means a lot of people writing the same broiler plate code.

Love the product and I have already built my version of the above code since yesterday as I assumed this would be your answer, I may even implement an IDataReader interface for the parquet reader today. The advantage of this library is .NET over java but it's not embracing the data aspect of working in .NET to implement normal programming paradigms.

I am totally fine with the serialize over data table, data reader for writing throwing an exception saying datatype not supported when it comes to those extra features as I know I won't see them in my workloads.

Thanks again for a great product, just my 2c which doesn't count as much :)

[aloneguid]

@jtgooding42 totally agree here, my use cases are also most of the times consist of flat data only. I have followed the similar approach when implementing DataFrame integration - just using primitive types. Would really appreciate you adding System.Data integration and happy to merge it into master here!

[jtgooding42]

I have some sample boiler plate code for datareader to parquet. I tested the table/row api, the new dictionary (untyped) and a home grown low level api, and the row api and the low level produce nearly same timings with low level a bit faster, the untyped was almost twice as long, I would love for ya to take a look and see if I did anything obviously wrong in each of the 3, or recommend perf improvements, I'm fine with the code being incorporated but would guess that it would require a fair bit of rework to be api friendly.
Lastly the new untyped api takes a dictionary<string, object>, not sure how to pass nulls in on that shouldn't it be dictionary<string, object?>.