georgelpreput/pushcart

Metadata based ingestion and transformation library for Databricks

pushcart's Issues

Load metadata from in-memory dataframe

In its current form the Metadata object can only be initialized from an existing csv using the from_existing_csv function. It would be beneficial to add the option that it can be loaded from an in-memory DataFrame (either pandas of Spark).

Some background: I would like to implement pushcart in Synapse Analytics, which handles files differently than, for instance, Databricks. Long story short, I can only use Spark to read from files. By adding the functionality described above I would be able to use pushcart, now I cannot.