QA/ Data loading in sparkling water

[RajatSablok]

QA/ Data loading in sparkling water

[mn-mikke]

Hi @RajatSablok,
When you convert Spark DataFrame to H2OFrame, the whole dataset is kept in H2O memory storage. Data is compressed column by column and distributed across H2O nodes. (Spark executors when using SW internal backend).

[RajatSablok]

Thanks for the reply @mn-mikke, is it possible to store the data in secondary memory (hard disk) instead of it being stored in RAM?

[mn-mikke]

Nope.

[RajatSablok]

Like in spark we have the option to change the storage location, do we have that sort of support in sparkling water? @mn-mikke

The reason I am still confused is, suppose we have a 1 TB file that needs to be processed, will we have to create a 4TB instance for that? This is my understad based on the information given here.

[mn-mikke]

yes, if you need all the columns of the dataset for training a model.,you will have to create a cluster of 4TB of memory (80x50GB for example). Usually, the dataset contains extra columns that are unimportant for model training. You can use Spark to reduce number of columns and thus memory requirements. or you can train algorithm on a random sample of data.

Note: H2O memory engine is designed to be super fast for machine algorithms. Everything is kept in memory since ML algorithms iterate over the dataset many times. Storing data to disk would harm training speed significantly. Spark engine comes from a different world of big data processing where dataset is walked through only once or very few times.

[RajatSablok]

Got it, thanks for the help. Appreciate it

[RajatSablok]

Hi @mn-mikke, I had a follow-up question based on this:

you can train algorithm on a random sample of data

Can we train the model in batches of data?
Example: If we have a 1 TB dataset, can we load and train the model 10 times, 100 GB at a time?