Save/Load H2O Models in spark

[h2oai]

Save/Load H2O Models in spark

[mmalohlava]

Hi Mikhail,

right now we can serialize and deserialize models directly via H2O - look at https://github.com/h2oai/h2o-3/blob/master/h2o-core/src/main/java/water/api/ModelsHandler.java#L181-L208

The code export/import model to/from a given location. You can use similar code from Sparkling Water.

Later i will provide nicer Scala API.

Please, let me know if it works for you.

[shahrozhaider26]

Can you please provide scala code for this?

[mmalohlava]

At least Java code for import:

List<Key> importedKeys = new ObjectTreeBinarySerializer().load(FileUtils.getURI(mimport.dir));
Model model = (Model) importedKeys.get(0).get();

Java code for export

Model model = getFromDKV("model_id", mexport.model_id.key());
List<Key> keysToExport = new LinkedList<>();
keysToExport.add(model._key);
keysToExport.addAll(model.getPublishedKeys());

new ObjectTreeBinarySerializer().save(keysToExport, FileUtils.getURI(mexport.dir));

[Tian-Su]

Hi @mmalohlava,
I tried the Scala code here:

sparkling-water/core/src/main/scala/water/app/SparklingWaterApp.scala

Lines 138 to 151 in a0ed174

	def exportH2OModel(model : Model[_,_,_], destination: URI): URI = {
	val modelKey = model._key.asInstanceOf[Key[_ <: Keyed[_ <: Keyed[_ <: AnyRef]]]]
	val keysToExport = model.getPublishedKeys()
	// Prepend model key
	keysToExport.add(0, modelKey)
	new ObjectTreeBinarySerializer().save(keysToExport, destination)
	destination
	}
	def loadH2OModel[M <: Model[_, _, _]](source: URI) : M = {
	val l = new ObjectTreeBinarySerializer().load(source)
	l.get(0).get().asInstanceOf[M]
	}

It seems both exportH2OModel and loadH2OModel work well! Thank you very much!

[jakubhava]

Hi, as @Tian-Su said, the ModelSerializationSupport trait contains methods exportH2OModel and loadH2OModel. These methods are preferred way how to export/load models.

If you need any more help, please feel free to reopen this issue or start new one.

Kuba

[BenRR]

Does ModelSerializationSupport support hdfs?

[mmalohlava]

@BenRR yes