Tensorflow can support pyspark API...?

[tensorflow]

Tensorflow can support pyspark API...?

[joyeshmishra]

Take a look here: https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector
Example notesbooks:
https://docs.databricks.com/_static/notebooks/spark-tensorflow-connector.html
https://github.com/GoogleCloudPlatform/cloud-dataproc/tree/master/spark-tensorflow

[skavulya]

Thanks @joyeshmishra!

@srinivasugaddam Here is a short example of how to use the connector with PySpark.

Run PySpark with the spark_connector in the jars argument as shown below:

$SPARK_HOME/bin/pyspark --jars target/spark-connector_2.11-1.8.0.jar

Here is the README example translated to Python.

from pyspark.sql.types import *

path = "test-output.tfrecord"

fields = [StructField("id", IntegerType()), StructField("IntegerCol", IntegerType()), StructField("LongCol", LongType()), StructField("FloatCol", FloatType()), StructField("DoubleCol", DoubleType()), StructField("VectorCol", ArrayType(DoubleType(), True)), StructField("StringCol", StringType())]
schema = StructType(fields)

test_rows = [[11, 1, 23, 10.0, 14.0, [1.0, 2.0], "r1"], [21, 2, 24, 12.0, 15.0, [2.0, 2.0], "r2"]]
rdd = spark.sparkContext.parallelize(test_rows)
df = spark.createDataFrame(rdd, schema)
df.write.format("tfrecords").option("recordType", "Example").save(path)

df = spark.read.format("tfrecords").option("recordType", "Example").load(path)
df.show()