Calling Scala Functions from .NET for Spark

[dotnet]

Calling Scala Functions from .NET for Spark

[rapoth]

@Ragavenderan: Thanks for raising this! Can you give us a sample Scala method and how you intend to use it within Spark?

[Ragavenderan]

Currently we have a jar file that contains the function definitions.
From pyspark - this is how we call the method -

from pyspark.sql import SparkSession, DataFrame
def load_curated_email(spark, category, startdate="", enddate="", location=""):

df = spark._jvm.com.microsoft.odinml.Extractor.loadCuratedEmail(spark._jsparkSession, category, startdate, enddate, location)
return DataFrame(df, spark)

[imback82]

The proper way to do this to introduce new datasource such that you can do:

var df = spark.Read().Format("your_format_here").Option("startdate","something").Load();

Have you considered this option?

Otherwise, we don't have a plan to support calling Scala methods from .NET other than RegisterJava for UDF here.

[zhiyong-gayang]

Hi @imback82
I assume the UDF can only be used in spark sql, right?
The scenario Raga shared is to call a scala method to return a DataFrame as the python code shows, since we don't want to duplicate the logic in C#, is there any way to achieve this goal?

[rapoth]

@garyyang2002: Yes, that's correct.

I'm afraid what you want is not possible by any simple means and is a use case we cannot support immediately. While you can call into regular Java functions through SparkSQL, the use case described here is to call a wrapper function that then invokes spark.read.format().load() which will return a Dataframe. This is a bit unconventional and is not the recommended way – that brings me to the next question – can you share some details regarding the loadCuratedEmail()? How complex would it be to write this one function in .NET?

[imback82]

The workaround is to use https://github.com/aelij/IgnoresAccessChecksToGenerator to access some internal classes (but this is not recommended since internal classes can break APIs).