Handle additional dependencies

[flinkml]

Handle additional dependencies

[francescofrontera]

Hi @Harish-Sridhar, could you show me your build.sbt or the dependencies file you're using?

Anyway, I tried to reproduce the error on my local machine, with this build.sbt:

ThisBuild / resolvers ++= Seq(
    "Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
    Resolver.mavenLocal
)

name := "flink-with-jpmml"

version := "0.1-SNAPSHOT"

organization := "io.ffrontera"

ThisBuild / scalaVersion := "2.11.12"

val flinkVersion = "1.6.2"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
  "io.radicalbit" %% "flink-jpmml-scala" % "0.6.3"
)

lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )

assembly / mainClass := Some("io.ffrontera.Job")

// make run command include the provided dependencies
Compile / run  := Defaults.runTask(Compile / fullClasspath,
                                   Compile / run / mainClass,
                                   Compile / run / runner
                                  ).evaluated

// stays inside the sbt console when we press "ctrl-c" while a Flink programme executes with "run" or "runMain"
Compile / run / fork := true
Global / cancelable := true

// exclude Scala library from assembly
assembly / assemblyOption  := (assembly / assemblyOption).value.copy(includeScala = false)

The error same not appear, when build the project.

Regards,
Francesco.

[spi-x-i]

@Harish-Sridhar any update on this?
Were you able to solve this?

[Harish-Sridhar]

Hi,

When I use the below SBT configuration, the build was successful, but when I run the program I was getting error about JPMML modules not existing in classpath.

name := "flink-jpmml-test"

version := "0.1"

scalaVersion := "2.11.11"

resolvers ++= Seq(
  "Radicalbit Releases" at "https://tools.radicalbit.io/artifactory/public-release/"
)

libraryDependencies ++= Seq("io.radicalbit" %% "flink-jpmml-scala" % "0.6.3",
  "org.apache.flink" %% "flink-streaming-scala" % "1.7.2",
  "org.apache.flink" %% "flink-scala" % "1.7.2"
)

But when I added the necessary dependencies it all went fine.
Following build configuration worked.

name := "flink-jpmml-test"

version := "0.1"

scalaVersion := "2.11.11"

resolvers ++= Seq(
  "Radicalbit Releases" at "https://tools.radicalbit.io/artifactory/public-release/"
)

libraryDependencies ++= Seq("io.radicalbit" %% "flink-jpmml-scala" % "0.6.3",
  "org.apache.flink" %% "flink-streaming-scala" % "1.7.2",
  "org.apache.flink" %% "flink-scala" % "1.7.2",
  "org.jpmml" % "pmml-evaluator" % "1.3.9",
// I needed to provided the below dependency because I was using Java10.
 "org.glassfish.jaxb" % "jaxb-runtime" % "2.3.2" 
)

[francescofrontera]

HI @Harish-Sridhar, I tried to run your github project without the listed dependencies:

name := "flink-jpmml-test"

version := "0.1"

scalaVersion := "2.11.11"

resolvers ++= Seq(
  "Radicalbit Releases" at "https://tools.radicalbit.io/artifactory/public-release/",
  //"Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
)

libraryDependencies ++= Seq("io.radicalbit" %% "flink-jpmml-scala" % "0.6.3",
  "org.apache.flink" %% "flink-streaming-scala" % "1.7.2",
  "org.apache.flink" %% "flink-scala" % "1.7.2",
  /*"org.jpmml" % "pmml-evaluator" % "1.3.9",*/
  /*"org.glassfish.jaxb" % "jaxb-runtime" % "2.3.2",*/
  "org.apache.kafka" % "kafka-clients" % "2.1.0",
  "com.storm-enroute" %% "scalameter" % "0.17"
  //, "org.slf4j" % "slf4j-simple" % "1.7.9"
)

mainClass in assembly := Some("org.hs.flink.jpmml.test.AppsDTree")

testFrameworks += new TestFramework(
  "org.scalameter.ScalaMeterFramework")

logBuffered := false

parallelExecution in Test := false

The run it's done correctly in my local machine. I recommend you to run sbt clean and reload before run the project.

Regards,
Francesco.