java.lang.InstantiationError: com.typesafe.scalalogging.Logger

[lightbend-labs]

java.lang.InstantiationError: com.typesafe.scalalogging.Logger

[rbellamy]

@analytically - looks like the folks over at reactive-kafka figured it out - akka/alpakka-kafka#99.

Basically, if your dependency tree includes the older scala-logging-slf4j artifact you'll see this exception.

Answer: exclude("com.typesafe.scala-logging", "scala-logging-slf4j_2.11")

[timo-aho]

I'll leave my solution for Databricks logging problem for further reference.

tl;dr:
Databricks uses old com.typesafe.scalalogging version 2.1.2. For newer ones, this is repaired.

Solution
Scalalogging uses org.slf4j.Logger in the background. Let's do our own Logger trait based on the latter:

package yourPackage

import org.slf4j.{Logger, LoggerFactory}

// Serializable logger
trait MyLogging {
  @transient
  protected lazy val logger: Logger =
    LoggerFactory.getLogger(getClass.getName)
}

Then let's use this trait instead of com.typesafe.scalalogging.LazyLogging.

trait YourSparkApp extends yourPackage.MyLogging
{
  def main(argList: Array[String]): Unit = {
      implicit val logger: Logger = this.logger // For implicit usage.
      logger.info("This should work.")
      ...
    }
}

Longer explanation
The currently newest Databricks 5.5 (Scala 2.11) uses old version of com.typesafe.scalalogging 2.1.2 as shown in
https://docs.databricks.com/release-notes/runtime/5.5.html#installed-java-and-scala-libraries-scala-2-11-cluster-version
To my knowledge, it is difficult to override this logger with other packages or versions in Spark. (I think there is an experimental flag for spark-submit.)

Logger should not be serialized, thus, it needs to be transient. However, 2.1.2 does not do this as shown here:
2.1.2: https://github.com/lightbend/scala-logging/blob/v2.1.2/scala-logging-slf4j/src/main/scala/com/typesafe/scalalogging/slf4j/Logging.scala#L28
As explained in this issue, in the newer versions, this is repaired. For instance 3.9.0: https://github.com/lightbend/scala-logging/blob/v3.9.0/src/main/scala/com/typesafe/scalalogging/Logging.scala#L27
Nevertheless, this currently does not help with Databricks because of the old library version used.

Hence, we need to create our own Logging trait based on org.sljf4j and inherit this (instead of com.typesafe.scalalogging.LazyLogging).
We do not even need to change dependencies in Maven or SBT because org.slf4j is already included in the scalalogging library.

[analytically]

Do you see the same using StrictLogging?

[skate056]

Yes, I see that same error intermittently using the StrictLogging interface.

java.lang.InstantiationError: com.typesafe.scalalogging.Logger
	at com.typesafe.scalalogging.Logger$.apply(Logger.scala:32)
	at com.typesafe.scalalogging.StrictLogging$class.$init$(Logging.scala:38)
	at com.hivehome.kafka.connect.sqs.SQSSourceTask.<init>(SQSSourceTask.scala:36)

I am also using 3.5.0

[rbellamy]

@analytically why was this issue closed? I don't see a workaround or a commit addressing this problem (which I am now seeing too).

[analytically]

I'm unable to reproduce it.