sbt/sbt-avro

sbt plugin for compiling Avro schemas, similar to sbt-protobuf

sbt-avro

Overview

sbt-avro is a sbt plugin for generating Java sources for Avro schemas. It also supports referencing schemas from different files.

Usage

Installing the plugin

Add the plugin according to the sbt documentation.

For instance, add the following lines to project/plugins.sbt:

addSbtPlugin("com.github.sbt" % "sbt-avro" % "3.4.3")

// Java sources compiled with one version of Avro might be incompatible with a
// different version of the Avro library. Therefore we specify the compiler
// version here explicitly.
libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.11.3"

Add the library dependency to build.sbt:

libraryDependencies += "org.apache.avro" % "avro" % avroCompilerVersion

Settings

Name	Default	Description
avroSource	sourceDirectory / avro	Source directory with *.avsc, *.avdl and *.avpr files.
avroSpecificRecords	Seq.empty	List of avro generated classes to recompile with current avro version and settings.
avroSchemaParserBuilder	DefaultSchemaParserBuilder.default()	.avsc schema parser builder
avroUnpackDependencies / includeFilter	All avro specifications	Avro specification files from dependencies to unpack
avroUnpackDependencies / excludeFilter	Hidden files	Avro specification files from dependencies to exclude from unpacking
avroUnpackDependencies / target	sourceManaged / avro / $config	Target directory for schemas packaged in the dependencies
avroGenerate / target	sourceManaged / compiled_avro / $config	Source directory for generated .java files.
avroDependencyIncludeFilter	source typed avro classifier artifacts	Dependencies containing avro schema to be unpacked for generation
avroIncludes	Seq()	Paths with extra *.avsc files to be included in compilation.
packageAvro / artifactClassifier	Some("avro")	Classifier for avro artifact
packageAvro / publishArtifact	false	Enable / Disable avro artifact publishing
avroStringType	CharSequence	Type for representing strings. Possible values: CharSequence, String, Utf8.
avroUseNamespace	false	Validate that directory layout reflects namespaces, i.e. com/myorg/MyRecord.avsc.
avroFieldVisibility	public	Field Visibility for the properties. Possible values: private, public.
avroEnableDecimalLogicalType	true	Use java.math.BigDecimal instead of java.nio.ByteBuffer for logical type decimal.
avroOptionalGetters	false (requires avro 1.10+)	Generate getters that return Optional for nullable fields.

Tasks

Name	Description
avroUnpackDependencies	Unpack avro schemas from dependencies. This task is automatically executed before avroGenerate.
avroGenerate	Generate Java sources for Avro schemas. This task is automatically executed before compile.
packageAvro	Produces an avro artifact, such as a jar containing avro schemas.

Examples

For example, to change the Java type of the string fields, add the following lines to build.sbt:

avroStringType := "String"

If you depend on an artifact with previously generated avro java classes with string fields as CharSequence, you can recompile them with String by also adding the following

Compile / avroSpecificRecords += classOf[com.example.MyAvroRecord] // lib must be declared in project/plugins.sbt

Packaging Avro files

Avro sources (*.avsc, *.avdl and *.avpr files) can be packaged in a separate jar with the source type and avro classifier by running packageAvro.

By default, sbt-avro does not publish this. You can enable it with

Compile / packageAvro / publishArtifact := true

Declaring dependencies

You can specify a dependency on an avro source artifact that contains the schemas like so:

libraryDependencies += "org" % "name" % "rev" classifier "avro"

If some avro schemas are not packaged in a source/avro artifact, you can update the avroDependencyIncludeFilter setting to instruct the plugin to look for schemas in the desired dependency:

libraryDependencies += "org" % "name" % "rev" // module containing avro schemas
avroDependencyIncludeFilter := avroDependencyIncludeFilter.value || moduleFilter(organization = "org", name = "name")

License

This program is distributed under the BSD license. See the file LICENSE for more details.

Credits

sbt-avro is maintained by the sbt Community. The initial code was based on a similar plugin: sbt-protobuf. Feel free to file issues or pull requests.

Contributors

• Ashwanth Kumar
• Ben McCann
• Brennan Saeta
• Daniel Lundin
• Juan Manuel Caicedo
• Julian Peeters
• Jérôme Wacongne
• Kellen Dye
• Martin Achenbach
• Michel Davit
• Mārtiņš Kalvāns
• Neville Li
• Oskar Jung
• Przemysław Dubaniewicz
• Ryan Koval
• Saket Kumar
• Vince Tse

sbt-avro's Issues

Setting packageAvro / publishArtifact is not respected

Plugin setting packageAvro / publishArtifact is not respected. the avro artifact is published in any case.

This is due to addArtifact that includes the artifact without any checks.

Generate classes from avro schema located in artifact or module

Hello!

I am trying to share an avro schema in the form of an artifact.

I have created a module that will be published containing the schema and another module that depends on the previous one and generates the classes from the schema within it.

However, I can not manage to configure the second module to generate the classes.
It feels like the avsc file is not detected, and I can not manage to configure the compiler to look inside the imported jar.

Now, I am not sure if this is a limitation in the tool or if I am missing something 😅 .

This is my setup:

lazy val `root` = (project in file("."))
  .settings(publish / skip := true)
  .aggregate(
    `avro-schema`,
    `app`
  )

lazy val `avro-schema` = (project in file("avro-schema"))
  .settings(
    organization := "com.app.avro",
    version := "0.1-SNAPSHOT"
  )

lazy val `app` = (project in file("app"))
  .settings(publish / skip := true)
  .dependsOn(`avro-schema`)
  // This does not work, not generating models:
  .settings(
    libraryDependencies ++= avroCompile,
    Compile / avroSource := (Compile / resourceDirectory).value / "avro" / "v1" / "model.avsc",
    Compile / avroStringType := "String"
  )

val avroCompile = Seq(
  "org.apache.avro" % "avro" % "1.11.2"
)

Is this achievable?
Thank you!

---

This is a minimal project that shows the issue:
avro-schema-module-test.zip

---

I also tried publishing the artifact and adding it like any other dependency without luck.

Intellij doesn't work with the plugin

Hello, thank you for the lib!

Using latest version of sbt-avo and avro-compile:

1. If I compile clean (no targets) project in Intellij it complains that Avro-generated classes are not found
2. If I first run avroGenerate then compile in Intellij - it complains that Avro-generated classes are defined 2 times

Any ideas how to solve that?
Thank you!

How does `excludeFilter` work?

We have been trying and failing to exclude a schema from sbt-avro. Currently we have this in our build.sbt:

(Compile / avroSource) := (Compile / sourceDirectory).value / "resources",
(Compile / avroUnpackDependencies / excludeFilter) := HiddenFileFilter || "target_schema.avsc",

the avroSource folder contains two schemas, source_schema.avsc and target_schema.avsc. They clash, so we only want to generate the source one.

Scala classes not being generated, although pointing to Avro schema with setting

I'm trying to use this repo sbt-avro in a Scala project to generate a Scala class from an Avro schema.

Here is the project structure:

multi-tenant/
    build.sbt
    project/
        plugins.sbt
    src/
        main/
            resources/
                avro/
                    Measurement.avsc
            scala/
                com.mycom.multitenant/
                    Producer

And here is the build.sbt (note that I try to point the sbt-avro plugin to the location of my Avro source files using the avroSource setting):

version := "1.0"

lazy val root = (project in file(".")).
  settings(
    inThisBuild(List(
      organization := "com.mycom",
      scalaVersion := "2.12.15"
    )),
    name := "multi-tenant",
    avroSource := new File("src/main/resources/avro"),
  )

libraryDependencies += "org.apache.avro" % "avro" % "1.11.0"

the plugins.sbt:

addSbtPlugin("com.github.sbt" % "sbt-avro" % "3.4.0")

// Java sources compiled with one version of Avro might be incompatible with a
// different version of the Avro library. Therefore we specify the compiler
// version here explicitly.
libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.11.0"

When building in IntelliJ (top-right green hammer button), no sources get generated. I have also tried writing compile in the sbt console with the same result.

Here are some logs which appear when I first start up the sbt console in IntelliJ. It may contain some clues:

[info] welcome to sbt 1.5.8 (Oracle Corporation Java 11.0.11)
[info] loading global plugins from /home/sahand/.sbt/1.0/plugins
[info] loading settings for project multi-tenant-build from plugins.sbt,idea.sbt ...
[info] loading project definition from /home/sahand/multi-tenant/project
[warn] Unrecognized repository Scala Plugin Bundled Repository, ignoring it
[info] loading settings for project root from build.sbt ...
[info] set current project to multi-tenant (in build file:/home/sahand/multi-tenant/)
[warn] there's a key that's not used by any other settings/tasks:
[warn]  
[warn] * root / avroSource
[warn]   +- /home/sahand/multi-tenant/build.sbt:10
[warn]  
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[info] Defining Global / ideaPort
[info] The new value will be used by Compile / compile, Test / compile
[info] Reapplying settings...
[info] set current project to multi-tenant (in build file:/home/sahand/multi-tenant/)

Seems like the plugin cannot pick up the avroSource setting. Is this a bug, or am I using the plugin wrong?

Support for older avro versions (i.e. < 1.8.2)

Are Avro versions older than 1.8.2 supposed to be supported?

I ask because it seems that compiler.setEnableDecimalLogicalType(enableDecimalLogicalType) is being set without checking the Avro compiler version and this seems to have been introduced in 1.8.2.

Example:
project sets:

addSbtPlugin("com.github.sbt" % "sbt-avro" % "3.4.2")

libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.8.0"

on sbt compile you can see the error: (Compile / avroGenerate) java.lang.NoSuchMethodError: org.apache.avro.compiler.specific.SpecificCompiler.setEnableDecimalLogicalType(Z)V

cannot find symbol when compiling

Hi, When compiling with auto-generated java source, the following error is generated:

sbt:Data Analytics> compile
[info] Compiling 7 Scala sources and 37 Java sources to /home/naitree/Desktop/projectname/target/scala-2.12/classes ...
[error] /home/naitree/Desktop/projectname/target/scala-2.12/src_managed/main/compiled_avro/com/company/projectname/schemaname/Detection.java:113:1: cannot find symbol
[error]   symbol:   class TimeConversion
[error]   location: class org.apache.avro.data.TimeConversions
[error]   protected static final org.apache.avro.data.TimeConversions.TimeConversion TIME_CONVERSION = new org.apache.avro.data.TimeConversions.TimeConversion();
[error] /home/naitree/Desktop/projectname/target/scala-2.12/src_managed/main/compiled_avro/com/company/projectname/schemaname/Detection.java:114:1: cannot find symbol
...

Looks like classpath is not set correctly. How to fix this?

My buld.sbt:

// ------- sbt-avro configurations --------
AvroConfig / useNamespace := true
AvroConfig / version := "1.8.2"

// ------- resolvers -------
resolvers += "confluent" at "http://packages.confluent.io/maven/"

// ------- root project configurations --------
lazy val root = (project in file("."))
    .settings(
        name := "Data Analytics",
        version := "0.1.0",
        libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "3.2.1" % "provided",
        libraryDependencies += "org.apache.hadoop" % "hadoop-aliyun" % "3.2.1" % "provided",
        libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.4" % "provided",
        libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "2.4.4" % "provided",
        libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.12" % "2.4.4",
        libraryDependencies += "org.apache.kafka" % "kafka-clients" % "2.4.0",
        libraryDependencies += "log4j" % "log4j" % "1.2.17" % "provided",
        libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.16" % "provided",
        libraryDependencies += "org.slf4j" % "slf4j-log4j12" % "1.7.16" % "provided",
        libraryDependencies += "org.apache.avro" % "avro" % "1.8.2" % "provided",
        libraryDependencies += "org.apache.spark" % "spark-avro_2.12" % "2.4.4",
        libraryDependencies += "za.co.absa" % "abris_2.12" % "3.1.1",
    )

Strangely, the unresolved symbols can be found and used in my scala code. It seems that only java code can not resolve these classes.

Setting 'avroSource' apparently ignored

The following configuration ignores avroSource, but not the other settings.

lazy val model = project
  .in(file("model"))
  .enablePlugins(SbtAvro)
  .settings(
    avroSource := (baseDirectory.value.getAbsoluteFile / "my_dir"),
    avroStringType := "CharSequence",
    avroFieldVisibility := "private",
    avroOptionalGetters := true,
    name := s"${baseName}-model",
    libraryDependencies ++= Libraries.model
  )

Am I doing something wrong here? I see avroSource is initialized in a different part than the rest of settings, and probably always overriden by the default sourceDirectory.value / "avro"

How to deal with `avro` & `avro-compiler` version?

More of a discussion stemming from #16 (comment)

• avro-compiler is used to compile .avsc to .java. It's a dependency of the plugin itself and needs to be overridden in plugins.sbt with dependencyOverrides.
• avro library is used by compiled .java files and a dependency of the user project.

Is there a way to set a single version string, that's picked up by both the plugin & user project?
The most obvious one I can think of is sys.props but that seems hacky.

`avroOptionalGetters` not available

Hi,
I wanted to use the avroOptionalGetters setting in my project, but sbt complained that it could not find it.
After checking this repo I see there hasn't been a release since the avroOptionalGetters feature was added.

Can you please release 3.2 which includes this feature? Thank you!

No processor claimed any of these annotations: org.apache.avro.specific.AvroGenerated

So the .avsc file I made has succesfully generated to an Avro Class. However, on sbt compile, I'm getting this error.
[warn] No processor claimed any of these annotations: org.apache.avro.specific.AvroGenerated
[error] warnings found and -Werror specified

I can't proceed unless I suppress or fix this warning.

PGP and Sonatype secrets

Originally posted by @RustedBones in #70 (comment)

Can the PGP and Sonatype secrets be added for this repo. I'd like to make a new release under the sbt organisation.

My release attempt failed here