Repository for java.* annotation-overlay files, hosted/maintained by us

[jspecify]

Repository for java.* annotation-overlay files, hosted/maintained by us

[msridhar]

Wasn't sure whether to comment on #98 or here, but I'll go ahead here for now. With NullAway, we're at the point where it would be pretty useful to be able to read in and use the nullability annotations present at https://github.com/jspecify/jdk and/or related sources. NullAway currently has support for its own homegrown astubx format, which we would have to extend in various ways if we want to support annotations on generic type arguments, upper bounds, etc. I'm wondering what the current thinking issue is on this topic and whether we still have hopes of doing some kind of shared repository / format.

@cpovirk I am curious how https://github.com/jspecify/jdk is used right now? I seem to recall an approach where code is built against the bytecodes of this annotated JDK after it is compiled; is that what is being used? Does this require some special build system integration?

Any other thoughts welcome :-) /cc @lazaroclapp @artempyanykh @kevinb9n

[cpovirk]

Yes, in broad strokes, we compile against an annotated JDK+android.jar based on https://github.com/jspecify/jdk. In principle, once you point your build system at such a JDK/android.jar, you don't need further integration. But I'm simplifying, and I may additionally be forgetting things :)

To get into more detail...

We don't actually "build" https://github.com/jspecify/jdk: Instead, we use https://github.com/google/turbine (the "header compiler" used by Bazel) to build a jar whose .class files contain the JDK's APIs only, no implementations. (That's enough, since all we care about is the annotations on the APIs.) Additionally, we wrote a tool that can take a normal .class file from any JDK (or android.jar) and spit out a copy with the annotations from the corresponding .class file from our API jar added in.

When the input "JDK" is a plain jar that serves as the bootclasspath (like android.jar), the actual rewriting process is straightforward: We can iterate over all the .class files in the jar, rewriting each. (And we do! Our Android builds, including Kotlin, have been seeing the jspecify/jdk annotations for a while now.) However, when the input is the modular image used by JDK releases from release 9 onward, the rewriting process is more complex. Our current approach is to use a jlink plugin (using a different approach than this one) to perform essentially the same "iterate and rewrite" task. We're actually still finishing up that work, so something could still go wrong, but it's been looking good. [update: We did successfully complete this.]

The JDK 9+ module system introduces another complication: When the annotations are used in the JDK classes, Java really wants the annotations themselves to be accessible from the JDK itself. So we're actually making the JSpecify annotation classes themselves available much as if they were JDK classes. (We do this for the non-modular case, too: Even then, it's important for the annotation classes to at least be present on the classpath/bootclasspath, and it makes sense to include them as JDK APIs for consistency with the modular case.) We put them in java.base, and we export them. (Our attempts to not export them did not go well :))

Note that, in all this, we're modifying only what the JDK APIs look like at compile time. The APIs seen at runtime (notably, through reflection) remain unannotated. This would of course be the case for the Android runtime whether we wanted it to or not, but it's also what we're setting up for our server-side Java code.

A result of that is that the runtime JDK is "missing" not only the usages of the JSpecify annotations in classes like java.util.List but also the declarations of the JSpecify annotations. That means that, for every app we build, we need to make sure that we still include the JSpecify library in the app itself if any code in the app uses it. That's the sort of thing that normally happens automatically because you need to declare a dependency on the library in order to use it. But that's not the case for us anymore: Because the JSpecify annotations appear alongside the usual compile-time JDK APIs, nothing forces users to declare a dependency on them in order to use them. So this is one place that we actually are looking into build-system integration: We want our build system to require users to declare a dependency on our JSpecify build target, even though the compiler would otherwise happily let them use the JSpecify annotations, only for the annotations to possibly be missing at runtime. (Another possible solution would be to just add the JSpecify dependency to all our apps automatically—or to just assume that we have enough common libraries that depend on it, so every app will always depend on it in practice.)

And there's actually one other bit of build-system integration: We're doing the JDK/android.jar rewriting as part of our build process (and relying on caching to make it essentially free), rather than pre-building the JDK and android.jar. As I said above, I'd expect for a pre-built copy to work. (And we may even end up with one if our caching isn't fully effective.) I think that part of the reason we did what we did is so that we could iterate quickly on fixes to the annotated JDK, but we have probably made most of the fixes that we need by this point.

Anyway, all this has ended up working pretty well as far as we can tell. We had to try a number of different approaches along the way, and we had to change a bunch of our Kotlin code to avoid errors (since we promote JSpecify diagnostics to errors, as might be becoming the default in 2.0—TBD, speak up somewhere if you have opinions :)), so we put together automation for that. We did also hit a couple Kotlin bugs, one of which we've reported so far and both of which we found reasonable workarounds for.

(@eamonnmcmanus , who has done the lion's share of this work, for any corrections or additions)

[msridhar]

Thanks so much, @cpovirk that's really helpful! We'll have to think about how we could adopt such an approach for NullAway. At the very least, though, we will almost certainly use jspecify/jdk as our "ground truth" as well, so that will be shared.

[eaftan]

I'm not inclined to provide annotations for the JDK as part of this project. I feel we should focus on specifying/providing the annotations and driving adoption but not actually annotating code ourselves.

There could be a different project that provides annotated stubs for common libraries.

[amaembo]

IntelliJ has its own format for external annotations and has JDK preannotated:
https://github.com/JetBrains/intellij-community/tree/master/java/jdkAnnotations
Not only nullability, but sideeffects, integral ranges, contracts, magic constants, etc. are preannotated there. If we decide to provide JDK annotation stubs, we may use these IntelliJ annotations as additional data source.

[kevinb9n]

I think our most important success metric is our adoption (whether by direct annotation or overlay file) by "key libraries". The JDK is the most key of all, so I think it's important to us that this happens. If it's important to our success, we might benefit from accepting responsibility for it (after all, success metrics that are outside your sphere of control are sadmaking).

Another point is that it would just basically be very sad if multiple repositories of this information sprang up, because there should be only one correct answer for which annotations apply to a particular API.

[kevinb9n]

(I continue to feel we couldn't reasonably avoid responsibility for this if we want to be successful.)