sourcegraph/scip-java

SCIP Code Intelligence Protocol generator for Java

Java, Scala, and Kotlin indexer for SCIP

Documentation	Link
Landing page	https://sourcegraph.github.io/scip-java
Getting started	https://sourcegraph.github.io/scip-java/docs/getting-started.html
Manual configuration	https://sourcegraph.github.io/scip-java/docs/manual-configuration.html
Contributing	https://sourcegraph.github.io/scip-java/docs/contributing.html
Design	https://sourcegraph.github.io/scip-java/docs/design.html

scip-java's Issues

Java 9+ Modules for LSIF Monikers

Information defined in the JPMS could be used to influence the emitting of monikers. To be investigated

Unable to generate LSIF dump for ReactiveX/RxJava

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

Added the createPom task and generated the pom.xml.

Then ran:

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot RxJava -out rxjava.lsif

Exception in thread "main" java.lang.UnsupportedOperationException: PartialSourcePosition only contains a CompilationUnit
	at spoon.reflect.cu.position.NoSourcePosition.getLine(NoSourcePosition.java:38)
	at DocumentIndexer.mkRange(DocumentIndexer.java:297)
	at DocumentIndexer.access$200(DocumentIndexer.java:16)
	at DocumentIndexer$DefinitionsVisitor.visitCtParameter(DocumentIndexer.java:113)
	at spoon.support.reflect.declaration.CtParameterImpl.accept(CtParameterImpl.java:53)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtConstructor(CtScanner.java:359)
	at spoon.support.reflect.declaration.CtConstructorImpl.accept(CtConstructorImpl.java:44)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtClass(CtScanner.java:330)
	at DocumentIndexer$DefinitionsVisitor.visitCtClass(DocumentIndexer.java:142)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:56)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtNewClass(CtScanner.java:596)
	at spoon.support.reflect.code.CtNewClassImpl.accept(CtNewClassImpl.java:23)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtLocalVariable(CtScanner.java:517)
	at DocumentIndexer$DefinitionsVisitor.visitCtLocalVariable(DocumentIndexer.java:118)
	at spoon.support.reflect.code.CtLocalVariableImpl.accept(CtLocalVariableImpl.java:48)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtBlock(CtScanner.java:294)
	at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:55)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtMethod(CtScanner.java:551)
	at DocumentIndexer$DefinitionsVisitor.visitCtMethod(DocumentIndexer.java:130)
	at spoon.support.reflect.declaration.CtMethodImpl.accept(CtMethodImpl.java:56)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtClass(CtScanner.java:330)
	at DocumentIndexer$DefinitionsVisitor.visitCtClass(DocumentIndexer.java:142)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:56)
	at DocumentIndexer.visitDefinitions(DocumentIndexer.java:51)
	at ProjectIndexer.index(ProjectIndexer.java:66)
	at Main.main(Main.java:15)

Occurrences for class type parameters are not emitted

It looks like we're not emitting type parameters on classes

https://github.com/sourcegraph/lsif-java/blob/48bce73664383381c85211a781a427924abae1a2/tests/snapshots/src/main/generated/com/airbnb/epoxy/BaseEpoxyTouchCallback.java#L8

-- obtained
++ expected
  interface BaseEpoxyTouchCallback<T extends EpoxyModel> {
+                                  ^ definition com/airbnb/epoxy/BaseEpoxyTouchCallback#[T]

Add documentation website

It would be nice if we can document lsif-java with a website instead of the readme. I personally like websites over READMEs because

• it's easier to dynamically generate content like version numbers for copy-paste installation instructions
• READMEs become difficult to navigate once they grow beyond a certain size

hovers for stdlib and cross repo symbols

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>sourcegraph/renovate-config)

Unable to generate LSIF dump for apache/pulsar

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot pulsar -out pulsar.lsif

Exception in thread "main" spoon.compiler.ModelBuildingException: The type PulsarKafkaProducerTest is already defined
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.reportProblem(JDTBasedSpoonCompiler.java:575)
	at spoon.support.compiler.jdt.TreeBuilderRequestor.acceptResult(TreeBuilderRequestor.java:27)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:86)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

JRE <15 compatibility

Currently, comsunsource based lsif-java uses Java 15 preview features. For this reason, it cant be run where JRE version is <15. However, it must be able to do so, as most people probably run JRE<15.

Solution:

• Migrate remaining Java code to Kotlin, which supports compiling to Java 8 bytecode while still having all the fancy fun features (and more) from Java 15

Provide index output for Maven (pom.xml) files

Not sure if this is the right place but I was hoping we could get the output to include indices for maven (pom.xml) files. For example, when you use a property like ${proto.version} it would point back to where the property is defined in the <properties> section.

Javadoc links support

Javadoc supports linking to classes/methods etc via {@link ... }. We should also support this in the LSIF output

See: https://docs.oracle.com/en/java/javase/11/docs/specs/doc-comment-spec.html#link

Cannot run program "lsif-semanticdb", No such file or directory

I used the following command to install lsif-java on my machine.

# macOS
curl -Lo lsif-java https://github.com/sourcegraph/lsif-java/releases/download/v0.2.0/lsif-java-x86_64-apple-darwin \
  && chmod +x lsif-java \
  && ./lsif-java --help

But when I tried to generate lsif index in my repository it failed, throwing execeptions descriped below.

info: lsif-semanticdb --out=/Users/xxx/dump.lsif --semanticdbDir=/Users/xxx/target/semanticdb-targetroot
java.io.IOException: Cannot run program "lsif-semanticdb" (in directory "/Users/xxx"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at os.proc.proc$lzycompute$1(ProcessOps.scala:128)
	at os.proc.proc$1(ProcessOps.scala:122)
	at os.proc.spawn(ProcessOps.scala:135)
	at moped.cli.SpawnableProcess.spawn(Processes.scala:161)
	at moped.cli.SpawnableProcess.call(Processes.scala:89)
	at com.sourcegraph.lsif_java.IndexCommand.process(IndexCommand.scala:67)
	at com.sourcegraph.lsif_java.IndexCommand.run(IndexCommand.scala:154)
	at moped.cli.Command.$anonfun$runAsFuture$1(BaseCommand.scala:14)
	at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.scala:17)
	at scala.util.Try$.apply(Try.scala:210)
	at moped.cli.Command.runAsFuture(BaseCommand.scala:14)
	at moped.cli.Application$.$anonfun$run$7(Application.scala:317)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:434)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.io.IOException: error=2, No such file or directory
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_io_IOException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at java.lang.ProcessImpl.forkAndExec(ProcessImpl.java)
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 22 more

Improve gradle project meta inference

Currently importing as an EclipseProject model. Importing as an IdeaProject (preliminarily) appears to be better at providing dependencies locations (which are fed into the -classpath flag). More investigation needed, maybe results will be merged?

Unable to generate dump on google/gson

https://github.com/google/gson

./build/install/lsifjava/bin/lsifjava -projectRoot /Users/chrismwendt/github.com/google/gson -out /Users/chrismwendt/github.com/google/gson/dump.lsif
Exception in thread "main" spoon.SpoonException: Modules are only available since Java 9. Please set appropriate compliance level.
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.forEachCompilationUnit(JDTBasedSpoonCompiler.java:445)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildModel(JDTBasedSpoonCompiler.java:421)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:365)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

lsif-java: cross-file references

Working for:

• type identifier in variable declaration

Unable to generate LSIF dump for apache/spark

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot spark -out spark.lsif

Exception in thread "main" org.eclipse.jdt.internal.compiler.problem.AbortCompilation: Pb(700) The class file ExternalSorter<K,V,C> contains a signature 'Lorg/apache/spark/util/collection/ExternalSorter<TK;TV;TC;>.SpilledFile$;' ill-formed at position 60
	at org.eclipse.jdt.internal.compiler.problem.ProblemHandler.handle(ProblemHandler.java:162)
	at org.eclipse.jdt.internal.compiler.problem.ProblemReporter.handle(ProblemReporter.java:2528)
	at org.eclipse.jdt.internal.compiler.problem.ProblemReporter.handle(ProblemReporter.java:2591)
	at org.eclipse.jdt.internal.compiler.problem.ProblemReporter.corruptedSignature(ProblemReporter.java:1704)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.getTypeFromTypeSignature(LookupEnvironment.java:1986)
	at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.createFields(BinaryTypeBinding.java:685)
	at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.cachePartsFrom(BinaryTypeBinding.java:547)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.createBinaryTypeFrom(LookupEnvironment.java:1040)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.createBinaryTypeFrom(LookupEnvironment.java:1021)
	at org.eclipse.jdt.internal.compiler.Compiler.accept(Compiler.java:308)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:249)
	at org.eclipse.jdt.internal.compiler.lookup.UnresolvedReferenceBinding.resolve(UnresolvedReferenceBinding.java:114)
	at org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding.resolveType(BinaryTypeBinding.java:231)
	at org.eclipse.jdt.internal.compiler.lookup.PackageBinding.getTypeOrPackage(PackageBinding.java:241)
	at org.eclipse.jdt.internal.compiler.lookup.Scope.getPackage(Scope.java:2980)
	at org.eclipse.jdt.internal.compiler.ast.QualifiedTypeReference.getTypeBinding(QualifiedTypeReference.java:110)
	at org.eclipse.jdt.internal.compiler.ast.JavadocQualifiedTypeReference.internalResolveType(JavadocQualifiedTypeReference.java:49)
	at org.eclipse.jdt.internal.compiler.ast.JavadocQualifiedTypeReference.resolveType(JavadocQualifiedTypeReference.java:92)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.resolveType(TypeReference.java:620)
	at org.eclipse.jdt.internal.compiler.ast.Javadoc.resolveReference(Javadoc.java:399)
	at org.eclipse.jdt.internal.compiler.ast.Javadoc.resolve(Javadoc.java:247)
	at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.resolve(TypeDeclaration.java:1236)
	at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.resolve(TypeDeclaration.java:1354)
	at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.resolve(CompilationUnitDeclaration.java:652)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:81)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:43)
	at Main.main(Main.java:15)

hoverResult vertex with empty hover content value created for current document

Result when indexing https://github.com/sourcegraph/lsif-java/tree/e5fc08538262d27d71448a555bd4d4d927dc4f5a/examples/small results in a hoverResult vertex with empty value field for hover content for the class name Main https://github.com/sourcegraph/lsif-java/blob/e5fc08538262d27d71448a555bd4d4d927dc4f5a/examples/small/src/main/java/Main.java#L3

Buck support

NullPointerException when trying to generate LSIF file

Thank you for your efforts!
If we try to generate the index file via a gitlab CI job, a NullPointerException is thrown:

$ ~/lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot . -out dump.lsif
Running JVM 11.0.6
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/datetime/DateTime.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/datetime/DateTimeFormats.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/test/java/myproject/shared/datetime/DateTimeTest.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/datetime/DateTimeType.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/datetime/TimeZoneData.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/filter/FilterCriterion.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/filter/FilterCriterionMatcher.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/test/java/myproject/shared/filter/FilterCriterionMatcherTest.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/test/java/myproject/shared/filter/FilterCriterionTest.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/filter/RegExpHelper.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/test/java/myproject/shared/filter/RegExpHelperTest.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/form/AbstractComponent.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/test/java/myproject/shared/form/AbstractComponentTestTemplate.java
Found /home/gitlab-runner/builds/mLw9xUbW/0/m.richter/mylibrary/src/main/java/myproject/shared/form/AbstractFormElement.java
Exception in thread "main" java.lang.NullPointerException
	at ProjectIndexer.index(ProjectIndexer.java:47)
	at Main.main(Main.java:16)

The next file would have been [...]/myproject/shared/form/ApplicationSettings.java and contentwise it does not have anything peculiar or different than the other files that would justify failure... no... hold on, My IDE tells me that the class inside is not used. This is not particularly surprising though, as this is only a library exposing some common functionality, but it does not show that for the other classes that are in the log above.

The project itself (can't share it, sorry) is a fairly standard gradle project of a java-only library to which i generate the pom.xml file as you documented in the README. The project has ~20kLOC of java.

Afaik we are running the latest version of lsif-java. I'll try to reproduce it locally as well and add a comment then.

Incorrect range for interface definition

https://github.com/sourcegraph/lsif-java/blob/756563ffca0a55d49141fc2c94d75af729c30744/tests/snapshots/src/main/generated/minimized/Interfaces.java#L3-L4

-- obtained
++ expected
  public interface Interfaces {
- //     ^^^^^^^^^^ definition minimized/Interfaces#
+ //               ^^^^^^^^^^ definition minimized/Interfaces#

Unable to generate LSIF dump for INRIA/spoon

I was curious about using this for spoon. JDT issue?

Exception in thread "main" java.lang.NullPointerException
	at spoon.support.compiler.jdt.JDTCommentBuilder$1.visitCtSwitch(JDTCommentBuilder.java:349)
	at spoon.support.reflect.code.CtSwitchImpl.accept(CtSwitchImpl.java:33)
	at spoon.reflect.visitor.CtInheritanceScanner.scan(CtInheritanceScanner.java:173)
	at spoon.support.compiler.jdt.JDTCommentBuilder$1.scan(JDTCommentBuilder.java:224)
	at spoon.support.compiler.jdt.JDTCommentBuilder.insertCommentInAST(JDTCommentBuilder.java:459)
	at spoon.support.compiler.jdt.JDTCommentBuilder.buildComment(JDTCommentBuilder.java:138)
	at spoon.support.compiler.jdt.JDTCommentBuilder.build(JDTCommentBuilder.java:102)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.lambda$buildModel$2(JDTBasedSpoonCompiler.java:433)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.forEachCompilationUnit(JDTBasedSpoonCompiler.java:450)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildModel(JDTBasedSpoonCompiler.java:432)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:365)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:43)
	at Main.main(Main.java:16)

Command:

~/lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot . -out dump.lsif

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Fallback to renovate.json file as a preset is deprecated, please use a default.json file instead.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• Update dependency org.scalameta:sbt-mdoc to v2.5.3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Pin eclipse-temurin Docker tag to b4c95d2
• Update dependency com.thesamet.scalapb:compilerplugin to v0.11.17
• Update dependency bazel_skylib to v1.7.1
• Update dependency golang to v1.22.4
• Update dependency net.bytebuddy:byte-buddy to v1.14.17
• Update dependency net.bytebuddy:byte-buddy-agent to v1.14.17
• Update dependency org.scalameta:munit to v1
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency ch.epfl.scala:sbt-scalafix to v0.12.1
• Update dependency com.github.sbt:sbt-ci-release to v1.5.12
• Update dependency org.projectlombok:lombok to v1.18.32
• Update dependency bazel to v5.4.1
• Update dependency com.google.protobuf:protobuf-java to v3.25.3
• Update dependency com.google.protobuf:protobuf-java-util to v3.25.3
• Update dependency com.lihaoyi:pprint to v0.9.0
• Update dependency org.scalameta:sbt-scalafmt to v2.5.2
• Update dependency org.xerial.sbt:sbt-pack to v0.20
• Update dependency sbt/sbt to v1.10.0
• Update dependency se.marcuslonnberg:sbt-docker to v1.11.0
• Update actions/checkout action to v4
• Update actions/setup-go action to v5
• Update actions/setup-java action to v4
• Update dependency bazel to v7
• Update dependency com.google.protobuf:protobuf-java to v4
• Update dependency com.google.protobuf:protobuf-java-util to v4
• Update dependency rules_proto to v6
• Update docker/login-action action to v3
• Update docker/setup-buildx-action action to v3
• Update eclipse-temurin Docker tag to v21

Detected dependencies

asdf

.tool-versions

• golang 1.17.5

website/.tool-versions

bazel

WORKSPACE

• bazel_skylib 1.0.3
• rules_proto 4.0.0-3.20.0
• com.google.protobuf:protobuf-java 3.15.6
• com.google.protobuf:protobuf-java-util 3.15.6
• org.projectlombok:lombok 1.18.22

bazelisk

.bazelversion

• bazel 5.2.0

dockerfile

Dockerfile

• eclipse-temurin 17

github-actions

.github/workflows/ci.yml

• actions/checkout v4
• actions/setup-java v4
• actions/checkout v4
• actions/setup-java v4
• actions/checkout v2
• actions/setup-java v3
• actions/checkout v2
• actions/checkout v2
• actions/checkout v2
• actions/setup-java v3

.github/workflows/labeler.yml

• github/issue-labeler v3.4

.github/workflows/mdoc.yml

• actions/checkout v3
• actions/setup-java v3

.github/workflows/pr-auditor.yml

• actions/checkout v4
• actions/setup-go v4

.github/workflows/project-board.yml

.github/workflows/release-docker.yml

• actions/checkout v3
• actions/setup-java v3
• docker/setup-buildx-action v1
• docker/login-action v1

.github/workflows/release-maven.yml

• actions/checkout v3
• actions/setup-java v3

.github/workflows/sourcegraph.yml

• actions/checkout v3
• actions/setup-java v3

npm

website/package.json

• docusaurus 1.14.7

sbt

build.sbt

• net.bytebuddy:byte-buddy 1.11.9
• net.bytebuddy:byte-buddy-agent 1.11.21
• org.scalameta:ascii-graphs 0.1.2
• org.projectlombok:lombok 1.18.22
• org.scalameta:munit 0.7.29
• com.lihaoyi:pprint 0.6.6

project/build.properties

• sbt/sbt 1.9.9

project/plugins.sbt

• org.xerial.sbt:sbt-pack 0.14
• se.marcuslonnberg:sbt-docker 1.9.0
• com.github.sbt:sbt-ci-release 1.5.10
• com.eed3si9n:sbt-buildinfo 0.10.0
• org.scalameta:sbt-scalafmt 2.4.6
• org.scalameta:sbt-mdoc 2.5.2
• ch.epfl.scala:sbt-scalafix 0.12.0
• com.sourcegraph:sbt-sourcegraph 0.4.3
• com.eed3si9n:sbt-assembly 0.15.0
• io.spray:sbt-revolver 0.9.1
• org.scala-debugger:sbt-jdi-tools 1.1.1
• com.thesamet.scalapb:compilerplugin 0.11.11

website/project/build.properties

• sbt/sbt 1.9.7

---

• Check this box to trigger a request for Renovate to run again on this repository

Unable to generate LSIF dump for netty/netty

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot netty -out netty.lsif

Exception in thread "main" spoon.compiler.ModelBuildingException: The type package-info is already defined
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.reportProblem(JDTBasedSpoonCompiler.java:575)
	at spoon.support.compiler.jdt.TreeBuilderRequestor.acceptResult(TreeBuilderRequestor.java:27)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:86)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

NullPointer exception if class extends generic class

Platform

Linux

Java info

openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10)
OpenJDK 64-Bit Server VM (build 11.0.8+10, mixed mode)

Description

NPE occurs with stack trace

Exception in thread "main" java.lang.NullPointerException
	at lsifjava.DocumentIndexer.getLocalizedName(DocumentIndexer.java:514)
	at lsifjava.DocumentIndexer.appendTypeParam(DocumentIndexer.java:505)
	at lsifjava.DocumentIndexer.mkClassDoc(DocumentIndexer.java:462)
	at lsifjava.DocumentIndexer.access$600(DocumentIndexer.java:19)
	at lsifjava.DocumentIndexer$DefinitionsVisitor.visitCtClass(DocumentIndexer.java:205)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:58)
	at lsifjava.DocumentIndexer.visitDefinitions(DocumentIndexer.java:55)
	at lsifjava.ProjectIndexer.index(ProjectIndexer.java:125)
	at lsifjava.Main.main(Main.java:19)

when class extends other generic class, e.g. this triggers error:

package org.example;

import java.util.ArrayList;

public class TestList<T extends Integer> extends ArrayList<T> {}

Unable to generate LSIF dump for Netflix/titus-control-plane

Linked spoon issue INRIA/spoon#3594

Remove system dependency on lsif-semanticdb

Currently, lsif-java shells out to lsif-semanticdb to convert SemanticDB files into LSIF. This system dependency complicates the installation for lsif-java because lsif-semanticdb needs to be available on the $PATH.

It would be nice if we can remove this system dependency by implementing the SemanticDB->LSIF conversion in Java. Ideally, I'd like lsif-java to be distributed as a standalone native binary with batteries included. We already embed the compiler plugin and the java agent jars as resources. WDYT @efritz @Strum355?

Gradle support

lsif-java: cross/same-file references (prev. same-file only)

Working for:

• variable type (incl class properties+args)
• variable uses (incl class properties+args)
• class instantiation
• method invoc
• method references (aka obj::method)

JDK 11+ support for javac plugin indexer

In Olaf's testing with graalvm JDK 11, there were compile issues with certain symbols from the com.sun.source packages not being resolved.
To support constructs beyond JDK 8 (records etc) we'll need to be able to compile with a newer JDK.

This was previously achieved in the Kotlin rewrite through compiling with runtime java version checks to avoid calling methods that don't exist in earlier versions.
Another approach we could try is with multi-release jars.

The issue of tools.jar not being a thing in JDK9+ is already taken care of by Olaf in the SBT plugin

Migrate usages of com.sun.tools.* to provided alternatives

As per JEP 396, Tools that use the com.sun.tools.javac.* packages to process source code. Such tools should instead use the javax.tools, javax.lang.model, and com.sun.source.* APIs, available since JDK 6.

As that package is used in multiple locations, it should be strongly considered to put time into migrating to alternatives which (hopefully) exist in the recommended packages

Lombok support

Lombok can be used to generate constructors/getters/setters etc through annotation processing at compile time. Without this, there will be errors emitted by javac about symbols not found/wrong args etc. While this doesn't cause any exceptions to be thrown (indexing still works but at somewhat reduced capabilities, to be measured), it should be investigated what the feasibility of doing lombok preprocessing would be and the gains.

georgewfraser/java-language-server#152 (comment)

lsif-java: emit definitions

Currently working for:

• methods
• classes
• variables

Unable to generate LSIF dump for intuit/karate

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot karate -out karate.lsif

Exception in thread "main" spoon.compiler.ModelBuildingException: The type SslTest is already defined
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.reportProblem(JDTBasedSpoonCompiler.java:575)
	at spoon.support.compiler.jdt.TreeBuilderRequestor.acceptResult(TreeBuilderRequestor.java:27)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:86)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:41)
	at Main.main(Main.java:15)

Improved support for Bazel

Related discussions here sourcegraph/sourcegraph#2982 (comment)

Unable to generate valid LSIF dump for spring-projects/spring-boot

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

/lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot spring-boot -out spring-boot.lsif

0 file(s), 0 def(s), 3 element(s)
Processed in 579ms[✱] ~/External

Type and Signatures are not emitted

These will be needed to reconstruct hovers in lsif-semanticdb. Everything needed is outlined in the semanticdb specs for Type and for Signature (under SymbolInformation).

Potential for numerous files to be missed on indexing

This is a multi (at least 2) tiered issue.

1. Spoons handling of finding source directories defined in pom.xml needs to be improved. Two issues found:
   1. If a module/pom file doesn't specify <sourceDirectory> and/or <testSourceDirectory>, it falls back to a default for whichever were missing (src/main/java and src/test/java respectively), even if they would be defined in an ancestor up the pom hierarchy. This issue at the time of writing has been addressed locally, PR incoming soon:tm:
      Ref: POM inheritance
   2. Spoon doesn't interpolate ${project.basedir}, which was encounted in some repos. PR mentioned above will also address this.
2. Theory: in multi-maven-project scenarios (>1 pom.xml file in a repo), in root pom files that don't act as aggregators (aka don't define a <module> block), maven projects that aren't the root project would be missed, and therefore not indexed. This could be addressed in lsif-java itself by comparing the discovered-by-Spoon projects and comparing it to the discovered-by-filesystem-tree-walking pom.xml files/projects, and adding the missing projects to the Spoon model's input resources.
   Ref: POM aggregation

Setup CI for Java 8

In #91, we added Java 11 support but removed our Java 8 tests.

Transform Javadoc hovers to Markdown hovers

Wrong ranges for lines with tab indentation

In the given file here, certain normal methods only return search based results while others give us semantic based results (albeit with "no hover info").

These are all completely basic methods taking and returning basic types (Long, String etc).

Fringe thought, is this related to @Entity or @Table annotations? There doesn't appear to be a pattern between getters vs setters however.

Java record hover text

In the traversed AST, records are represented as their desugared class equivalent. Using the record_components field of the class decl symbol, we can reconstruct a hover text identical/similar enough to the original declaration

Unexpected hover data for Enum value

Example: /*public static final*/ SAMPLE /* \u003d new Enum() */ /*enum*/

Implement SymbolInformation.overrides

Currently, "find references" only returns results for the exact symbol over the cursor. The results don't include methods that override that symbol, or the supermethod of a symbol.

• implement the new SymbolInformation.overrides feature that got added to SemanticDB scalameta/scalameta#2233
• extend lsif-semanticdb conversion to include overrides in the "find references" results

reference before definition emitted causes miss

If a reference to a definition is encountered in a single file before the definition is emitted, the reference is not emitted

Gradle support

Move away from getting users to have to generate a Maven project from a Gradle project with fiddly Groovy scripts. We demand native support 💢

Direction will be dependent on results of RFC 238

Convert SemanticDB docstrings into LSIF hover messages

This ticket will require changes inside and outside of this repo

• scalameta/scalameta: add the documentation field to the upstream SemanticDB spec.
• sourcegraph/lsif-semanticdb: use the new documentation field (will happen in rewrite to Java)
• use new documentation field when emitting docstrings

lsif-java on master no longer indexes Spoon

As of 2de8444 a commit introduced a regression that causes lsif-java to spin forever (>5 hours on GH action) when indexing the Spoon repository. Example. I've reverted the lsif-java-action to use the last known good commit for indexing Spoon, since it currently uses lsif-java in their CI: sourcegraph/lsif-java-action@11dd91b. Context from spoon thread.

Since lsif-java is under development, I won't harp on this, but what is the current thinking on testing (to catch/avoid regressions) and in future, support stable versions (so we can revert to safe commit versions, in case of regressions)?

missing classes with same names

Environment JDK 1.8

I'v been using lsif-java to generate a lsif dump to our java project, but I found that some classes were not been indexed correctly. These classes have same class names but in different packages.
I have been create a simple project to recreate this scenario. see https://github.com/zhouang777/multi-package-demo/tree/master.

---

I have dig into code and found something might be useful.

1. "typeByFile" in ProjectIndexer already missing a class. It should be 2 TestClass in the map.
   

2. I have tried to disable the spoon environment "ignoreDuplicateDeclarations". It comes to an spoon/JDT error

Exception in thread "main" spoon.compiler.ModelBuildingException: The type TestClass is already defined
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.reportProblem(JDTBasedSpoonCompiler.java:575)
	at spoon.support.compiler.jdt.TreeBuilderRequestor.acceptResult(TreeBuilderRequestor.java:27)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:86)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:255)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:403)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:362)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:330)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:113)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:96)
	at spoon.Launcher.buildModel(Launcher.java:766)
	at ProjectIndexer.index(ProjectIndexer.java:43)
	at Main.main(Main.java:16)

It seems that spoon use only simple class name to identifier a type, not with package name prefixed. Not sure what is the root cause.

Do you guys have any suggestions?

Maven support

Bring across Maven support from Spoon based lsif-java (aka from Spoon itself) as that appeared to work pretty well for the majority of cases

sbt support

We have sbt-sourcegraph that does some of the work but it still requires manual configuration and the steps are not documented. Opening this issue to track progress on adding automatic inference.

• Automatically enable semanticdb-javac in sbt-sourcegraph
• Support sbt v1.0-v1.3, or report helpful error message
• Configurable version of SemanticDB and mtags
• Support manual addCompilerPlugin()
• #293
• #294

Unable to generate LSIF dump for apache/skywalking

Platform

macOS Mojave 10.14.6 (18G103)

Java info

openjdk 13.0.1 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Steps to reproduce

./lsif-java/build/install/lsifjava/bin/lsifjava -projectRoot skywalking -out skywalking.lsif

Exception in thread "main" java.lang.NullPointerException
	at DocumentIndexer.identifierRange(DocumentIndexer.java:312)
	at DocumentIndexer.access$1100(DocumentIndexer.java:16)
	at DocumentIndexer$ReferencesVisitor.visitCtInvocation(DocumentIndexer.java:188)
	at spoon.support.reflect.code.CtInvocationImpl.accept(CtInvocationImpl.java:44)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtBlock(CtScanner.java:294)
	at spoon.support.reflect.code.CtBlockImpl.accept(CtBlockImpl.java:55)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.visitCtConstructor(CtScanner.java:362)
	at spoon.support.reflect.declaration.CtConstructorImpl.accept(CtConstructorImpl.java:44)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:173)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:165)
	at spoon.reflect.visitor.CtScanner.scan(CtScanner.java:139)
	at spoon.reflect.visitor.CtScanner.visitCtClass(CtScanner.java:330)
	at spoon.support.reflect.declaration.CtClassImpl.accept(CtClassImpl.java:56)
	at DocumentIndexer.visitReferences(DocumentIndexer.java:55)
	at ProjectIndexer.index(ProjectIndexer.java:70)
	at Main.main(Main.java:15)