neo4j-contrib/neo4j-sproc-compiler

Compile-time annotation processor to verify Neo4j procedure|functions validity

Neo4j stored procedure compiler

This is a annotation processor that will verify your stored procedures at compile time.

While most of the basic checks can be performed, you still need some unit tests to verify some runtime behaviours.

What does it do?

Once the stored procedure compiler is added into your project classpath (see Maven/Gradle instructions below), it will trigger compilation failures if any of the following requirements is not met:

• @Context fields must be public and non-final
• all other fields must be static
• Map record fields/procedure parameters must define their key type as String
• @Procedure|@UserFunction class must define a public constructor with no arguments
• @Procedure method must return a Stream
• @Procedure|@UserFunction parameter and record types must be supported
• @Procedure|@UserFunction parameters must be annotated with @Name
• @UserFunction cannot be defined in the root namespace
• all visited @Procedure|@UserFunction names must be unique*

*A deployed Neo4j instance can aggregate stored procedures from different JARs. Inter-JAR naming conflict cannot be detected by an annotation processor. By definition, it can only inspect one compilation unit at a time.

DSV export

The compiler can also export metadata to DSV (Delimiter-Separated Values) files. To make it work, you need to specify the following options (by passing -A${CONFIGURATION_KEY}=${value} to the Java compiler (where ${CONFIGURATION_KEY} is one of the keys listed below):

• GeneratedDocumentationPath: mandatory - the folder path in which the files are going to be generated in. If not specified, the export won't be done.

• Documentation.FieldDelimiter: optional (default: ,) - the delimiter of values within a row

• Documentation.DelimitedFirstField: optional (default: false) - the first column of each row (headers included) is prefixed with the configured separator

• Documentation.QuotedFields: optional (default: true) - whether column values are quoted or not

• Documentation.ExportGrouping: optional (default: SINGLE) - comma-separated values of grouping strategy. The data is exported to a single place (SINGLE), to a single place per enclosing package (PACKAGE), to a single place per enclosing class (CLASS). For now, "place" may mean one or two files, depending on Documentation.ExportSplit.

• Documentation.ExportSplit: optional (default: NONE) - whether to split data by kind (KIND, e.g. procedure vs. function) or not (NONE).

• Documentation.ExportedHeaders: optional (default: *) - delimiter-separated values which define custom ordering and filtering of the available headers. They are not to be quoted (regardless of Documentation.QuotedFields setting), have to be separated by the configured delimiter (see Documentation.FieldDelimiter) and the first header must not be prefixed, even if Documentation.DelimitedFirstField is enabled. The available headers are (in default order):

  • type: 'procedure' or 'function' for now
  • name: procedure/function logical name
  • description: optional procedure/function description
  • execution mode: configured execution mode
  • location: fully qualified method name defining the procedure/function
  • deprecated by: optional replacement of the procedure/function

Use the processor

Maven

SNAPSHOT repository

If you do not plan to test the development version, you can skip this section.

Add to <repositories> section:

   <repository>
      <id>sonatype-snapshot-repo</id>
      <name>Sonatype SNAPSHOT repository</name>
      <layout>default</layout>
      <releases>
         <enabled>false</enabled>
      </releases>
      <snapshots>
         <enabled>true</enabled>
      </snapshots>
      <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
   </repository>

Dependency

Add the dependency simply as follows:

<dependency>
   <groupId>net.biville.florent</groupId>
   <artifactId>neo4j-sproc-compiler</artifactId>
   <version><!-- check last release on https://search.maven.org --></version>
   <scope>provided</scope>
   <optional>true</optional>
</dependency>

Gradle

SNAPSHOT repository

If you do not plan to test the development version, you can skip this section.

Just add to your repositories:

maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }

Dependency

Add to your dependencies:

compileOnly group: 'net.biville.florent', name: 'neo4j-sproc-compiler', version:'/* check last release on https://search.maven.org */'

neo4j-sproc-compiler's Issues

Introduce an option to disable @Context warnings

functions / procedures with the same name appear only once in the CSV output

perhaps a map by name somewhere that should include type?

Generate @Context warnings

• @Context on types other than GraphDatabaseService, Log

[3.1] Support user-defined functions

See neo4j/neo4j#7851

Validate record field types

A record can only have public fields, and they must be of the following type:

• String
• Long
• long
• Double
• double
• Number
• Boolean
• boolean
• Path
• Node
• Relationship
• java.util.Map<String,Object>
• java.util.List of any type in this list, including java.util.List
• Object

Validate @Context fields

The field must be public and non-final and only those types are allowed:

• GraphDatabaseService
• org.neo4j.logging.Log

The project cannot be built

Running mvn clean package works fine, but mvn clean install fails because it cannot find GPG:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:1.6:sign (sign-artifacts) on project neo4j-sproc-compiler-parent: Unable to execute gpg command: Error while executing process. Cannot run program "gpg": error=2, No such file or directory -> [Help 1]

Should this be a mandatory step? Can't it be limited to releases?

[3.1] @PerformsWrite can only be used with Mode.DEFAULT

[3.1] Procedure|UDF name definition via `name` prevails over the rest

List parameter types are not verified

... whereas parameter types are verified for Maps.

Change test-jar packaging

The idea of producing a test-jar from processor module is to extract all procedures/functions supposed to be processed without errors and invoke them in integration tests.

While the current approach works, it's hacky at best and not well supported by the IDE.
The test procedures should belong to src/test/java.

I see two ways of doing this:

• having a separate module that produces a JAR made of valid procedures/functions
• move the valid procedures/functions into some sub-directory of src/test/java (not src/test/resources like now) and properly create a test JAR with only these classes.

Expose option to customize file grouping|splitting

Three grouping options (that can be added up -> n executions for n selections):

• ONE_FILE: everything is written to a single file
• ONE_FILE_PER_PACKAGE : 1 file is generated per package
• ONE_FILE_PER_CLASS : 1 file is generated per class

Two splitting options:

• NONE: procedure and functions are in the same file
• TYPE_SPLIT: procedure and functions are in different files

Procedure must have 1 public no-arg constructor

Procedures (and functions) can have a custom name

Currently, the unicity validation steps rely on the default name, which may not be correct as the procedure can define a custom name.

In Neo 3.1, functions and procedures have 3 ways of naming (in ascending order of priority):

• the default one (package + method name)
• value()
• name()

Fix type export

There is a glitch with exported types:

• return types are fully qualified
• parameter types are simple and raw (no generics)

Instead, it should be simple raw names.
In the case of user-defined procedures, the record type should not be printed but all its fields' simple and raw types.

The generic information should be printed if and only if the exporter is configured to do so.

[Neo4j 3.2+] Support @UserAggregationFunction

or... find a way to rely only on Neo4j's built-in (and up-to-date) processor.

Map values should be processed the same as List values

... despite what documentation says.
That also includes #12.

Demo+doc

Demo integration with Maven, Gradle and Intellij

[Neo4j 3.4+] Support byte array and temporal types

or... find a way to rely only on Neo4j's built-in (and up-to-date) processor.

Improve CSV processing

• do not output parameter type qualified names (use simple names instead)
• output empty strings instead of "N/A" for missing values
• introduce type field : function or procedure
• switch from CSV to DSV where the delimiter is a compiler option
• allow custom export of columns

Isolate DSV processing as a custom Maven module

Ideally, all the original (and now outdated) annotation processing code (now built in Neo4j) should go away.

Fix package name collision

Until #44 is solved, the only possible generation strategy is as follows: one file per package is generated. However, the filename corresponds to the simple package name and not its fully qualified name.

This obviously causes problems if different subpackages share the same simple name.

Write an integration with test-harness and annotation processor

Migrate to JUnit 5

compile-testing may be an issue, maybe not (see google/compile-testing#155)

Introduce boolean option to delimit first column

Asciidoctor requires for instance for tables:

|header1|header2
|value1|value2

Currently, we can generate only that:

header1|header2
value1|value2

Node, Path and Relationship are valid procedure input types

despite what documentation states.
This also applies to List and Map values (see #13 for Map value processing).

Add missing return type to signature

Introduce boolean option to control whether fields are quoted or not

Validate sproc unicity (package+method name) within compilation unit

Expose option to not output columns for which all values are empty

Enforce UTF8 when writing DSV

DsvSerializer uses FileWriter1 which means the system default locale will determine the encoding. We discovered this when running in a Docker container where the locale was POSIX, and any non-ASCII characters used for delimiters or in Strings in proc annotations were garbled. It would be better to enforce UTF8, or make the encoding configurable if there is a use case for other encodings.

The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.2

Validate procedure input type

Only those types are valid:

• String
• Long
• long
• Double
• double
• Number
• Boolean
• boolean
• java.util.Map<String,Object>
• java.util.List of any type in this list, including java.util.List
• Object