tomokinakamaru/silverchain

Fluent API generator

silverchain's Issues

Feature request: multiplication syntax

Part 1 of 4

Sometimes, a fluent API has methods where the number of parameters of a method matches those of a another method called earlier in the chain:

AggregateBuilder
{
    Aggregate
    properties(
        String nameA,
        String nameB
    )
    withValues(
        Object valueA,
        Object valueB
    )
    ;

    Aggregate
    properties(
        String nameA,
        String nameB,
        String nameC
    )
    withValues(
        Object valueA,
        Object valueB,
        Object valueC
    )
    ;
}

In the action class, one can then direct the overloads of each method to the same private varargs method:

@Override
public void properties(String nameA, String nameB)
{
    setProperties(nameA, nameB);
}

@Override
public void properties(String nameA, String nameB, String nameC)
{
    setProperties(nameA, nameB, nameC);
}

private void setProperties(String... names)
{
    // implementation
}

@Override
public Aggregate withValues(Object valueA, Object valueB)
{
    return build(valueA, valueB);
}

@Override
public Aggregate withValues(Object valueA, Object valueB, Object valueC)
{
    return build(valueA, valueB, valueC);
}

private Aggregate build(Object... values)
{
    // implementation
}

Note: with its 2 to 3 parameters, this example is a shortened version of my original use case, an API that offers overloads with 2 to 8 parameters.

While the pattern works fine, it is quite repetitious both inside the AG and in Java, and the average Silverchain user will probably not come up with it on their own.

Obviously, Silverchain could offer a more compact way to achieve the same result. I imagine a parameter multiplication syntax which may look roughly as follows:

AggregateBuilder
{
    Aggregate
    $N=[2,3]
    properties($N × String name)
    withValues($N × Object value);
}

(Note the use of Unicode × instead of ASCII *. Although the parser could probably use * as well and distinguish the two meanings, I fear that humans might have trouble if it did.)

Silverchain would create the same chain interfaces/classes as in the manual version above, but all overloads of each method (properties() or withValues()) would use the same varargs method in the action class:

@Override
public void properties(String... names)
{
    // ...
}

@Override
public Aggregate withValues(Object... values)
{
    // ...
}

The beauty of this is obviously that I can change the $N=[2,3] to $N=[2,8] or even $N=[2,20] without having to add any boilerplate AG or Java code - it all stays the same.

Part 2 of 4

While writing this, I suddenly thought "why limit this feature to method calls that each have N parameters? Why don't we also allow N successive method calls?" Granted, I don't have a real-world use case for this (my AggregateBuilder quite intentionally always works via a two-method chain) - but maybe it is something that's worth pursuing.

So we could generalize the idea of parameter multiplication syntax to expression multiplication syntax, and also allow using it for method calls as follows:

BazBuilder
{
    Baz
    $N=[1,10]
    $N × initColumn(Object columnName)
    (
        addRow()
        $N × setCell(Object value, Color background);
    )+
    build()
}

@Override
public void initColumn(String... columnName)
{
    // ...
}

@Override
public void setCell(Object... value, Color... background)
{
    // ...
}

Part 3 of 4

If you didn't guess it yet, I see no reason why Silverchain should restrict each chain to have exactly one multiplier. Maybe there are use cases for having a chain with $I=[1,6] $J=[2,3].

Part 4 of 4

The variants from Part 1 and 2 (multiplying parameters & method calls) could even be mixed in the same chain: when setColumns() was called with N parameters, one has to call setCell() N times.

BazBuilder2
{
    Baz
    $N=[1,10]
    setColumns($N × Object columnName)
    (
        addRow()
        $N × setCell(Object value, Color background);
    )+
    build()
}

Finals

So, what do you think about this, do you like it? Does it look useful to you, as well? And maybe the most important question: can all this be implemented (and with reasonable effort)?

I'm the first to admit that especially Parts 2 to 4 look a bit over the top. But then again, Silverchain offering powerful features like this may be exactly the thing that inspires developers to even attempt to create much richer fluent APIs than usual, with less effort.

Java CI always fail in forked repositories

https://github.com/tomokinakamaru/silverchain/actions/runs/1431361698

Use ANTLR

• Use ANTLR
• Refactor AG grammar & parser-related code

Publish to Maven Central

Building on PR #53, I'd like to ask for Silverchain to be published to Maven Central.

Terminology notes: Maven Central (a.k.a. the Central Repository) is operated by the company "Sonatype". For individuals, publishing to Maven Central happens via the OSSRH ("Open Source Software Repository Hosting") servers.

We will need the following things:

• One-time setup for @tomokinakamaru as the one in control of and performing releases:

  • Get a free account on OSSRH: Sign up page

  • Claim the namespace com.github.tomokinakamaru io.github.tomokinakamaru (valid for .silverchain and any other personal repositories)

    • Steps
      1. Create a 'New Project' ticket
      2. Prove ownership by creating an empty temporary GitHub repository with the same name as the ticket number, e.g. OSSRH-12345 which resides at https://github.com/tomokinakamaru/OSSRH-12345

  • Personal OpenPGP key for code signing

    • Sonatype provides detailed explanations for Working with PGP Signatures, but for starters, you only need these three steps:
      1. Installing GnuPG
      2. Generating a Key Pair
      3. Distributing Your Public Key

  • Set up gradle for code signing & publishing

    1. Export your secret key ring file:

       gpg --keyring secring.gpg --export-secret-keys \
         > ~/.gnupg/secring.gpg
       

    2. Locate your Gradle home directory (default is $USER_HOME/.gradle) and create/open the file gradle.properties
    3. Determine the key ID (gpg -K) and set the signing properties:

       signing.keyId=24875D73
       signing.password=secret
       signing.secretKeyRingFile=/Users/me/.gnupg/secring.gpg
       

    4. Add OSSRH credentials as follows:

       ossrhUsername=your-jira-id
       ossrhPassword=your-jira-password
       

• Extending the Gradle build script to support publishing
  • In an upcoming PR

Later, I'll provide more details on how to actually perform releases. I promise it's way simpler than the above. 😉

Valid parameter name is rejected

When one of the methods of an .ag chain rule uses the parameter name name1, Silverchain fails with the following message:

Encountered " <NUMBER> "1 "" at line 219, column 16.
Was expecting:
    ")" ...

Please make sure that any parameter name which is valid for Java is accepted by Silverchain, as well.

Generic methods

I'm struggling to properly express generic methods in the .ag file. The Java signature of the action class methods should end up like this:

<T> void addPlugin(Class<T> targetType, Plugin<? super T> plugin)

Judging from the examples and the grammar definition, I'd guess that Silverchain does not yet support type parameters on methods, but only on classes.

Please add support for this to the .ag syntax.

Note: the Java tutorial chapter "Generic Methods" lists a few other variants that should be supported as well. (Just ignore the non-void return type.)

public boolean containsAll(Collection<?> c);
public boolean addAll(Collection<? extends E> c);
public static <T> void copy(List<T> dest, List<? extends T> src)

Of particular interest is the simple SomeType<?> which I'd need as well - while the raw type works fine, I'd like to avoid suppressing the associated warning in my action class.

Reorganize generated Java code

For a block org.example.Foo { ... }, generate

• Interface for state N: public org.example.intermediates.FooN (N = 0, 1, ...)
• Class for state N: non-public org.example.FooNImpl
• Action interface: non-public org.example.FooAction

In this file layout, the library author would not have to add any public types of their own and the only public types added by Silverchain all reside in the intermediates subpackage, so they don't "spam" the list of "regular" classes which are the fluent API entrypoints of the library. This file layout also hepls Silverchain users to generate well-documented fluent APIs more easily.

Support importing class names in .ag files

Compared to Java method signatures, .ag files are quite verbose because one has to use fully qualified class names.

For illustration, let's pretend the matrix.ag used conventional package names:

org.example.math.MatrixBuilder<;R extends org.example.math.Size, C extends org.example.math.Size> {
  org.example.math.Matrix<R, C> random() row(R row) col(C col);
}

org.example.math.Matrix<R extends org.example.math.Size, C extends org.example.math.Size; NEW_C extends org.example.math.Size> {
  org.example.math.Matrix<R, C> plus(org.example.math.Matrix<R, C> matrix);
  org.example.math.Matrix<R, NEW_C> mult(org.example.math.Matrix<C, NEW_C> matrix);
}

(For a real world example, see the AG file in my PureTemplate project.)

How about adding an AG language construct to import class names which allows using their simple class names? The AG file could then look like this:

import org.example.math.MatrixBuilder;
import org.example.math.Size;
import org.example.math.Matrix;

MatrixBuilder<;R extends Size, C extends Size> {
  Matrix<R, C> random() row(R row) col(C col);
}

Matrix<R extends Size, C extends Size; NEW_C extends Size> {
  Matrix<R, C> plus(Matrix<R, C> matrix);
  Matrix<R, NEW_C> mult(Matrix<C, NEW_C> matrix);
}

Notice how the latter looks almost like Java. 😄

Obviously, it would make no difference whether I import my own classes or things like java.util.List. Also, unlike the Java compiler, Silverchain would not have to do any actual classpath checking: if an import declaration in my AG is wrong, it will end up as-is in the generated Java sources, and then the compiler will complain there.

The implementation could be really simple:

• Import lines are only allowed at the start of the file (= before the first rule) to mimic Java and to keep AG files tidy.
• Whenever a type name in a rule (regardless if it's a type parameter, argument or return value) matches the final segment of one of the imported fully qualified names, use that instead.
• Complain if the final segments are not unique, e.g. if I accidentally import both java.util.List and java.awt.List. Just like with Java, I could only import one of them; the other name would need to be fully qualified.
• Wildcard imports (import java.util.*) are not supported because they would require Silverchain actually checking the classpath.

What do you think?

Feature Request: customizable type names for states

For an API entry point called Foo, Silverchain currently creates state interfaces called FooN and classes called FooNImpl (N = 0, 1, ...).

As discussed in chat, I believe that having numbers in a class name is unusual enough that it should help API users remember that these intermediate types are somewhat special, i.e. that keeping references to them in variables threatens the forward compatibility of their own code.

However, I now think that a fluent API could emphasize the fragility of its intermediate type names even more by incorporating non-latin Unicode characters. As Java identifiers can include any Unicode character that is a letter, digit or currency symbol in any language, one can certainly find a character that signals "this is unusual". For example, my current favorite is ⴵ because it is as large as a capital letter (and thus unlikely to be overlooked) and resembles an hourglass (fitting the "call in progress" nature of a method chain).

That said, Silverchain should not make that choice for API developers, but default to the safe, (hopefully) non-controversial FooN style. I propose adding an optional configuration setting called stateNameFormat which is a MessageFormat pattern:

stateNameTemplate	Interface Name	Class Name
{0}{1} (default setting)	Foo7	Foo7Impl
{0}ⴵ{1}	Fooⴵ7	Fooⴵ7Impl
{0}{1,number,0000}	Foo0007	Foo0007Impl
State{1}Of{0}	State7OfFoo	State7OfFooImpl

Silverchain would not have to validate anything here; if the resulting name is not a valid Java identifier or results in non-unique type names, the Java compiler will complain anyway:

stateNameTemplate	Interface Name	Class Name	Remark
Fun	Fun	FunImpl	Non-unique
{0}	Foo	FooImpl	Non-unique
{1}	7	7Impl	Invalid identifier: starts with a digit
{0}⛔{1}	Foo⛔7	Foo⛔7Impl	Invalid identifier: '⛔' is not a letter/digit/currency

Cryptic errors on missing/wrong braces

Silverchain fails to parse the following AG files, which is correct. However, the error messages are overly generic and vague. I included some examples of what I would consider a helpful message.

// fails with the following error:
// line 13:15 no viable alternative at input 'Foo{voidfirstsecond'
//
// fix:
// replace "first" with "first()"
//
// ideas for helpful messages:
// - line 13:14 expected '(', got ' '
// - line 13:15 expected '(', got 'second'
// - line 13:9 expected METHOD_SIGNATURE, got TYPE_NAME
//
Foo {
    void first second() third();
}

// fails with the following error:
// line 15:15 no viable alternative at input 'Foo{voidfirst(secondA('
//
// fix:
// replace "first" with "first()"
//
// ideas for helpful messages:
// - line 15:15 unexpected opening brace
// - line 15:8 expected TYPE_NAME, got METHOD_SIGNATURE
// - line 15:8 expected PARAMETER, got METHOD_SIGNATURE
//
Foo {
    void
    first
    (
        secondA()
        |
        secondB()
    )
    third();
}

Feature request: Fragment support

I have several APIs where I need to repeat the exact same expression in several places. Take this (simplified) example from PureTemplate:

GroupLoader
{
    Group

    // Allow only one call of each with*() method, but in any order, and mixed with other, unrestricted methods
    ( importTemplates() | registerModelAdaptor() | registerAttributeRenderer() )*
    {
        (
            withDelimiters()?
            ( importTemplates() | registerModelAdaptor() | registerAttributeRenderer() )*
        ),
        (
            withErrorListener()?
            ( importTemplates() | registerModelAdaptor() | registerAttributeRenderer() )*
        ),
        (
            withLegacyRendering()?
            ( importTemplates() | registerModelAdaptor() | registerAttributeRenderer() )*
        )
    }

    build();
}

It's plain to see that the lines starting with ( importTemplates() ... are identical. Note that in the original version , those lines are much longer (even with the recent addition of import declarations).

For this and other situations, Silverchain could offer a feature that lets me define an alias for an expression that I can reuse throughout my AG file. I think "fragment" might be a good name. I'm not sure what the syntax should be - as usual we should try to balance "being obvious", i.e. Java-like, with achieving compact syntax.

Here's one idea for defining and using such a fragment:

$UNRESTRICTED_OPTIONS = ( importTemplates() | registerModelAdaptor() | registerAttributeRenderer() )*;

GroupLoader
{
    Group

    // Allow only one call of each with*() method, but in any order, and mixed with other, unrestricted methods
    $UNRESTRICTED_OPTIONS
    {
        ( withDelimiters()? $UNRESTRICTED_OPTIONS ),
        ( withErrorListener()? $UNRESTRICTED_OPTIONS ),
        ( withLegacyRendering()? $UNRESTRICTED_OPTIONS )
    }

    build();
}

This idea imitates Java field declaration and definition, uses ALL_CAPS (which may or may not be enforced by Silverchain) like Java constants and prefixes the name with a $ (which should be enforced) to reduce the chance of collisions with class/method/parameter names.

Alternatively, one might use existing keywords as part of the syntax. I came up with private static final $UNRESTRICTED_OPTIONS = ..., intentionally mimicking a Java constant, but that feels "fake" to me because none of the three keywords really do what they do in Java here - and it's really long-winded.

Lastly, we could invent new keywords, something like fragment $UNRESTRICTED_OPTIONS = .... One could say that version is bad because it's not Java-like, or great because the keyword makes its purpose obvious. I have a slight preference for the latter. However, for consistency, we should then definitely consider having keywords elsewhere in AG, for example class GroupLoader { } and rule MyReturnValue Method1() Method2();.

Note: implementation-wise, any usage of a fragment would be equivalent of pasting the expression at that position, but wrapped in parentheses to make sure the fragment is atomic. This would even allow things like $UNRESTRICTED_OPTIONS*.

Looking forward to read your thoughts on this!

Methods with array parameter can't be generated

While testing #48, I added a method signature with an array parameter to my AG file:

bar(String[] strings)

When run, Silverchain prints the following error:

Encountered " <NAME> "bar "" at line XX, column YY.
Was expecting:
    ";" ...

Interestingly, it doesn't complain about the String[] which clearly is the problem here, but about "bar" (line/column number matches that identifier as well).

Note: Personally, I'm not really using array arguments, but for completeness' sake, Silverchain should support them.

Cryptic error on rules without "tail" method

Due to sloppy editing, at one point I ended up with an AG whose structure looked as shown below (except that the rule ended after the *). I was confused because I could not see a "conflict" anywhere, then figured out what the cause was.

// fails with the following error:
// Conflict: String#L12C5, secondA()#L15C9, secondB()#L17C9
//
// fix:
// replace "//third()" with "third()"
//
// ideas for helpful messages:
// - line 21:4 rule must have non-repeatable tail method so it can return String#L16C5
//          ^-- points to the semicolon ending the rule
//
Foo {
    String
    first()
    (
        secondA()
        |
        secondB()
    )
    *
    //third()
    ;
}

I guess if the error message had said something about what I would (informally?) call a "tail" method, I'd have understood immediately.

Make AG syntax more like Java

• Make AG syntax more like Java
  • Allow numbers in a name
  • Type declaration: Foo: ... → Foo { ... }
  • Type parameter declaration Foo[T] → Foo<T>
  • Type parameter bounds: <: → extends and :> → super
  • Return type:
    • m1() ... mx() Foo; → Foo m1() ... mx();
    • m1() ... mx(); → void m1() ... mx();
  • Repeat operator: {1,10} → [1,10]
• Update docs
• Remove --language

Document type parameter syntax

#39 (comment) says

There are now three kinds of type parameters:

1. Ones listed in a type declaration

   1. Ones listed before ; or without ;
      These type parameters are shared in a chain expression and are included in the generated type declaration. For example, if you give an input like Foo[T]: …, Silverchain generates Foo<T>.
   2. Ones listed after ;
      These type parameters are shared in a chain expression but are not included in the generated type declaration. For example, If you give an input like Foo[;T]: ..., Silverchain generates Foo (without the type parameter). The examples mapbuilder.ag and listutil.ag are their usecases.

2. Ones listed in a method declaration (New)
   These type parameters are not shared in a chain expression. They are only used in the method.

This information should be documented somewhere. It doesn't fit in README.md or doc/tutorial.md, so maybe it should be the start of a doc/ag-reference.md.

Improve maintainability

#develop

• internal
  • frontend
    • parser: Build Ag AST from text
      • Implementation
      • Tests
      • Check modifiers
      • Annotate with API guardian
      • Annotate ANTLR-generated files with @API(status = API.Status.INTERNAL) if possible
    • checker: Check Ag AST
      • Implementation
      • Tests
      • Check modifiers
      • Annotate with API guardian
    • rewriter: Rewrite Ag AST
      • Implementation
      • Tests
      • Check modifiers
      • Annotate with API guardian
    • Frontend: Build, Check, Rewrite Ag ASTs
      • Implementation
      • Tests
      • Check modifiers
      • Annotate with API guardian
  • middleware

    • graph

      • data: Data structure & visitors
        • Review Implementation
        • Check modifiers
        • Annotate with API guardian
        • Create GraphWalker (likeParseTreeWalker)
      • builder: Build graphs from Ag AST
        • Review Implementation
        • Check modifiers
        • Annotate with API guardian
        • Tests
      • checker: Check graphs
        • Review Implementation
        • Check modifiers
        • Annotate with API guardian
        • Tests
      • rewriter: Rewrite graphs
        • Review Implementation
        • Check modifiers
        • Annotate with API guardian
        • Tests

    • java

      • data: Data structure
      • builder: Build Java ASTs from graphs
      • checker: Check Java ASTs
      • rewriter: Rewrite Java ASTs
  • backend
    • data: Data structure
    • builder: Build File from Java ASTs
    • checker: Check Files
    • generator: Generate .java files
• API
  • Silverchain
  • SilverchainException
  • SilverchainWarning

---

• silverchain.command
  • [x] Use picocli
• silverchain.diagram
  • Use learnlib.automatalib?
• silverchain.generator
  • [ ] AST-based code generation
• silverchain.javadoc
• silverchain.parser
  • [ ] Remove hand-written AST classes
• silverchain.validator
  • [x] Merge JavaValidator & Validator
  • [x] Remove validations that can be checked with Java compiler
• silverchain.warning
• silverchain

Methods with "throws" clause can't be generated

A new API I added to my AG file contains several methods declaring a checked exception:

foo() throws java.io.IOException

When run, Silverchain prints the following error:

Encountered " <NAME> "throws "" at line XX, column YY.
Was expecting:
    ";" ...

Javadoc support

In my experience, Javadoc is not well-suited for documenting fluent APIs - users simply don't get a nice overview of the possible chains. That said, the code completion in my IDE would benefit greatly from Javadoc on the individual methods. For hand-written fluent APIs, this can be done.

Silverchain cannot generate Javadoc comments, of course - they still must be written by a human. However, a library author certainly should not modify the Silverchain-generated sources (in my project, those are not even checked in).

How about adding new feature so that when generating any method in a stateX interface, Silverchain copies the Javadoc comment for that signature from somewhere? I don't care much from where - it could be one or several Java interfaces, it could be the action class (MelodyAction in the tutorial) or it could be something else entirely.

Method type parameter shared in chain expression

Generic methods as introduced by #39 work fine, but I just noticed that they are shared with the chain (i.e. have a generic return value), which bloats Javadoc and potentially increases the number of states.

Test AG file

import com.example.FooBuilder;
import com.example.Foo;

import java.util.function.Function;

FooBuilder
{
    Foo
    addConverter<T>(Class<T> targetClass, Function<T, String> converter)*
    build();
}

Expected result

interface IFooBuilder {

  <T> IFooBuilder addConverter(Class<T> targetClass, java.util.function.Function<T, String> converter);

  com.example.Foo build();
}

Actual result

interface IFooBuilder {

  <T> state1.FooBuilder<T> addConverter(Class<T> targetClass, java.util.function.Function<T, String> converter);

  com.example.Foo build();
}

Methods with varargs parameter can't be generated

A new API I added to my AG file contains methods which use varargs:

quux(String... strings)

When run, Silverchain prints the following error:

Encountered " <NAME> "quux "" at line XX, column YY.
Was expecting:
    ";" ...

Interestingly, it doesn't complain about the String... which clearly is the problem here, but about "quux" (line/column number matches that identifier as well).

NPE when referencing undefined fragment

Referencing a fragment that is not defined (e.g. due to a typo) causes the following NPE:

java.lang.NullPointerException
    at silverchain.parser.adapter.ASTBuilder.visitFragmentReference (ASTBuilder.java:454)
    at silverchain.parser.adapter.ASTBuilder.visitRuleAtom (ASTBuilder.java:214)
    at silverchain.parser.adapter.ASTBuilder.visitRuleElement (ASTBuilder.java:199)
    at silverchain.parser.adapter.ASTBuilder.visitRuleFactor (ASTBuilder.java:188)
    at silverchain.parser.adapter.ASTBuilder.visitRuleTerm (ASTBuilder.java:178)

Silverchain should produce a readable error message in this case, e.g. "$FOO is not defined at #L1C2".

Misleading location reported for "no viable alternative"

Example (invalid) AG:

import java.util.List;
import java.util.function.UnaryOperator;

$ADJUSTMENTS = adjusting<F>(List<F> list, UnaryOperator<F> adjuster);

// something else

// more code

Silverchain rejects this with the following error:

Could not generate API: line 9:0 no viable alternative at input '$ADJUSTMENTS=adjusting<F>(List<F>list,UnaryOperator<F>adjuster);// something else// more code'

The location reported is misleading: it points to the end of the file, but should have been something like 4:25 (the <F>) or at least 4:16 (the adjusting<F>).

Especially for large AG files, it takes a while to figure out that it really rejects something at the start of the expression in the error message.

Feature request: multiline comments

In addition to // single line comments, Silverchain should support /* multiline comments */ as well.

I tried to use them to comment out sections of my AG file while debugging a problem, but I imagine it would be useful in other cases as well.

Use consistent format when pointing out AG locations

Silverchain error messages use different formats for AG code locations:

Conflict: String#L12C5, secondA()#L15C9, secondB()#L17C9

line 13:15 no viable alternative at input 'Foo{voidfirstsecond'

This should be made consistent, no matter what the cause of the problem is.

Feature request: shorthand syntax for "each method only once"

In my API, I have a group of methods that can be called in any order, but each of them only 0 to 1 times.

For methods A, B and C, this can be solved with the following AG code:

(
    ( A() B()? C()? ) |
    ( A() C()? B()? ) |
    ( B() A()? C()? ) |
    ( B() C()? A()? ) |
    ( C() A()? B()? ) |
    ( C() B()? A()? )
)?

Compared to implementing this manually, using Silverchain like this is a vast improvement! Still, the solution above is really cryptic and would need a comment explaining the author's intention. Also, if there are more than three methods, or if one needs to add/remove a method, things get cumbersome.

So I had this crazy idea that Silverchain could offer a special shorthand syntax for this scenario, maybe something like this:

( A() ~ B() ~ C() )

The syntax would effectively be syntactic sugar and could even be implemented as a kind of simple preprocessor that transforms the shorthand into the equivalent solution.

Granted, this feature is obviously not "mainstream", but for those that use it, the readability and maintainability of their AG file would increase a lot. Also, it would be another nice demonstration of the kind of added value that people get out of Silverchain.

What do you think?