btc-ag/service-idl

Xtext-based Service IDL (Interface Definition Language) and Code Generators for Protobuf, C++, Java and .NET

This repository contains a Service IDL defined in Xtext, and several code generators for C++, Java, .NET and Google Protocol Buffers.

It is currently experimental, and not yet suitable for general use.

Obtaining the standalone command line generator

Releases of the standalone command line generator are available via GitHub or via Artifactory.

Release of the Eclipse plug-ins are currently not made available as an Eclipse update site. This may change in the future.

Usage

See Introduction

Dependencies of generated code

Note: All generated C++ code, and the Proxy/Dispatcher/Codec modules generated by the .NET and Java code generators, currently depend on proprietary libraries. These may be made open-source in the future.

Target technology	Main dependency	Supported versions
C++	BTC.CAB.ServiceComm[.SQ]	cpp.servicecomm: 0.10, 0.11, 0.12
.NET	BTC.CAB.ServiceComm.NET	dotnet.servicecomm: 0.6, 0.7
Java	BTC.CAB.ServiceComm.Java	java.servicecomm: 0.3, 0.5

The default version is always the most recent version. An older version may be specified for the command line generator with the -versions parameter, e.g.

java -jar com.btc.serviceidl.plainjava-1.0.0-SNAPSHOT.jar input.idl -outputPath out -cppProjectSystem cmake -versions cpp.servicecomm=0.10

Currently generated .NET code is targeting .NET Framework version 4.6 or any compatible version.

Configuration when using the Eclipse plug-ins

Configuration of the above-mentioned settings is also possible when using the Eclipse plug-ins, although no UI is provided for that.

Properties files can be placed in parallel to the IDL file, either at directory scope (.generator) or at file scope (foo.idl.generator). If both are present, first the settings from the directory scope properties are applied, and then the settings from the file scope properties, which may override the above.

The output directories cannot be configured via the properties files at the moment.

The command line runner also interprets these files. Command line options override both the directory and file scope properties in that case.

An example file would look like this:

languages = java,cpp
cppProjectSystem = cmake
projectSet = full
versions = cpp.servicecomm=0.10

Contributing

See the contribution guidelines.

Frequently Asked Questions

See FAQ.

service-idl's Issues

Set up tests that generate and build a Java project

Support generating only specific modules

E.g. generate only a ServiceAPI, Codec and Proxy

Set up tests that generate and build a C++ project

Support targeting multiple ServiceComm versions in the .NET generator

Generate redundancy facades

Generation only for specific technology causes IllegalArgumentException

Calling the standalone generator with parameter -javaOutputPath <path> causes an exception:

Validating IDL input.
IDL input is valid.
Disabling ODB generation, this is unsupported with CMake project system
Configuring generation of java artifacts to D:/Diverse/Idea/netcalc/target/netcalc-generated
Exception in thread "main" java.lang.IllegalArgumentException: A slot with name 'cpp' has not been configured.
        at org.eclipse.xtext.generator.JavaIoFileSystemAccess.getFile(JavaIoFileSystemAccess.java:174)
        at org.eclipse.xtext.generator.JavaIoFileSystemAccess.generateFile(JavaIoFileSystemAccess.java:99)
        at com.btc.serviceidl.generator.protobuf.ProtobufGenerator.generateProtobufFile(ProtobufGenerator.xtend:163)
        at com.btc.serviceidl.generator.protobuf.ProtobufGenerator.doGenerate(ProtobufGenerator.xtend:103)
        at com.btc.serviceidl.generator.IdlGenerator.doGenerate(IdlGenerator.xtend:82)
        at org.eclipse.xtext.generator.GeneratorDelegate.doGenerate(GeneratorDelegate.java:43)
        at org.eclipse.xtext.generator.GeneratorDelegate.generate(GeneratorDelegate.java:34)
        at com.btc.serviceidl.generator.Main.tryRunGenerator(Main.java:223)
        at com.btc.serviceidl.generator.Main.mainBackend(Main.java:114)
        at com.btc.serviceidl.generator.Main.main(Main.java:71)

Created on behalf of @sebastianhaberey

Make specification of output directories more flexible

Goal: it should not be necessary to copy the generated files to different locations

This will require specifying distinct output directories for each (enabled) target technology. Implicit addition of java/dotnet/cpp prefixes should then be removed.

Support failable keyword in the .NET generator

Support configuration of package structure depending on language

Had a talk with @GerrietReents about this a couple days ago. Since the existing package structure may well be different depending on the language, it would be nice to have some sort of configuration file or option where you could specify packages per language.

Properly configure indentation settings for Xtend code formatter

Requires resolution of https://bugs.eclipse.org/bugs/show_bug.cgi?id=405956

Register service faults with custom attributes in generated Java code

Reconsider and document design decision on specifying service faults for failable types

As currently implemented, service faults are specifying along with the definition of a failable type, rather than with the operations that raises them.

The pros and cons of these approaches, and the resulting design decision, should be explicitly documented. If necessary, the IDL definition and generators should be adapted.

The directory name of a generated Java module should correspond with its Maven artifactId

After refactoring, C++ generator fails to resolve library dependencies

Found this issue in the C++ generator after refactoring: when code is generated, the business logic fails to resolve corresponding lib files for a header. Result: in generated file Dependencies.cpp we have this line:
#pragma comment(lib, "") leading to the linker error fatal error LNK1146: no argument specified with option '/DEFAULTLIB:' Reason: after refactoring paths are no longer represented as strings but as IPath objects. In LibResolver.getCABLibs we check for existing key-value mapping by key.toString (with key is the path = OK), but then fail to retrieve the value because of using cab_lib_mapper.get(key) which delivers a null reference (but no error, since Java HashSet may contain null entries), finally resulting in the empty string in Dependencies.cpp. I will provide a fix for this problem.

It should be possible to define a struct inside an interface with the same name

Currently, this leads to a problem at least in Java, because of the following behaviour:

If there's a message in the file with the same name, "OuterClass" will be appended to the outer class's name. For example, if foo_bar.proto contains a message called FooBar, the outer class will become FooBarOuterClass. (https://developers.google.com/protocol-buffers/docs/reference/java-generated)

This is not accounted for in the generated Java code.

Set up automated release process

Set up tests that generate and build a .NET project

Error if struct member name contains underscore

The error below was thrown during compilation of the generated source code. Removing the underscores in member names fixed it.

[INFO] Compiling 22 source files to D:\Diverse\Idea\netcalc\target\netcalc-generated\java\btc.cab.netcalc\target\classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[236,17] cannot find symbol
  symbol:   method setP_mw(java.lang.Double)
  location: variable builder of type com.btc.cab.netcalc.protobuf.Types.ComplexPower.Builder
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[237,17] cannot find symbol
  symbol:   method setQ_mvar(java.lang.Double)
  location: variable builder of type com.btc.cab.netcalc.protobuf.Types.ComplexPower.Builder
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[246,17] cannot find symbol
  symbol:   method setMagnitude_kv(java.lang.Double)
  location: variable builder of type com.btc.cab.netcalc.protobuf.Types.ComplexVoltage.Builder
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[247,17] cannot find symbol
  symbol:   method setAngle_degree(java.lang.Double)
  location: variable builder of type com.btc.cab.netcalc.protobuf.Types.ComplexVoltage.Builder
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[775,33] cannot find symbol
  symbol:   method getP_mw()
  location: variable typedData of type com.btc.cab.netcalc.protobuf.Types.ComplexPower
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[776,35] cannot find symbol
  symbol:   method getQ_mvar()
  location: variable typedData of type com.btc.cab.netcalc.protobuf.Types.ComplexPower
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[786,41] cannot find symbol
  symbol:   method getMagnitude_kv()
  location: variable typedData of type com.btc.cab.netcalc.protobuf.Types.ComplexVoltage
[ERROR] /D:/Diverse/Idea/netcalc/target/netcalc-generated/java/btc.cab.netcalc/src/main/java/com/btc/cab/netcalc/protobuf/TypesCodec.java:[787,41] cannot find symbol
  symbol:   method getAngle_degree()
  location: variable typedData of type com.btc.cab.netcalc.protobuf.Types.ComplexVoltage

Add support for Bitfield types

Add support for unsigned integer types

Ensure naming conventions for .proto files are adhered to

https://developers.google.com/protocol-buffers/docs/style specifies naming conventions:

Use CamelCase (with an initial capital) for message names – for example, SongServerRequest.

Use underscore_separated_names for field names – for example, song_name.

Use CamelCase (with an initial capital) for enum type names and CAPITALS_WITH_UNDERSCORES for value names

The naming convention used in the IDL file must be mapped to this. Either there must be fixed naming conventions for the IDL file, or these must be configurable.

Replace antrun with xolstice/protobuf-maven-plugin

Consider using the xolstice/protobuf-maven-plugin instead of the antrun plugin as this results in way cleaner pom files.

Also make sure the .proto files will be placed in the src/main/proto folder.

It should be checked if code generation produces conflicting files

The regular JavaIoFileSystemAccess doesn't check for overwritten files. This should be changed, e.g. by providing a wrapper that at least checks if the same file isn't written twice during the same execution of the generator.

Register service faults with custom attributes in generated C++ code

Generation of events for ServiceComm 0.11 and 0.12 fails

When generating an IDL file with events such as BTC.GA.FieldLink.idl, the generated code targeting ServiceComm 0.11 or 0.12 does not compile. It fails with an error like

[ERROR-] D:\NLS\serviceidl-integrationtests\output-263\cpp\modules/Archive/Proxy/include/CArchiveAPIProxy.h(22): fatal error C1083: Cannot open include file: 'ServiceComm/Commons/include/CSharedMessage.h': No such file or directory [D:\NLS\serviceidl-integrationtests\output-263\cpp\build\B\5\BTC.GA.Archive.Proxy.vcxproj]

Support configuration of distributionManagement for a generated parent POM

Language specific artifact servers should be configurable, for example for maven pom files.

Support external types

We would like to use an already existing external type in the IDL. In our specific case it's an ID class that belongs to another API and serves to identify objects from that API.

Support marking service operations as idempotent

Currently, it is unknown whether commands (as opposed to queries) are idempotent or not. This could be marked explicitly on each operation. Alternatively, non-idempotent commands could be declared as unsupported.

Open questions:

• Who should use this information? Is it just an annotation for better understanding of the developer? Can it be used to improve generation of proxies or dispatchers? It might be used if the generator is extending to generate redundancy facades (see #79) to select the appropriate error handling strategy.

Generate .proto files using com.google.protobuf.DescriptorProtos.DescriptorProto.Builder

Currently, the .proto files are written literally. However, there is the com.google.protobuf.DescriptorProtos.DescriptorProto.Builder, which can be used to create a Protobuf Specification programmatically, which were easier to understand and safer to use. Changes this would improve maintainability of the Protobuf generator. The generated Protobuf metamodel could then also be used in the other generators.

Add support for DateTime/Timestamp type

Support generating only for specific target technologies via the command-line generator

Register service faults with custom attributes in generated .NET code

Remove unnecessary cmake dependencies

The CMakeLists.txt generator currently outputs a fixed set of external dependencies. This should be changed such that each module has only the required dependencies.

Use MsBuild.ProtocolBuffers to call protoc in generated .NET csproj files

See https://www.nuget.org/packages/MsBuild.ProtocolBuffers/ resp. https://github.com/warrenfalk/msbuild-protocol-buffers

Separate skeleton and non-skeleton code into different target directories

The code generators generate both skeleton code intended for manual editing (such as a service implementation) and non-skeleton code which should not be modified manually (such as the Service API, Proxy and Dispatcher).

To make clear which files belong to which category, they should be generated into separate target directories (Xtext itself uses src vs. src-gen directories). In addition, skeleton files, at least when they have been modified manually, should not be overwritten.

Generate an aggregate parent POM for all generated Java modules

At the moment, the user must manually run the maven builds in the correct order, which may become very tedious. Given a parent pom, maven would automatically consider the module dependencies to establish a proper build order.

Support targeting multiple ServiceComm versions in the Java generator

Generate OO-API and corresponding mappers along with the Service API

I know you will probably close this issue right away, but since the other day, when you explained ServiceIDL's concept of the automatically generated Service API vs the manually defined, local OO-API, this didnt leave my head, so I just want to write it down.

Basically, what we subconciously hoped for when we decided to use ServiceIDL was that we would write an IDL, "push the button" and start working with the Java generated entities on the Java side and the C++ generated entities on the other side. We didn't expect that we had to manually write an OO-API to use locally, and the corresponding mapper that maps the OO-API to the Service-API - on both sides.

I'm sure there are convincing technical reasons for having to to this manually, but from a mere user perspective, if I had a magic wand, I'd still love to be able to use the generated entities right away in my code along with my own entities instead of having to map them first.

Support specification of explicit values for enum elements

Allow generation parameters to be specified in a configuration file rather than on the command line

Update .NET generator to Google.Protobuf 3.6.1 (or newer)

Add option to generate target types according to CAB conventions

This in particular means mapping the IDL types as follows:

• string -> BTC::Commons::Core::String
• sequence<...> -> BTC::Commons::Core::AutoPtr<BTC::Commons::Core::ForwardConstIterable>

Also, the CAB smart pointer types should be used.

Add tests for com.btc.serviceidl.formatting2.IdlFormatter

Remove Maven resolver from generated pom.xml

C# generator produces syntactically invalid code

In generated C# code there is an issue making the code syntactically invalid, so that the generated project does not compile. In the generated codec file there is a statement if (plainData === null) as well as if (encodedData === null); this line was probably introduced while migrating former Xtend code to more recent versions by find-replace ( === is the proper way to make a null reference check in Xtend, but this code part is located in a template and finally becomes C# code). I will provide a fix for this.

Support for map-like structures

I saw in the language definition that there's plans for a tuple<> type, but it's not yet supported. Since we have a few maps in our entities it would be very useful.

Version information should be specified on a larger scope than individual interfaces

Currently, only interfaces can be versioned. Other definitions within an IDL file are always assigned the default version "0.1.0" by the code generators.

Probably, the possibility to version interfaces individually should be deprecated/removed.

Coverage measurement is broken

Coverage measurement is broken. At the moment, no coverage is recorded at all.

In a previous state, coverage was determined for the test modules, but not for the production code, and correctly uploaded to https://codecov.io/gh/btc-ag/service-idl

Support targeting multiple ServiceComm versions in the C++ generator

Create a single .proto specification that is used by all technologies