quotient-im/gtad

Generate Things from API Descriptions

gtad's Issues

Proper scope for local schema definitions

As a part of #16 implementation, a case should be supported when there are two schema definitions with the same title. Keeping them both on the top level leads to a name conflict while only keeping the first definition will fails if the two define different structures (and we cannot guarantee them being the same structure, even though the Matrix Swagger seems to be good enough in that respect).

Support variadics aka 'exploding'

Because matrix-org/matrix-spec-proposals#1414 uses them (and also explode is a thing in OpenAPI 3).

Rename to GTAD

Support content-repo.yaml

content-repo.yaml defines headers that should be present in the request and the response. The request headers make it to the output but hyphens are not erased from their names, resulting in invalid identifiers. The response headers just don't make it to Printer.
Also, content-repo is the only file using content type different from application/json, meaning that plain byte stream will be used for the result.
There may probably be other caveats.

Types map adds ^ and $ to its regexes

std::regex_match checks that the entire string matches the regex. This in particular means that // doesn't work for "everything else", it only matches an empty string, which is not intuitive.

Recurring data definitions land in improper directories

As of ccfe0d8, if there's a reference inside a file to another file, that other one is correctly found and parsed by Printer receives wrong directory to output the respective result to. This leads to rather (un)funny things like some definitions landing in jobs/generated, while their proper place would be in jobs/generated/definitions, etc.

Fully reproduce 0.1 functionality using templates

This means generation of the easiest cases: calls that get a simple structure or just primitive types on input and return nothing but 200. Ban/unban for Matrix are good examples.

Support titles and deeper parsing of schemas

So far anything more complex than a flat list of properties is converted to an opaque object (matching QJsonObject in Qt), both for parameters and for response properties. Also, schema title attribute is completely skipped though it could serve as a type name.

OpenAPI 3.0 support

Transition of Matrix CS API to OpenAPI 3 somewhat suddenly got traction at matrix-org/matrix-spec-proposals#1446, so GTAD has to support OpenAPI 3 too.

Error when building

[ 82%] Building CXX object CMakeFiles/gtad.dir/analyzer.cpp.o
/home/adam/repos/gtad/analyzer.cpp:25:22: error: 'placeholders' is not a namespace-name
 using namespace std::placeholders;
                      ^~~~~~~~~~~~
/home/adam/repos/gtad/analyzer.cpp:25:34: error: expected namespace-name before ';' token
 using namespace std::placeholders;
                                  ^
make[2]: *** [CMakeFiles/gtad.dir/build.make:89: CMakeFiles/gtad.dir/analyzer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/gtad.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Again, not familiar with normal cpp toolchain. If more info is needed than above, I can run commands to get it.

Support exporting a context structure from Model that could be used in a template

Try to be as close to the swagger-codegen list of template variables as possible

Allow to substitute parameter/`VarDecl` names

This is a better fix for a problem with parameters which names coincide with reserved words of a target language. For C++ and Matrix CS API, such conflicting names are signed, unsigned and default (default is most conflicting because in addition it nearly-clashes with OpenAPI's own default attribute that stores a default value for a certain parameter). A workaround has been to pre-substitute all such words in YAML; however, its well-known drawback (it replaces all such words, regardless of the place) backfired when GTAD gained support for (you guessed it) default values. Providing a way to substituting parameter names (and only them) somewhere around env (but under a separate config section) looks a better alternative.

Support additionalProperties

Case in point: admin.yaml that uses additionalProperties to map device names to device info.

Support response structure inlining

Similar to inlineBody for request bodies, there should be inlineResponse for responses, with the same rationale - when a structure is included (via $ref) from another file, the fact of inclusion should be completely transparent for the actual marshalling. Case in point also matches that for inlineBody: if a top-level schema is a $ref, there should be a top-level data structure representing this schema in the code; however, this data structure is entirely absent in the actual JSON, so should be explicitly rolled-up from actual JSON top-level data.

Sub-schema name is not used when mapping array types

Case in point: getDevices in device_management.yaml; instead of using inner Device schema, the array is mapped to an opaque JSON array.

Make a minimal example with templates (exporting at least a couple of variables) work

This is just an internal milestone. No plan for anything really working yet.

Allow to use different templates for models and operations

Right now GTAD doesn't distinguish between templates for models and for operations. This leads to each template having a big {{#models}} section followed by a big {{#operations}} section with very little common part. Instead of a single array, mustache/templates should become a map with model and operation being arrays of template file names.

Parse and expose descriptions to Mustache

As the number of supported Matrix calls/structures increased, documentation for all the generated code becomes very convenient, if not crucial.

All elements of schema objects are (incorrectly) required

JSON Schema also has the required attribute, and by default a data item in a schema is not required. GTAD 0.5 disregards this and treats all parts as required.

Avoid processing same structures over and over

As of now, if the same data structure ($refed, e.g.) is encountered several times in the API description it gets parsed, and has code generated, every time anew. So far GTAD generates the same code for externally ($ref) defined schemas; however, if In/Out direction is applied to those schemas, things may get broken (imagine the same structure $refed both from request and from response).

The proper way to fix it would probably be to accumulate all data models in memory before dumping them all to Printer. This will enable extending those models as they get encountered in the API description (e.g. add Out to already known In direction).

enumerations support

Support analysing enum attributes in parameter descriptions and replacing strings with enumeration types. To stay neutral to the target language this might need introducing one more renderer in gtad.yaml syntax.

Custom Mustache delimiters cannot be used in type definitions

This becomes a problem with languages like Julia - consider Vector{Int64} for an array of integers.

Support multiple inheritance via allOf

allOf is used throughout Matrix specs but doesn't involve multiple entries under it. Still it would be good to support multiple entries.

Support oneOf

As a part of moving towards OpenAPI 3

Switch to Mustache 4.0

Blank lines elimination allows for much more readable templates.

Parse "consumes" node

GTAD already parses "produces" OpenAPI key, "consumes" should be done in a similar way.

Write a README.md

Right now the project is completely unusable from outside because there is no documentation regarding even how to run it, let alone example config files, templates etc.

Types (and maybe some other items) translation in Analyzer

This might not be the prerequisite for Mustache-based generation to work, and probably we'd even get away with reproducing the whole 0.1 stuff without this, but that will definitely be needed for swagger-codegen Mustache files.

Support Matrix CS API 0.4.0

The new release of Matrix CS API has schemas that are compositions of another type and an additional field in the body schema. The following problems should be solved:

• A combination of $ref and a non-$ref entity under allOf. In the particular cases of CS API it can be remedied by pulling properties of the inner object to the level of the main structure (and that's actually a more transparent description). However, not all (theoretical) cases can be resolved that way, so GTAD has to learn that allOf is allowed to have non-$ref entries.
• Even if the properties are pulled as described above, GTAD fails to flatten the request body schema, instead creating a type definition that depends on itself (corresponding to struct Type : Type in C++). It should further pull out properties out of the schema and turn them into individual parameters of the request.

Fix CI failure on OS X

https://travis-ci.org/KitsuneRal/gtad/jobs/286150894

Don't even generate files for intermediate trivial schemas

The idea is to optimise out generation of files that contain a single alias definition. Just use the aliased type instead in referring files.

clang-format generated files

Have a sensible configuration file for clang-format in the source tree.

Doesn't mark an in schema as an out one too

Case in point: search.yaml where SearchJob::Result (the output schema) uses SearchJob::EventContext (the input schema). EventContext is not marked as out, leading to the missing FromJson converter.

Generate minimal stubs for data definitions

The criteria is successful compilation of all files generated for Matrix CS API.

TypeUsage -> TypeDefinition+TypeUsage

Translator::mapType() now has to make a copy of TypeUsage (with all its attributes), which is very inefficient given how many times types should be resolved while analysing API specs. The situation can be solved easily for non-parameterised types by pointing to the type instead of copying. For parameterised types it's more complicated since that pointer would need to be accompanied with type parameters. As noted in the comment in Translator::mapType(), TypeUsage should become a "smart handle" to an instance of a newly-made TypeDefinition structure that would own all the stuff TypeUsage now has, except innerTypes.

Rewire Printer to use mustache templates instead of hand-written code generation.

Drop all that SrcFormatting etc. code and replace it with a simple template engine wrapper. Might even work to drop Printer altogether, not sure yet though.

Don't alias types

There was an idea that trivial schemas should still be named if they are named in the API spec. That idea turned to be ineffective, as all such aliases are rather misleading than hinting. So remove this.

GTAD ignores $refs (even those not overridden)

As of 0.5, GTAD ignores $references unless the catch-all block in object is written in a very specific way excluding $ref'ed file name patterns. In case of a usual // it just intercepts such objects and uses a generic object (such as QJsonObject) for them. The proper way would be to distinguish between objects defined inline and $refs, having separate type maps for them (with $ref possibly falling back to object).

Proper processing of additionalProperties

Turns out that additionalProperties and properties can legitimately coexist, as the case of definitions/auth_data.yaml shows. GTAD only processes additionalProperties when there are no base types or other properties, which is incorrect (even overriding all properties with a single structure for additionalProperties would be better).

Update: there's more to this issue - GTAD doesn't process additionalProperties if they appear on the top level of a standalone schema (such as freshly added application-service/protocol_metadata.yaml).

Support body in response

As of f3d4f3b, GTAD can parse object schemas, including those in the response. Next steps are:

• Move the response structure from CallClass to Call in the model (thereby accepting cases when the two calls having the same name might have different response types - but having two calls with the same name is forbidden by the Swagger spec, anyway).
• Remove the code that checks the response for non-triviality and move the whole block that analyzes the result schema after a Call object creation.
• In Printer, use the (now useable) response structure to fill in correct context values
• In libqmatrixclient code, use the context values (the current blocks of code refer to variables that don't exist in Swagger's intermediate JSON).

Support all responses along with non-empty body

Even if libQMatrixClient doesn't need it (yet), it's still useful to parse definitions/error.yaml from CS API and use it to provide structures elaborating on the error.

Support passing at least the presence of security data to templates

Originally intended as a Matrix-specific patchwork, now there looks to be an option to supply at least the information about the fact of security definitions being specified in the call.

Support non-trivial responses with no body

No body error responses do not require any special treatment other than storing the HTTP code (which should normally be outside of GTAD-generated code anyway). So probably it boils down to parsing those codes without digging into returned schemas (a matter of #32).

Pass information on whether an object schema is used for input, or output, or in both directions

This is needed to make JSON converters optional because some of them (e.g. toJson for a generic Event object) might not even be well-formed, let alone needed.

The order of imports is unstable

Since the list of imports is internally an unordered_set, the order of entries in it may change between runs, even if input files are intact. This breaks build reproducibility and adds unnecessary diffs if generated files are kept under version control.

Support default values

Default parameter values have been ignored in API files so far; they should be taken and inserted into the model. Respectively, Printer should gain ability to dump default values so that Mustache templates could adorn them with additional text if necessary. This implies introducing type attributes that can work as Mustache partials - similar to what env variables already do.

Multitype support

Multitype is used in presence.yaml of Matrix CS API; having a QJsonObject instead of QVariant (or even a QString but we'll need parameter special-casing for that) in that particular case is not even correct because it's actually either a string or nothing; not an object.

Add necessity of cloning submodules for building gtad

Not being super familiar with norms of cpp development, it took me a bit of time to figure out that the errors cmake was spitting at me were because I hadn't cloned the project's submodules. Brief line in README would clarify this for people in the future.

Allow to intercept types specified through $ref or title

Sometimes one just wants to manually write a certain type (notably, Event). GTAD should allow to specify such type (as named, or reffed, in the API description) and provide a mapping for it in analyzer/types section

There's no way to stop inheriting higher-level values down to inner structures

Especially aggravating with doc-comments that are stored in description GTAD-list objects; if an inner entity doesn't have its own description, the upper-level description takes the place of the absentee, which is really not what's wanted. A fix would probably involve removing the actual list from the hierarchy to some wrapper section, so that trying to get to it on the lower level would stop on checking out an existing-but-empty object instead of no-object.