eclipse-langium/langium

Next-gen language engineering / DSL framework

langium's Issues

[cli] Add a 'generate' command

The current CLI functionality should be moved into a command so it's invoked with langium generate instead of just langium.

Rationale:

• The command is more descriptive of what it does.
• It enables us to add more commands in the future if we find other use cases for the CLI.

Measure the gain of the stream API

I'm sure that lazy operations are faster than eager operations. But adding a new API is costly (in the sense of complexity, maintenance, ...), so I asked myself what we gain with the stream API (in numbers).

We should measure the gain somehow. Maybe adding the stream API is premature optimization. Making Langium faster is always possible afterwards.

Rename 'value' to 'ref' in Reference

I propose to rename the value property of Reference to ref.

Rationale: this property will be accessed very, very often in user code. It will be accessed for every single cross-reference. The name ref is shorter and more descriptive because it shows that the respective AST path is a reference and not a plain value.

Rethink DI

CAUTION: There are still some caveats in the types, I will fix them but already wanted to showcase you the general idea of the API here.

I still feel the urge of rethinking our DI spike and the use of classes. Here is my suggestion on how to wire the Langium parts while maintaining the following features:

• dependency injection (achieved by binding a curried function to an instance of Langium)
• cyclic references are supported (e.g. linking, scoping and typesystem reference each other)
• 100% transparency/documentation of the langium API in one place
• allowing the user to extend the API/adding new objects that will be injected and can be used everywhere

Top level framework API

Here we have an overview what is Langium capable of. Provided the types are well documented in this central location, the user will get a good overview.

export type Langium = {
    linkingProvider: LinkingProvider,
    scopingProvider: ScopingProvider,
    validationProvider: ValidationProvider
}

// just examples
export type LinkingProvider = () => void
export type ScopingProvider = () => void
export type ValidationProvider = () => void

Functional dependency injection

These generic types and one generic function for injection is basically all we need.

// I think this self recursive type still needs to be tweaked a bit.
// We need to ensure that we only have types `Inject<..., ...>`,
// so `K in keyof Langium` might not be sufficient.
export type LangiumAPI<L extends LangiumAPI<L>> = {
    [K in keyof Langium]: Inject<L, Langium[K]>
}

export type Inject<L extends LangiumAPI<L>, T> = (api: L) => T

export function createLangium<L extends LangiumAPI<L>> (api: L): Langium {
    return Object.fromEntries(
        Object.entries(api).map(([key, value]) => [key, value(api)])
    ) as Langium;
}

Great defaults which can be reused

By exactly knowing our domain, we will provide one or more defaults for our users. This could be even an optional lib to keep the Langium core small.

export function defaultLangiumAPI<L extends LangiumAPI<L>>() { // TODO: declare self-recursive type
    return {
        linkingProvider: (api: L) => () => {},
        scopingProvider: (api: L) => () => {},
        validationProvider: (api: L) => () => {}
    };
}

Typical use-site

The user typically needs to customize the defaults when developing non-trivial languages.

interface MyLangiumAPI extends LangiumAPI<MyLangiumAPI> {
    myCustomProvider: MyCustomProvider
}

type MyCustomProvider = (api: MyLangiumAPI) => unknown

 // look ma, custom api is injected into the default ScopingProvider API
const scopingProvider = (api: MyLangiumAPI) => () => {
    const { myCustomProvider } = api;
    // ...
}

// a user defined feature
const myCustomProvider = (api: MyLangiumAPI) => () => {
    // ...
}

const langium = createLangium<MyLangiumAPI>({
    ...defaultLangiumAPI(), // use the defaults
    scopingProvider, // re-define the default scoping provider
    myCustomProvider // add a custom provider
});

It would be great if we could consider such an JS-like approach instead of just transferring to Langium what we did in Xtext.

---

☝️ please take a look @msujew @spoenemann

Use .langium as grammar file extension

We should use a file extension that avoids collisions with other tools.

It would also be nice to move the grammar file into the src folder.

Use Lerna to manage internal dependencies

Story

As a Langium maintainer I want to have fast development turnaround cycles.

For example I don't want to release a new version of the core to make changes available to the vscode extension.

Acceptance criteria (to be discussed)

• Lerna was added to Langium
• The vscode-extension module depends on the core

Generate an AST visitor in addition to the AST

Often times I want to just go through some objects of the AST (e.g. for validation) without streaming the whole AST tree. For this, we should think about generating an AST visitor which we can use to override certain visitor methods.

Consider making datatype rules explicit

Xtext detects datatype rules (parser rules that produce data types like strings instead of AST nodes) and marks them with a special color in the editor. We should consider using an explicit keyword to mark such rules instead, e.g. value.

Centralize template string functionality

Story

As a maintainer I want to keep application logic in a central place. The vscode extension should use core template string functionality to calculate indentation of template strings.

Acceptance criteria

• the entry point of the core package only exports the tagged template s.
• the core/generator package exports the internal functionality to compute template string indentation
• the vscode-extension package uses the internal indentation functionality of the sub-module core/generator

Only add extra tooling if there is a use case

Story

As an open source project we should reduce the onboarding barrier by using standard tools that come out of the box. Only add extra tools if there is a real use-case.

npm got many updates during the last years and is known now to do the same as yarn. Also non-functional improvements happened, like an increase in speed and caching of previously downloaded dependencies.

Acceptance criteria

• remove yarn.lock and add package-lock.json

Use modern ES features and target older platforms

Story

As a developer, I want to use the newest ECMAScript features and support older environments at the same time.

Acceptance criteria

• The tsconfig lib is set to an actual ECMAScript version or even ESNext if that is possible (we need to ensure that the bundler/transpiler keep in line with the polyfills over time when updating the TypeScript dependency).
• We use Microbundle to produce ES5 or ES6 artifacts. The package json contains sections for targets. We need to decide on all or some of the targets umd, esm (es5/6), esm (modern), cjs, ...

Questions: does the package.json need to contain an engines section. If we want to use recursive fs.mkdir, for example

{
  "engines": {
    "node": ">=10.12.0"
  },
}

Add langium-vscode as recommended extension

The generator-langium should provide a list of recommended extensions that includes langium-vscode.

Make entry rule explicit

I don't like the fact that Xtext assumes the first rule to be the entry rule. I would prefer to make this explicit in Langium, e.g. with a keyword main or root. We could offer a code action that adds this keyword to the first rule to simplify migration from Xtext.

Move tests to separate folder

Currently test files are included in the src folder (in __test__ subfolders). We decided to have them in a separate test folder.

Update dependencies

Hi @msujew, can we update our dependencies, especially Chevrotain, or is there a reason we are 2 major versions behind?
Thx!

Improve order of proposals

We should order the completion proposals by type: first cross-references, then word-keywords (/\w+/), then non-word-keywords (delimiters).

As a second ordering key we could apply alphabetical ordering.

Commit VS Code setting to use workspace TS version

Story

Some IDEs might use a global or IDE specific TypeScript version by default. As a maintainer I want to ensure that all committers use the same TypeScript version.

Acceptance criteria

• .vscode/settings.json contains a settings for using the workspace TS version

Dependencies

• #11 we need to hoist common dependencies first because the vs code setting references node_modules

${workspace_root}/.vscode/settings.json:

{
    "typescript.tsdk": "node_modules/typescript/lib"
}

Rename the default branch from `master` to `main`

See https://github.blog/changelog/2020-10-01-the-default-branch-for-newly-created-repositories-is-now-main/

I know, this seems not to be important compared to all the business features we want to implement. But it would be nice to follow the new GitHub standard.

Add template-string to Langium core

Story

As maintainers we want to bundle standalone functionality within the core library.

Acceptance criteria

• The code of the project danieldietrich/rich-string is copied into core/src/generator, including the tests

Dependencies

• #4 Existing unit tests are moved to core
• #5 Code compiles, requires new ES features

Parser generator bug for alternatives with cardinality

The following rule contains an alternative wrapped in a * cardinality

Model:
	(persons+=Person | greetings+=Greeting)*;

However, the parser generator only contains the or instruction, completely ignoring the cardinality symbol.

Transpile source during build to match target

Currently, the microbundle output isn't conform to our target. I'm not sure if BabelJS is required here. Please note that TypeScript can be compiled using BabelJS without using tsc.

Make LangiumModule and the directory structure congruent

some services need to be moved from ./services/ to somewhere else

Check if npm workspaces are a Lerna substitute

It would be nice if we do not need yarn as extra dependency but if npm workspaces do not work for us, we could use yarn to slove the Lerna project setup problems.

https://docs.npmjs.com/cli/v7/using-npm/workspaces

/cc @spoenemann @msujew

Prefix is not considered in completion proposals

The prefix is not considered: when I start writing something where a cross-reference or keyword is expected, all proposals are shown regardless of the current prefix.

Langium version: 0.1.0-next.00ebfd9
Package name: langium

Steps To Reproduce

1. Open langium grammar
2. Put cursor in the middle of a cross-reference
3. ctrl+space

The current behavior

All proposals are shown.

The expected behavior

Proposals that don't match the current prefix should be omitted.

Make service groups consistent with directory structure

We should make the structure of service API groups consistent with the directory structure. In particular, we should create a group service with subgroups for the LSP services.

While we're at it, we could think about another name for the LSP services to avoid using the term "service" too much (we also use it for injected service classes).

It is not totally clear in which group to put the generated services. I'd propose a group generated.

Serialize grammar JSON as proper tree

Currently grammars are serialized to JSON with a generic decycling approach. This should be changed so the JSON tree corresponds exactly to the AST structure. This could be generalized to serialize all Langium ASTs.

Which config could be moved to the parent dir?

• eslint searches the configuration in the parent dir
• ./tsconfig.json could extend ../tsconfig.json
• etc.

This would only make sense if all projects have roughly the same shape, e.g. being node projects.

Parser throws TypeError in domainmodel example

When typing import in domain model example language a TypeError is logged:

TypeError: Cannot read property 'object' of undefined
    at Parser.construct (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:175:26)
    at Parser.<anonymous> (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:84:31)
    at ChevrotainWrapper.invokeRuleWithTry (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:118:33)
    at ChevrotainWrapper.wrappedGrammarRule [as Domainmodel] (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:132:38)
    at Parser.parse (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:57:38)
    at DefaultDocumentBuilder.build (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/documents/document-builder.js:17:41)
    at /Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/service/language-server.js:39:25
    at CallbackList.invoke (/Users/dhuebner/git/langium/node_modules/vscode-jsonrpc/lib/common/events.js:55:39)
    at Emitter.fire (/Users/dhuebner/git/langium/node_modules/vscode-jsonrpc/lib/common/events.js:117:36)
    at /Users/dhuebner/git/langium/node_modules/vscode-languageserver/lib/common/server.js:146:38
Parser exception thrown! TypeError: Cannot read property 'object' of undefined
    at Parser.assign (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:203:34)
    at Parser.subrule (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:133:22)
    at ChevrotainWrapper.<anonymous> (/Users/dhuebner/git/langium/examples/domainmodel/out/language-server/generated/parser.js:43:22)
    at ChevrotainWrapper.doSingleRepetition (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:386:16)
    at ChevrotainWrapper.manyInternalLogic (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:319:29)
    at ChevrotainWrapper.manyInternal (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:296:21)
    at ChevrotainWrapper.many (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_api.js:37:21)
    at ChevrotainWrapper.wrapMany (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:273:14)
    at Parser.many (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:96:22)
    at /Users/dhuebner/git/langium/examples/domainmodel/out/language-server/generated/parser.js:42:18
TypeError: Cannot read property 'object' of undefined
    at Parser.construct (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:175:26)
    at Parser.<anonymous> (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:84:31)
    at ChevrotainWrapper.invokeRuleWithTry (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:118:33)
    at ChevrotainWrapper.wrappedGrammarRule [as Domainmodel] (/Users/dhuebner/git/langium/node_modules/chevrotain/lib/src/parse/parser/traits/recognizer_engine.js:132:38)
    at Parser.parse (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/parser/langium-parser.js:57:38)
    at DefaultDocumentBuilder.build (/Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/documents/document-builder.js:17:41)
    at /Users/dhuebner/git/langium/examples/domainmodel/node_modules/langium/lib/service/language-server.js:39:25
    at CallbackList.invoke (/Users/dhuebner/git/langium/node_modules/vscode-jsonrpc/lib/common/events.js:55:39)
    at Emitter.fire (/Users/dhuebner/git/langium/node_modules/vscode-jsonrpc/lib/common/events.js:117:36)
    at /Users/dhuebner/git/langium/node_modules/vscode-languageserver/lib/common/server.js:146:38

Langium version: 0.0.0
Package name: langium

Steps To Reproduce

1. Start DomainModel example as extension
2. open a new .dmodel file
3. Type import (error is also logged on contant assist)

Link to code example: will be merged soon see /pull/94

The current behavior

Parser throws an error

The expected behavior

Parser should stop doing it

Enhance Reference interface to hold a node instead of string

In some cases it is necessary to get a CstNode for a particular reference (e.g. find reference)

Current state:
AstNodeReference -> Reference

Reference holds readonly $refName: string; , refName in this case is the token string which represent a cross reference, we can instead store this information as CstNode like this:

export interface Reference<T extends AstNode = AstNode> {
    readonly ref?: T;
    readonly $refNode: CstNode;
    readonly $refString: string;
}

To get the string just call $refNode.text

Create first example project

Would like to start with Xtext domain model example:

• Domain model

Support configuration in external file

We should support configuration via an external json file additionally to the embedded config in package.json. By doing that we can easily provide a json schema in langium-vscode so users get editor support for the config (is that possible for package.json extensions?)

We should have a default for the file name, but also allow passing a config file path as argument.

Candidates for default name:

• langiumconfig.json
• langium-config.json
• .langiumconfig.json
• .langium-config.json
• .langiumrc.json

I like langium-config.json most. IMO .langiumrc.json is not really applicable because rc means "run commands".

Improve Stream.concat behavior

As stated in here, there is still some work left to do regarding the stream.concat implementation.

Remove Chevrotain API from public Langium API

... by composing parsers instead of inheriting from the Chevrotain parser.

Class constructor LangiumParser cannot be invoked without 'new'

Langium version: 0.0.0
Package name: langium

Steps To Reproduce

1. generate domainmodel example
2. start extension
3. open new *.dmodel file
4. language server crashes

[Info  - 2:49:46 PM] Connection to server got closed. Server will restart.
Debugger listening on ws://127.0.0.1:6009/646ca63e-0fb7-4653-912b-6b90cf296d93
For help, see: https://nodejs.org/en/docs/inspector
/Users/dhuebner/git/langium/examples/domainmodel/out/language-server/generated/parser.js:38
        var _this = _super.call(this, tokens, services) || this;
                           ^

TypeError: Class constructor LangiumParser cannot be invoked without 'new'
    at new Parser (/Users/dhuebner/git/langium/examples/domainmodel/out/language-server/generated/parser.js:38:28)
    at Parser (/Users/dhuebner/git/langium/examples/domainmodel/out/language-server/generated/module.js:8:42)
    at _resolve (/Users/dhuebner/git/langium/packages/langium/lib/dependency-injection.js:71:53)
    at Object.get (/Users/dhuebner/git/langium/packages/langium/lib/dependency-injection.js:37:29)
    at new DefaultDocumentBuilder (/Users/dhuebner/git/langium/packages/langium/lib/documents/document-builder.js:12:32)
    at DocumentBuilder (/Users/dhuebner/git/langium/packages/langium/lib/default-module.js:33:44)
    at _resolve (/Users/dhuebner/git/langium/packages/langium/lib/dependency-injection.js:71:53)
    at Object.get (/Users/dhuebner/git/langium/packages/langium/lib/dependency-injection.js:37:29)
    at Object.startLanguageServer (/Users/dhuebner/git/langium/packages/langium/lib/service/language-server.js:37:48)
    at Object.<anonymous> (/Users/dhuebner/git/langium/examples/domainmodel/out/language-server/main.js:8:11)
[Error - 2:49:47 PM] Connection to server got closed. Server will not be restarted.

Set up continuous integration (CI) using GitHub Actions

Story

As a committer I want to ensure that PRs don't break the code base, including tests.

Acceptance criteria

• We use GitHub Actions to build and test the projects on push or PR to branch main.

The triggers could look like this:

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

---

Dependencies

Probably we need to update our GitHub plan until we will be able to use GitHub Actions.

Consider disabling enum rules

Instead of implementing enum rules as TypeScript rules, we could consider using datatype rules instead (see #69). We could have a special detection in the AST generator that creates a type declaration for a datatype rule that consists of an alternative of several keywords.

Example:

value MyEnumRule: 'value1' | 'value2' | 'value3';

could lead to this TypeScript declaration:

type MyEnumRule = 'value1' | 'value2' | 'value3'

Generate header comment for generated files

/******************************************************************************
 * This file was generated by langium-cli {version}.
 * DO NOT EDIT MANUALLY!
 ******************************************************************************/

Add a unit test library to our projects

Story

As a maintainer I want to ensure with unit tests that ongoing development in a team does not break existing functionality.

Acceptance criteria

• jest is integrated in the build process
  whitebox testing of internal functions.
• we measure the code coverage
• we use ts-jest to support TypeScript in our tests

Note: Adding tests is postponed to #7

Add a VS Code Extension

Story

As a user of I want to integrate Langium in my VS Code development environment.

Acceptance criteria

• We created a new project vscode-langium
• The project contains a VS Code extension with the id langium
• The extension contains Daniel's generator support spike. Later we will add a language service for the Langium grammar.

Revert core/tsconfig option downlevelIteration

That change was made in #28 because of a compiler problem. We need to investigate & fix it.

Add ESLint to our projects

Story

As a maintainer of the project I want to ensure that we keep up the quality of our project by automatically checking certain rules.

Acceptance criteria

• We created a central eslint config that is used by all per projects. (update: there exists a concept for shared configs but that's a different story)
• The linter is running on every project build.

Note: All developers need to integrate eslint in their development environment by using vs code extensions: eslint (and optionally error lense).

Grammar access is always generated with all imports even if unnecessary

Langium version: 0.0.0
Package name: langium-cli

Steps To Reproduce

1. Create a language without any actions.
2. Watch the linter/compiler error on an unused import.

The current behavior

The Action import is always generated in the grammar-access.ts even when it is not used.

The expected behavior

Only imports that are actually used are generated in the grammar-access.ts.

Add CONTRIBUTING.md

Story

As a maintainer I want to make certain design guidelines transparent to all contributors.

Acceptance criteria

• CONTRIBUTING.md was created at the root of the project

Notes

Some guidelines are:

• the entry point, e.g. src/index.ts, defines the public API. More specifically, it contains only export directives but not code.
• consumers of the library only use public API, e.g. import { s } from 'langium'
• when we use internal dependencies, we are allowed to import non-public API, like import { indent } from 'langium/generator/smart-string'. Tree-shaking helps us to bundle only code we use.

Issue templates and labels

Now that more discussion around Langium is happening in the issues, we should start thinking about how we would triage them.

Therefore, we should probably start using issue template and appropriate labels for issues and PRs.

Yeoman generator not working on windows correctly

Langium version: 0.1.0-next.00ebfd9
Package name: generator-langium

Steps To Reproduce

1. Run the yeoman generator for langium on a Windows machine.
2. Observe the error message.

The current behavior

Running the generator currently results in the following error message:

npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path C:\Users\Mark\Source\Repos\langium-test/package.json
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\User\Source\Repos\langium-test\package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\User\AppData\Local\npm-cache\_logs\2021-06-16T13_04_30_619Z-debug.log
Error langium

Command failed with exit code 4294963238: npm install --prefix C:\Users\User\Source\Repos\langium-test\hello-world

The expected behavior

No error message and everything working smoothly ;)

Switch to ES6 target

We currently have set ES5 as target for the TypeScript compiler. NodeJS supports almost all of ES6 since version 10 (and most of it in earlier versions as well). We should switch the compiler target to get more readable code and better performance.

Make ReferenceFinder work for references

Right now one can only find references for a declaration.
As soon as Declaration finder is merged, we can use it to get a declaration for a reference and than look up references for it.

Generate more specific $container references

We could analyze for every generated AST node in which other node types they can occur, and add a corresponding more specific $container reference overriding the generic $container?: AstNode. This would make it easier to work with the ASTs.

Configure eslint to check the license header

Eslint should check that a valid license header is on top of the ts file

• should we check other files as well?
• how will generated files be handled?

Make LangiumParser stateless

The LangiumParser implementation currently holds state in class fields. We should refactor this to a stateless service, or if that is not possible change to factory injection.