ocamlpro/superbol-studio-oss

Open-Source part of SuperBOL Studio, including the Visual Studio Code extension and its LSP server

superbol-studio-oss's Issues

Check and fix auto-indent on free format

Check that auto-indent is correctly implemented on most COBOL files in free format.

Colorization of Copybooks

Currently, Copybooks are not colorized at all.

FR: distribute the language-server along with the .vsix

A slightly longer plan might be to ship the binary within the vsix extension file.

Originally posted by @nberth in #75 (comment)

This issue is about doing so, using platform-specific extensions.

More details are found at https://code.visualstudio.com/api/working-with-extensions/publishing-extension#platformspecific-extensions and of course - for the client side - https://github.com/microsoft/vscode-vsce.

There would be one version for each "target" you want plus one extra without a target (which would need an external install and possibly configuration to reach it as it is now).

Implement auto-indent on fixed format

Fixed format must take care not only of the left margin, but also not to push the end of line in the right margin.

Add Superbol: Restart Language Server command

We should be able to restart the language server when it gets stuck in a buggy state (similar to what is available on the OCaml Language Server).

Related: #81

Move auto-indent configuration to per-project file

This must done after having moved the LSP per-project configuration to a generic per-project file (#47)

Documentation for alpha-release users

Documentation should explain:

• the list of features of the extension
• the known limitations of the extension:
  • no support for GnuCOBOL's comma-separated arguments to functions
• how to install it from binary
• how to install it from sources
• how to use most common features

Incorrect folding

I tested all worldcities files (which are written in free format) on Visual Studio Code Version 1.85.2. Here are several common errors related to folding:

1. identification division folds the whole source code except for worldcities5 and worldcities6 (but I assume it is because of syntax errors). In all cases, it goes far beyond the identification division;
2. for the file section in data division, the fold button appears at the wrong position, one line after (on the next fd sentence, typically);
3. same thing for working-storage section, the fold button appears at the next definition when there're at least two definitions but it does not appear at all for a single declaration;
4. in worldcities0.cbl L83-92:

working-storage section.

01  command-file-name pic x(128) value 'worldcities0.sh'.
01  techtonics.
    03  pic x(128) value 'export FILES=$HOME/worldcityfiles'.
    03  pic x(128) value
        'cobc worldcities0.cbl -x -W -fdebugging-line'.
    03  pic x(128) value './worldcities0 $FILES/CA.txt'.
    03  pic x(128) value 'rm worldcities0'.
    03  pic x(128) value 'end'.

it looks like the continuation line 'cobc worldcities0.cbl -x -W -fdebugging-line'. (L89) is mistakenly taken for a sub-level variable, making a fold button appear at the 03 pic x(128) value just before (L88). This happens for continuations lines in general;

5. it may be related but in worldcities5.cbl L161:

           03  run-line.
               05  display-count pic z,zzz,zzzb.
               05  display-elapsed-seconds
                   redefines display-count pic zz,zz9.99b.
               05  display-message pic x(62).

there is a fold button on run-line, as expected and on display-elapsed-seconds as explained in the last point but also on display-count which folds redefines display-count pic zz,zz9.99b..

Remark

In case of "conflicts" between fold positions, there is a sudden increase of CPU usage. For instance, in the following code:

working-storage section.

01  command-file-name pic x(128) value 'worldcities0.sh'.
01  techtonics.
    03  pic x(128) value 'export FILES=$HOME/worldcityfiles'.
    03  pic x(128) value
        'cobc worldcities0.cbl -x -W -fdebugging-line'.
    03  pic x(128) value './worldcities0 $FILES/CA.txt'.
    03  pic x(128) value 'rm worldcities0'.
    03  pic x(128) value 'end'.

The fold button for working-storage section. (mistakenly) appears at 01 command-file-name pic x(128) value 'worldcities0.sh'. (L85) but if we define sub-levels for command-file-name:

working-storage section.

01  command-file-name pic x(128) value 'worldcities0.sh'.
    03  ...
    03  ...
    03  ...
01  techtonics.
    03  pic x(128) value 'export FILES=$HOME/worldcityfiles'.
    03  pic x(128) value
        'cobc worldcities0.cbl -x -W -fdebugging-line'.
    03  pic x(128) value './worldcities0 $FILES/CA.txt'.
    03  pic x(128) value 'rm worldcities0'.
    03  pic x(128) value 'end'.

Then a fold button should have also appeared for it, being in conflict with the former one. But such conflicts should not happen anyway and I assume it should be solved by correcting folding.

Implement autodetection of source format "as in GnuCOBOL"

Move file indentation out of `Cobol_indent`

          Now that I think about it, the code for effectively indenting a file should just not belong to `cobol_indent`; there, we should just stick to the indentation logic/algorithm. By that approach, `Superbol_free_lib` may just rely on `lsp` to apply the text editions.

(Apologies if there was an indent_file function already there; it should have been removed before).

Originally posted by @nberth in #97 (comment)

Move LSP configuration to a per-project file

We should create a library superbol_project for this per-project file that is common to LSP but also other possible tools (linter, builder, etc.).

Clarify and handle `CONSTANT`/`78`-level entries

the format for qualified names does not accept any literal in the position of the qualifiers (XX in the example)

... which is the reason for the error message. A qualification needs an identifier.
You may be able to improve that by postponing, in this case you can raise a better error message like

syntax error, unexpected constant name 'XX`, expecting qualification

But let's check what the standard says. We have 8.4.6 "scope of names" that specifies under "8.4.6.2.2 Scope of condition-names, constant-names, ..."

A constant-name [...] described with a GLOBAL clause is a global name. All data-names and screen-names subordinate to a global name are global names. A constant-name, data-name, [...} declared in a source element for an object definition, whether factory or instance, is a global name. [...]

When a condition-name, constant-name, data-name, [...] is not a global name, it is a local name.

The requirements governing the uniqueness of the names allocated by a single source element to be condition-names, constant-names, data-names, [...] are explained elsewhere in these specifications. (See 8.3.2.2, User-defined words.)

Which then specifies:

Within a source element, a given user-defined word may be used as only one type of user-defined word
with the following [...exception:] the same name may be used as any of the following types of user-defined words:

• constant-name
• data-name
• property-name
• record-key-name
• record-name

... but as 13.10.4.1 specifies

[...] the effect of specifying constant-name-1 in other than this entry is as if literal-1 [... is] written where constant-name-1 is written

I don't think the rules are clear. I'll ask the working group by mail.
Side note: Wikipedia says that

Association Française de Normalisation [AFNOR ... is a] member body for France at the International Organization for Standardization (ISO).

It is not clear to me which of those sub-organizations, if any, takes care of programming languages. I suggest you check with them as they don't have someone in the working group and may also not commented on the last versions.
You may be able to volunteer for that and by this also get access to both the documents of the working group (which were sadly moved to a non-public area which can only be accessed by ISO accounts (also given out to international member bodies) and to the working group itself.

Originally posted by @GitMensch in #53 (comment)

Lost ability to properly parse TOML files that do not end with a newline character

In these cases we get a Ez_toml.Types.Error (_, 0, 0) exception.

exceptions on unsaved files

It seems the extension tries to do something on the file system, while the active editor should get the content from the editor window (= also handles unsaved changes).

The easiest way to reproduce:

• [CTRL]+[N] to create a new editor, as expected it is set to "Plain Text"
• manually change to COBOL
• see errors

[Error - 20:24:27] Request textDocument/semanticTokens/full failed.
  Message: Caught exception: Not parsing copybook at untitled:Untitled-4
  Code: -32603 
[Error - 20:24:27] Request textDocument/foldingRange failed.
  Message: Caught exception: Not parsing copybook at untitled:Untitled-4
  Code: -32603 

Error at installation

After installing superbol in /usr/local/bin and adding the extension from the vsix file, I get the following error in VSC:

[Error - 11:12:04] Cobol Server client: couldn't create connection to server.
Error: Unsupported server configuration {
    "command": "",
    "args": [
        "lsp"
    ]
}
	at MYHOME/.vscode/extensions/ocamlpro.superbol-0.1.0/_dist/superbol_vscode_platform.bc.js:18089:33

Generalize `goto-definitions` and `find-references`

Extend the current support for definitions in WORKING-STORAGE to other sections of the DATA DIVISION.

Achieving this requires:

• Some preliminary refactoring and simplification of the parse-tree definitions (to be moved into a new cobol_ptree library);
• Rework of cobol_data and cobol_typeck;
• Maybe some minor adaptations in cobol_lsp.

REDEFINES keyword in LSP: problem with case sensitivity?

In several worldcities files such as worldcities3.cbl and worldcities4.cbl, there are identical errors on REDEFINES statements in WORKING-STORAGE SECTION. It seems to be related to case sensitivity.

For example, when using the VSCode extension for worldcities3.cbl line 150-153:

01  filler redefines continents.
    03  filler occurs 7.
        05  continent-name pic x(16).
        05  continent-code pic xx.

continents is highlighted in red with the following error message:

Unexpected target name 'continents' in REDEFINES clause for FILLER item; expected 'CONTINENTS'

Another example with worldcities4.cbl line 135-153:

01  start-edge redefines polygon.
    03  filler occurs 5.
        05  start-city-name pic x(32).
        05  start-latitude pic s9(3)v9(6).
        05  start-longitude pic s9(3)v9(6).
    03  filler.
        05  filler pic x(32).
        05  filler pic s9(3)v9(6).
        05  filler pic s9(3)v9(6). 
 
01  end-edge redefines polygon.
    03  filler.
        05  filler pic x(32).
        05  filler pic s9(3)v9(6).
        05  filler pic s9(3)v9(6). 
    03  filler occurs 5.
        05  end-city pic x(32).
        05  end-latitude pic s9(3)v9(6).
        05  end-longitude pic s9(3)v9(6).

with error:

Unexpected target name 'polygon' in REDEFINES clause for item 'start-edge'; expected 'POLYGON'

add nice logo for superbol-vscode-platform.vsix

currently none is specified so the default logo is used, the website https://get-superbol.com/ does have one (but it may not be what you someday end with and may not be the one that's good for the extension either...

recognize `WITH DEBUGGING MODE`

While I consider that a low-priority issue, this just occurred to me as a cool feature which may be relative easy to implement in the language server:

• If debugging mode is not active:
  • mark lines with debugging indicator as comment lines (currently always done)
  • mark "use for debugging" sections as comment lines (currently never done)
• if it isn't: swap around

Maybe there's a way to mark them different to a comment, ideally like #ifdef 0 parts in C?

Currently, that mode would only be active depending on the DEBUGGING MODE phrase of the SOURCE-COMPUTER paragraph

add artifacts (superbol-free and vsix) to github workflow

This way people can easily test work-in-progress, without the need to checkout + build themselves.
This is possibly something for either @nberth or @lefessan.

The REAMDE may reference this on the installation part.

debug extension out of date

as https://github.com/OCamlPro/superbol-vscode-debug has no issue tracker, I'm posting it here

• it should be updated from upstream (especially because the external term settings)
• the term settings need to be updated in the docs https://ocamlpro.github.io/superbol-studio-oss/sphinx/debug.html

additional (as this isn't tracked) the "TODO: add screenshots" should be handled somedayTM

and one note: I highly suggest to move the code-coverage part out of the extension (it likely would be language agnostic, no?) or, if it cannot do "more" than the existing ones after the creation part is mainly a build task, may consider to drop that in favor of an (optional?) dependency on one of the existing coverage extensions.
Language agnostic will also have the benefit that it can be used for mixed-language coverage (GnuCOBOL modules CALLing into C modules and for GC devs possibly also into libcob).

Documentation on Vscode marketplace

handling of conditional compilation (and `EXEC SQL`)

This sample program was shared on the IBM extension:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. TSTSQLCA.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  HOWMUCH PIC S9(9) COMP-5.
       >>define isDB2 as b'1'
       >>if isDB2
           exec sql include sqlca end-exec.
       >>end-if
       PROCEDURE DIVISION.
       MAIN.
           exec sql
             select count(*) into :HOWMUCH
             from SYSIBM.SYSDUMM1
           end-exec
           if SQLCODE not = 0 and SQLCODE not = +100
             display "Oups !"
           end-if
           .
       BYE.
           goback.

I'm "quite sure" that the LSP also has a problem here (it should error if the >>define line is removed, and pass otherwise (and of course, know that HOMMUCH is used within the SQL statement).

Add `check syntax` sub-command for `superbol-free`

Provide a web-only version of the extension by compiling the server into WASM

Performance issues with the Javascript+effects version seem to be too hard to deal with at the moment. Another way towards a "pure" web extension could be to compile the LSP server into WASM using wasm_of_ocaml (or another such compiler).

Originally posted by @nberth in #93 (comment)

(re)start LSP after change of `superbol.path`

Currently one needs to reload all related windows.

Indenter: respect FormattingOptions

When formatting a document there are many FormattingOptions that can be specified, and are currently ignored by the indenter.

We should support these (at least tabSize, and possibly insertSpaces, seem relevant — not sure if the ones related to whitespace and newline trimming must be supported or are handled by a separate text formatter).

Rigid order of paragraphs in configuration section

In the following program:

environment division.
configuration section.
repository. function all intrinsic. 
special-names. C01 is one-page.
input-output section.
file-control.

the keyword special-names produces an Invalid Syntax error with the VSCode extension. However, if we exchange the repository paragraph and the special-names one:

environment division.
configuration section.
special-names. C01 is one-page.
repository. function all intrinsic. 
input-output section.
file-control.

there is no error.

It seems to me that the LSP requires the special-names paragraph to occurs strictly after a repository paragraph. However, both are accepted by GnuCOBOL.

Fix left-alignment of IBM test files

cf e986871#commitcomment-137790264

hard dependency on tamasfe.even-better-toml

As far as I see that's for better supporting the superbol configuration - Could this extension be "suggested" only instead of the hard dependency? vscode declined to install the superbol extension because the toml one was not installed.

bug: words are "recognized" with the default "C style"

While the COBOL word recognition was added to the debugging part (registerEvaluatableExpressionProvider) it is missing in the plain editor. I don't know if this is about the tm syntax or an lsp part, but if I put the cursor into SOME-VAR, then the complete word should be highlighted, not only the part before/after the hyphen.

Fix typos in debug extension readme and related docs

pointing to https://ocamlpro.com/superbol/ which requires authentication

note: the debug extension readme also has some typos, i suggest pasting that to https://www.deepl.com/write to get those fixed; but https://ocamlpro.github.io/superbol-studio-oss/sphinx/_sources/debug.rst.txt (I did not found the place to send a PR for fixing that) has more typos

superbol-free-linux-x64: version GLIBC_2.28 / GLIBC_2.35 not found

First test on the extension, did not work. Reason is seen in the language server output pane:

Testing with CentOS7

extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64: /usr/lib64/libm.so.6: version `GLIBC_2.29' not found (required by extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64)
extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64: /usr/lib64/libm.so.6: version `GLIBC_2.35' not found (required by extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64: /usr/lib64/libc.so.6: version `GLIBC_2.34' not found (required by extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64)
extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64: /usr/lib64/libc.so.6: version `GLIBC_2.28' not found (required by extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64)

Testing with RockyLinux 9.2:

extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64: /usr/lib64/libm.so.6: version `GLIBC_2.35' not found (required by extensions/ocamlpro.superbol-0.1.0/_dist/superbol-free-linux-x64)

Testing with Debian:

extensions/ocamlpro.superbol-0.1.0-linux-x64/_dist/superbol-free-linux-x64: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by extensions/ocamlpro.superbol-0.1.0-linux-x64/_dist/superbol-free-linux-x64)
extensions/ocamlpro.superbol-0.1.0-linux-x64/_dist/superbol-free-linux-x64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by extensions/ocamlpro.superbol-0.1.0-linux-x64/_dist/superbol-free-linux-x64)

Can you create the distributed binary (CI?) on a CentOS7 environment? This seems to be compatible to nearly all environments in use.

LSP: add pendant for cobc's `-fmf-comment` and auto-enable it for `$ SET`

This will fix the syntax error currently raised in: test/testsuite/microfocus/www.csis.ul.ie/Indexed/DirectReadIdx.CBL

Consider ExtensionContext.storageUri for private data

While running inside VScode, we could use ExtensionContext.storageUri to store data instead of having extra _superbol directories everywhere.

LSP: Notify client with configuration summary whenever a document is opened

This makes me realize the LSP server is not verbose enough as to what its current configuration is. Maybe some notification with a summary of relevant configurations could be shown when a file is opened (or the server is restarted).

Originally posted by @nberth in #86 (comment)

syntaxes/COBOL.tmLanguage.json

Please note, this this grammar is not for the GnuCOBOL dialect but is for Micro Focus/OpenText based compilers.

It includes many reserved words that are not appropriate for GnuCOBOL and will cause inconsistencies with the lsp.

provide outline view

so far that is empty "no symbols found", but the LSP surely has information for that

Improve completion

Currently, the LSP does not help for completion of words:

• First version: add keyword context-free completion
• Second version: add context sensitive completion (i.e. depending on the location)

Disable auto-indent in fixed format

For alpha release, auto-indent should be disabled on fixed format, and only in the next release shall it be implemented (#45)

Fix wrong list of source formats available for `sourceFormat` client configuration.

At the moment this shows a list of dialects.
Changes should be located in src/lsp/superbol_free_lib/vscode_extrension.ml.

Improve accounting of columns in the presence of tabulation characters

At the moment, any tabulation character counts as one towards the right margin in fixed format code. Better accounting is needed to properly detect the portions of text that exceed the right margin (and to take tab-width configuration into account).

Handle debugging-mode floating indicator (`>>D`)

In the GnuCOBOL doc, a -fdebugging-line option is defined for debugging. The D indicator (in the 7th column) is recognised by the LSP but not the >>D directive (which exists both in fixed and free format but according to GnuCOBOL sources it is strictly limited to COB2002 for free format). Such directives appear in the GnuCOBOL test suite.

I see that in the preprocessor's parsing of compiler directives, we parse expressions beginning by >>, followed by an optional space and finishing by some words used for compiler directives. However, >>D is different as it cannot have a space character in-between. For the fixed format, I thought about adding a case:

and fixed_cdir_line marker state          (* `>>`-prefixed compiler directive *)
  = parse
  | 'D'
      {
        fixed_debug_line state lexbuf
      }
  | ' '? cdir_word_suffix
      {
        Src_lexing.cdir_word ~ktkd:gobble_line ~knom:fixed_nominal
          ~marker (Src_lexing.flush_continued state) lexbuf
      }
  | epsilon
      {
        newline_or_eof (Src_lexing.flush_continued state) lexbuf
      }

But it would probably be a too ad-hoc fix.

Bundle COB_CONFIG_DIR

So that dialects work.

debugging extension runs npm compile which fails

 *  Executing task: npm run compile 

> [email protected] compile
> make compile

make: *** No rule to make target 'compile'.  Stop.

 *  The terminal process "/bin/bash '-c', 'npm run compile'" terminated with exit code: 2. 
 *  Terminal will be reused by tasks, press any key to close it. 

Minor issue in RST generation with `command_texi2rst.ml`

If I understand correctly, the program in src/lsp/superbol_free_lib/command_texi2rst.ml generates the sphinx/commands.rst file. It is a minor issue but the generated file generate warnings because it does not perfectly follow the Sphinx syntax:

superbol-studio-oss/sphinx/commands.rst:115: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:177: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:180: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:183: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:186: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:258: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:300: WARNING: Bullet list ends without a blank line; unexpected unindent.
superbol-studio-oss/sphinx/commands.rst:327: WARNING: Inline emphasis start-string without end-string.
superbol-studio-oss/sphinx/commands.rst:329: WARNING: Inline emphasis start-string without end-string.
superbol-studio-oss/sphinx/commands.rst:333: WARNING: Inline emphasis start-string without end-string.
[...]

It seems to me caused by misplaced * symbols, wrong indentation and such.

SourceFormat as an enum

          this and the entries below should use an enum type, possibly with an additional string option

Originally posted by @GitMensch in #72 (comment)

Planning: when (if at all) should the LSP at least parse NIST85 without errors?

NIST85 has a lot of obsolete stuff in, most important is the line continuation including word continuation.

Then it has things that some preparsers stumble over (but that is also commonly used "somehow" in production code): fixed format with "comment" text in line 1-6, 73+ and minor ones like comment paragraphs (AUTHOR and friends).

It also has a lot of special PICTURE clauses and strange formatted code.

As I don't know what the state is: Is the LSP already able to parser all of NIST85 without errors?

... and a special thing: can it handle the programs that GnuCOBOL compiles and executes in its testsuite?
For the later one may use autofonce and set COBC to a shell that first sends the LSP to those files (expecting no output) and afterwards cobc. ... and of course something similar may be done with the NIST85 autofonce'd version (or again by exporting COBC to a shell that executes both).

`make build-deps` does not install `autofonce_lib`

which is required to run make superbol-free

clarification needed: What is the state of the debugger?

I can create a debug configuration but using it seems to not do anything at all and the debug console has zero output.