valyrie/vul

playing around with writing a compiler in ocaml -- extremely wip -- don't email me about it

vul's Issues

Move TODOs and notes into github issues

Move scattered TODOs and notes into a centralized location (github issues) for tracking purposes.

Track source file in token

Tokens do not currently track which file they originate, in. This makes recovering the text which produced a token impossible.

Output to stdout / stderr

Hook compiler/parser.ml into the cmdline tool

The current lexer/parser cannot be tested without first doing the plumbing work between it and the cli in vul.ml

The verbose flag (-v, --verbose) is unused

Currently, the verbose flag does nothing. This should increase the verbosity of the compiler, causing it to emit additional diagnostic information.

Cons reduction should be right-recursive

Signed numeric literals

Identifier token

Prefixed numbers starting with a _ should be invalid

Currently, the lexer accepts prefixed numbers that start with a _, such as 0h_A and 0y_. This behavior is erroneous, particularly in the second case, where it is possible to construct an integer literal without any digits.

Suffixed integer literals

Additional flexibility when specifying integer literals via suffixing of the form C053h would be a nice quality of life addition to the lexer.

Inconsistent code formatting

Currently, code formatting is not enforced in any consistent way -- it would be preferable to have some sort of autoformatter on commits, or at least a formatting discipline documented somewhere.

Read from standard input

A repl-mode could be provided by adding additional machinery to read from standard input

Compiler.Ast module

AST types should be lifted out into a dedicated module

Signed numeric literals

It is impossible to generate a signed numeric literal in the lexer. This should be remedied by permitted a prefix of - or + to a numeric literal during lexing.

_ as a wildcard identifier

The usage of _ as a wildcard identifier is desirable for destructing and pattern matching where the specific value of a field is irrelevant. This would make future implementation of a match ... with ... construct much nicer.

String literals

Byte and raw string literals

Support for special strings, such as python's raw string syntax, would be desirable.

At the moment, I've only figured out the following special strings:
raw strings of the form r"text", which do not parse escape sequences
non-terminated strings of the form b"text", which do not have a terminating \0 appended

Wildcard identifier

Symbolic constants

Symbolic constants of the form :symbol and :"symbol" would be quite nice to have when writing code for which the precise value of a constant doesn't matter, only that it is unique.

Symbolic literals

Move symbolic expression logic outside of Compiler.Ast

Ast.Expr is doing double-duty as both an AST representations and symbolic expressions. This should be broken up.

Parenthesis tokens, nonterminals

Orphaned structural token nonterminal

integer literals in non-decimal bases

add sexp and module artifacts

Artifacts are needed to print the sexp of a module, plus dump a module's symbol table

Implementing the parser

The parser, and anything downstream of it, have yet to be implemented; but once the lexer is complete enough, the parser should be the next target.

Cmdline interface does not properly report invalid switches

Cmdline interface, when passed an invalid switch, tries to interpret it as a filename. This is incorrect, and invalid switches should be detected and reported as such.

Output.t should handle missing directories

When created from a path, Output.t tries to open that path. If the path doesn't exist, it fail with a cryptic message about being unable to open the output file.

In this case, it would be good if the error raised an exception that explained the particular issue to the user, and perhaps gave them a hint as to how to fix it?

Cmdline interface needs hints

Currently, the cmdline interface prints quite cryptic errors when the user makes a mistake invoking the tool.
Ideally, vul.ml could print some sort of help text for these sorts of cases, guiding the user towards the correct usage (even if that's just telling them to run vul -h)

lexer logic does not properly strip whitespace in lex_token

looks in the dispatching logic of lex_token don't reference the version of the input which has been stripped of leading whitespace -- these leads to confusing, inconsistent, and buggy parses

print_expr functionality

Numeric literals

integer literal type suffixes

Remove old parser

explain atifact syntax

Quote special form

cons nonterminal

Implementation of basic sexp parser

Before more advanced work on the parser can happen, support for s-expressions needs to be finished.
More advanced syntax will be built on top of this.

Fix cons construction

Cons isn't being constructed in a way that enables differentiation between (x y) and (x (y))

Quoted identifiers

Unit type

The lexer needs to be able to generate the unit type ()

Quoted special identifier support

It is impossible to encode identifiers which contain special characters, such as (. Implementation of a lexical construct in the form of i"identifier" would remedy this.

Hexadecimal, Octal, and Unicode escape sequences

String escape sequences are incredibly limited, and it's impossible to encode characters via hexadecimal escape, octal escape, or unicode codepoint escapes.

Support for fractional literals in the lexer

Numeric literals with fractional components are entirely unsupported. This makes any potential work with floating point or other fractional number formats impossible.

Single line and block comments