ratstail91/toy

A toy programming language.

Toy

This is the Toy programming language interpreter, written in C.

Special thanks to http://craftinginterpreters.com/ for their fantastic book that set me on this path.

Nifty Features

• Simple C-like syntax
• Bytecode intermediate compilation
• Optional, but robust type system (including opaque for arbitrary data)
• Functions and types are first-class citizens
• Import external libraries
• Fancy slice notation for strings, arrays and dictionaries
• Can re-direct output, error and assertion failure messages
• Open source under the zlib license

Building

For Windows(mingw32 & cygwin), Linux and MacOS, simply run make in the root directory.

For Windows(MSVC), Visual Studio project files are included.

Note: MacOS and Windows(MSVC) are not officially supported, but we'll do our best!

Tools

Run make install-tools to install a number of tools, including:

• VSCode syntax highlighting

Syntax

import standard; //for a bunch of utility functions

print "Hello world"; //"print" is a keyword

var msg = "foobar"; //declare a variable like this

assert true, "This message won't be seen"; //assert is another keyword

//-------------------------

fn makeCounter() { //declare a function like this
	var total: int = 0; //declare a variable with a type like this

	fn counter(): int { //declare a return type like this
		return ++total;
	}

	return counter; //closures are explicitly supported
}

var tally = makeCounter();

print tally(); //1
print tally(); //2
print tally(); //3

License

This source code is covered by the zlib license (see LICENSE.md).

Patrons via Patreon

• Seth A. Robinson

toy's Issues

Segfault reports and other issues

Don't let it be said that I don't listen to feedback. These will be necessary fixes for 1.0.

TODO

Slashes seem to be badly done, and back/forward slashes are incorrect.

Arrow keys tend to lead to unknown input values.

Forgive the dirty text, but null appears to be an error as a parameter value.

Likely related to 32-bit integers.

This was unexpected.

Will need to switch off the color codes on windows.

Inconsistent behaviour?

FIXED

"type" is not a function, but something is going on here.

Compiler error here. See #57

I don't know what's going on here. This is now caught by another bugfix.

Bad length. This now appears to be correct.

Bounds check?

This might be correct, but is related to slashes being backwards.

Bad input.

Bounds check.

Another bad input.

Probably off-by-one error.

The language only supports ascii, as far as I can tell.

This may need to be worked on at some point in the future, especially if it gains traction - for now, it's fine though.

Print ignores functions

The built-in print does not print anything at all (not even a line break) if the argument is of
type function. I expected it to either print a debug representation (like <function>) or throw an error that there is no suitable representation.

Example:

print("alpha");
print(x => x);
print("omega");

This prints:

alpha
omega

String Reference Counting for Toy strings/identifiers

One of the possible solutions for the performance and memory usage issues could be to replace the current string usage patterns with a refstring library - that is, a library that shares string pointers via refcounting.

typedef struct refString {
    char* cstring; //ref pointer is equal to a char**
    int refcount;
    int length;
} refString;

The aspects needed would be:

• refstrings are immutable
• custom allocator hook
• createRefString(char* cstring)
• deleteRefString(refString*)
• lengthRefString(refString*)
• copyRefString(refString*)
• copyCString(char* cstring)
• toCString(refString*)

Forgot to write tests for importing libraries

typeof nested values in compounds loses const-ness

print typeof about;
    <[<string const>:<any const>] const>
print typeof about["author"];
    <string>

Here, the nested value "author" loses the const-ness, because types are stored separately from literals proper.

AST Inspector Needed

I need to write an AST inspector, so I can print out ASTs when debugging.

This would be a separate tool, but would definitely help in locating bugs. Currently GDB is my only way to find bugs in the ASTs :/

I can't figure out how to get the shared library to build for Linux

Compound Library Needed

A library for manipulating compounds - arrays and dictionaries. Exactly what is in it comes from the most commonly used JS functions...

Difficult to describe bug found - possible dot notation issue

var parent: int = null;
fn _getParent(self) { return 42; }
fn _init(self) { parent = self.getParent(); }

null.init(); //Undeclared variable "getParent"
_init(null); //Undeclared variable "getParent"

fn _quit(self) { parent = _getParent(self); }
null.quit();
print parent; //42

The exact cause is not known - it's possible that dotted functions aren't having sub-functions dotted.

64 bit int and floats

The union types are no longer a feature - refactor the type system to be simpler without it.

The type system was planned from the beginning to have union-ish types, but that's no longer the case (they're too bloody hard to implement for too small a gain).

I can simplify things now and just have an enum representing the types.

Rewrite the timer library

Bytecode longer than 65535 bytes will break the JUMP opcodes

All control flow uses the jump opcodes, but jumps are absolute, so in the case that the resulting code region is too long, the behaviour is undefined.

Printing the rest argument's type has a bad output

Ternary Operator Needed

loadAndRunScript() and loadAndRunScriptBytecode() in runner

Remove the _index() function?

It's annoying me that it's there, as it doesn't really fill a need. It's also obscure as to what it's for.

For anyone wondering, it gets called by the indexing notation:

array[0] = 1;

It was a pain to code, and might be hiding subtle bugs due to how it works, and how many ways it can be used.

It's gotta go before 1.0.

Calling type() on a const segfaults

To reproduce:

var a: int const = 42;
a.type();

Result: Segfault

Expected Result: Not that

A lot of error cases leak memory

Find and patch these as much as possible.

Add a keyword for close the REPL

Perhaps using quit or exit may do the trick.

Infinite recursion causes a segfault

fn f() {
  return f();
}

f(); //segfault

TOKEN_EOF printed twice in verbose mode

This is because of the overarching structure of the program, and the need to re-probe the tokens after completion.

Since this is is such a minor display bug used for debugging, I'm marking this as wontfix.

Missing advance after while?

Hi,

I was reading the code and I stumbled across the function makeNumber(...) inside lexer.c
Inside the function, you peek if the current char is a '.' meaning the number you are lexing is a float, but after advancing you run a while loop that does nothing and if tested it would probably run an infinite loop.

the lines of code I'm referring to
if (peek(lexer) == '.') { advance(lexer); while(isDigit(lexer)); }

Arrays/dictionaries can't have calls to functions within them

var a = [ ([].length()) ];

This currently makes the compiler panic - but no segfault as of now.

Originally reported in #56.

There may be integer division issues (currently, the remainder is discarded).

Is this the desired behaviour?

Move functions like `compileString`, et. al. into a utility file

As it is, I'm copying them around within airport - that's not ideal.

Line numbers in compilation errors begin at 0

[Line 0] Error at '': Expression statements not yet implemented

Lines start at 1 in text files. They should be identical in compiler errors

Chained assignment statements currently don't work

External Script Runner Library Needed

I need some way for Toy to call external scripts, analogous to modules in other languages.

Theoretically, calling external scripts might be something that a host program will want to disallow. As such, it should be implemented as an optional external library. It might also be necessary to limit file system access for these scripts via the host.

import runner;

//load the bytecode and environment
var script: opaque = loadScript('assets:/scripts/example.toy");

//run the script on a clean environment, making it dirty
script.runScript();

//deep-copies the value of a global variable within the dirty environment
var x = script.getVariable("x");

//calls this fn within the dirty environment, returning the value
var result = script.callFn("getX"); //can take parameters

//cleans the dirty environment
script.resetScript();

//frees the loaded script - must be called after every load!
script.freeScript();

It might be possible to use some of the groundwork in airport's node system for some of this... but actually I want to keep the two systems separate. As such, the runner library will utilize a unique opaque tag of some kind. I might need a utility to register new tags for the opaque tag system.

Add the opaque data type

How did I forget this as a feature?

"opaque" data represents some kind of data within C that Toy can't understand, and only passes from one place to another. An example of this would be a pointer to an EngineNode in airport.

var node = makeNode(); //node is an opaque value
processNode(node);

Terminal IO Library Needed

Platform-independent types used in the internals

Currently, there are size checks for different datatypes within common.c, but this isn't ideal. I'd like to change all of the variable types to match the following pattern:

• s32 -> 32 bit signed int
• u16 -> 16 bit unsigned short
• b8 -> 8 bit boolean
• f32 -> 32 bit float
• etc.

These could be based on the types in stdint.h

This is low priority, but will need to be done before version 1.0.0.

Assert what version is being used in the script

Seemingly unrelated error message appearing from else-clause

//BUG: this causes a strange error message
if (true) {
	;
}
else {
	;
}

Allow the "for" to have empty clauses

for (var i = 0; i < 10; /* EMPTY */) {
    i++;
}

I've done stuff like this more than once.

Typecasting to null gives a bad error message

This is because this error propagates all the way to the interpreter. It needs to be caught long before there, somehow.

Check for memory leaks

Redirect Lexer, Parser, Compiler Error Messages

Currently, only the Interpreter's messages are redirected.

The parser mustfail tests will spit out errors into the log otherwise.

Nested compound indexing doesn't currently work.

var a = [
  [1, 2, 3]
];

print a[0]; //works
print a[0][0]; //error

Define, very rigidly, the optimisation levels available and what they do

Right now:

-O0 does nothing - the AST resembles the source code completely.
-O1 is the default - it calculates numbers (and negates them when needed) and removes groupings where necessary.
-O2?
-O3?

Will need to switch off the color codes on windows.

Support for custom allocators

I would like to propose adding support for a user-provided custom allocation scheme. I would just have a user callback for malloc, realloc, and free, or a single callback from which the intention can be determined, as Lua does.

Here's an example of a custom allocator callback for the LuaVM which doesn't do anything other than log allocation metrics:

void *LuaVM::_luaAlloc(void *ud, void *ptr, size_t osize, size_t nsize) {
  (void)ud;
  (void)osize; /* not used */

  if (nsize == 0) { // free
    if (ptr) {
      AllocMetadata *pMeta = _luaPtrToMetadata(ptr);
      _allocStats.freeEvent(pMeta->luaType, pMeta->size);
      free(pMeta);
    } else {
      osize += 47;
    }
    return nullptr;

  } else {
    if (ptr) { // realloc
      AllocMetadata *pMeta = _luaPtrToMetadata(ptr);
      _allocStats.reallocEvent(pMeta->luaType, osize, nsize);
      pMeta->size = nsize;
      pMeta = static_cast<AllocMetadata *>(
          realloc(pMeta, nsize + sizeof(AllocMetadata)));
      return _metadataToLuaPtr(pMeta);

    } else { // alloc
      AllocMetadata *pMeta =
          static_cast<AllocMetadata *>(malloc(sizeof(AllocMetadata) + nsize));
      pMeta->size = nsize;
      pMeta->luaType = static_cast<int>(osize);
      _allocStats.allocEvent(pMeta->luaType, pMeta->size);
      return _metadataToLuaPtr(pMeta);
    }
  }
}

Notice a few things: 
1) single callback for all three operations (malloc, realloc, free), to save VM space
2) userdata pointer gets passed back, so you can have some sort of stateful allocation scheme
3) the underlying "type" associated with the allocation can be figured out in this callback
    * might be prettier to pass it directly
    * allows user to provide custom allocation schemes per type
    * allows for finer grained profiling 

Types are duplicated in the literal cache

This seems to affect at least the "any" type.

Exports Region Can Be Safely Removed

The import and export keywords are supposed to access the "exports region", which is a dictionary of globally available variables... this was intended to allow the host program and Toy scripts to pass variables back and forth.

What I wasn't expecting, was that the "engine nodes" that make up the host game engine can actually go into the dirty interpreter and retrieve the variables manually when it's first executed. The direct upshot of this is, a function like onDraw(node: opaque) can be called without needing to explicitly export it first, leading to some nicer looking scripts.

Add to this whole issue that import is also used to import libraries... and I think the exports region feature can be safely removed.

Indexing an array with a non-number should cause an error

Extract the optimisations out into their own layer

At the current time, the only optimisation that really exists is constant folding - but that can be pulled out of the parser and inserted into an AST optimising step.

print type x;

var x = 5;
print type x; //prints 5

I was expecting an error.

Why are array and dictionary reserved words?

var x: typeof t = int;

The above does not work, because IDC.

It theoretically shouldn't be too hard to implement.