skx/critical

A simple/minimal TCL interpreter, written in golang

Steve Kemp

• I've been programming for over half my life, I'm comfortable creating, developing, maintaining, and improving software written in C, C++, Emacs Lisp, Perl, Ruby, Java, Shell, TCL, etc.
  • Most of my new projects are written in Golang.
  • But also I've been writing code in Z80 assembly language, to run under CP/M and the humble ZX Spectrum 48k.
• My interests primarily revolve around compilers, interpreters, domain-specific languages, and virtual machines.
  • Examples of scripting languages include a simple BASIC, a simple FORTH, a simple Lisp, and a simple TCL.
  • DSLs are great tools for automation, etc.
• Location: Helsinki, Finland.
• Occupation: Sysadmin / Devops / Cloud-person.

critical's Issues

Implement `return`

This is related to #5, but I ran out of time last night.

Handle comments properly

Right now our parser/lexer doesn't do anything special with comments.

The end result is the following lines don't behave as you'd expect:

# This is a comment
// This is also a comment

What they actually do is invoke the TCL command "#", or "//", and pass the whole line as arguments. In practice that works out - the comment primitive just does nothing and all appears well.

However there is one problem, these two lines cause syntax errors:

#Note the lack of space between "#" and "Note"?
//Same here.

When executed you'll see this:

frodo ~/critical $ ./critical bad.tcl 
Error running program:unknown command '#Note':{token{Type:IDENT Literal:#Note} [token{Type:IDENT Literal:the} ...

Or this:

frodo ~/critical $ ./critical bad.tcl 
Error running program:unknown command '//Same':{token{Type:IDENT Literal://Same} [token{Type:IDENT Literal:here.}]}

Here we're obviously trying to run the word/proc/command "//Same", and "#Note". Neither exist, so they throw an error.

Fix this by just swallowing the comments in the lexer, and getting rid of the comment proc in the interpreter.

Support proper handling for "{" and "}"

This should return the contents, between them, unprocessed

So "if { $a < 4 }" will return the literal text $a < 4 when this is hit.

If I can fix this then the complexity of if and while will go away, and that means that we can add proc to define user-functions.

It won't be a massive change, but I guess adding a parser/ package would be the sanest way to go.

Argument counts are broken?

The following code runs:

proc foo {} {
   if { 1 } {
      return "steve"
   } else {
       return "bar"
   }
    puts "I'm still here ..?"
}

foo 1

Obviously foo should take zero arguments, but calling like that fails:

$ ./critical a.tcl
Error running program:function argument mismatch, foo takes 1 arguments, 0 supplied

This is the first of two issues with that same piece of code. See also #16.

Improve test-coverage

Since adding floating-point support, in #20, the test-case coverage has slipped.

(Mostly because we try to return an int/float as appropriate. Handle both cases and ensure our test coverage is back to 100%.

Allow users to define functions.

Something like this:

proc square {x} { * $x $x }
square 7

This might mean we need to add continue, break, and return. We'll cross those bridges when we come to them..

Allow resetting the state ..

I wrote a naive benchmark test:

	//
	// Source code of the script we're going to run
	//
	src := `return 34`

	//
	// return values
	//
	var err error
	var out string

        // create
	i := New(src)

	//
	// Now run the thing in a loop
	//
	b.ResetTimer()
	for n := 0; n < b.N; n++ {



		// execute
		out, err = i.Evaluate()
	}
	b.StopTimer()

It doesn't work. Because we essentially run:

• New()
• Evaluate()
• Evaluate()
• ..
• Evaluate()

The first call to the Evaluate method invokes the parser, which then consumes the program. That means future executions do nothing.

We should move the Parse step to the constructor to allow the interpreter to re-run the same script multiple times.

source a source file

How to source a tcl file?

Error running program:unknown command 'source':{token{Type:IDENT Literal:source} [...]

Add a simple "standard library".

I had to rewrite @antirez's code from this:

proc fib {x} {
    if {<= $x 1} {
        return 1
    } else {
        + [fib [- $x 1]] [fib [- $x 2]]
    }
}

To this:

proc fib {x} {
    if { expr $x <= 1 } {
        return 1
    } else {
        return [expr [fib [expr $x - 1]] + [fib [expr $x - 2]]]
    }
}

i.e. Replace { <= $x 1} with { expr $x <= 1}, because I don't have the prefix-forms present.

But of course we could add them:

proc + {a b} { expr $a + $b}
..
proc <= {a b} { expr $a <= $b}

Create an embedded STDLIB to do that, and define repeat as well as other functions.

Add an example of embedding this in another app

I hacked together a simple FORTH:

• https://github.com/skx/foth/

And demonstrated that here:

• https://github.com/skx/turtle

Do something similar here, to show how to embed.

Return from inside `if` fails

The following code, as used in #15, has a second failure:

proc foo {} {
   if { 1 } {
      puts "steve"
      return "steve"
   } else {
       puts "bar"
       return "bar"
   }
    puts "I'm still here ..?"
}

foo 1

Running this shows:

$ ./critical a.tcl
I'm still here ..?

Result:I'm still here ..?

The issue here is that either branch should have resulted in the return being executed - but it wasn't. So the execution hit the puts at the foot of the code.

TLDR; return is broken inside if. (Possibly while and for too?)

Full TCL port ?

https://gitlab.com/cznic/tcl/-/blob/master/go.mod

Would be interesting what this can do

Maybe adding lists isn't so hard ..?

• Introduce an Object class
  • We'll have three objects, number, string, and list
    • list will hold Objects so it can contain strings, integers, or other lists.
• Update the environment package so the variables are now map[string] *Object

Annoyingly at that point we'll almost certainly need to change our function signatures for HostFunction to read:

function func(i *Interpreter, args []*Object) (Object, error)

Once that is done we'll have:

• The interpreter allowing different types.
• The built-in functions, such as list, returning those types.

The other complication will be that we'll need to validate types on our primitives, not just argument-counts.

e.g. exit will need to ensure it has "string|number". Similarly the return value of interpreter.Evaluate and interpreter.Eval will need to be *Object,error rather than string,error

So significant churn, but the actual steps are pretty simple.

Implement `break` and `continue` for while-loops

We don't have for yet, so I think those are the only places these make sense.

(Much like adding return #9 we'll just define a special kind of Error and use that to flag the loop handling in the builtin_while.go file.)

Weird errors printing strings

This is almost certainly going to turn out to be related to expansion.

Consider this program, to print a string five times:

repeat 5 { puts " * " }

Works just fine:

$ ./critical newline.tcl 
 * 
 * 
 * 
 * 
 * 

Result: * 

Now remove the spaces:

repeat 5 { puts "*" }

And suddenly:

frodo ~/critical $ ./critical err.tcl 
*
*
*
*
*
Error running program:function argument mismatch, * takes 2 arguments, 0 supplied

Replacing * with +, or similar, produces the same error message.

Add linting

We don't have any CI process setup against this repository; add it.

At the least we're seeing failures now:

frodo ~/Repos/github.com/skx/tcl $ ~/go/lint 
interpreter/interpreter.go:310:10: if block ends with a return statement, so drop this else and outdent its block
interpreter/interpreter.go:316:10: if block ends with a return statement, so drop this else and outdent its block
interpreter/interpreter.go:322:10: if block ends with a return statement, so drop this else and outdent its block
interpreter/interpreter.go:328:10: if block ends with a return statement, so drop this else and outdent its block
interpreter/interpreter.go:369:9: if block ends with a return statement, so drop this else and outdent its block
interpreter/interpreter.go:469:3: this value of err is never used (SA4006)
lexer/lexer.go:29:2: field decimal is unused (U1000)
frodo ~/Repos/github.com/skx/tcl $ 

Create fuzz-tester for lexer, and parser.

We don't yet have 100% test-coverage, so that should be done first, but then we should have fuzz-tests for both packages.

Early termination error?

Need to check with a real TCL to see if this is expected, but the following program surprised me by returning 5 and not printing anything:

5
puts "OK"

I suspect that we're seeing the 5, considering it a literal, and returning from our evaluation loop.