opendarkbasic/opendarkbasic

Cross-platform reimplementation of the DarkBASIC Pro programming language. Code is compiled to machine code using LLVM.

OpenDarkBASIC

This project is a modern re-implementation of the DarkBASIC Professional language and SDK. It consists of a compiler and a runtime. The compiler is built using FLEX and BISON to parse the language into an AST and uses LLVM for code generation. The runtime provides common runtime functionality and a framework for plugins to register commands.

OpenDarkBASIC supports both the original DarkBASIC Pro SDK and the ODB SDK (reimplementation). Of course, the original SDK will only work on Windows.

Building

You will need to install following dependencies:

• CMake 3.13 or later
• FLEX 2.6 or later
• BISON 3.7 or later
• A C++17 compliant compiler
• LLVM 10.0 or later

For the Windows peeps out there, you can get up to date FLEX and BISON binaries from here. You can unzip the release anywhere you want (I put it under C:\Program Files (x86)). To get CMake to find them, you have to add the path to the executables to your PATH.

Mac OS users will want to get up to date versions of cmake, llvm, bison and flex using homebrew:

brew install cmake llvm bison flex

And then add the paths to llvm, bison and flex to CMAKE_PREFIX_PATH when calling cmake:

cmake -DCMAKE_PREFIX_PATH="/opt/homebrew/opt/llvm/bin:/opt/homebrew/opt/bison/bin:/opt/homebrew/opt/flex/bin" ../

By default the project is built in release mode. If you want to develop on it, you will want to set it to debug mode.

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug ../
cmake --build . -- -j $(nproc)

On Windows you can specify the arch using the -A option to CMake if they're using VS2019 or later:

cmake -A x64 ../

If you're using an earlier version of Visual Studio, then the architecture is an argument to the -G option. You can type cmake --help to list all available generators.

cmake -G "Visual Studio 14 2015 Win64" ../

Other interesting CMake options:

Option	Default	Description
ODBCOMPILER_LIB_TYPE	SHARED	Build odbc as either SHARED or STATIC
ODBCOMPILER_BISON_COUNTER_EXAMPLES	OFF	Provide counter examples when sr/rr conflicts occur in the grammar
ODBCOMPILER_DOT_EXPORT	ON	Enable Graphviz DOT export capability. Unit tests will also export all ASTs
ODBCOMPILER_VERBOSE_BISON	OFF	Makes the bison very noisy
ODBCOMPILER_VERBOSE_FLEX	OFF	Output every token to stderr
ODBCOMPILER_TESTS	ON	Build unit tests
ODBCOMPILER_LLVM_ENABLE_SHARED_LIBS	OFF	Link with a shared library build of LLVM
ODBSDK_LIB_TYPE	SHARED	Build the SDK library either as SHARED or STATIC

LLVM

Linux

LLVM can usually be installed directly from your distributions repositories. For example, Ubuntu users can simply install llvm-dev, and CMake will detect it. It can also be build from source.

macOS

Untested. Binaries seem to be available here, so that may work. If not, you could build from source.

Windows

Unfortunately, development binaries don't exist for Windows, so you'll need to build from source.

Building from source

If you want a quick list of instructions to build LLVM from source with the minimum required components, follow these instructions:

1. git clone https://github.com/llvm/llvm-project -b llvmorg-11.0.0 (for LLVM 11.0)
2. cd llvm-project
3. mkdir build && cd build
4. cmake ../llvm -DCMAKE_INSTALL_PREFIX=./install -DLLVM_INCLUDE_TESTS=0 -DLLVM_INCLUDE_BENCHMARKS=0 -DLLVM_TARGETS_TO_BUILD="ARM;X86;AArch64"
   • If on Windows, add these options: -DLLVM_COMPILER_JOBS=$(nproc) -Thost=x64
   • If on Linux, it is recommended to use ninja build by adding this option: -GNinja
5. cmake --build . --target install (this took about 15 minutes on a Ryzen 2700X)
6. LLVM binaries will be installed to path/to/llvm-project/build/install.
   • Pass -DLLVM_DIR=path/to/llvm-project/build/install/lib/cmake/llvm to CMake when configuring OpenDarkBASIC to point it to your binaries.

Running

You can run the unit tests by executing:

cd build/bin
./odbc_tests

There is some sample DarkBASIC code in the folder dba-sources in the root directory which you can try and compile.

In this example we'll parse the file iced.dba, which is an old DarkBASIC Classic sample clocking in at around 1.3k lines of code. Here's the full command required to generate an executable:

cd build/bin
./odbc \
    --parse-dba ../../dba-sources/iced.dba \
    -o iced.exe

Most likely, though, this will fail because the project is still under heavy development. You can, however, try to output one of the intermediate stages. For example, you could generate a graph of the AST using Graphviz:

cd build/bin
./odbc \
    --parse-dba ../../dba-sources/iced.dba \
    --dump-ast-dot \
    | dot -Tpdf > out.pdf

Now you can open out.pdf with your favorite PDF viewer and see a visual representation of the program's structure.

Fuzzing

mkdir build-afl
sudo mount -t tmpfs -o size=16384m afl-ramdisk build-afl
cd build-afl
CC=afl-gcc CXX=afl-g++ AS=afl-as cmake -DCMAKE_BUILD_TYPE=Release -DODBCOMPILER_TESTS=OFF -DODBCOMPILER_LLVM_ENABLE_SHARED_LIBS=ON ../
make -j$(nproc)
cd bin
cp ../../scripts/fuzz_odbc.sh .
./fuzz_odbc.sh

opendarkbasic's Issues

Implement codegen for arrays

SourceLocation last_column is off by 1

CommandIndex: addCommand() and findConflicts() should be combined

odbc can't find DLLs on Windows - Restructure output directories

Suggest the following structure for both build and install trees:

build/bin/
├── dbp-sdk/
│   ├── plugins/
│   └── thirdparty-plugins/
├── odb-sdk/
│   ├── plugins/
│   |   └── test-plugin.dll
|   └── thirdparty-plugins/
├── include/
│   ├── odb-compiler
│   └── odb-sdk
├── lib/
│   ├── odb-compiler.lib
│   └── odb-sdk.lib
├── odbc
├── odbc_tests
├── odbi
├── odb-sdk.dll
└── odb-compiler.dll

CI Todo

• install gperf
• install and run valgrind
• only run CI on PRs

Add support for pointer types in parser

Implement codegen for global variables and constants

Generate an ELF or MachO executable using LLD

Once #70 is merged, this can be done

Increment/Decrement arrays and UDTs

When a plugin tries to add a command which already exists, it should be replaced.

The Matrix1Util_07 defines a number of functions that already exist in DBP, such as CLAMP, ABS, etc. The description of that plugin is:

"This plug-in provides faster alternatives to some of the standard DBPro functions, and fills in a few missing ones"

This implies that a conflicting command should replace the original command. The current behaviour is to generate an error.

Examples of conflicts which I believe should be replacements:

[sdk] Command `clamp F(FFF)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_07.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProCore.dll`
[sdk] Command `max F(FF)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_07.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProCore.dll`
[sdk] Command `min F(FF)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_07.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProCore.dll`
[sdk] Command `object scale x F(L)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_18.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProBasic3DDebug.dll`
[sdk] Command `object scale y F(L)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_18.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProBasic3DDebug.dll`
[sdk] Command `object scale z F(L)` redefined in library `C:\Users\dga\Work\DBPro\Compiler\plugins-user\Matrix1Util_18.dll`
[sdk] Command was first declared in library `C:\Users\dga\Work\DBPro\Compiler\plugins\DBProBasic3DDebug.dll`

Allow the codegen target triple to be changed, and implement a stub EngineInterface for ODB

Implement EngineInterface for the ODB SDK. Also update the CLI to allow the target triple to be changed from i386-pc-windows-msvc

The parser generates a broken error message after encountering an invalid command

If I try to parse the following DBP code:

invalid command 1, 2
print "real command"

it generates the following error message:

[db Parser] dba-sources/game.dba:2:7: syntax error, unexpected not, expected end of file
  | print "real command"
  |       ^~~~~~~~~~~~~~

I'd expect the error message to be something like: syntax error: unknown command "invalid command ".

Find out precedence of unary operators in original DBP

Not entirely sure how to test this, but currently the logical not operator and the unary - operator may not have the correct precedence (relative to other binary operators).

It is also unclear what the precedence rules are among unary operators themselves. I would assume - is higher than not such that not -a is parsed as not (-a).

Add support for UDT declarations and "as udttype" for variables

Support for FreeBSD

Is it possible to build ODB on FreeBSD?

swapChild() assumption of how dynamic_cast is wrong -> need to assert for nullptr

odbc fails to compile code with windows line endings -> revert 7922682 and add a rule for "\r"

Add support for semi-colon in parser

Build bison internally using CMake

Short option argument is not parsed correctly when argument is optional

/home/dga/Work/opendarkbasic/cmake-build-debug/bin/odbc --dump-commands -i dba-sources/odb.dba -o test/test
Error: Unrecognized command line option `test/test`

CLI arguments should be grouped into phases and executed in a predefined order

Fix location info when doing more than 1 lookahead

Trailing comments cause parser issues

This code:

print "Hello" `comment
print "World"

gives this error:

[db parser] dba-sources/test.dba:2:1: syntax error: unexpected command, expected end of file
2 | print "World"
  | ^~~~~

Split AST node classes up into their own files

Enable some LLVM optimisations

Currently the LLVM module is turned straight into object code. We should run some optimizations on it such as dead code elimination, const propagation, loop unrolling, function inlining, etc.

Add ast::Node::toString()

This should non-recursively return a string representation of the node. For example:

Block::toString() -> "Block(5)"
ScopedAnnotatedSymbol::toString() -> "ScopedAnnotatedSymbol(GLOBAL, FLOAT, \"var\")"

Support matching all builtin names to construct keywords

Add support for restore/read/data statements in parser

Autodetect SDK type based on SDK root

If the SDK root contains DBPro DLL's, then we can assume the sdk-type is dbpro, otherwise odb

Bitwise not should be unary

Disable colors if stderr is being piped

"break" keyword in DBP does not break from for loops. "exit" does.

Calls to setParent() should occur after assignment, otherwise old value is in an inconsistent state during destruction

Convert "1" and "0" literals to ByteLiteral instead of BooleanLiteral

Add support for declaring zero dimension arrays (dim arr()), array pointer refs (arr()) and undim

It seems like x = arr() is syntax for returning the base memory address of the array. It also looks like it's possible to assign new memory to arrays:

type Thingy
  Array as dword
endtype
 
type ThingyWithinAThingy
  a as integer
  b as float
  c as byte
endtype
 
ThingA as Thingy
ThingA.Array=make memory(SizeOf("ThingyWithinAThingy")*10)      `dim ThingA.Array(9) as ThingyWithinAThingy
fill memory ThingA.Array,0x00,SizeOf("ThingyWithinAThingy")*10  `I'm sure the memory is filled with garbage, so we'd better clear it.
 
end
 
function SizeOf(type$)
  type$=get type pattern$(type$,0)
  Size=0
  for a=1 to len(type$)
    char$=mid$(type$,a)
    if char$="L" or char$="F" or char$="S" or char$="D" then inc Size,4 `This is assuming that "strings" are just pointers.  I don't actually know, but it would make the most sense.
    if char$="W" then inc Size,2
    if char$="Y" then inc Size,1
    if char$="O" or char$="R" then inc Size,8
  next a
endfunction Size

Here's another example I found:

type Thingy
  Array as dword
endtype
 
ThingA as Thingy
ThingA.Array=make memory(4*10)      `dim ThingA.Array(9) as integer
fill memory ThingA.Array,0x00,4*10  `I'm sure the memory is filled with garbage, so we'd better clear it.
 
SetIntegerElement(ThingA.Array,0,1234)
SetIntegerElement(ThingA.Array,1,5678)
print GetIntegerElement(ThingA.Array,0)
print GetIntegerElement(ThingA.Array,1)
 
suspend for key
end
 
`Getter/Setter Functions
function GetIntegerElement(Address as dword,Index)
  inc Address,4*Index
  Element=*Address
endfunction Element
 
function SetIntegerElement(Address as dword,Index,Element)
  inc Address,4*Index
  *Address=Element
endfunction

Add select/case statements back

Implement codegen for UDTs

Adding Global before a variable declaration causes parse issues later.

I was hitting an issue with the current parser. It seems that this parses fine:

OPlayer as Integer = 1
Make Object Sphere OPlayer , 1

but adding Global breaks it:

Global OPlayer as Integer = 1
Make Object Sphere OPlayer , 1

C:\Users\dga\Work\DBP\Snippets\Inertia\_Temp.dbsource:3:6: syntax error, unexpected SYMBOL, expecting AS
  Make Object Sphere OPlayer , 1

This case also doesn't seem to work:

Global OPlayer As Integer
OPlayer = 1
Make Object Sphere OPlayer , 1

C:\Users\dga\Work\DBP\Snippets\Inertia\_Temp.dbsource:3:6: syntax error, unexpected SYMBOL, expecting AS
  Make Object Sphere OPlayer , 1

Commands which are a superset of other commands are not parsed correctly

move camera 1, 0.5
move camera up 1, 0.5

This gets parsed as two move camera commands, with the 2nd one having a first argument up.

[db parser] dba-sources/test.dba:2:19: syntax error: unexpected integer literal, expected end of file
2 |    move camera up 1, 0.5

Get codegen to work on Linux

This will involve implementing the generateCommandCall and generateEntryPoint functions correctly in ODBEngineInterface

Compile code from stdin

Remove the dependency on the platform libraries in the generated object file.

Currently, the object file generated by odbc depends on a number of platform services such as:

• LoadLibraryA
• GetProcAddress
• puts
• etc

This is to implement features such as initialising the DBP engine, loading plugins at runtime, getting function pointers for commands.

This approach means that to create a fully linked executable, we need to link against kernel32.lib and possibly other libraries, which means that we need access to an MSVC installation. We'd also need libstdc++ / libc++ dev libraries installed on Linux/macOS.

An alternative approach would be to ship a shared library as part of OpenDarkBASIC. I'll call this odb-dbp.dll (for the DBP DLLs) or odb.dll/odb.so for the ODB SDK. This library would provide a number of functions that the DBP executable would require, such as:

• loadPlugin - a shim for dlopen / LoadLibraryA. Could do something clever with paths as well.
• getFunctionPtr - a shim for dlsym / GetProcAddress. Could implement caching as part of this.
• initialiseEngine - code that initialses the DBP engine (probably not applicable for the odb SDK). Currently this is hard coded in the code generator, but it would be far easier to maintain in C++.
• print functions - these would be useful to emit debug output in the executable.

The idea is that a binary OpenDarkBASIC distribution would ship a copy of odb-dbp.dll and odb.dll (on Windows), and odbc would link the generated game executable with that DLL / shared object file ONLY. This can be done without any dependency on any other compiler or toolchain on the system, and would enable cross compilation, assuming the required odb.dll/.so file is available to link against.

Add support for hints / multiple colors for warning and error messages

It would be nice to have a class that lets you combine multiple SourceLocation instances together with hints attached to locations, which can then be used to format a message along the lines of:

[db parser] test:1:26: syntax error: unexpected byte, expected string
1 | global dim arr$(2, 3) as byte
  |               ^          ^~~~
  |               |
  |               Annotation `$' implies string
  

SourceLocation should also support setting the color so when multiple SourceLocations are involved, it's easy to distinguish them.

Some cases should also be formatted a little differently, such as with binary operators:

[db parser] test:1:9: syntax error: Invalid operands
1 | result = 3.4 && 0xFF
  |          ~~~ ^^ ~~~~

Remove GotoSymbol, Command*Symbol, etc. and let IR do symbol resolution

Add a toString() method on all AST nodes so it is easier to

Implement codegen for strings

Add support for initializers for arrays

The command index should not allow keywords to be added.

It was possible for DBP plugins to have built-in keywords (such as "do" and "loop") as commands in the string table. As a result, the parser will emit a warning if it encounters a keyword that is also registered as a command.

I recently made a change which ensures that such commands are never added to the command index. The command index should have a check that causes a fatal error if a plugin contains a built in keyword as a command.

Refactor Keyword class

Commands are parsed incorrectly if a command contains another valid command in its name.

If we have the following code (where text, set text font and center text are valid commands):

set text font "Arial"
text 20, 20, "Hello"
center text 200, 200, "Centered"
wait key

This fails to parse with the following error:

[db Parser] dba-sources/simple.dba:2:14: syntax error, unexpected symbol, expected `=`
2 | text 20, 20, "Hello"
  |              ^~~~~~~

Another failing example. In this case ink and ink foreground are valid commands in the command index:

ink foreground rgb(255, 255, 0)
print "yellow"
wait key

[db Parser] dba-sources/simple.dba:3:1: syntax error, unexpected command, expected end of file
3 | wait key
  | ^~~~

I'm not sure what exactly is happening here, but it seems that the parser is entering some strange state when it thinks it's matched a smaller command (such as text or ink), but really it should be matching a longer command (such as ink foreground). Unfortunately, this is blocking me from parsing any reasonably complex project.

Add enum to BinaryOp and UnaryOp, and remove derived types