teal-language / cyan Goto Github PK

View Code? Open in Web Editor NEW

57.0 57.0 6.0 573 KB

The Teal build system and project manager

License: MIT License

Makefile 0.83% Lua 99.17%

cyan's People

Contributors

Stargazers

Watchers

Forkers

svermeulen jlplabs crnflke hishamhm kayibea thisisgame

cyan's Issues

Generate html docs and host them with github pages

Probably something similar to how I'm doing it myself with my ltreesitter module

`--prune` should be able to remove directories

Ensure tests run on windows

Additionally, find some way to run CI for windows as leafo's github actions do not support it. So either fix that upstream or find a different method.

maybe use chocolatey - luarocks package is pretty outdated, we use the 3.0 format so that wouldn't work
manually install everything - would work, but I'd have to learn what windows-latest comes with on github actions. for example can I curl/wget a luarocks binary? and add it to the PATH
See if there's another github action that supports windows

Disable colors when output is not a tty + support `$NO_COLOR`

When using stuff like vim's makeprg using tools to filter the output of a build, the escape sequences really muddy up the output.

Additionally, NO_COLOR is an extremely trivial thing to support that would be in the same code path.

Warnings in out of project files should not affect the current project

Easy example:

cyan init --werror unused
echo "local x: number" >> module.tl
echo "require'module'" >> src/main.tl
cyan build

Which results in a build error due to the unused variable in the module.

allow loading .tl files for scripts

The API documentation is unavailable

Hello, 👋

It seems the link from the README https://teal-language.github.io/cyan/ is down. According to https://github.com/teal-language/cyan/actions/runs/4219037124, the page has expired.

I tried to clone the repo and run make docs, but it fails with

make docs
tl --quiet -I src gen --check src/cyan/graph.tl -o build/cyan/graph.lua
========================================
2 errors:
src/cyan/fs/path.tl:5:28: module not found: 'lfs'
src/cyan/fs/path.tl:274:30: assignment in declaration did not produce an initial value for variable 'err'
========================================
1 error:
src/cyan/fs/init.tl:5:28: module not found: 'lfs'
make: *** [Makefile:12: build/cyan/graph.lua] Error 1

Luarocks support

How could I have proper build and package support with luarocks using this build system?

discrepancy between `tl gen` and `cyan build`

This comes up when I forget to do a make bootstrap. When using regular tl gen/default make target to compile cyan, there are no newlines at the ends of files, but when compiled with cyan build/make bootstrap there are newlines.

TODO: is this a tl or cyan issue?

ci has been failing for centuries, figure out why

Installing teal seems to be the issue?

cyan dev-1 depends on tl (not installed)
Installing https://luarocks.org/dev/tl-dev-1.rockspec

Error: Failed installing dependency: https://luarocks.org/dev/tl-dev-1.rockspec - Failed cloning git repository.
Error: Process completed with exit code 1.

build: Add way of only checking

In tl build there is -p --pretend to run build then stop before compiling. This is just a nice feature to have

Ability to ignore certain files in the build directory

I currently commit my build directory into git and have a README.md file with the contents:

**WARNING**: Do not edit the files in this directory. The files are generated from [teal](https://github.com/teal-language/tl). For the original source files please look in the [teal](../teal) directory.

However, this file causes cyan to ~~error out~~ emit a warning.

It would be ideal if cyan could simply ignore non-*.tl files, or alternatively have a way to explicitly ignore particular files.

Feature request: Copy any lua files found in source directory into build directory

I find that it is common for some projects to have some portion of the code in lua. It can be convenient to have these lua files saved alongside teal files in the same source directory. I wonder if cyan could also copy those files over to the output directory too? Or provide an option to do that? I'm doing this as a second step after cyan build --prune but it's not ideal since it copies the lua files every time

build: add --prune option to delete unexpected files in build directory

Related to the last patch, since we can detect these files we should provide a flag to remove them with --prune

path normalization needed to avoid duplicated builds?

I tried to run cyan build on teal-language-server without having other teal-types definitions around. I got this output (will paste it in full here, but point to a specific thing below)

cyan build      
    Error 5 type errors in src/rpc.tl
      ...    ./src/util.tl 2:29 no type information for required module: 'dkjson'
      ...    ./src/util.tl 73:19 cannot index a value of unknown type
      ...    src/rpc.tl 3:29 no type information for required module: 'dkjson'
      ...    src/rpc.tl 71:22 cannot index a value of unknown type
      ...    src/rpc.tl 83:21 cannot index a value of unknown type
     Info Type checked src/uri.tl
    Error 3 type errors in src/methods.tl
      ...    ./src/rpc.tl 3:29 no type information for required module: 'dkjson'
      ...    ./src/rpc.tl 71:22 cannot index a value of unknown type
      ...    ./src/rpc.tl 83:21 cannot index a value of unknown type
    Error 19 type errors in src/document.tl
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/command.tl 5:33 no type information for required module: 'argparse'
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/command.tl 34:23 unknown type argparse.Command
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/command.tl 55:34 unknown type argparse.Parser
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/command.tl 57:27 cannot index something that is not a record: argparse.Parser (an unknown type)
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/init.tl 5:28 module not found: 'lfs'
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/init.tl 18:24 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/init.tl 26:43 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/init.tl 124:11 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 112:15 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 170:15 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 176:15 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 182:15 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 194:30 assignment in declaration did not produce an initial value for variable 'err'
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 194:48 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 196:27 in return value: got <unknown type>, expected string
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 209:18 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 381:27 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/fs/path.tl 383:27 cannot index a value of unknown type
      ...    /System/Aliens/LuaRocks/share/lua/5.4/cyan/log.tl 9:36 no type information for required module: 'inspect'
    Error 2 type errors in src/handlers.tl
      ...    src/handlers.tl 4:28 no type information for required module: 'lfs'
      ...    src/handlers.tl 30:20 cannot index a value of unknown type
    Error 2 type errors in src/init.tl
      ...    ./src/handlers.tl 4:28 no type information for required module: 'lfs'
      ...    ./src/handlers.tl 30:20 cannot index a value of unknown type

What caught my attention was that src/rpc.tl and src/handlers.tl were compiled twice (once as src/ and once as ./src), leading to the duplicated error messages

`cyan gen --output` fails with some relative paths

$ cyan gen blah.tl --output ../blah.lua

fails by 'failing' to parse the path ../blah.lua since the current iteration of the path module doesn't account for this.

this shines a light on a more structural problem with the path module itself that needs to be addressed

Custom luarocks build backend

Having a custom backend would make writing and, more importantly, autogenerating rockspecs much easier as we can just reuse all the code for build and make sure to install the files to the correct place.

Additionally, autogeneration would barely be necessary, since we would only need tlconfig.lua to build the project and know where all the files are.

"post build" steps that operate on all of the files

Is it possible to make a script that uses all the resulting lua files and makes a result from them? I want to be able to run this tool on each lua file, which makes a corresponding C file, so then I can link it all to one executable

Add build.tl functionality (or equivalent)

This is the last feature to put cyan at feature parity with the tl script.

My initial thoughts to implement this:

If we're to just imitate the feature as is, I don't think the option should exist to change the name of the build script or,
maybe adding some sort of script_dir config option and there could be some special apis exposed to it depending on the name of the script e.x.

-- scripts/build.tl
before {
   exec = function()
      print("This is run before the build")
   end,
}

after {
   exec = function()
      print("This is run after the build")
   end,
}

(Maybe this type of thing should be called a hook?)
and there should be a way of telling the build system that these steps depend on files or generate/update files, perhaps something like

-- scripts/build.tl
generate {
   -- takes an array of whatever this record is:
   {
      -- given the inputs and outputs, the build system can determine
      -- when this should be rerun. While this is more cumbersome,
      -- we can't monitor the filesystem (at least not easily) for changes
      -- made by this code
      inputs = { "foo/bar.tl" },
      outputs = { "foo/generated.tl" },

      exec = function(inputs: {string}, outputs: {string})
          -- just feed the fields into the args so you
          -- dont have to repeat yourself and maybe mess it up
         local i = io.open(inputs[1], "r")
         local o = io.open(outputs[1], "w")
         o:write(i:read("*a"), "\n\n-- :D")
         i:close()
         o:close()
      end,
   },
}

Documentation

Finish writing the documentation for the current api as it stands and the documentation for the fields in tlconfig.lua as well as some examples

Files in build directory without corresponding sources should be reported as warnings/errors

Consider the following sequence of events:

user has the following project:

src/
   foo.tl
   bar.tl
build/
   foo.lua
   bar.lua

user renames bar.tl -> baz.tl and rebuilds resulting in:

src/
   foo.tl
   baz.tl
build/
   foo.lua
   bar.lua
   baz.lua

If the user has tests or other random files that require the compiledbar module, they could still work, and are likely to have subtle errors caused by this.

We should report that a file was found in the build_dir that was not expected and maybe add a flag to purge the build_dir of any unexpected files

Add verbosity flags for debugging

Not only would this reduce some print debugging, but would help for people reporting issues as well

lua files could get overwritten if source_dir == build_dir

Unlikely to be the cause of an actual bug, but is needless work.

Solution: don't write out when source file == destination file

cyan build: -b, --build-dir and -s, --source-dir flags?

Would it be a good idea to add those flags -b, --build-dir and -s, --source-dir flags to cyan build to make it more of a drop-in replacement to tl build? I'm looking forward to removing tl build in the near future and directing users to cyan build instead.

New version published to luarocks

Could we get a new version published to luarocks? The official version does not have the --prune option which is really useful

`cyan build -q` did not produce quiet output when used alongside other options

cyan --help says -q --quiet works like -v quiet, but when I tried cyan build -q --build-dir ../build and cyan build --quiet --build-dir ../build, I still got the "Info" output, but when running cyan build -v quiet --build-dir ../build, they did not show.

Dependencies are sometimes not processed in the correct order

Context:

Writing a parser in Teal, using Cyan to build it.

token.tl -- defines tokens and map strings to/from integers
parser.tl -- scans source code and emits tokens

Versions:

> tl --version
0.15.1
> cyan version
      Info Cyan version: 0.3.0
       ... Teal version: 0.15.1
       ...  Lua version: Lua 5.4

I am not redeclaring globals. Teal agrees (and often Cyan agrees).

Teal's view:

> tl check src/parser.tl 
========================================
Type checked src/parser.tl
0 errors detected -- you can use:

   tl run src/parser.tl

       to run src/parser.tl as a program

   tl gen src/parser.tl

       to generate parser.lua
> tl check src/token.tl 
========================================
Type checked src/token.tl
0 errors detected -- you can use:

   tl run src/token.tl

       to run src/token.tl as a program

   tl gen src/token.tl

       to generate token.lua

Cyan's view:

ebc> cyan build
      Info Type checked src/token.tl
      Info Type checked src/parser.tl
      Info Wrote build/token.lua
      Info Wrote build/parser.lua

Sometimes Cyan Disagrees.

Say I modify the token.tl file to add a new token (but NOT change any types). Sometimes Cyan complains as shown:

> cyan build
      Info Type checked src/parser.tl
     Error 2 type errors in src/token.tl
       ... src/token.tl 37:8
       ...    37 | global TOK <total>: {Tokstr:TokenType} = {
       ...       |        ^^^
       ...       | cannot reassign to <total> global: TOK
       ... 
       ... src/token.tl 61:8
       ...    61 | global TOKSTR: {TokenType:Tokstr} = {
       ...       |        ^^^^^^
       ...       | cannot redeclare global with a different type: previous type of TOKSTR is {TokenType : Tokstr}

One way to "fix" the error is to change the type name from "TOKSTR" to "TOK_STR", save the file, build it (the error is gone), go back in and rename back to "TOKSTR", build it (the error is still gone).

Issue is related to case-sensitive differences?

I have an enum named 'Tokstr'.
I have a map named 'TOKSTR'.

> grep -in tokstr src/token.tl 
8:local type Tokstr = enum
37:global TOK <total>: {Tokstr:TokenType} = {
61:global TOKSTR: {TokenType:Tokstr} = {

I can post the token file if that is useful.

Failed to build: `attempt to index a nil value (field 'emptytable_type')`

cyan build
      Warn in config:
       ... Unknown key 'aot_options'
      Info Type checked src/module.tl
      Info Type checked src/nested/module.tl
     Error Error executing command
       ...    ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:7351: attempt to index a nil value (field 'emptytable_type')
       ... stack traceback:
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:7351: in upvalue 'assert_is_a'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:9182: in local 'cbkind_after'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3629: in upvalue 'recurse'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3436: in local 'fn'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3621: in upvalue 'recurse'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3552: in local 'fn'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3621: in upvalue 'recurse'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3436: in local 'fn'
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3621: in function <...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:3599>
       ...      (...tail calls...)
       ...      ...ocuments/teal-to-native/lua_modules/share/lua/5.4/tl.lua:10430: in function 'tl.type_check'
       ...      (...tail calls...)
       ...      ...native/lua_modules/share/lua/5.4/cyan/commands/build.lua:161: in local 'process_node'
       ...      ...native/lua_modules/share/lua/5.4/cyan/commands/build.lua:198: in field 'exec'
       ...      ...ts/teal-to-native/lua_modules/share/lua/5.4/cyan/cli.lua:170: in function <...ts/teal-to-native/lua_modules/share/lua/5.4/cyan/cli.lua:169>
       ...      [C]: in function 'xpcall'
       ...      ...ts/teal-to-native/lua_modules/share/lua/5.4/cyan/cli.lua:169: in main chunk
       ...      [C]: in function 'require'
       ...      ...lua_modules/lib/luarocks/rocks-5.4/cyan/0.3.0-1/bin/cyan:2: in main chunk
       ...      [C]: in ?

Error: Build error: Failed building.

This only happens when I have a .lua file in my src, I have an appropriate .d.tl to go along with it aswell

False positives on reporting "out of project" type errors on Windows

We currently report when there are type errors outside of the current project with an explicit marker, but there are false positives that hit on windows. This is probably due to the path library not being extensively tested on Windows.