dimitriv / ziria Goto Github PK
View Code? Open in Web Editor NEWA domain-specific-language and compiler for low-level bitstream processing.
A domain-specific-language and compiler for low-level bitstream processing.
# make
EXTRAOPTS='--bounds-check ' ../../scripts/preprocesscompile-gcc.sh array1.wpl array1.out
array1.wpl
Running WPL compiler...
Compiling C code (GCC)
~/wt/projects/Ziria/tests/backend ~/wt/projects/Ziria/tests/backend
gcc -std=c99 -msse3 -c test.c -o test.o
gcc -o driver bit.o numerics.o params.o test.o driver.o wpl_alloc.o utils.o buf_bit.o buf_numerics32.o buf_numerics16.o ext_math.o ext_arr.o -lm
~/wt/projects/Ziria/tests/backend
./array1.out --input=file \
--input-file-name=array1.infile \
--input-file-mode=dbg \
--output-file-name=array1.outfile \
--output-file-mode=dbg
Parse error when loading debug file.make: *** [array1.outfile] Error 1
rm array1.out
Add char type, make TString == array of char, and make them both first class citizens
The following Ziria code:
while (k < length(tbs_table) && tbs_table[k] != trblklen) {
print k, " ";
k := k + 3;
}
compiles into the following C code:
bounds_check(81, kln11_111 + 0, "ln26_");
while (kln11_111 < 81L && _local_arr_ln4_115[kln11_111] !=
trblklenln11_109) {
printf("%ld", kln11_111);
UNIT;
printf("%s", " ");
UNIT;
printf("%s", "");
UNIT;
kln11_111 = kln11_111 + 3L;
bounds_check(81, kln11_111 + 0, "ln26");
__while116 = UNIT;
}
Note that the last bounds_check is incorrect, as it refers to the condition tbs_table[k] != trblklen in while loop which never gets executed when k > length(tbs_table), so we shouldn't invoke bounds_check exception here.
A forgotten then in if-then-else results in error:
unexpected "\n"
instead of
expecting then
(which I believe used to be the case).
The instructions for setting up WinDDK and Sora are clear enough, but the details of the assumed underlying *nix-like environment (make, gcc, etc.) are missing.
var a:int32;
var b:int16;
a := a+b;
Currently, this fails because the types of a and b don't match. But we can safely automatically cast b into int32 and compile the code. Add automatic casting wherever it makes sense.
Test test_real_rx.blk in WiFi fails in MAC branch
Add the following extension to parser/type-checker:
let (a : int32) = 5 in
which will force type of let-bound variable to something.
print array function is inefficient when applied on function that returns an array.
Consider println pss(3), where pss(3) :: Int -> Arr[62] Int
The code generated by Ziria is:
for (int __print_cnt_96 = 0; __print_cnt_96 < 62;
__print_cnt_96++) {
calign complex16 __print_val_97;
mem_idx99 = wpl_get_free_idx(pheap_ctx);
pss_ln6_81(__retcall_pss_ln7_98, 62, 3, 1000, &pi_ln34_4);
wpl_restore_free_idx(pheap_ctx, mem_idx99);
__print_val_97 = __retcall_pss_ln7_98[__print_cnt_96];
printf("(%ld,%ld)", __print_val_97.re, __print_val_97.im);
printf("%s", ",");
}
So we actually call pss function 62 times in this case! (which is potentially also incorrect).
Depending on how serious we are about this might be good to have a parse-pretty-print-parse round trip test.
Warning: some variables may be initialized from external functions (e.g. zero_complex16), so this has to be take into account as well.
Add assert to verify a condition only in debug mode (e.g. when it is compiled with --assert-on switch)
In the following example:
let external g(b : int) : arr int in
let f(a:arr int) =
var b : arr[length(a)] int;
b := g(3);
return b
in
read[int] >>> emits f({1,2}) >>> write[int]
the return type of f() is correctly resolved to be arr[3] int, but the return type of g() is not resolved (though I don't see a reason why it couldn't be).
For instance allow: takes (3+5) and similarly for EArrRead/Write and ranges in general.
Add enumerate type to the language
Currently, bound check violation only displays a line number and sizes, e.g.:
Bounds check violation: ln26_
Size = 81
Access = 82
It would be nice to print the file as well (as there may be many nested includes) and variable.
I'd like to be able to write:
let f(input : arr bit) =
return input[0, length(input)/3]
in
Restructure various calls from Ziria generated code to make the API more clear. For example, wpl_global_init() initializes the heap using wpl_init_heap and generates LUTs. So if I want to run wpl_go() many times in a C loop, I need to do only one one global init but I need to run wpl_init_heap in each loop. This should be simplified and clarified.
Also, provide better return of parameters between MAC(C) and PHY(Ziria). For example, if Ziria code is a computer that returns a struct, this struct should be returned by wpl_go. Also, allow MAC to query Ziria's compile-time parameters, such as input/output types.
Note: Each subarray in a matrice of bits should be byte-aligned
(starting at the beginning of a byte and padded to a length divisible by 8)
Add constant size arrays with variable values. For example,
let y = {x1, x2} in
currently does not compile, but seems to have very clean sematics:
var tmp:arr[2] : int;
tmp[0] := x1; tmp[1] := x2;
let y = tmp
For example:
let f(a:arr int) =
let x1 = 5;
let x2 = 4;
let b = {x1, x2} in
let c = b[1] in
return c
in
read[int] >>> emit f({1,2}) >>> write[int]
When doing standalone
pipelining, we get a funny readQueue
operation, which does not block when the queue is empty but instead jumps to its consume continuation. This is because when the queue is empty, the standalone
thread hasn't put anything there yet, but in order to fill the queue, that thread needs some input which the main thread is responsible for delivering - so we jump to produce that input, then try the read again.
Now, if we're reading a finite input source, the main thread may tear through it, dumping all of it into the standalone
thread's input queue, essentially reaching EOF. If this happens and the standalone
thread still hasn't produced any output, the main thread will try to produce more input for that thread, realize that the input is all gone, and then happily exit. However, the main thread was responsible for reading the other thread's output and doing something with it before writing it into the output buffer - since it already exited, this will now never happen, so the output gets truncated.
We need to figure out a way to keep the main thread running for as long as it's needed.
the following:
read [int] >> write [int]
which involves a typo, does not fail during parsing but much later. At least in feature/cleanup-comp-parser.
The current non-blocking internal read operation is very naïve: if there's output, do something with it, otherwise jump upstream to produce some input for the other thread, then try reading again. This will probably not do very good job of keeping the other thread from starving, so a better strategy is needed.
Implement a print thread and queue all requests for it to print, to avoid slowing down the main threads
Consider
let x = 1
let x = 2
let comp main = ...
This will be accepted without any warning, with the second x
shadowing the first. But presumably this was not intended.
The following code:
var a : arr[5] complex16;
a[0].re := 0;
fails during parsing.
The following code does not parse:
let f(a:arr int) =
var c : arr[2] complex16 := {complex16{re=0; im=0}, complex16{re=0; im=0}};
return 0
in
read[int] >>> emit f({1,2}) >>> write[int]
since static initialization of complex array seems not allowed at the moment.
struct3.wpl test fails in backend because of memory allocation error:
WPL allocator out of memory, try increasing heap size!
Add C/Matlab-like switch/case statement.
Add buffer read/write for arbitrary data types, so that we can write read[struct AAA] >>> f >>> write[struct BBB].
Error message for missing argument is not informative. For example, if function is defined
to have two arguments (e.g. f(a,b)) and I call it with one f(a), I get:
rx.blk
"rx.blk" (line 42, column 45):
unexpected ")"
expecting letter or digit, ".", "[", operator or ","
ExitFailure 1
Makefile:49: recipe for target 'rx.c' failed
whereas it should have told me that I failed to supply the second argument.
But I have also see instances where a similar error message was reported correctly.
Add support for different endianness through a compiler switch
Although the array passing semantics is not entirely clear, it is sort of assumed to be by reference. And here is a related bug:
arr_shift(params.channelCoeffs, -agc_shift);
where
let arr_shift(a : arr complex16, agc_shift : int) =
if (agc_shift > 0) then {
v_shift_left_complex16(a,a,agc_shift);
} else {
v_shift_right_complex16(a,a,-agc_shift);
}
in
It uses the implicit assumption that the arrays are passed by reference. But here it is not the case. Probably because params is a struct. So we get:
blink_copy(aln60__rx245, __bnd_fst_ln186__rx107params.channelCoeffs,
64 * sizeof( complex16));
if (-agc_shiftln119__rx159 > 0L) {
__if_rx246 = __ext_v_shift_left_complex16(aln60__rx245, 64,
aln60__rx245, 64,
-agc_shiftln119__rx159);
} else {
__if_rx246 = __ext_v_shift_right_complex16(aln60__rx245, 64,
aln60__rx245, 64,
--agc_shiftln119__rx159);
}
And clearly the original memory location does not get changed.
More interestingly, if the arr_shift function is manually inlined, the projection is recognized as an array and passed directly by reference.
Currently we can define let comp locally within a let comp but we cannot do the same for let. For example, we cannot write:
let f(a:arr int) =
let g(x:int) =
return x
in
return g(0)
in
read[int] >>> emit f({1,2}) >>> write[int]
See commented code in file typeinfer.wpl (in tests/backend). When included, the code compiles by Ziria but fails in C. Instead, Ziria should have reported type checking ambiguity.
See https://gist.github.com/kosmikus/f3f4cd27601dc381d008.
Recording this here mostly so that we don't forget about it ( @kosmikus don't delete that gist for now :)
Hey, I found a nasty vectorization bug. Look at this code:
let comp ttt1() =
times LEN {
x <- takes 8;
}
In
It should take 250000 * 8 = 2000000 items. And indeed it does, without vectorization:
EXTRAOPTS='' ../../scripts/preprocesscompile-vs.sh test.zir test.out
Total input items (including EOF): 2000000 (64000000 B), output items: 0 (0 B)
But with vectorise it doesn’t:
EXTRAOPTS='--vectorize --autolut --native-mitigators' ../../scripts/preprocesscompile-vs.sh test.zir test.out
Total input items (including EOF): 250000 (8000000 B), output items: 0 (0 B)
Basically, what seems to happen is that it vectorises times into an inner loop and an outer loop, but it takes only once per inner loop (instead of inner loop times)!
I’ve checked it in branch WiFi/cca in /tests/bugs (make sure you run ttt1() and not ttt() which doesn’t have the same bug as it doesn’t vectorise).
Introduce one-off initialization code (something like static in C). For example:
let comp test() =
once{ dadada };
rest;
in
repeat{test()}
should execute dadada only once, preferably at the init stage before even starting reading the stream. Keyword once is just the first thought, any suggestion is more than welcome.
The following code won't compile
let comp test() =
let f(x:int) = return x in
x <- take;
y <- take;
in
Instead, we have to write:
let comp test() =
let f(x:int) = return x in
{
x <- take;
y <- take;
}
in
which is tedious, and we don't have to do it if we remove let f.
There is an unexpected thing happening when parsing computation language.
The following code
let comp A() =
if true then
{
...
}
in
fails with:
unexpected conditional used as command
expecting operator or "else"
But if I put extra braces around if then it works.
Equate in types a scalar type with an array of the same type with one element.
e.g. allow
let f(a : arr[1] int) = ...
f(1)
Currently, we only allow:
x <- take.
We should also allow
a := take;
where a is a declared variable.
The program below is obviously incorrect (index array too large) but it gives a Haskell error message (Prelude.(!!): index too large) instead of something from Ziria.
let f(a:arr int) =
let b = {3,4,5} in
let c = b[5] in
return c
in
read[int] >>> emit f({1,2}) >>> write[int]
similarly:
The following code:
let table = {3, 10, 7, 12, 19, 42, 7, 16, 7, 18, 11, 20, 5, 22, 11, 24, 7, 26, 41, 84, 103} in
reg1[0] := reg1[table[-5]];
causes compiler error:
Prelude.(!!): negative index
because compiler wants to inline without checking the index...
Have the top-level be a list of declarations, so that we can write
fun foo(..) {
..
}
instead of
fun foo(..) {
..
} in
This is more natural and opens up the possibility of separate compilation. Only problem is that this will require the definition of a label
main = ...
which would replace the final . This means that we need an AST transformation to turn the list of declarations, one of which is for main
, into the big expression that is it expecting now (or an equivalent change to the compiler).
var a : arr[10] int16;
for i in [0, 10] {
a[i] := i;
}
This fails compilation because i is assumed to be int32. But this should be more flexible. Maybe we can assume i to be int16, and automatic casting (see other feature request) can convert it into int32 where needed. Also, we might want to add:
for (i:int32) in [0, 10] ...
for an explicit declaration.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.