Support for malayalam characters

[sreyas-sreelal]

Support for malayalam characters

[Sreyas-Sreelal]

Here's a simple example in pure Malayalam keyword.
https://github.com/Sreyas-Sreelal/malluscript/blob/malayalam-characters/examples/pure_malayalam.ms

[de-sh]

Some errors I faced

>> pwoli_sadhanam രണ്ട്;
thread 'main' panicked at 'byte index 16 is not a char boundary; it is inside 'ര' (bytes 15..18) of `pwoli_sadhanam രണ്ട്;`', src/libcore/str/mod.rs:2069:5

Rust Users Forum#1909 is a discussion I can point to and the one pointer I could figure out is that using encoding might help us solve this problem.

[Sreyas-Sreelal]

Yes.I have started the work on malayalam-characters,I'll merge it to master after the work is done :)

[Sreyas-Sreelal]

Ok this comes with some limitation.The encoding of source code to ASCII,can lead to encoding of string literals too.This should be avoided.One of possible solution that comes to my mind is to create a literal table,before tokenizing the source unit,and replace the literals with reference to corresponding value in table.