kosarev/escapeless

Efficient binary encoding for large alphabets

escapeless's Issues

Couple of Questions

Question 1: Is the Python port written in Python 2? Because when running it, it came up with some errors that upon searching pointed me to people trying to run Python 2 code in Python 3.

Question 2: Is there any way we could get a bit of documentation on how to use this? I get a base of what it's looking for, but even with the testers, it's hard to see what's happening.

Is this UTF8-friendly?

Was wondering if this will work with UTF8 (and other Unicode encoding)

Have a Python 3 implementation

Suggestion: special cases

Hi there,

I am using a variant of this encoding in a project that runs on an ancient sub-1 MHz 8-bit machine, with data over serial port and extremely low bandwidth. I say a variant because I keep all excluded symbols in the 0...n range and exploit this fact.

I needed to make some optimisations and one applies to the general format of this encoding scheme and thus this project:

When marking the symbols used in a block, I keep track of the maximum byte found. If max < (0xff - n), you can safely pick the last n possible bytes as substitutions and move on without checking if they're taken. Obviously this is content-dependent, but it does trigger.

The other optimisation will not work with your code, but in case anyone else might want to do what I'm doing:

Because all of my "take outs" are in 0...n, when encoding I also find the minimum byte in a block, and if min > n, then there is no need for substitution at all - in this case I set at least the two first bytes of the replacement list to the same value. The receiver then checks if first two replacements are the same byte, and if they are, it knows that there is no substitution required.

This encoding scheme is a great find by the way, my project would have been impossible to implement if it wasn't for this. It's a case where I need to send IP packets via a serial link that does not support hardware flow control (so SLIP will not work), but it still requires explicit flow control that must be controlled by the application (XON/XOFF will not work either), plus extra control characters (block acknowledgment and separator / end of packet = forced end of block). With escapeless I could easily implement my own flow control and still send arbitrary data using escapeless 128/4.

Provide examples of use

#2 suggests that the implementations and tests are probably not enough to figure out how to use the code.