Use in production?

[hxim]

Use in production?

[rafael2k]

Hi! The use case is for very low bitrate email transmission over HF skywave radio transmission [1]. So the processing overhead is really worth. 10% gain is very relevant. Thanks for the checksum suggestion, I will apply it to be in the safe side. Thanks!!
: )

[1] https://www.rhizomatica.org/hermes/

[rafael2k]

I have a feeling the errors might be very infrequent, right? I got no errors in the tests I carried up to now (including emails with base64 attachments or even binary encoded image and audio). So you are right, may be just drop the data in case of bad checksum in the rx side in order to avoid compression / decompression in the sender side.
Btw, here is the code which does the e-mail magic:
https://github.com/Rhizomatica/hermes-net/tree/main/uuxcomp

[GotthardtZ]

Generally paq8px is not recommended for any production use. It's an experimental software, a playground so to speak.
Though it is quite stable these days, this software doesn't go through thousands or millions of test runs. So really nothing is guaranteed.

If it runs fine for you, and you tested it for your use case and it has a significant advantage vs. the risk why not to use it, right?
You just need to make sure that you mitigate the risk as much as you can and you have a "backup" solution or "alerting mechanism" in case of a failure.

Maybe paq actually does checksum internally and tells you if there ways an issue. I am not sure.

It doesn't have any checksum checking. You'll need to impelement sending a checksum over the line and veryfining it at the receiving end. To be even safer, you'll need to measure decompression runtime at the receiving end and if it is higher than an expected upper limit, you'll need to kill the decompression process. In the past it happened that decompression was stuck due to a bug.
I believe with these two extra measures, your risk is mitigated to an absolute minimum.

[GotthardtZ]

but I just write a binary file and compress it, right?

Yes.

No conversion to string is better, correct?

Correct.

To boost compression, you may insert a dummy byte with a value which occurs very rarely in your data. Paq8px will then detect that there is a repeating columnar pattern there and so it will pick up the record size easily.
If your data already has such a (nearly) constant column (=unique byte in any of your fields), then there is nothing to do, record size detection will be using it.

[rafael2k]

Tks! The records are constant size and indeed, the first field of the record is uint32_t epoch time, in which the MSB byte most likely will not change for some time!

[rafael2k]

So, I integrated to my solution to grab sensors data here: https://github.com/Rhizomatica/hermes-sensors
For now, I'll just let the user inform in case of some kind of corruption, as I believe it is stable enough with this kind of well formatted data.

[rafael2k]

Thanks for all the support. I do consider paq8px to be the highest compression state-of-the-art codec I know about to be able to be applied to the real world use-cases. Cheers and thanks for the great software!

[L3P3]

If the huge processing overhead is really compensated by saved transfer time, I would already be surprised.
Apart from that, there have been bugs in the past that caused files to incorrectly decompress. Of course, you can just decompress and checksum the file before transfer. If your decompression result is correct, it will be on the receiving end.
Also, you can just transfer the sha sum along with the paq file and verify the decompression result on the receiving end. This way, there might be transfer failures but there will not be incorrect results. 😉

[L3P3]

Great. And forget about my first checksum proposal, which is a waste of cpu/time. Just verify the checksum on the receiving end using sha... 👍
If the checksum mismatches, you could automatically resend the data using xz. 🤔

[L3P3]

Yeah, I assume that it is quite reliable. But in my opinion, we cannot guarantee an always correct decompression.
For some other applications, there are scientifical methods to prove its correctness. Maybe paq actually does checksum internally and tells you if there ways an issue. I am not sure.

[rafael2k]

If it safer if the data to be compresses has no framing? Lets say, I have a table of 50000 x (8x floats + 1x uint32_t + 1 uint8_t) to be compressed, but I just write a binary file and compress it, right? No conversion to string is better, correct?
I'm getting wonderful compression rates.
: )

[GotthardtZ]

Rafael2k, you may close this issue if you believe you got what you needed.