Comments (12)
Hi! The use case is for very low bitrate email transmission over HF skywave radio transmission [1]. So the processing overhead is really worth. 10% gain is very relevant. Thanks for the checksum suggestion, I will apply it to be in the safe side. Thanks!!
: )
[1] https://www.rhizomatica.org/hermes/
from paq8px.
I have a feeling the errors might be very infrequent, right? I got no errors in the tests I carried up to now (including emails with base64 attachments or even binary encoded image and audio). So you are right, may be just drop the data in case of bad checksum in the rx side in order to avoid compression / decompression in the sender side.
Btw, here is the code which does the e-mail magic:
https://github.com/Rhizomatica/hermes-net/tree/main/uuxcomp
from paq8px.
Generally paq8px is not recommended for any production use. It's an experimental software, a playground so to speak.
Though it is quite stable these days, this software doesn't go through thousands or millions of test runs. So really nothing is guaranteed.
If it runs fine for you, and you tested it for your use case and it has a significant advantage vs. the risk why not to use it, right?
You just need to make sure that you mitigate the risk as much as you can and you have a "backup" solution or "alerting mechanism" in case of a failure.
Maybe paq actually does checksum internally and tells you if there ways an issue. I am not sure.
It doesn't have any checksum checking. You'll need to impelement sending a checksum over the line and veryfining it at the receiving end. To be even safer, you'll need to measure decompression runtime at the receiving end and if it is higher than an expected upper limit, you'll need to kill the decompression process. In the past it happened that decompression was stuck due to a bug.
I believe with these two extra measures, your risk is mitigated to an absolute minimum.
from paq8px.
but I just write a binary file and compress it, right?
Yes.
No conversion to string is better, correct?
Correct.
To boost compression, you may insert a dummy byte with a value which occurs very rarely in your data. Paq8px will then detect that there is a repeating columnar pattern there and so it will pick up the record size easily.
If your data already has such a (nearly) constant column (=unique byte in any of your fields), then there is nothing to do, record size detection will be using it.
from paq8px.
Tks! The records are constant size and indeed, the first field of the record is uint32_t epoch time, in which the MSB byte most likely will not change for some time!
from paq8px.
So, I integrated to my solution to grab sensors data here: https://github.com/Rhizomatica/hermes-sensors
For now, I'll just let the user inform in case of some kind of corruption, as I believe it is stable enough with this kind of well formatted data.
from paq8px.
Thanks for all the support. I do consider paq8px to be the highest compression state-of-the-art codec I know about to be able to be applied to the real world use-cases. Cheers and thanks for the great software!
from paq8px.
If the huge processing overhead is really compensated by saved transfer time, I would already be surprised.
Apart from that, there have been bugs in the past that caused files to incorrectly decompress. Of course, you can just decompress and checksum the file before transfer. If your decompression result is correct, it will be on the receiving end.
Also, you can just transfer the sha sum along with the paq file and verify the decompression result on the receiving end. This way, there might be transfer failures but there will not be incorrect results. 😉
from paq8px.
Great. And forget about my first checksum proposal, which is a waste of cpu/time. Just verify the checksum on the receiving end using sha... 👍
If the checksum mismatches, you could automatically resend the data using xz. 🤔
from paq8px.
Yeah, I assume that it is quite reliable. But in my opinion, we cannot guarantee an always correct decompression.
For some other applications, there are scientifical methods to prove its correctness. Maybe paq actually does checksum internally and tells you if there ways an issue. I am not sure.
from paq8px.
If it safer if the data to be compresses has no framing? Lets say, I have a table of 50000 x (8x floats + 1x uint32_t + 1 uint8_t) to be compressed, but I just write a binary file and compress it, right? No conversion to string is better, correct?
I'm getting wonderful compression rates.
: )
from paq8px.
Rafael2k, you may close this issue if you believe you got what you needed.
from paq8px.
Related Issues (17)
- Add build instructions for linux HOT 10
- Direct buffer compression HOT 5
- Python interface? HOT 1
- Not a real issue but a request HOT 3
- cannot find .exe for windows HOT 2
- Text detection HOT 1
- Not issue but a request. ###How can I increase the compression speed? HOT 6
- Release new Windows Builds and/or update Windows Build Instructions with requirements HOT 2
- vTune Profile HOT 1
- Add info on max input file size HOT 6
- cannot build in linux HOT 2
- Cannot compile on debian HOT 2
- Warnings at compile time HOT 5
- Optimizing text compression HOT 2
- Linux compile problem HOT 1
- Compression Failed on -1 parameter HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from paq8px.