only-cliches / noproto Goto Github PK
View Code? Open in Web Editor NEWFlexible, Fast & Compact Serialization with RPC
License: MIT License
Flexible, Fast & Compact Serialization with RPC
License: MIT License
Awesome work here! Reviewing the documentation, I don't see an immediately obvious option for being able to specify that a field may be one of a set of types. In Rust this would typically look like:
enum Variants {
ObjA(ObjA),
ObjB(ObjB),
}
I see that enum
s can be used for a known collection of ordered strings, and I see that there is also the any
type. Would the idea be to use a combination of those items within a struct?
Hey ๐ Another cool project of yours ๐
The README mentions that the library is WASM-ready. But is there any example of how to use it from Javascript?
16 bit addresses are more compact but limit the address space to 64KB, switching to 32 bit addresses will allow buffers of up to 4GB in size.
When calling NP_RPC_Factory::new_bytes() with an empty byte slice, it causes a thread panic.
I am new to Rust, so excuse my question: I would have expected the function to fail with an NP_Error.
I am loading the rpc spec in bytes from a remote endpoint, so I wanted to check for validity.
On line https://github.com/only-cliches/NoProto/blob/master/bench/src/run_bench_flatbuffers.rs#L40, using the FlatBufferBuilder::reset
method should greatly improve your FlatBuffers write performance. To do this, you'll need to make sure to-reuse a FlatBufferBuilder
instead of creating a new one each time through the loop.
The benchmarks right now compare NoProto to formats that are fairly different - either the data carries along a schema (json), or the schema is precompiled (protobuf).
I think flexbuffers and avro are probably much better comparisons, as they're both formats where the schemas can be loaded dynamically. I think it would be very interesting to see comparisons to those two.
Hello!
I found weird bug. I tried to wrap NP_Buffer in struct like this:
use no_proto::{NP_Factory, buffer::NP_Buffer, error::NP_Error, memory::NP_Memory_Owned};
struct Test {
buffer: NP_Buffer<NP_Memory_Owned>
}
impl Test {
fn new(field: i64) -> Result<Self, NP_Error> {
let factory = NP_Factory::new(r#"
struct({fields: {
field: i64()
}})
"#)?;
let mut test = Self {
buffer: factory.new_buffer(None)
};
test.set_field(field)?;
Ok(test)
}
fn get_field(&self) -> Result<Option<i64>, NP_Error> {
self.buffer.get::<i64>(&["field"])
}
fn set_field(&mut self, field: i64) -> Result<bool, NP_Error> {
self.buffer.set(&["field"], field)
}
}
fn main() -> Result<(), NP_Error> {
let test = Test::new(0)?;
assert_eq!(test.get_field()?, Some(0));
Ok(())
}
On Windows 10 this code:
test.get_field()
returns Ok(None)
error: process didn't exit successfully: 'target\debug\np_test.exe' (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
On Ubuntu (WLS) this code always crashes with
Segmentation fault (core dumped)
Without struct everything work fine.
Would be nice to use no_proto in places were trying to be as minimal as possible (like web assembly)
It looks like that RPC is done differently in 0.10, but is not quite implemented yet. Is that correct?
Could you summarize the principle differences between 0.10 and 0.9.6? It is difficult to see in the code, because of the move of the old source tree.
Thank you!
You cannot build with any version of no_proto, the library does not work. It throws more than 71 serious bugs in version 9.6.0.
In the readme, in comparisons against each of Avro, Protobuf, JSON, and Flatbuffers, each comparison lists "Safely handle untrusted data" as a benefit of Noproto. I can't speak for Avro or FlatBuffers, but Protobuf is regularly used to handle untrusted data, and JSON is obviously used by basically everyone to handle untrusted data. So, these bullet points seem incorrect.
Later on, the readme says this:
When to use Flatbuffers / Bincode / CapN Proto
If you can safely compile all your data types into your application, all the buffers/data is trusted, and you don't intend to mutate buffers after they're created, Bincode/Flatbuffers/CapNProto is a better choice for you.
I can't speak for Faltbuffers or Bincode, but Cap'n Proto is explicitly designed for and heavily used in sandboxing scenarios involving unstruted data. In particular, Cap'n Proto was originally built to serve as the protocol that Sandstorm.io uses to communicate with untrusted apps running in secure sandboxes. It is believed to be secure for such use cases.
The statement also seems to imply that Cap'n Proto can only operate on schemas known at compile time. This is incorrect: Cap'n Proto features a mechanism for loading schemas dynamically and using them to operate on types with a reflection-like API. Incidentally, Protobuf features a very similar API. Practical experience has shown that these features are almost never used: people almost always know their schemas at compile time, and the benefits of compile-time type checking and optimizations are large. But dynamic schema loading can be useful in some niche scenarios.
Finally, regarding "mutation": Cap'n Proto actually supports mutation, with caveats. Fixed-width fields can be mutated in-place, but variable-width data will require new allocations which are always added to the end of the message. Removing an object will leave a zero'd "hole" in the data, which can only be reclaimed by making a full-tree copy. In principle, a Cap'n Proto implementation could use a complex memory allocation algorithm that is able to reuse holes in-place for later allocations of same or smaller size, but at present no implementation bothers to attempt this. This problem is generally fundamental to zero-copy formats, due to the need to allocate contiguous memory for each message. If NoProto has solved it somehow, it would be great to see some explanation of how; I couldn't immediately find it in the docs.
It would be nice to add Veriform to the comparisons list.
I'd love to see the bincode
crate included in the benchmarks and comparisons.
Hi,
I try to use NoProto and Sled as a simple DB and figured it would make sense to parse the Schema only once.
However, its all across an async code base and the inserts and updates would come only after some awaits happened.
This is seems to be a roadblock with NoProto, as NP_Factory
is not Send
.
I basically
Compiling my simple PoC gives me:
error: future cannot be sent between threads safely
--> src/main.rs:82:38
|
82 | let server = Server::bind(&addr).serve(make_service);
| ^^^^^ future returned by `proxy` is not `Send`
|
= help: within `no_proto::NP_Factory`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<no_proto::schema::NP_Schema>`
note: future is not `Send` as this value is used across an await
I guess an Arc
instead of an Rc
(in NP_Factory
and NP_SchemaKinds
) would solve it.
Or am I doing it wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.