bincode-org / bincode Goto Github PK
View Code? Open in Web Editor NEWA binary encoder / decoder implementation in Rust.
License: MIT License
A binary encoder / decoder implementation in Rust.
License: MIT License
Deserialising a simple struct with a PathBuf
inside fails with the error: SyntaxError
Can be reproduced by this Gist:
https://gist.github.com/neon64/03230cb40ff3bafed8ee#file-bincode_pathbuf-rs
#![feature(plugin)]
#![feature(custom_derive)]
#![plugin(serde_macros)]
extern crate serde;
extern crate bincode;
use std::path;
use bincode::SizeLimit;
use bincode::serde::{serialize, deserialize};
#[derive(Serialize, Deserialize)]
struct MyData {
path: path::PathBuf
}
#[test]
fn it_works() {
let data = MyData {
path: path::Path::new("foo").to_path_buf()
};
let ser = serialize(&data, SizeLimit::Infinite).unwrap();
let de: MyData = deserialize(&ser).expect("bincode failed");
}
It seems like it is just limited to bincode, as serde_json works for example.
Cargo.toml
description still mentions rustc_serialize as a backend.
These are useful for conservatively serializing &str
and &[T]
then deserializing to Box<str>
and Box<[T]>
, which are analogous to String
and Vec<u8>
.
Have you considered storing the lengths of strings and sequences as Varints? The fixed 8-byte encoding of the length adds quite some space overhead when you store many shorter strings.
Just an idea. I could create a pull request if you like it.
Example:
extern crate rustc_serialize;
extern crate bincode;
use std::io::Cursor;
#[derive(RustcEncodable)]
struct FakeVec {
len: u64,
byte: u8
}
fn main() {
let x = bincode::encode(&FakeVec { len: 0xffffffffffffffffu64, byte: 1 }, bincode::SizeLimit::Bounded(10)).unwrap();
let y : Vec<u8> = bincode::decode_from(&mut Cursor::new(&x[..]), bincode::SizeLimit::Bounded(10)).unwrap();
println!("{:?} {:?}", y, x);
}
Output:
oom-bincode % ./target/debug/oom-bincode
zsh: illegal hardware instruction (core dumped) ./target/debug/oom-bincode
gdb backtrace:
Core was generated by `./target/debug/oom-bincode'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x00007f0712686ad0 in oom::hfcd4a1f5b5903c6dX5a ()
warning: Missing auto-load scripts referenced in section .debug_gdb_scripts
of file /home/cody/oom-bincode/target/debug/oom-bincode
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) bt
#0 0x00007f0712686ad0 in oom::hfcd4a1f5b5903c6dX5a ()
#1 0x00007f071266c2a4 in oom_bincode::vec::Vec<T>::with_capacity (capacity=18446744073709551615)
at /home/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcollections/vec.rs:223
#2 0x00007f0712670ca4 in fnfn (d=0x7ffd68fec418, len=18446744073709551615)
at /home/cody/.multirust/toolchains/nightly/cargo/registry/src/github.com-1ecc6299db9ec823/rustc-serialize-0.3.14/src/serialize.rs:458
#3 0x00007f071266f40c in oom_bincode::reader::DecoderReader<'a, R>.Decoder::read_seq<collections::vec::Vec<u8>,closure> (self=0x7ffd68fec418,
f={union Result<collections::vec::Vec<u8>, bincode::reader::DecodingError> (struct (&mut bincode::reader::DecoderReader<std::io::cursor::Cursor<&[u8]>>, usize))} 0x7ffd68fec360) at /home/cody/.multirust/toolchains/nightly/cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.3.0/src/reader.rs:339
#4 0x00007f071266f220 in oom_bincode::serialize::Vec<T>.Decodable::decode<bincode::reader::DecoderReader<std::io::cursor::Cursor<&[u8]>>> (d=0x7ffd68fec418)
at /home/cody/.multirust/toolchains/nightly/cargo/registry/src/github.com-1ecc6299db9ec823/rustc-serialize-0.3.14/src/serialize.rs:457
#5 0x00007f071266f1bf in oom_bincode::decode_from<std::io::cursor::Cursor<&[u8]>,collections::vec::Vec<u8>> (r=0x7ffd68fec568, size_limit=...)
at /home/cody/.multirust/toolchains/nightly/cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.3.0/src/lib.rs:138
#6 0x00007f0712669e3f in oom_bincode::main () at src/main.rs:15
#7 0x00007f07126814a9 in rust_try_inner ()
#8 0x00007f0712681496 in rust_try ()
#9 0x00007f071267ec04 in rt::lang_start::h054bbe55c321aa2c7Gw ()
#10 0x00007f0712672255 in main ()
(gdb)
This is so it is possible to parse from a large (e.g. ring) buffer and know when the current thing we are decoding has ended and another message has begun.
I was about to update the library to the latest version of rustc-serialize
(now with associated types), and I noticed the last commit.
Is it necessary, considering that the size limit can be easily enforce through the writer -- e.g. with a BufWriter
?
bincode::serde::deserialize_from(&mut file, SizeLimit::Infinite).expect("bincode deserialize error");
thread '<main>' panicked at 'bincode deserialize error: InvalidEncoding(InvalidEncoding { desc: "invalid tag when decoding Option", detail: Some("Expected 0 or 1, got 11") })', ../src/libcore/result.rs:709
I'm going to attempt to build a repro case, but it might take me a while. Thought maybe this would mean more to you than me.
This issue was automatically generated. Feel free to close without ceremony if
you do not agree with re-licensing or if it is not possible for other reasons.
Respond to @cmr with any questions or concerns, or pop over to
#rust-offtopic
on IRC to discuss.
You're receiving this because someone (perhaps the project maintainer)
published a crates.io package with the license as "MIT" xor "Apache-2.0" and
the repository field pointing here.
TL;DR the Rust ecosystem is largely Apache-2.0. Being available under that
license is good for interoperation. The MIT license as an add-on can be nice
for GPLv2 projects to use your code.
The MIT license requires reproducing countless copies of the same copyright
header with different names in the copyright field, for every MIT library in
use. The Apache license does not have this drawback. However, this is not the
primary motivation for me creating these issues. The Apache license also has
protections from patent trolls and an explicit contribution licensing clause.
However, the Apache license is incompatible with GPLv2. This is why Rust is
dual-licensed as MIT/Apache (the "primary" license being Apache, MIT only for
GPLv2 compat), and doing so would be wise for this project. This also makes
this crate suitable for inclusion and unrestricted sharing in the Rust
standard distribution and other projects using dual MIT/Apache, such as my
personal ulterior motive, the Robigalia project.
Some ask, "Does this really apply to binary redistributions? Does MIT really
require reproducing the whole thing?" I'm not a lawyer, and I can't give legal
advice, but some Google Android apps include open source attributions using
this interpretation. Others also agree with
it.
But, again, the copyright notice redistribution is not the primary motivation
for the dual-licensing. It's stronger protections to licensees and better
interoperation with the wider Rust ecosystem.
To do this, get explicit approval from each contributor of copyrightable work
(as not all contributions qualify for copyright, due to not being a "creative
work", e.g. a typo fix) and then add the following to your README:
## License
Licensed under either of
* Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
* MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
at your option.
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any
additional terms or conditions.
and in your license headers, if you have them, use the following boilerplate
(based on that used in Rust):
// Copyright 2016 bincode developers
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
It's commonly asked whether license headers are required. I'm not comfortable
making an official recommendation either way, but the Apache license
recommends it in their appendix on how to use the license.
Be sure to add the relevant LICENSE-{MIT,APACHE}
files. You can copy these
from the Rust repo for a plain-text
version.
And don't forget to update the license
metadata in your Cargo.toml
to:
license = "MIT/Apache-2.0"
I'll be going through projects which agree to be relicensed and have approval
by the necessary contributors and doing this changes, so feel free to leave
the heavy lifting to me!
To agree to relicensing, comment with :
I license past and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.
Or, if you're a contributor, you can check the box in this repo next to your
name. My scripts will pick this exact phrase up and check your checkbox, but
I'll come through and manually review this issue later as well.
When a encoding a structure that contains a fixed length array the length of the array is also encoded. For example, this code:
extern crate rustc_serialize;
extern crate bincode;
use bincode::SizeLimit;
use bincode::rustc_serialize::encode;
#[derive(RustcEncodable)]
struct TestStruct {
a : u16,
b : [u8; 6],
}
fn main() {
let test = TestStruct {a: 0xffff, b: [1, 2, 3, 4, 5, 6]};
let encoded: Vec<u8> = encode(&test, SizeLimit::Infinite).unwrap();
println!("header: {:?}", encoded);
}
Produces this output:
header: [255, 255, 0, 0, 0, 0, 0, 0, 0, 6, 1, 2, 3, 4, 5, 6]
This behaviour makes it hard to use bincode for unpacking externally defined binary formats (e.g ELF headers). I suppose this library isn't really designed for that, but it works well enough for BE formats so it would be nice to see this fixed in the next major version.
Rust 1.15 is out, so that means we can have 1.0 now, right? :)
Will this library be suitable serialization in rust, and then deserialization in another language? The use case I have in mind is running expensive numerical simulations in rust, saving the result using e.g. bincode, and then performing some data analysis in python.
See #95 (comment).
Sure! Accepting a ?Sized type parameter for io::Read and io::Write means that it will work with &mut io::Read and &mut io::Write trait objects which are dynamically sized.
use std::fs::File;
use std::io::Read;
fn main() {
let mut data = File::open("important.bin").unwrap();
let trait_object: &mut Read = &mut data;
deserialize_from(trait_object); // requires ?Sized
}
fn deserialize_from<R>(_: &mut R) where R: Read {}
And accepting a ?Sized type parameter for &T means that it can work with dynamically sized implementations of Serialize, which includes str and [T] among others.
fn main() {
serialize("&str"); // requires ?Sized
}
trait Serialize {}
impl Serialize for str {}
fn serialize<T>(_: &T) where T: Serialize {}
I have to admit I'm not 100% sure whether this problem is on bincode or flate2 (Zlib Encoder/Decoder)'s side, but I'm having trouble decoding a file I made on Linux in Windows 10. The file opens fine on the Linux install (and it's the same physical box so no weird hardware issues). Same version of Rust on both (nightly March 9, 2016). The struct is:
pub enum RecordingState {
Action(Action),
Reset(AleSystemState),
ChangeGame(Rom,f32,i32),
}
pub struct Rom {
pub rom_path: PathBuf,
pub data: Vec<u8>,
}
And AleSystemState is encoded/decoded as a Vec<i8>
. The file in question is a series of individual encodes of RecordingState
s (not a Vec, a series of individual ones). The first one is always a ChangeGame(Rom,f32,f32)
instruction. Action is just pub struct Action(pub i32);
.
I managed to track the error down to the line:
let (rom,repeat_prob,seed) = match rustc_serialize::decode_from(&mut self.decoder,
bincode::SizeLimit::Infinite)
.expect("Could not decode ROM as initial replay step") {
ChangeGame(rom,repeat_prob,seed) => (rom,repeat_prob,seed),
_ => panic!("Initial value in recording isn't ChangeGame")
};
When I hit that line I do NOT get a normal Rust panic, instead I get:
fatal runtime error: out of memory
Illegal instruction
self.decoder
is a ZlibDecoder<BufReader<File>>
. The compression I encode with is Best
.
The file that's breaking is here:
https://www.dropbox.com/s/ry9zhzlanvxaqkr/data.games?dl=0
It looks like my project is on 0.4.0
, but I observe the same behavior after bumping to 0.5.1
even after re-encoding/decoding.
I'm trying to reproduce this with a smaller example, especially in order to determine if this is due to flate2 or this library. I'm only guessing bincode because of the size limit. I haven't had any luck triggering it so far though.
uint
is always serialized and deserialized as u64
. int
, however, uses write_be_int
/read_be_int
methods. This makes int
serialization potentially platform-dependent.
Got another bincode build issue.
/home/pez/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.0.13/src/reader.rs:126 self.read += cast(count).unwrap();
^~~~~~~~~~~~~~~~~~~~
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
Could not compile `bincode`.
Look at the current master, it seems you have fixed this but it's not sync up to Cargo.io
I would greatly appreciate if you would automate a sync between your master and Cargo.io so you don't forget. I'm running a startup and it causes productivity problems :(
readme currently uses encode/decode, methods are called serialize/deserialize now
also consider using serede as example
When the user has bounded the size during serialization, instead of checking the size bounds while writing to the out buffer, run a separate pass through the serialization API to check what the size would be before writing starts.
This means that all oversized messages will be caught before they pollute the output target.
I noticed you have fixed bugs with the new nightly but looks like you haven't propogated the changes to Cargo.
.rs:8:1: 8:51 error: can't find crate for rustc_serialize
/home/jp/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.0.12/src/lib.rs:8 extern crate "rustc-serialize" as rustc_serialize;
See #95 (comment).
Right now you treat structs like tuples, because they are laid out the same way in bincode's format. During serialization that works mostly fine (since serializers usually pass struct fields in the order in which they are declared). During deserialization you deserialize the struct through visit_seq
. Structs are meant to be deserialized through visit_map
.
See some context here: serde-rs/serde#177
I think there could be room for improvement in serde wrt struct serialization to make sure that the serializer can choose the order of the elements it receives instead of the struct's Deserialize
implementation.
cc @erickt
I see some Option::expect
calls in serialize_seq
and serialize_map
that should return errors instead.
let len = len.expect("do not know how to serialize a sequence with no length");
let len = len.expect("do not know how to serialize a map with no length");
Could the serialize_into
and deserialize_from
methods return the number of bytes written/read respectively? This would both handy and more efficient, since you already have to compute these values in these methods, and it thus saves an extra call to serialized_size
.
New rust version makes a breaking change:
error: the parameter type `T` may not live long enough [E0310]
help: consider adding an explicit lifetime bound `T: 'static`...
I didn't see this on the breaking changes list, but I ended up having to modify my code as well, though wasn't thrilled with using static. Anyways, just giving you a heads up! Thanks for your support :D
With rustc (1.0.0-beta.3 (5241bf9c3 2015-04-25) (built 2015-04-25))
I just created a dependency with bincode = "*"
in my code. Now cargo build is saying
/Users/XXX/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.1.2/src/lib.rs:6:1: 6:21 error: unstable feature
/Users/XXX/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.1.2/src/lib.rs:6 #![feature(unicode)]
^~~~~~~~~~~~~~~~~~~~
note: this feature may not be used in the beta release channel
error: aborting due to previous error
Could not compile bincode
.
It looks like 0.4.0 doesn't list a version range for serde, causing builds to fail because Cargo is pulling in version 0.7.0.
Like if I wanted to read the segment commands of Mach-O from a file into structs.
warning: feature has been added to Rust, directive not necessary
/Users/sven/.cargo/registry/src/github.com-1ecc6299db9ec823/bincode-0.0.1/src/lib.rs:5 #![feature(struct_variant)]
^~~~~~~~~~~~~~
Serde 0.9.0-rc1 is out (serde-rs/serde#668) and there are some breaking changes.
It would be great to have a support for it. Thanks.
@TyOverby Do you mind if I publish 0.5.0 now on crates.io?
I'm not sure if this is the right place for this, but
I am unable to use bincode to serialize and deserialize structs where the serialize
and deserialize
traits are implemented instead of derived.
For example, I copied the Point struct, and the serialize and deserialize implementations used in the "Serliazation without macros" section of the readme here, https://github.com/serde-rs/serde#serialization-without-macros and attempted to use bincode to serialize and deserialize a Point variable. I was unable to do so and got this error
thread '
' panicked at 'calledResult::unwrap()
on anErr
value: SyntaxError',
../src/libcore/result.rs:738
Process didn't exit successfully:target/debug/tcpLocal
(exit code: 101)
If I instead derive the serialize
and deserialize
traits, I can serialize and deserialize the Point variable fine. Using the copied implementations of serialize
and deserialize
, I can serialize and deserialize from JSON just fine.
Is this expected? Did I miss something where only serializing with macros is supported in bincode?
My Cargo.toml looks like this:
[dependencies]
bincode = "*"
However it cannot build because it is trying to fetch an old version of binorder: v.0.2.14
I've noticed in your Cargo.toml it states it should be using v0.3.1 but it's ^v0.2.9 on the Cargo web site.
https://crates.io/crates/bincode
I've tried doing cargo update and that didn't help.
It is often necessary for binary protocols.
A la:
#[derive(RustcEncodable, RustcDecodable, PartialEq)]
struct Entity {
#[bits(4)]
x: u8,
#[bits(4)]
y: u8,
}
The fields of this struct can go right in the Error enum
As a new rustacean, I've read elsewhere that Serde is preferred over rustc_serialize. Would you please add a full example of using bincode with serde?
New release made some breaking changes, and it'd be great to have support.
"binary-encode" doesn't advertise the fact that there is a decoder and it also sounds like it is dealing with 1s and 0s.
Strings are encoded with a prefix of their byte length, but the decoder takes that length and reads that many char
s (aka unicode codepoints/Unicode scalar values). E.g. å
is 2 bytes in UTF-8 so encoding "å"
will write [2, 0xXX, 0xYY]
(I don't know the exact encoding of å), and then decoding will first read the 2 bytes of the å
and then try to read another codepoint from the stream, even though there's no string data left.
Trying to run tests with nightly rustc 1.11.0-nightly (bb4a79b08 2016-06-15)
.
tests/test.rs:405:68: 405:68 error: mismatched types [E0308]
tests/test.rs:405 #[derive(RustcEncodable, RustcDecodable, Serialize, Deserialize, Debug)]
^
tests/test.rs:405:57: 405:68 note: in this expansion of try! (defined in <std macros>)
tests/test.rs:405:57: 405:68 note: in this expansion of #[derive_Deserialize] (defined in tests/test.rs)
tests/test.rs:405:68: 405:68 help: run `rustc --explain E0308` to see a detailed explanation
tests/test.rs:405:68: 405:68 note: expected type `serde::Deserialize`
tests/test.rs:405:68: 405:68 note: found type `serde::Deserialize`
tests/test.rs:405:57: 405:68 note: the lifetime 'a as defined on the block at 405:56...
tests/test.rs:405 #[derive(RustcEncodable, RustcDecodable, Serialize, Deserialize, Debug)]
^~~~~~~~~~~
tests/test.rs:405:68: 405:68 note: ...does not necessarily outlive the static lifetime
As far as I can tell this error points RefBox
's lifetimes.
Is this something that could be implemented, or does it mean my code is doing something wrong?
I'm trying to deserialize a chrono::NaiveDateTime
which bincode serialized just fine.
Once RFC 213 is implemented, we should allow optional endiannes to be added to the bincode encoders and decoders.
bincode::encode(something);
bincode::encode::<LittleEndian>(something);
See http://carllerche.github.io/bytes/bytes/trait.MutBuf.html
(And same with Deserializer
/ Buf
/ io::Read
)
This needs to find replacement methods for reading ints/floats.
Servo depends on bincode and it would be nice if releases were made more regularly.
bincode attempts to read a tag when there isn't one. Rather than reading a tag, the value should be compared to the value of each of the variants.
The current error messages have an unfriendly Display representation. For example SerializeError in 9e47b71:
impl fmt::Display for SerializeError {
fn fmt(&self, f: &mut fmt::Formatter) -> Result<(), fmt::Error> {
match *self {
SerializeError::IoError(ref err) => write!(f, "IoError: {}", err),
SerializeError::Custom(ref s) => write!(f, "Custom Error {}", s),
SerializeError::InfiniteSequence => write!(f, "InfiniteSequence"),
SerializeError::SizeLimit => write!(f, "SizeLimit"),
}
}
}
Let's audit all the error messages in this crate and make sure everything has a self-explanatory message that a human can read and understand what went wrong.
Example of some code that fails to run:
https://gist.github.com/iqualfragile/743cc39ac4199e0f669bae13b778474e
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.