Comments (4)
We're thinking of switching to the read_buf
API (tracking issue, RFC) for this, once that is stabilized. Does that cover your use-case?
from bincode.
We're thinking of switching to the
read_buf
API (tracking issue, RFC) for this, once that is stabilized. Does that cover your use-case?
Unfortunately, no. read_buf
is to avoid initialization. Maybe I did not describe it clearly. I'd like to provide some code samples.
For example, I have a struct:
#[derive(Serialize, Deserialize)]
struct MyData {
// Serde will treat Vec<u8> as an array.
// So, we need `serde_bytes` here.
// Or, we can use String as an example.
#[serde(with = "serde_bytes")]
bytes: Vec<u8>,
// ..other fields
}
I memory-map a file into memory:
let mmap = memmap2::MmapMut::map_mut(&file)?;
let ptr = mmap.as_mut_ptr();
let len = mmap.len();
let reader = MyReader { ptr, len };
std::mem::forget(mmap);
Then I implement a custom BincodeReader:
struct MyReader {
ptr: *mut u8,
len: usize,
}
impl<'a> BincodeRead<'a> for MyReader {
fn get_byte_buffer(&mut self, length: usize) -> bincode::Result<Vec<u8>> {
if self.len < length {
// ...error handling stuff
}
// construct the Vec just on the mapped memory
let vec = unsafe { Vec::from_raw_parts(self.ptr, length, length) };
self.ptr = unsafe { self.ptr.add(amt) };
self.len -= amt;
Ok(vec)
}
// ...other methods
}
Finally,
let data: MyData = bincode::deserialize_from_custom(reader);
This will work if we are using a custom global allocator that ignores deallocations in mapped heap region (otherwise it will corrupt). You can see that data.bytes
will right point to the mapped memory: it is completely copy-free, and the copying from kernel file buffer to userspace is also eliminated.
from bincode.
I think I understand what you want for your use case.
Unfortunately the bincode trait can't return Vec
directly any more because we want to make embedded systems a first class citizen for bincode. I don't see an easy way to add a specialized function that returns a Vec
only if alloc
is enabled, this sounds like it'd break horribly if bincode is somewhere in a complex dependency tree.
It would be nice if we could do something like
trait CustomAllocator {
fn allocate_vec_in_place(...);
}
impl<T, A> bincode::Decode for Vec<T, A>
where T: bincode::Decode,
A: CustomAllocator
{
// ...
}
impl<T> bincode::Decode for Vec<T, Global>
where T: bincode::Decode
{
// ...
}
But that
- sounds like it needs implementation specialization
- sounds very specific to your use case and not something we can implement globally.
For now I think the best solution would be to have your own Vec
wrapper:
struct CustomAllocVec<T>(Vec<T, YourAllocator>);
impl<T: Decode> Decode for CustomAllocVec<T> {
// you can do your custom logic here
}
But that would require having control over all vecs in your dependency tree
from bincode.
But that
- sounds like it needs implementation specialization
- sounds very specific to your use case and not something we can implement globally.
Yes, you're right. Maybe we could do it with something that is similar to #[serde(with="...")]
. It would be nice if we could achieve proxying using bincode's encode and decode, e.g.:
#[derive(Encode, Decode)]
struct MyData {
#[bincode(decode_with="zero_copy_decode_vec")]
bytes: Vec<u8>,
}
It is somewhat out of current topic, though. And we can still do it by implementing custom decode for the whole struct or with bincode::serde
.
from bincode.
Related Issues (20)
- Deserialize external tag to separate type variable HOT 1
- Arc/Rc handling clarification HOT 1
- Is there a way to patch 1.x? HOT 1
- Add codec support for chrone HOT 3
- Unclear why serialized_size() yields different results HOT 1
- Does bincode guarantee a 1:1 relationship between encodings and values? HOT 3
- bincode::de::Deserializer::with_reader() doesn't seem to work HOT 1
- Issue when decoding a simple Enum with no member values.
- make config::DefaultOptions::new `const` HOT 1
- Add note about Maps to readme HOT 7
- Cannot deserialize VecDeque with structure of Decimals HOT 4
- Discriminant size in `fixint` serialization of `Option<T>` HOT 3
- Build pipeline fails HOT 1
- Support retrieving the bytes written from encode HOT 6
- Decode is only implemented for Arc<str> and not Rc<str>
- deserialize into allocator HOT 2
- falling test with serde::skip_serializing_if HOT 2
- Are initial "buffer size" bytes in bincode2 decode_from_slice really required? HOT 4
- BorrowDecode not implemented for slices of borrowed elements
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bincode.