avl / savefile Goto Github PK
View Code? Open in Web Editor NEWAn easy to use library to save arbitrary rust data-structures to disk (or serialize to any other stream)
License: MIT License
An easy to use library to save arbitrary rust data-structures to disk (or serialize to any other stream)
License: MIT License
Loading data stored by 0.16 can lead to data-corruption when loading it using 0.17 betas.
This must be fixed before 0.17 is released.
Removes the need to keep two versions in a Cargo.toml
in sync.
See https://github.com/clap-rs/clap/blob/ca855c651c4fbd89accc40b20d442967f1e79942/Cargo.toml#L89
For example, rules for removal of field are wrong.
Some missing words.
There's some nonidiomatic english.
Some of the user facing methods are not documented (save, load).
Some structs which should not be used by user are documented and show prominently in docs.
No hyperlinks to types from text.
Any thoughts on adding an encryption layer? I'm not thinking anything fancy, just simple symmetrical encryption with a preshared key.
I'm inquiring because I'm going to try to use this for state dumps of my applications (and in the future use this to transfer data between applications and datacenters) and encryption natively would remove complexity from my future plans.
Something like this (assuming your demo code):
save_encrypted_file("save.bin", 0, player, "my symmetrical key is this").unwrap();
savefile-derive should probably always derive the traits from the savefile crate, even if the caller has:
1: Not imported the symbols from the savefile crate
2: Created their own symbols with colliding names, like a Serialize trait, for instance.
I think there are probably bugs here right now.
When compiling savefile, there are a few compiler warnings. I think all of them are benign, but this should definitely be fixed.
Would it be in scope for this project?
#[default_trait="TraitName"] is not implemented yet.
This is a way to be able to provide arbitrary default values of complex types, to fields which did not exist in older versions of protocol.
Savefile uses a whole lot of attributes right now, all with the prefix 'savefile_' in their names.
It would probably be better if instead the attribute was called 'savefile', and then different arguments to this single attribute were used.
This would be a breaking change without a huge tangible benefit. There are a few ways forward
This is not a real issue.
Given
#[derive(Clone, Debug, Default, Savefile)]
enum ControAction {
..
}
#[derive(Clone, Debug, Default, Savefile)]
struct ControllerConfig {
input_map: InputMap<ControAction>
// input_map: Vec<KeyCode, ControlAction>
}
// Where
// KeyCode is from https://docs.rs/bevy/latest/bevy/input/keyboard/enum.KeyCode.html
// InputMap is from https://docs.rs/leafwing-input-manager/0.5.2/leafwing_input_manager/input_map/struct.InputMap.html
I couldn't find a way to save structs which invole external structs and got various erros from my attempts.
# for input_map: InputMap<ControAction>
the trait `_IMPL_SAVEFILE_SERIALIZE_FOR_SaveData::_savefile::Serialize` is not implemented for `InputMap<ControlAction>`
# for input_map: Vec<KeyCode, ControlAction>
the trait `_IMPL_SAVEFILE_SERIALIZE_FOR_SaveData::_savefile::Serialize` is not implemented for `std::vec::Vec<(bevy::prelude::KeyCode, ControlAction)>`
the trait `WithSchema` is not implemented for `bevy::prelude::KeyCode`
Is saving external structs supported?
Document the rules for applying the #[versions] tag the right way.
See PR #32
Document how to change type of a field from one version to the next.
I will PR.
Savefile does not presently support recursive data-structures.
To be precise, the problem is with the schema-function. The schema-function tries to provide a simple tree which describes the data format. However, this fails if the data is recursive in nature.
Consider this data structure:
struct TreeNode(u32, Vec<TreeNode>);
The schema for TreeNode is basically a tuple of (u32,[TreeNode]). But savefile will try to evaluate the schema down to primitives, and will thus evaluate the schema of TreeNode again, yielding (u32,[(u32,[TreeNode])]), then (u32,[(u32,[(u32,[TreeNode])])]) etc in infinity.
Savefile is previously documented as not supporting cyclic data structures. But as now discovered, any recursive data structure, even without actual cycles, causes problems.
I can see 3 possible ways forward:
1: Just document that recursive data structures are not supported. Possibly add some sort of recursion-limit to detect the problem.
2: Add support for recursive data structures.
3: Possibly add full support for serializing/deserializing arbitrary graphs, including graphs with cycles
I am working with really large files and multi-threading can really help me out if implemented into Savefile.
I'm not really sure just how Savefile does the preparation of the data before writing it into a file because I have not read the code, but I imagine it would be really simple using the rayon crate.
Saving the files in the computer currently takes about 5-10 minutes as they are really large, so multithreading could REALLY improve this for saving and loading these types of files multiple times. Perhaps even you could make that optional somehow, some kind of feature that contains a save_file_par(...)
function that could work.
Doing something like this casuses segfault:
let boxed: Box = Box::new(ExampleImplementation{});
let conn = unsafe { AbiConnection::::from_boxed_trait(::ABI_ENTRY, boxed).unwrap() };
Hello, I want to have a reference to another derived struct. For example
#[derive(Savefile)]
pub struct A {
pub hello: String,
}
#[derive(Savefile)]
pub struct B<'a> {
pub world: &'a A,
}
What I get is the trait `savefile::Deserialize<'_>` is not implemented for `&'a LedPattern
and the same error for the rest of the required traits.
Any pointers?
See CVE-2023-22895
Hello again.
I have noticed that when savefile
is deserializing arrays, it uses the stack in unreasonable amounts, making it effectively impossible to use in spawned threads without setting an abnormally large stack size.
A minimal example:
#![feature(bench_black_box)]
std::thread::Builder::new()
.stack_size(2 * 1024 * 1024)
.spawn(|| {
println!(
"[u64; 16_000]: {} bytes",
std::mem::size_of::<[u64; 16_000]>()
);
savefile::save_file("debug.savefile", 0, &[0u64; 16_000]).unwrap();
println!("Saved, loading after 1 second.");
std::thread::sleep(std::time::Duration::from_secs(1));
let y: [u64; 16_000] = savefile::load_file("debug.savefile", 0).unwrap();
std::hint::black_box(y);
})
.unwrap();
std::thread::park();
In the example, I spawn a thread with a 2 MiB stack size, which should be more than enough, considering the array we're dealing with, as it will show when running, is barely 128 000 bytes.
Serializing the array works fine, no issues, but the deserialization results in a stack overflow (at least on my machine).
Is this expected behaviour?
It seems the standard Rust test framework now uses the identifier 'ignore' as an attribute.
This means it isn't very wise of Savefile to also use this attribute-name.
All savefile attributes should probably be renamed to contain 'savefile_' as a prefix in their name.
This will be a very breaking change.
I expect to make this change for the 0.4-release.
It uses the description from savefile-derive . Some typo/error somewhere.
Here are some of the features cited at the top of the main library file
#![allow(incomplete_features)]
#![cfg_attr(feature = "nightly", feature(test))]
#![cfg_attr(feature = "nightly", feature(specialization))]
#![cfg_attr(feature = "nightly", feature(integer_atomics))]
#![cfg_attr(feature = "nightly", feature(const_generics))]
However, integer_atomics have been stabilized in Rust 1.34 and from what I've seen, min_const_generics is sufficient to have Serialize impls for arrays. And it has been stabilized in Rust 1.51.
I'd also be interested to know where test feature is used in the code as I couldn't find it.
Rust RFC 3373: Avoid non-local definitions in functions was accepted and it's implementation at rust-lang/rust#120393 found that this crate would be affected by it.
To be more precise users of this crate would be affected by it, in the form of a warn-by-default lint: non_local_definitions
. This is because the derive macros from this crate use impl
in a local context, const _IMPL_SAVEFILE_REPRC_FOR_???
:
savefile/savefile-derive/src/lib.rs
Lines 1135 to 1136 in 2c5b36b
Fortunately a simple fix exist for this crate, by using a const-anon instead of named one:
#[allow(non_upper_case_globals)]
- const #dummy_const: () = {
+ const _: () = {
Be-aware, there are many instances of this pattern in the code, don't forget to replace them all.
I would suggest applying some form of the patch above as well as releasing a patch version of this crate, as to have a fix available for users as soon as possible.
cc @avl
It would be great to have an example of a custom serialization, using WithSchema
, Serialize
, and Deserialize
. I'd like to serialize an enum that has a PathBuf
in it, but it's tricky to figure out how this is supposed to work.
There's a bug that means that if the read of the file isn't chunked exactly the same way as the write, random junk will be read.
I'm unsure what the consequences are in practice, but since I believe no one is using 0.4.0, the solution is simply to upgrade to 0.4.1.
0.4.1 has a stronger test bench, hopefully meaning there are no similar bugs in 0.4.1.
Support for the std BTreeSet type would be nice.
As an example, the following fails to compile.
#[derive(Savefile)]
struct Data {
data: std::collections::BTreeSet<u32>,
}
Thanks!
The following code:
#[derive(Savefile)]
struct Foo<T> {
pub bar: T
}
yields errors, that T doesn't implement Serialize
, Deserialize
, etc., but this should be possible - implementation, that works only when T fulfills the constraints should be derived.
Eg. in following code:
#[derive(Clone)]
struct Foo<T> {
pub bar: T
}
Clone
derive macro generates this implementation: impl<T: core::clone::Clone> core::clone::Clone for Foo<T> { /*... */}
where T is constrained in the implementation instead of erroring out. Savefile derive macro should behave similarly.
Savefile-derive currently fails if applied to a unit struct.
I.e, the following does not compile:
#[derive(Savefile)]
struct MyUnitStruct;
"Serializing" such a struct is of course super-simple (since it's stateless). So this should be supported. It's just a mistake.
I have this corrupted file, that I'm trying to load using the following code:
let map_loader = load_encrypted_file("./tgr/7.bin", 0, "mykeyhere");
Loading this file causes the following error:
memory allocation of 11159284083750323767 bytes failedAborted (core dumped)
It seems to me this should throw an error and not cause a core dump.
EDIT. It looks any binary file will cause this. I used dd if=/dev/urandom of=7.bin bs=1024 count=3
to make one, and it core dumped like above
Savefile uses the 'failure' crate to do its internal error handling.
This makes the user experience suboptimal, if the user doesn't also use 'failure'.
This should be reworked, so that the Error type used by Savefile implements std::error::Error, presumably making it more compatible with any error handling crate.
"Don't pay for what you don't use"
Bzip2 is good, but packaging a full bzip2 encoder/decoder even if you don't use compression is a bit cumbersome.
Same goes for encryption for example.
I suggest putting theses behind conditional features.
Cargo's default features can be use to keep backward's compatibility.
I am using savefile 0.8.3 on win10. With the official example, I modified a little like this:
use std::collections::VecDeque;
use savefile::prelude::*;
#[macro_use]
extern crate savefile_derive;
pub struct MyPathBuf {
path: String,
}
use savefile::prelude::*;
impl WithSchema for MyPathBuf {
fn schema(_version: u32) -> Schema {
Schema::Primitive(SchemaPrimitive::schema_string)
}
}
impl Serialize for MyPathBuf {
fn serialize<'a>(&self, serializer: &mut Serializer<'a>) -> Result<(), SavefileError> {
self.path.serialize(serializer)
}
}
impl Deserialize for MyPathBuf {
fn deserialize(deserializer: &mut Deserializer) -> Result<Self, SavefileError> {
Ok(MyPathBuf {
path: String::deserialize(deserializer)?,
})
}
}
#[derive(Savefile)]
struct Player {
name: String,
strength: u32,
inventory: VecDeque<MyPathBuf>, //Vec<String>,
}
fn main() {
let player = Player {
name: "Steve".to_string(),
strength: 42,
// inventory: Vec::with_capacity(100),
inventory: VecDeque::with_capacity(100),
};
}
But it raised error like --- the trait bound
MyPathBuf: Introspectis not satisfied required because of the requirements on the impl of
Introspectfor
VecDequerequired for the cast to the object type
dyn Introspect``
Does it mean I need to implement "introspect"? If so , how?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.