Giter Site home page Giter Site logo

json's Introduction

Serde JSON   Build Status Latest Version Rustc Version 1.36+

Serde is a framework for serializing and deserializing Rust data structures efficiently and generically.


[dependencies]
serde_json = "1.0"

You may be looking for:

JSON is a ubiquitous open-standard format that uses human-readable text to transmit data objects consisting of key-value pairs.

{
    "name": "John Doe",
    "age": 43,
    "address": {
        "street": "10 Downing Street",
        "city": "London"
    },
    "phones": [
        "+44 1234567",
        "+44 2345678"
    ]
}

There are three common ways that you might find yourself needing to work with JSON data in Rust.

  • As text data. An unprocessed string of JSON data that you receive on an HTTP endpoint, read from a file, or prepare to send to a remote server.
  • As an untyped or loosely typed representation. Maybe you want to check that some JSON data is valid before passing it on, but without knowing the structure of what it contains. Or you want to do very basic manipulations like insert a key in a particular spot.
  • As a strongly typed Rust data structure. When you expect all or most of your data to conform to a particular structure and want to get real work done without JSON's loosey-goosey nature tripping you up.

Serde JSON provides efficient, flexible, safe ways of converting data between each of these representations.

Operating on untyped JSON values

Any valid JSON data can be manipulated in the following recursive enum representation. This data structure is serde_json::Value.

enum Value {
    Null,
    Bool(bool),
    Number(Number),
    String(String),
    Array(Vec<Value>),
    Object(Map<String, Value>),
}

A string of JSON data can be parsed into a serde_json::Value by the serde_json::from_str function. There is also from_slice for parsing from a byte slice &[u8] and from_reader for parsing from any io::Read like a File or a TCP stream.

use serde_json::{Result, Value};

fn untyped_example() -> Result<()> {
    // Some JSON input data as a &str. Maybe this comes from the user.
    let data = r#"
        {
            "name": "John Doe",
            "age": 43,
            "phones": [
                "+44 1234567",
                "+44 2345678"
            ]
        }"#;

    // Parse the string of data into serde_json::Value.
    let v: Value = serde_json::from_str(data)?;

    // Access parts of the data by indexing with square brackets.
    println!("Please call {} at the number {}", v["name"], v["phones"][0]);

    Ok(())
}

The result of square bracket indexing like v["name"] is a borrow of the data at that index, so the type is &Value. A JSON map can be indexed with string keys, while a JSON array can be indexed with integer keys. If the type of the data is not right for the type with which it is being indexed, or if a map does not contain the key being indexed, or if the index into a vector is out of bounds, the returned element is Value::Null.

When a Value is printed, it is printed as a JSON string. So in the code above, the output looks like Please call "John Doe" at the number "+44 1234567". The quotation marks appear because v["name"] is a &Value containing a JSON string and its JSON representation is "John Doe". Printing as a plain string without quotation marks involves converting from a JSON string to a Rust string with as_str() or avoiding the use of Value as described in the following section.

The Value representation is sufficient for very basic tasks but can be tedious to work with for anything more significant. Error handling is verbose to implement correctly, for example imagine trying to detect the presence of unrecognized fields in the input data. The compiler is powerless to help you when you make a mistake, for example imagine typoing v["name"] as v["nmae"] in one of the dozens of places it is used in your code.

Parsing JSON as strongly typed data structures

Serde provides a powerful way of mapping JSON data into Rust data structures largely automatically.

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize)]
struct Person {
    name: String,
    age: u8,
    phones: Vec<String>,
}

fn typed_example() -> Result<()> {
    // Some JSON input data as a &str. Maybe this comes from the user.
    let data = r#"
        {
            "name": "John Doe",
            "age": 43,
            "phones": [
                "+44 1234567",
                "+44 2345678"
            ]
        }"#;

    // Parse the string of data into a Person object. This is exactly the
    // same function as the one that produced serde_json::Value above, but
    // now we are asking it for a Person as output.
    let p: Person = serde_json::from_str(data)?;

    // Do things just like with any other Rust data structure.
    println!("Please call {} at the number {}", p.name, p.phones[0]);

    Ok(())
}

This is the same serde_json::from_str function as before, but this time we assign the return value to a variable of type Person so Serde will automatically interpret the input data as a Person and produce informative error messages if the layout does not conform to what a Person is expected to look like.

Any type that implements Serde's Deserialize trait can be deserialized this way. This includes built-in Rust standard library types like Vec<T> and HashMap<K, V>, as well as any structs or enums annotated with #[derive(Deserialize)].

Once we have p of type Person, our IDE and the Rust compiler can help us use it correctly like they do for any other Rust code. The IDE can autocomplete field names to prevent typos, which was impossible in the serde_json::Value representation. And the Rust compiler can check that when we write p.phones[0], then p.phones is guaranteed to be a Vec<String> so indexing into it makes sense and produces a String.

The necessary setup for using Serde's derive macros is explained on the Using derive page of the Serde site.

Constructing JSON values

Serde JSON provides a json! macro to build serde_json::Value objects with very natural JSON syntax.

use serde_json::json;

fn main() {
    // The type of `john` is `serde_json::Value`
    let john = json!({
        "name": "John Doe",
        "age": 43,
        "phones": [
            "+44 1234567",
            "+44 2345678"
        ]
    });

    println!("first phone number: {}", john["phones"][0]);

    // Convert to a string of JSON and print it out
    println!("{}", john.to_string());
}

The Value::to_string() function converts a serde_json::Value into a String of JSON text.

One neat thing about the json! macro is that variables and expressions can be interpolated directly into the JSON value as you are building it. Serde will check at compile time that the value you are interpolating is able to be represented as JSON.

let full_name = "John Doe";
let age_last_year = 42;

// The type of `john` is `serde_json::Value`
let john = json!({
    "name": full_name,
    "age": age_last_year + 1,
    "phones": [
        format!("+44 {}", random_phone())
    ]
});

This is amazingly convenient, but we have the problem we had before with Value: the IDE and Rust compiler cannot help us if we get it wrong. Serde JSON provides a better way of serializing strongly-typed data structures into JSON text.

Creating JSON by serializing data structures

A data structure can be converted to a JSON string by serde_json::to_string. There is also serde_json::to_vec which serializes to a Vec<u8> and serde_json::to_writer which serializes to any io::Write such as a File or a TCP stream.

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize)]
struct Address {
    street: String,
    city: String,
}

fn print_an_address() -> Result<()> {
    // Some data structure.
    let address = Address {
        street: "10 Downing Street".to_owned(),
        city: "London".to_owned(),
    };

    // Serialize it to a JSON string.
    let j = serde_json::to_string(&address)?;

    // Print, write to a file, or send to an HTTP server.
    println!("{}", j);

    Ok(())
}

Any type that implements Serde's Serialize trait can be serialized this way. This includes built-in Rust standard library types like Vec<T> and HashMap<K, V>, as well as any structs or enums annotated with #[derive(Serialize)].

Performance

It is fast. You should expect in the ballpark of 500 to 1000 megabytes per second deserialization and 600 to 900 megabytes per second serialization, depending on the characteristics of your data. This is competitive with the fastest C and C++ JSON libraries or even 30% faster for many use cases. Benchmarks live in the serde-rs/json-benchmark repo.

Getting help

Serde is one of the most widely used Rust libraries, so any place that Rustaceans congregate will be able to help you out. For chat, consider trying the #rust-questions or #rust-beginners channels of the unofficial community Discord (invite: https://discord.gg/rust-lang-community), the #rust-usage or #beginners channels of the official Rust Project Discord (invite: https://discord.gg/rust-lang), or the #general stream in Zulip. For asynchronous, consider the [rust] tag on StackOverflow, the /r/rust subreddit which has a pinned weekly easy questions post, or the Rust Discourse forum. It's acceptable to file a support issue in this repo, but they tend not to get as many eyes as any of the above and may get closed without a response after some time.

No-std support

As long as there is a memory allocator, it is possible to use serde_json without the rest of the Rust standard library. Disable the default "std" feature and enable the "alloc" feature:

[dependencies]
serde_json = { version = "1.0", default-features = false, features = ["alloc"] }

For JSON support in Serde without a memory allocator, please see the serde-json-core crate.


License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

json's People

Contributors

andrisak avatar arcnmx avatar bennofs avatar chanced avatar deankarn avatar dflemstr avatar dtolnay avatar erickt avatar frewsxcv avatar homu avatar imp avatar japaric avatar jplatte avatar jwilm avatar killercup avatar laktak avatar little-arhat avatar lnicola avatar lucacasonato avatar lucretiel avatar ndmitchell avatar nox avatar oli-obk avatar osiewicz avatar overdrivenpotato avatar rufflewind avatar s-panferov avatar srijs avatar vegai avatar xanewok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

json's Issues

Improve ergonomics: ToValue, impl FromStr, ToString

Recently I've moved my projects to Serde and I found it amazing, but it has several inconveniences:

  1. FromStr trait is not implemented for Value, so I can't use str.parse()
  2. rustc-serialize has ToJson trait which improves ergonomics a lot.
  3. ToString is not implemented for Value.

I've made several pull requests:

  1. #64 - create and impl ToJson trait
  2. #63 - impl Display and FromStr traits

I believe that this will improve ergonomics. This will also improve compatibility with rustc-serialize and simplify migration.

Ability to step past errors in StreamDeserializer if input is valid JSON

The current behavior of StreamDeserializer is that if it hits an error, that same error is returned in perpetuity which is typically not what you want. (If that is what you want, it is easy to implement that behavior atop what I propose below.)

Suppose we have a stream deserializer that expects to read a stream of Strings. If it receives the input "a" {"b": 0} "c", a reasonable thing to expect as output would be "a" (error) "c" because the second JSON object failed to deserialize to a String.

On the other hand, if the input is not a valid JSON stream I believe the StreamDeserializer should retain its current behavior because there is no way for it to regain its bearings in an invalid JSON stream. For example if the input is "a" {"b{{"c" "d", I would expect the output to be "a" (error) (error) (error...).

Bad error code propagated for missing field when deserializing a struct

When deserializing a structure from a JSON string that's missing a field, serde_json propagates an ExpectedSomeValue error code variant, not the MissingField error code variant.

The program below shows the behavior.

Expected output (roughly speaking):

Got: "missing field x" at line 3 column 5
OK, got 'missing field' x

Actual output:

Got: "expected value" at line 3 column 5
BAD, got 'expected some value'

Here's the program:

extern crate serde;
extern crate serde_json;

#[derive(Debug)]
struct Foo {
    x: i32,
    y: i32,
}

impl serde::Deserialize for Foo {
    fn deserialize<D>(d: &mut D) -> Result<Self, D::Error> where D: serde::Deserializer {

        enum Field {
            X,
            Y,
        }

        impl serde::Deserialize for Field {
            fn deserialize<D>(d: &mut D) -> Result<Self, D::Error> where D: serde::Deserializer {

                struct Visitor;

                impl serde::de::Visitor for Visitor {
                    type Value = Field;

                    fn visit_str<E>(&mut self, value: &str) -> Result<Self::Value, E>
                        where E: serde::de::Error
                    {
                        match value {
                            "x" => Ok(Field::X),
                            "y" => Ok(Field::Y),
                            _ => Err(E::unknown_field(value)),
                        }
                    }
                }

                d.visit(Visitor)
            }
        }

        struct Visitor;

        impl serde::de::Visitor for Visitor {
            type Value = Foo;

            fn visit_map<V>(&mut self, mut visitor: V) -> Result<Foo, V::Error>
                where V: serde::de::MapVisitor
            {
                let mut x = None;
                let mut y = None;

                loop {
                    match try!(visitor.visit_key()) {
                        Some(Field::X) => {
                            x = Some(try!(visitor.visit_value()));
                        }
                        Some(Field::Y) => {
                            y = Some(try!(visitor.visit_value()));
                        }
                        None => { break; }
                    }
                }

                try!(visitor.end());

                let x = match x {
                    Some(x) => x,
                    None => try!(visitor.missing_field("x")),
                };

                let y = match y {
                    Some(x) => x,
                    None => try!(visitor.missing_field("y")),
                };

                let v = Foo {
                    x: x,
                    y: y,
                };

                Ok(v)
            }
        }

        static FIELDS: &'static [&'static str] = &["x", "y"];
        d.visit_struct("Foo", FIELDS, Visitor)
    }
}

fn main() {

    let s = r#"{
        "y": 17
    }"#;

    let got = serde_json::from_str::<Foo>(&s).unwrap_err();

    println!("Got: {}", got);

    match got {
        serde_json::Error::SyntaxError(code, _line, _column) => {
            match code {
                serde_json::ErrorCode::MissingField(field) => {
                    println!("OK, got 'missing field' {}", field);
                }
                serde_json::ErrorCode::ExpectedSomeValue => {
                    println!("BAD, got 'expected some value'");
                }
                _ => { println!("BAD, got some other error code variant"); }
            }
        }
        _ => { println!("BAD, Got unexpected error variant"); }
    }
}

Remove escape_bytes in 0.8.0

I don't see a reason that this was part of the public API in the first place. GitHub search does not show any callers. The implementation has never been remotely correct (it is somewhat better as of #58 but still can produce invalid JSON).

This would be a breaking change.

Stream Deserializer not handling EOF Correctly

The stream deserializer currently does not return none on the first EOF from the input stream, instead it takes two. This is related to the way the deserializer does it's peeking.

Why can't deserializer use the std Peekable.

extern crate serde_json;

use serde_json::*;
use std::io::{self, Read};

fn main() {
    let stdin = io::stdin();
    let stream_deserializer: StreamDeserializer<Value, _> = StreamDeserializer::new(stdin.bytes());

    for v in stream_deserializer {
        match v {
            Ok(value) => {
                println!("{:?}", value);
            },
            Err(_) => println!("Encountered a json parsing error, closing"),
        }
    }
}

Running this you'll notice that it takes two control-D's to close the program.

How to handle a tagged union?

I'm looking to generate de/serialization for json tagged union messages from an external rest api, eg.
{ type: "error", message: "wrong"} or { type:"ok", response: { a: 1, b: 2}}.

It looks like this can't be done with macros, but is this the correct way to do it?

Obviously it would be much easier if the json matched this:

#[derive(Serialize, Deserialize, Debug)]
enum Message {
   Ok{ resp: Response }
   Error{ message: String}
}

Why 3 types of number?

Just an innocent question, I'm curious to here the reason for three types (F64, U64, and I64) of what is ultimately typed in JSON as a number. This may betray the fact that I was never a low level programmer, but even if there is a low level reason I'd like to know it. Thanks for the great open source project 😊

Invalid JSON produced

I was playing around with custom Serialize implementation, and granted this is not a sane thing, but I would expect this to not produce the JSON string {1,2} as it currently does.

#![feature(custom_derive, plugin)]
#![plugin(serde_macros)]

extern crate serde;
extern crate serde_json;

#[derive(Deserialize, Debug)]
struct Point {
    x: i32,
    y: i32,
}

impl serde::Serialize for Point {
    fn serialize<S>(&self, serializer: &mut S) -> Result<(), S::Error>
        where S: serde::Serializer
    {
        serializer.visit_struct("Point", PointMapVisitor {
            value: self,
            state: 0,
        })
    }
}

struct PointMapVisitor<'a> {
    value: &'a Point,
    state: u8,
}

impl<'a> serde::ser::MapVisitor for PointMapVisitor<'a> {
    fn visit<S>(&mut self, serializer: &mut S) -> Result<Option<()>, S::Error>
        where S: serde::Serializer
    {
        match self.state {
            0 => {
                self.state += 1;
                Ok(Some(try!(serializer.visit_seq_elt(&self.value.x))))
            }
            1 => {
                self.state += 1;
                Ok(Some(try!(serializer.visit_seq_elt(&self.value.y))))
            }
            _ => {
                Ok(None)
            }
        }
    }
}

fn main() {
    let point = Point { x: 1, y: 2 };
    let serialized = serde_json::to_string(&point).unwrap();
    println!("{}", serialized);

    let serialized = "{\"x\": 6, \"y\": 12}";
    let deserialized: Point = serde_json::from_reader(serialized.as_bytes()).unwrap();
    println!("{:?}", deserialized);
}

Meta

rustc 1.8.0-nightly (303892ee1 2016-01-30)

Dependency on serde_json 0.6 broken

Because there wasn't a 0.6 release after 5661c78, crates depending on 0.6 are now broken because they try to upgrade to serde 0.7.

Can you release serde_json 0.6.1 please? :)

add pointer_mut

It should be possible to add a pointer_mut(&str) method to Value. This would allow to add and modify values. (Maybe even a delete method) I failed to add such a method because of borrow checker.

#41 (comment)

Unify the number variants in Value

I am not convinced by the justification in #48 (comment). Instead of three I64, U64, F64 variants I would much rather use one Number variant backed by an arbitrary-precision number. We can keep as_i64, as_u64, as_f64 for convenience.

Consider using JSON pointers

JSON values provide a convenience to look up nested values called Value.lookup. The path is given separated by dots. It is impossible to escape dots in key names with the current method. I propose to use JSON pointers as described in RFC6901 as they provide a standard way to access nested values.

If you want to use JSON pointers I can also implement the change.

Incorrect result when using custom serialize_with method on struct member

I have encountered what I believe to be a bug. I am using 0.7's ability to specify a custom serialize_with function. I have a struct with a string and an enum wrapping either a u64 or a vector of the same. When serializing a struct with the vector, I get a comma after the field name and colon, but before the array. If I rearrange my struct, so that the enum wrapping the vector comes first, before the other field, it behaves as expected.
See the following gist
problem code

Improve syntax error messages for enums and structs

I believe that the error messages for enums and structs can be made much more useful.

Unknown field "foo" while parsing aFooBar, expected one of "a", "b", "c" (required), "d", "e", "f" (optional).

or

Unknown variant "foo" while parsing aFooBar, expected one of "a", "b", "c".

These error messages would make it much easier to fix typos and/or locate documentation.

Consider serializing map integer keys as strings

Right now serde_json rejects serializing maps with integer keys because semantically speaking, JSON only supports maps with string keys. There are workarounds in serde 0.7 with the new #[serde(serialize_with="...", deserialize_with="...)] (see this gist), but it still can be annoying if this keeps causing problems.

Is there any real value about erroring out on non-key values?

Parsing floats is slow

These two lines are extremely expensive:

selection_003

We can entirely avoid the division and multiplication by using a lookup table, or some other better approach.

[Builder] Need Examples: How to make types known in context?

I am just in the starting phase of learning Rust and am looking to add logic after I can successfully compile this example JSON from the builder. It seems that the examples only cover the simplest use cases and leaves you in the dark about assembling complex JSON documents using all of JSON features.

Example Code:

extern crate serde_json;

use serde_json::builder::ObjectBuilder;
use serde_json::Value;

fn main() {
    let json = ObjectBuilder::new()
        .insert("key", "12345678901234567890")
        .insert_object("message", |builder| {
            builder
                .insert("html", "<p>Example HTML content</p>")
                .insert_array("to", |builder| {
                    builder
                        .push_object(|builder| {
                            builder
                                .insert("email", "[email protected]")
                        })
                })
                .insert("important", false)
                .insert("track_clicks", Value::NULL)
                .insert_array("tags", |builder| {
                    builder
                        .push("password-resets")
                })
                .insert_array("recipient_metadata", |builder| {
                    builder
                        .push_object(|builder| {
                            builder
                                .insert("rcpt", "[email protected]")
                                .insert_object("values", |builder| {
                                    builder
                                        .insert("user_id", 123456_f64)
                                })
                        })
                })
        })
        .insert("async", false)
        .insert("ip_pool", "Main Pool")
        .unwrap();
}
forge@ubuntu:~/code/contact-us-rust$ rustc  -V
rustc 1.9.0 (e4e8b6668 2016-05-18)
forge@ubuntu:~/code/contact-us-rust$ clear; cargo build && cargo run
�[3;J
   Compiling contact-us v0.1.0 (file:///home/forge/code/contact-us-rust)
src/main.rs:20:41: 20:52 error: no associated item named `NULL` found for type `serde_json::Value` in the current scope
src/main.rs:20                 .insert("track_clicks", Value::NULL)
                                                       ^~~~~~~~~~~
src/main.rs:22:21: 23:49 error: the type of this value must be known in this context
src/main.rs:22                     builder
src/main.rs:23                         .push("password-resets")
src/main.rs:26:21: 34:27 error: the type of this value must be known in this context
src/main.rs:26                     builder
src/main.rs:27                         .push_object(|builder| {
src/main.rs:28                             builder
src/main.rs:29                                 .insert("rcpt", "[email protected]")
src/main.rs:30                                 .insert_object("values", |builder| {
src/main.rs:31                                     builder
               ...
src/main.rs:28:29: 29:79 error: the type of this value must be known in this context
src/main.rs:28                             builder
src/main.rs:29                                 .insert("rcpt", "[email protected]")
src/main.rs:31:37: 32:71 error: the type of this value must be known in this context
src/main.rs:31                                     builder
src/main.rs:32                                         .insert("user_id", 123456_f64)
error: aborting due to 5 previous errors
error: Could not compile `contact-us`.

To learn more, run the command again with --verbose.

It's not clear:

  • how to use NULL
  • how to make the type of values known in this context

Possibly better names for the `as_*` functions

It's probably just me, but I've always been confused when trying to recall the names of the as_* functions on Value.

I first try to use .as_str, because that name is popular in std for methods returning a &str. Then I fail, and next I try .as_string and it succeeds. The problem is, after that I try to call .as_integer because that was the logic I used in the previous stage ("use the fully qualified name"). Then I fail again, so instead I try .as_number because that's the terminology used in the JSON spec. And I fail once more. At this point I give up and open the browser to read the documentation, realizing the method was .as_i64.

It seems that those names were inherited from rustc-serialize. I can see the rationale used to decide the current names, which corresponds to the names of the enum variants (I64, U64, F64, String, Boolean). But personally, I think using .as_<resulting_type> as the names has several advantages:

  • One can know intuitively what type will be returned. .as_str returns &str, .as_i64 returns i64, and .as_bool returns bool, and so on.
  • If we add more "helper" methods in the future, such as .as_i32 or .as_i16, it would be straightforward. Even as_str (the one returning &str) and as_string (the one returning String) could coexist!
  • They are shorter!

The only (but possibly big) disadvantage is, of course, it is a breaking change. (And I missed the 0.7 timeframe due to my laziness!) But this could be alleviated by introducing a deprecation period.

What's your thoughts on this?

serialize_u64 is slow

It currently looks like this:

fn serialize_u64(&mut self, value: u64) -> Result<()> {
    write!(&mut self.writer, "{}", value).map_err(From::from)
}

I tested the following variation and it made an astonishing 20% difference in the bench_log benchmarks (294 MB/s vs 244 MB/s).

if value == 0 {
    self.writer.write_all(b"0").map_err(From::from)
} else {
    let mut scratch = [0u8; 20];
    let mut len = 0;
    while value > 0 {
        scratch[19 - len] = (value % 10) as u8 + b'0';
        value /= 10;
        len += 1;
    }
    self.writer.write_all(&scratch[20-len..20]).map_err(From::from)
}

That is enough to motivate looking for a better solution, or contributing a fix to Rust for whatever is making write! slow.

Serialize optional key/value sets

In the spaceapi-rs crate, we're implementing the Space API. The crate basically provides the structs for all types in the spec, as well as the serialization and deserialization code.

Right now we're using rustc-serialize. The problem is how rustc-serialize handles optional values. In the Space API spec, there are some values that must be either absent, or contain a value. So "phone": "012 345 67 89" is fine, not specifying a phone is also fine, but "phone": null is invalid according to the spec. (You could consider the value a non-nullable type.)

rustc-serialize always serializes an Option::None to null. So we created our own enum called Optional that can be either Value(T) or Absent. The unfortunate result of that is that we have to write custom serialization code for each and every one of our types: https://github.com/coredump-ch/spaceapi-rs/blob/master/src/status.rs#L175-L188

Skimming through the documentation, I didn't see a way to do this more easily with serde. Did I miss something? Or if not, would it be possible to implement something like that in serde?

json_macros dropping fields

Hello! I have an API that is returning JSON with certain fields missing:

{ "one":45, "two":5, "four":17 }
{ "one":81, "three":35 }

My rust file looks something like:

#[derive(Debug, PartialEq, Serialize, Deserialize, Clone)]
pub struct JsonStruct {
  pub one: Option<usize>,
  pub two: Option<usize>,
  pub three: Option<usize>,
  pub four: Option<usize> }

I'm not sure if Option is the way to model missing fields, but any missing field all the fields afterwards are parsed as None. For example, when I try to parse the first example, the data returned has the fourth field parsed as None. For the second data example, I would expect the second and fourth field to be None, but the third field is also None. Great library, thanks!

ObjectBuilder: Rename unwrap?

Correct me if I'm wrong, but ObjectBuilder::unwrap doesn't actually panic!(), right? If so, I think it should be renamed to to_value or something like that.

Need to escape ASCII control characters

Serializing control characters such as "\x01" results in these characters being passed through, ie "\"\x01\"" rather than the correct "\"\\u0001\"". The JSON spec is quite clear that these characters must be escaped, and at least one JSON implementation (Swift's) rejects the unescaped variant as invalid JSON.

I'm happy to provide a patch, not sure if you prefer that or to write your own.

Parser cannot read arbitrary precision numbers

http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf specifies in the second paragraph of the introduction that JSON is agnostic about numbers, and simply represents them as a series of digits.

However, serde-json parses anything with a decimal point as a Rust f64, which causes numbers to be read incorrectly. There is no way to avoid this because this behaviour is chosen as soon as a decimal point is encountered. This makes it impossible to use serde-json to interoperate with financial software using JSON.

Unexpected brackets in to_value of a newtype struct

#![feature(custom_derive, plugin)]
#![plugin(serde_macros)]

use std::collections::BTreeMap;

extern crate serde_json;

#[derive(Serialize, Deserialize)]
struct Bar(BTreeMap<String, i32>);

fn main() {
    let mut b = Bar(BTreeMap::new());
    b.0.insert(String::from("foo"), 123);

    // prints {"foo":123}
    println!("{}", serde_json::to_string(&b).unwrap());

    let mut outer = BTreeMap::new();
    outer.insert(String::from("outer"), serde_json::to_value(&b));

    // prints {"outer":[{"foo":123}]}
    println!("{}", serde_json::to_string(&outer).unwrap());
}

I would expect the second print to be {"outer":{"foo":123}}.

Missing Vec field in a struct is not reported as an error

The following program should result in an error when deserializing the string “{}” because the bar field is missing. However, the call to visitor.missing_field returns an Ok result.

Expected output:

CHECK

Actual output:

CHECK
thread '<main>' panicked at 'called `Result::unwrap_err()` on an `Ok` value: Foo { bar: [] }', ../src/libcore/result.rs:787
Process didn't exit successfully: `target/debug/serde-test` (exit code: 101)

Program source code:

extern crate serde;
extern crate serde_json;

#[derive(Debug)]
pub struct Foo {
    pub bar: Vec<String>,
}

impl serde::Deserialize for Foo {
    fn deserialize<D>(d: &mut D) -> Result<Self, D::Error>
        where D: serde::Deserializer
    {
        enum Field {
            Bar,
        }

        impl serde::Deserialize for Field {
            fn deserialize<D>(d: &mut D) -> Result<Field, D::Error>
                where D: serde::Deserializer
            {
                struct Visitor;

                impl serde::de::Visitor for Visitor {
                    type Value = Field;

                    fn visit_str<E>(&mut self, value: &str) -> Result<Field, E>
                        where E: serde::de::Error
                    {
                        match value {
                            "bar" => Ok(Field::Bar),
                            _ => Err(E::unknown_field(value)),
                        }
                    }
                }

                d.visit(Visitor)
            }
        }

        struct Visitor;

        impl serde::de::Visitor for Visitor {
            type Value = Foo;

            fn visit_map<V>(&mut self, mut visitor: V) -> Result<Self::Value, V::Error>
                where V: serde::de::MapVisitor
            {
                let mut bar = None;
                loop {
                    match try!(visitor.visit_key()) {
                        Some(Field::Bar) => {
                            bar = Some(try!(visitor.visit_value()));
                        }
                        None => {
                            break;
                        }
                    }
                }

                try!(visitor.end());

                let bar = match bar {
                    Some(x) => x,
                    None => {
                        println!("CHECK");
                        try!(visitor.missing_field("bar"))
                    }
                };

                Ok(Foo { bar: bar })
            }
        }

        static FIELDS: &'static [&'static str] = &["bar"];
        d.visit_struct("Foo", FIELDS, Visitor)
    }
}

fn main() {
    serde_json::from_str::<Foo>("{}").unwrap_err();
}

Missing field in a struct is reported as an InvalidType error code when deserializing from Value

Serde reports the ErrorCode::InvalidType variant when deserializing from a Value and a field is missing. It should report the ErrorCode::MissingField variant—same as when deserializing from a JSON-formatted string.

The program below shows the problem.

extern crate serde;
extern crate serde_json;

#[derive(Debug)]
struct Foo {
    field_1: i32,
}

impl serde::Deserialize for Foo {
    fn deserialize<D>(deserializer: &mut D) -> Result<Self, D::Error>
        where D: serde::Deserializer
    {
        enum Field {
            Field1,
        }

        impl serde::Deserialize for Field {
            fn deserialize<D>(deserializer: &mut D) -> Result<Self, D::Error>
                where D: serde::Deserializer
            {
                struct Visitor;

                impl serde::de::Visitor for Visitor {
                    type Value = Field;

                    fn visit_str<E>(&mut self, value: &str) -> Result<Self::Value, E>
                        where E: serde::de::Error
                    {
                        match value {
                            "field_1" => Ok(Field::Field1),
                            _ => Err(E::unknown_field(value)),
                        }
                    }
                }

                deserializer.deserialize(Visitor)
            }
        }

        struct Visitor;

        impl serde::de::Visitor for Visitor {
            type Value = Foo;

            fn visit_map<Vis>(&mut self, mut visitor: Vis) -> Result<Self::Value, Vis::Error>
                where Vis: serde::de::MapVisitor
            {
                let mut field_1 = None;

                loop {
                    match try!(visitor.visit_key()) {
                        Some(Field::Field1) => {
                            field_1 = Some(try!(visitor.visit_value()));
                        }
                        None => {
                            break;
                        }
                    }
                }

                try!(visitor.end());

                let field_1 = match field_1 {
                    Some(field_1) => field_1,
                    None => try!(visitor.missing_field("field_1")),
                };

                Ok(Foo { field_1: field_1 })
            }
        }

        static FIELDS: &'static [&'static str] = &["field_1"];
        deserializer.deserialize_struct("Foo", FIELDS, Visitor)
    }
}

fn main() {
    let json_value = serde_json::builder::ObjectBuilder::new().unwrap();
    let json_encoded = serde_json::to_string(&json_value).unwrap();

    match serde_json::from_str::<Foo>(&json_encoded) {
        Err(serde_json::Error::Syntax(serde_json::ErrorCode::MissingField("field_1"), 1, 2)) => (),
        x @ _ => {
            panic!("Got unexpected result (from_str): {:?}", x);
        }
    }

    match serde_json::from_value::<Foo>(json_value) {
        Err(serde_json::Error::Syntax(serde_json::ErrorCode::MissingField("field_1"), _, _)) => (),
        x @ _ => {
            panic!("Got unexpected result (from_value): {:?}", x);
        }
    }
}

Expected output: (success)

Actual output:

thread '<main>' panicked at 'Got unexpected result (from_value): Err(Syntax(invalid type: Unit, 0, 0))', src/main.rs:91
Process didn't exit successfully: `target/debug/serde_json_test` (exit code: 101)

Enhancement: associated functions for deserialization

Let me know if there is a better venue to discuss possible changes to serde.

Currently, the easiest way to do serialization is to do e.g.:

let obj: MyType=serde_json::from_slice(&my_buf);

I think it would look “cleaner” if you could write it as an associated function on a type, e.g.:

let obj=MyType::from_json_slice(&my_buf);
// cf. MyType::new()

There's currently the Deserialize trait which does this but constructing a Deserializer is very verbose. One way to implement this would be:

trait JsonDeserialize: serde::Deserialize {
    fn from_json_slice(v: &[u8]) -> Result<Self,JsonError> {
        serde_json::from_slice(v)
    }
    // etc. for _str, _iter, _reader
}

impl<T: serde::Deserialize> JsonDeserialize for T {}

Thoughts?

Panic on input that is not valid UTF8

This should be an error not a panic.

fn main() {
    let data = &[b'"', b'\xed', b'\xa0', b'\x80', b'"'];
    serde_json::from_slice::<String>(data).ok();
}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 0 }'

Caused by this unwrap.

Not compiled on Rust 1.5.0

Error:
serde_json-0.7.0/src/lib.rs:120:1: 120:19 error: use of unstable library feature 'core': the libcore library has not yet been scrutinized for stabilization in terms of structure and naming (see issue #27701)
serde_json-0.7.0/src/lib.rs:120 extern crate core;
rustc --version: rustc 1.5.0
cargo --version: cargo 0.8.0-nightly (b3273e5 2016-01-11)
Please fix this

Missing () field in a struct is not reported as an error

If a () struct field is missing when deserializing JSON input, no error is reported. This means there's no distinction between () and Option<()>.

This is related to issue #29, which is about missing Vec<_> fields going unreported, and issue #22, which is about serde_json reporting missing fields incorrectly in general.

The following program demonstrates the behavior for the () case.

Expected output: (none)

Actual output:

thread '<main>' panicked at 'Got unexpected OK result: Ok(Foo { a: () })', main.rs:91
Process didn't exit successfully: `/home/craig/art/serde_json-cmbrandenburg/issue_x_test/target/debug/issue_x_test` (exit code: 101)
extern crate serde;
extern crate serde_json;

#[derive(Debug)]
struct Foo {
    a: (),
}

impl serde::Deserialize for Foo {
    fn deserialize<D>(d: &mut D) -> Result<Self, D::Error>
        where D: serde::Deserializer
    {

        enum Field {
            A,
        }

        impl serde::Deserialize for Field {
            fn deserialize<D>(d: &mut D) -> Result<Self, D::Error>
                where D: serde::Deserializer
            {

                struct Visitor;

                impl serde::de::Visitor for Visitor {
                    type Value = Field;

                    fn visit_str<E>(&mut self, value: &str) -> Result<Self::Value, E>
                        where E: serde::de::Error
                    {
                        match value {
                            "a" => Ok(Field::A),
                            _ => Err(E::unknown_field(value)),
                        }
                    }
                }

                d.visit(Visitor)
            }
        }

        struct Visitor;

        impl serde::de::Visitor for Visitor {
            type Value = Foo;

            fn visit_map<V>(&mut self, mut visitor: V) -> Result<Foo, V::Error>
                where V: serde::de::MapVisitor
            {
                let mut a = None;

                loop {
                    match try!(visitor.visit_key()) {
                        Some(Field::A) => {
                            a = Some(try!(visitor.visit_value()));
                        }
                        None => {
                            break;
                        }
                    }
                }

                try!(visitor.end());

                let a = match a {
                    Some(x) => x,
                    None => try!(visitor.missing_field("a")),
                };

                let v = Foo { a: a };

                Ok(v)
            }
        }

        static FIELDS: &'static [&'static str] = &["a"];
        d.visit_struct("Foo", FIELDS, Visitor)
    }
}

fn main() {
    let s = "{}";
    let got = serde_json::from_str::<Foo>(&s);
    match got {
        Err(serde_json::Error::SyntaxError(serde_json::ErrorCode::MissingField(field),
                                           line,
                                           column)) => {
            assert_eq!(("a", 1, 2), (field, line, column));
        }
        Ok(_) => {
            panic!("Got unexpected OK result: {:?}", got);
        }
        _ => {
            panic!("Got unexpected error result: {:?}", got);
        }
    }
}

Stream Deserializer not handling u64 overflow.

In a simple use of the stream deserializer we have, a number larger than 18446744073709551615 will not result in an error.

Here is the output of our program.

Running `target/debug/xstream`
18446744073709551615
18446744073709551615
18446744073709551616
1
18446744073709552000
1

This should return:

Running `target/debug/xstream`
18446744073709551615
18446744073709551615
18446744073709551616
error parsing JSON
1
1

Sample serialize/deserialize implementation

Hi !
I've been looking on the internets how to implement my own serialize/deserialize on a type (particularly on the crate time, for TimeSpec or Tm), so I can seamlessly parse a date to a json string (using strftime) ... but I couldn't find a simplified example on how to implement that on a type and how.

¿Is there any place to look for such information?

Thanks for reading and for making serde ! ♥

'arithmetic operation overflowed' when reading "-0" as JSON

Running this program:

extern crate serde_json;
fn main() {
    serde_json::from_str::<serde_json::Value>("-0").unwrap();
}

causes:

thread '<main>' panicked at 'arithmetic operation overflowed', [...]/.cargo/registry/src/github.com-88ac128001ac3a9a/serde_json-0.6.0/src/de.rs:289

whereas I expected it to run to completion successfully with no output.

GitCop experience is unfriendly to a new contributor

I made a small doc fix PR and received a GitCop email. I have a few issues with it:

  • there are no contribution guidelines in any prominent place, and the location the email pointed me to certainly had none.
  • none of the commit messages by the primary author follow the format GitCop said to use, and I did check before making my commit
  • the tone and wording of the email is unfriendly

This isn't a great experience for a new contributor!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.