UnexpectedVariant on encoded data

[penso]

UnexpectedVariant on encoded data

[VictorKoenders]

Looks like they're using #[serde(untagged)]: https://github.com/informalsystems/tendermint-rs/blob/main/rpc/src/id.rs#L11

We do not get this information from serde, and this results in types that bincode cannot properly serialize and deserialize. A full list can be found here: https://docs.rs/bincode/2.0.0-rc.3/bincode/serde/index.html#known-issues

Unfortunately this is not something we can fix in bincode. More information here: #130 (comment)

[VictorKoenders]

This should only fail if there is a bug in bincode, or if your encode/decode logic is not correct. The best way to debug this is to make your encode/decode struct as small as possible to where it still generates an issue. You can also use at the spec to manually encode your type and see if that matches what bincode generates

[penso]

I'll try to encode a subset of the external crate struct to see what generate the issue, but since I'm not doing any specific encoding/decoding I'd say it's a bug in bincode.

Would creating a repo with a minimal example of the fail help you?

[VictorKoenders]

If you've truly found a bug in bincode it would be great to see a minimal reproducible example, yes

[penso]

@VictorKoenders here for you https://github.com/penso/bincode-fails (just use cargo run)

Made it as simple as possible:

use std::{fs::File, io::Read, str::from_utf8};

use anyhow::Context;
use serde::{de::DeserializeOwned, Deserialize, Serialize};
use tendermint_rpc::endpoint::{block, block_results};

#[derive(Serialize, Deserialize, Debug)]
pub struct BlockAndTransactions {
    pub block: block::Response,
    pub block_results: block_results::Response,
}

fn main() -> anyhow::Result<()> {
    let data: BlockAndTransactions = read_data()?;

    let config = bincode::config::standard();
    let encoded: Vec<u8> = bincode::serde::encode_to_vec(&data, config)?;
    // THIS IS THE LINE THAT FAILS
    let (_decoded, _): (BlockAndTransactions, _) =
        bincode::serde::decode_from_slice(&encoded, config)?;

    Ok(())
}

pub fn read_data<T: DeserializeOwned>() -> anyhow::Result<T> {
    let filename = "1206335.json.xz";

    let file = File::open(filename)?;
    let mut decompressor = lzma::LzmaReader::new_decompressor(file)?;

    let mut s = Vec::new();
    decompressor.read_to_end(&mut s)?;

    let string = from_utf8(&s)?;
    Ok(serde_json::from_str::<T>(string).with_context(|| "Can't decode json data")?)
}

[penso]

Ah bummer, thanks for looking at this. Will try to get them to remove this tag then.