Giter Site home page Giter Site logo

yas's Introduction

CI

YAS

Yet Another Serialization

-Time

  • YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1, benchmark 2)
  • YAS is header only library
  • YAS does not depend on third-party libraries or boost
  • YAS require C++11 support
  • YAS binary archives is endian independent

Supported the following types of archives:

  • binary
  • text
  • json (not fully comply)

Supported the following compilers:

  • GCC : 4.8.5, ... - 32/64 bit
  • MinGW: 4.8.5, ... - 32/64 bit
  • Clang: 3.5, ... - 32/64 bit
  • Intel: (untested)
  • MSVC : 2017(in c++14 mode), ... - 32/64 bit
  • Emscripten: 1.38 (clang version 6.0.1)

Samples

The easiest way to save and load some object or vars is to use the yas::save() and yas::load() functions like this:

#include <yas/serialize.hpp>
#include <yas/std_types.hpp>

int main() {
    int a = 3, aa{};
    short b = 4, bb{};
    float c = 3.14, cc{};
    
    constexpr std::size_t flags = 
         yas::mem // IO type
        |yas::json; // IO format
    
    auto buf = yas::save<flags>(
        YAS_OBJECT("myobject", a, b, c)
    );
    
    // buf = {"a":3,"b":4,"c":3.14}
    
    yas::load<flags>(buf,
        YAS_OBJECT_NVP("myobject"
            ,("a", aa)
            ,("b", bb)
            ,("c", cc)
        )
    );
    // a == aa && b == bb && c == cc;
}

The IO type can be one of yas::mem or yas::file. The IO format can be one of yas::binary or yas::text or yas::json.

The YAS_OBJECT()/YAS_OBJECT_NVP()/YAS_OBJECT_STRUCT()/YAS_OBJECT_STRUCT_NVP() macro are declared here, example use is here.

More examples you can see here.

TODO:

  • protobuf/messagepack support
  • limits
  • objects versioning

Support the project

You can support the YAS project by donating:

  • BTC: 12rjx6prAxwJ1Aep6HuM54At9wDvSCDbSJ
  • ETH: 0x62719DDEc96C513699a276107622C73F6cAcec47

Serialization for the following types is supported:

Projects using YAS

  • Ufochain: a mimblewimble implementation of crypto currency using X17r algorithm
  • Kvant: Kvant - is an original project using the consensus of MimbleWimble, due to which maximum anonymity and security were achieved
  • zkPoD-lib: zkPoD-lib is the underlying core library for zkPoD system. It fully implements PoD (proof of delivery) protocol and also provides a CLI interface together with Golang bindings
  • Litecash: Litecash is the next generation scalable, confidential cryptocurrency based on an elegant and innovative Mimblewimble protocol
  • K3: K3 is a programming language for building large-scale data systems
  • vistle: Software Environment for High-Performance Simulation and Parallel Visualization
  • LGraph: Live Graph infrastructure for Synthesis and Simulation
  • Beam: BEAM is a next generation scalable, confidential cryptocurrency based on an elegant and innovative Mimblewimble protocol
  • libfon9: C++11 Cross-platform infrastructure for Order Management System
  • iris-crypt: Store Node.js modules encrypted in a package file
  • cppan: Class members annotations for C++
  • GeekSys company: GeekSys is using YAS to serialize features from images

yas's People

Contributors

ahartmetz avatar aumuell avatar daldegam avatar gigte avatar ml1nk avatar nixman avatar pfeatherstone avatar pmed avatar theunwisewolf avatar tusharpm avatar tzlaine avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yas's Issues

License?

This code is seemingly unlicensed. If you want it be used by other developers/companies, you should add a license.

Ah never mind, I see you have a license included in the code. I didn't notice because there is no license in the top level of the repo.

problem with std::map

when I use yas to serialize with std::unordered_map or std::map, the result is not a standard JSON string, and not compatible with other JSON formatted application.
so my question is whether it's by design for some reason or just a bug?

About macro __cplusplus

In Visual Studio 2017 , the macro __cplusplus is still 1997, so in <yas/detail/config/config.hpp>, check __cplusplus will throw error.

Unable to serialize boost types (boost/compiler interactions?)

There have been some changes in recent compiler/boost versions that cause build failures with YAS_SERIALIZE_BOOST_TYPES set.

Here's a full build trace, when trying to compile the included base tests, as follows:

$.../yas/tests/base> g++ -std=c++1y -DYAS_SERIALIZE_BOOST_TYPES -I ../../include main.cpp 
In file included from ../../include/yas/detail/preprocessor/enum_params.hpp:19:0,
                 from ../../include/yas/detail/preprocessor/preprocessor.hpp:48,
                 from ../../include/yas/detail/io/information.hpp:50,
                 from ../../include/yas/binary_oarchive.hpp:42,
                 from main.cpp:40:
../../include/yas/detail/preprocessor/repeat.hpp:43:58: error: expected constructor, destructor, or type conversion before ‘(’ token
 # define YAS_PP_REPEAT_1_I(c, m, d) YAS_PP_REPEAT_1_ ## c(m, d)
                                                          ^
../../include/yas/detail/preprocessor/repeat.hpp:38:35: note: in expansion of macro ‘YAS_PP_REPEAT_1_I’
 # define YAS_PP_REPEAT_1(c, m, d) YAS_PP_REPEAT_1_I(c, m, d)
                                   ^
../../include/yas/detail/preprocessor/cat.hpp:29:32: note: in expansion of macro ‘YAS_PP_REPEAT_1’
 #    define YAS_PP_CAT_I(a, b) a ## b
                                ^
../../include/yas/detail/preprocessor/cat.hpp:22:30: note: in expansion of macro ‘YAS_PP_CAT_I’
 #    define YAS_PP_CAT(a, b) YAS_PP_CAT_I(a, b)
                              ^
../../include/yas/detail/preprocessor/repeat.hpp:29:24: note: in expansion of macro ‘YAS_PP_CAT’
 # define YAS_PP_REPEAT YAS_PP_CAT(YAS_PP_REPEAT_, YAS_PP_AUTO_REC(YAS_PP_REPEAT_P, 4))
                        ^
../../include/yas/serializers/binary/boost/boost_fusion_vector_serializers.hpp:122:2: note: in expansion of macro ‘YAS_PP_REPEAT’
  YAS_PP_REPEAT( \
  ^
../../include/yas/serializers/binary/boost/boost_fusion_vector_serializers.hpp:130:1: note: in expansion of macro ‘YAS__BINARY__GENERATE_SERIALIZE_BOOST_FUSION_VECTOR_FUNCTIONS’
 YAS__BINARY__GENERATE_SERIALIZE_BOOST_FUSION_VECTOR_FUNCTIONS(FUSION_MAX_VECTOR_SIZE)
 ^
main.cpp:450:1: error: expected ‘}’ at end of input
 }
 ^
main.cpp:450:1: error: expected ‘}’ at end of input
In file included from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40:0,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
../../include/yas/detail/tools/utf8conv.hpp: In instantiation of ‘void yas::detail::to_utf8(D&, const S&) [with D = boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >; S = boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >]’:
../../include/yas/detail/tools/utf8conv.hpp:100:19:   required from here
../../include/yas/detail/tools/utf8conv.hpp:21:28: error: invalid use of incomplete type ‘const class boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >’
  for ( auto it = src.begin(); it != src.end(); ++it ) {
                            ^
In file included from ../../include/yas/detail/tools/utf8conv.hpp:11:0,
                 from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
/usr/include/boost/container/container_fwd.hpp:189:7: note: declaration of ‘class boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >’
 class basic_string;
       ^
In file included from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40:0,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
../../include/yas/detail/tools/utf8conv.hpp:21:34: error: invalid use of incomplete type ‘const class boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >’
  for ( auto it = src.begin(); it != src.end(); ++it ) {
                                  ^
In file included from ../../include/yas/detail/tools/utf8conv.hpp:11:0,
                 from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
/usr/include/boost/container/container_fwd.hpp:189:7: note: declaration of ‘class boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >’
 class basic_string;
       ^
In file included from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40:0,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
../../include/yas/detail/tools/utf8conv.hpp:24:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘std::__cxx11::basic_string<char>::value_type {aka char}’)
    dst += (std::string::value_type)nchar;
        ^
../../include/yas/detail/tools/utf8conv.hpp:26:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0xC0 | (nchar >> 6));
        ^
../../include/yas/detail/tools/utf8conv.hpp:27:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | (nchar & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp:31:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0xE0 | (nchar >> 12));
        ^
../../include/yas/detail/tools/utf8conv.hpp:32:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | ((nchar >> 6) & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp:33:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | (nchar & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp:35:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0xF0 | (nchar >> 18));
        ^
../../include/yas/detail/tools/utf8conv.hpp:36:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | ((nchar >> 12) & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp:37:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | ((nchar >> 6) & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp:38:8: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’ and ‘int’)
    dst += (0x80 | (nchar & 0x3F));
        ^
../../include/yas/detail/tools/utf8conv.hpp: In instantiation of ‘void yas::detail::from_utf8(D&, const S&) [with D = boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >; S = boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >]’:
../../include/yas/detail/tools/utf8conv.hpp:107:21:   required from here
../../include/yas/detail/tools/utf8conv.hpp:47:28: error: invalid use of incomplete type ‘const class boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’
  for ( auto it = src.begin(); it != src.end(); ++it ) {
                            ^
In file included from ../../include/yas/detail/tools/utf8conv.hpp:11:0,
                 from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
/usr/include/boost/container/container_fwd.hpp:189:7: note: declaration of ‘class boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’
 class basic_string;
       ^
In file included from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40:0,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
../../include/yas/detail/tools/utf8conv.hpp:47:34: error: invalid use of incomplete type ‘const class boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’
  for ( auto it = src.begin(); it != src.end(); ++it ) {
                                  ^
In file included from ../../include/yas/detail/tools/utf8conv.hpp:11:0,
                 from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
/usr/include/boost/container/container_fwd.hpp:189:7: note: declaration of ‘class boost::container::basic_string<char, std::char_traits<char>, boost::container::new_allocator<char> >’
 class basic_string;
       ^
In file included from ../../include/yas/serializers/binary/std/std_wstring_serializers.hpp:40:0,
                 from ../../include/yas/serializers/std_types_serializers.hpp:65,
                 from main.cpp:52:
../../include/yas/detail/tools/utf8conv.hpp:70:7: error: no match for ‘operator+=’ (operand types are ‘boost::container::basic_string<wchar_t, std::char_traits<wchar_t>, boost::container::new_allocator<wchar_t> >’ and ‘std::__cxx11::basic_string<wchar_t>::value_type {aka wchar_t}’)
   dst += tmp;
       ^

I've tried this on the latest version of yas, with multiple boost versions (1.57, 1.59, 1.60), and the most recent compiler versions (gcc-5.3.0, clang-3.7.0), all with the same results. Compiling without YAS_SERIALIZE_BOOST_TYPES however, works fine.

std::map and c++11 in Xcode

Just a little note for anyone having issues with std::map on Xcode; I'm finding that to be able to serialize std::map in Xcode, I need to bring up the C++ Language Dialect project setting to C++14 [-std=c++14].

std::vector and other structures I've used so far worked fine in C++11.

How to handle class private data members

Hi,

suggest you have a class with a private member and public setter and getter functions for it.
Assuming you are not allowed to modify the class implementation. How would a serialize template as free function look like for the deserialization / load part ?

class A
{
public:
  A() = default;
  int getPrivate() const {return i;}
  void setPrivate(int new_i) {i = new_i;}
private:
  int i;
};

template <typename Archive>
void serialize(Archive& ar, A& a)
{
  if ( Archive::is_writable() ) 
  {
        // save / serialize
        ar & a.getPrivate();
  }
  else
  {
      // load / deserialize
      ar & ??
    }
}

Thanks in advance

MSVC compile errors with /Zc:__cplusplus

I'm using Visual Studio 2019. When I add option /Zc:__cplusplus (fixes __cplusplus macro value) it refuses to compile. I get strange errors like
yas\include\yas\types\std\pair.hpp(63,1): error C2975: '_Val': invalid template argument for 'std::integral_constant', expected compile-time constant expression

I did a quick test with file examples/one_memfn/main.cpp

This compiles:
cl main.cpp /I"../../include" /EHsc /std:c++17 /permissive-

And this does not:
cl main.cpp /I"../../include" /EHsc /std:c++17 /permissive- /Zc:__cplusplus

This probably has something to do with fnv1a function in include\yas\detail\tools\fnv1a.hpp . If you switch to the code for __cplusplus < 201402L yas compiles just fine.

android / linux headers are mismatching

I'm using yas::binary_iarchiveyas::file_istream and yas::binary_oarchiveyas::file_ostream to, respectively, WRITE on Android (ARM v7a) and to READ on Linux (x86_64).

Android binary file contains header "yas5800", but when I try to read it from Linux it is throwing an exception:

what():  .../yas/include/yas/detail/io/information.hpp(198): incompatible archive version

If I both WRITE and READ on Linux the binary file contains header "yas0105" and works.

Why?

License?

What's the license for the code?

unknow seg. fault inside a lambda thread

static volatile stopped = false;

auto task = std::thread([&] {
	yas::file_ostream os(yas_filename.c_str(), yas::file_mode::file_trunc);
	yas::binary_oarchive<yas::file_ostream> oa(os);

	for (unsigned int k = 0; !stopped; k++) {
		features f{
			k: k,
			feature: {cv::Point2f(0,0)};
		oa & f;

		if (k % 100 == 0)
			os.flush();
	}

	os.flush();
};

task.join();
I/DEBUG   (   98): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   (   98): Build fingerprint: 'rk31sdk/rk31sdk/rk31sdk:4.2.2/JDQ39/eng.lsf.20131030.091733:eng/test-keys'
I/DEBUG   (   98): Revision: '0'
I/DEBUG   (   98): pid: 5854, tid: 5856, name: dataset_client  >>> ./dataset_client <<<
I/DEBUG   (   98): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad
I/DEBUG   (   98):     r0 0000003b  r1 40367d70  r2 00000003  r3 deadbaad
I/DEBUG   (   98):     r4 40143228  r5 00000000  r6 4013a02c  r7 401366c3
I/DEBUG   (   98):     r8 01b7adb0  r9 00100000  sl 01b7adbc  fp 00000016
I/DEBUG   (   98):     ip 00000000  sp 40367d98  lr 401229c9  pc 40106ffc  cpsr 00000030
I/DEBUG   (   98):     d0  e7e6e5e4e3e2e1e0  d1  efeeedecebeae9e8
I/DEBUG   (   98):     d2  f7f6f5f4f3f2f1f0  d3  fffefdfcfbfaf9f8
I/DEBUG   (   98):     d4  0000000000000000  d5  bb50b5cf00000000
I/DEBUG   (   98):     d6  42fb333342fd5c29  d7  000000003ecccc00
I/DEBUG   (   98):     d8  0000000000000000  d9  0000000000000000
I/DEBUG   (   98):     d10 0000000000000000  d11 0000000000000000
I/DEBUG   (   98):     d12 0000000000000000  d13 0000000000000000
I/DEBUG   (   98):     d14 0000000000000000  d15 0000000000000000
I/DEBUG   (   98):     d16 40511eb856333333  d17 405f400000000000
I/DEBUG   (   98):     d18 405f800000000000  d19 0000000000000000
I/DEBUG   (   98):     d20 0000000000000000  d21 0000000000000000
I/DEBUG   (   98):     d22 0000000000000000  d23 0000000000000000
I/DEBUG   (   98):     d24 0000000000000000  d25 0000000000000000
I/DEBUG   (   98):     d26 0000000000000000  d27 0000000000000000
I/DEBUG   (   98):     d28 0000000000000000  d29 0000000000000000
I/DEBUG   (   98):     d30 0000000000000000  d31 0000000000000000
I/DEBUG   (   98):     scr 20000010
I/DEBUG   (   98):
I/DEBUG   (   98): backtrace:
I/DEBUG   (   98):     #00  pc 0000effc  /system/lib/libc.so
I/DEBUG   (   98):     #01  pc 00011dd3  /system/lib/libc.so (dlfree+1506)
I/DEBUG   (   98):     #02  pc 0000cf13  /system/lib/libc.so (free+10)
I/DEBUG   (   98):     #03  pc 002617a8  /data/local/dataset_client (std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > std::num_get<wchar_t, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > >::_M_extract_int<long>(std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::ios_base&, std::_Ios_Iostate&, long&) const+208)
I/DEBUG   (   98):
I/DEBUG   (   98): stack:
I/DEBUG   (   98):          40367d58  00000001
I/DEBUG   (   98):          40367d5c  401366c3  /system/lib/libc.so
I/DEBUG   (   98):          40367d60  01b7adb0  [heap]
I/DEBUG   (   98):          40367d64  40122a33  /system/lib/libc.so
I/DEBUG   (   98):          40367d68  002de6b0
I/DEBUG   (   98):          40367d6c  00000007
I/DEBUG   (   98):          40367d70  40367d6c
I/DEBUG   (   98):          40367d74  00000001
I/DEBUG   (   98):          40367d78  401362d6  /system/lib/libc.so
I/DEBUG   (   98):          40367d7c  00000005
I/DEBUG   (   98):          40367d80  40367da4
I/DEBUG   (   98):          40367d84  00000035
I/DEBUG   (   98):          40367d88  40143228
I/DEBUG   (   98):          40367d8c  00000000
I/DEBUG   (   98):          40367d90  df0027ad
I/DEBUG   (   98):          40367d94  00000000
I/DEBUG   (   98):     #00  40367d98  30313431
I/DEBUG   (   98):          40367d9c  001915b0  /data/local/dataset_client (set_string_number+684)
I/DEBUG   (   98):          40367da0  9b9a9998
I/DEBUG   (   98):          40367da4  20404040
I/DEBUG   (   98):          40367da8  524f4241
I/DEBUG   (   98):          40367dac  474e4954
I/DEBUG   (   98):          40367db0  494c203a
I/DEBUG   (   98):          40367db4  203a4342
I/DEBUG   (   98):          40367db8  50414548
I/DEBUG   (   98):          40367dbc  4d454d20
I/DEBUG   (   98):          40367dc0  2059524f
I/DEBUG   (   98):          40367dc4  52524f43
I/DEBUG   (   98):          40367dc8  49545055
I/DEBUG   (   98):          40367dcc  49204e4f
I/DEBUG   (   98):          40367dd0  6c64204e
I/DEBUG   (   98):          40367dd4  65657266
I/DEBUG   (   98):          ........  ........
I/DEBUG   (   98):     #01  40367ec0  01b7adb0  [heap]
I/DEBUG   (   98):          40367ec4  01b7add8  [heap]
I/DEBUG   (   98):          40367ec8  00269704  /data/local/dataset_client (std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > std::num_get<wchar_t, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > >::_M_extract_int<long>(std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::ios_base&, std::_Ios_Iostate&, long&) const+44)
I/DEBUG   (   98):          40367ecc  01b7adbc  [heap]
I/DEBUG   (   98):          40367ed0  00269704  /data/local/dataset_client (std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > std::num_get<wchar_t, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > >::_M_extract_int<long>(std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::ios_base&, std::_Ios_Iostate&, long&) const+44)
I/DEBUG   (   98):          40367ed4  40104f15  /system/lib/libc.so (free+12)
I/DEBUG   (   98):     #02  40367ed8  00012f51  /data/local/dataset_client
I/DEBUG   (   98):          40367edc  002697ac  /data/local/dataset_client (std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > std::num_get<wchar_t, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> > >::_M_extract_int<long>(std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::istreambuf_iterator<wchar_t, std::char_traits<wchar_t> >, std::ios_base&, std::_Ios_Iostate&, long&) const+212)
I/DEBUG   (   98):```

Doc and Benchmarks

I'd love to give this lib a try, it sounds very exciting but I don't know what to think about it yet. So it would very much help (not only me I venture) if you could create some nice concise (but complete) documentation with a top-2-bottom approach and add documentation to the examples too, because without doc the examples are pretty much useless ;)

And though your claims about the speed comparison look impressive I think your lib would attract way more people if you had a proper benchmark test with 2-3 contender libs that others could reproduce and confirm.

Question : compile time check that type is serialisable

Is it possible to have a compile time check that a type is yas-serialisable?
To be specific, I am looking to do the following:

template<typename Object, typename std::enable_if<YAS_SERIALISABLE<Object>::value>::type = 0>
yas::shared_buffer serialise_yas(const Object& obj)
{
return yas::save<yas::mem | yas::binary>(obj);
}

template<typename Object, typename std::enable_if<!YAS_SERIALISABLE<Object>::value>::type = 0>
yas::shared_buffer serialise_yas(const Object& obj)
{
yas::shared_buffer buf;
printf("%s not yas-serialisable\n", typeid(Object).name());
return buf;
}

using fopen_s instead of fopen

I'm getting following error when including <yas/serialize.hpp>:

Error C4996 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. S

IDE: VS2019

compatibility after struct changed

hi,

After a add a data member to a yas serializable struct, and deserialize it with old buffer. what's the defined behavior, thrown or abort or load old fields anyway?

HDF5 file format support for serialization

Hi!

Do you have any plans to support HDF5 binary format?
HDF5 is portable and well suited to store big scientific data. Serializing to HDF5 would be a huge plus for many purposes.
There was an effort to implement HDF5 support in original boost::serialization, but it seems that purpose is forgotten. Here's the link: https://github.com/dk1978/serialization.git
Maybe you will look into it and estimate if this code can be adopted for YAS?

Maximum number of objects to be deserialized (Error)

Hello, I'm running some tests with YAS.

But I have a problem.

It happens only when I'm going to deserialize an object, and has more than 21 variables.

I'll leave a code here to reproduce the error.

#include <yas/serialize.hpp>
#include <yas/std_types.hpp>

/***************************************************************************/
struct type {
	type() {}

	int var1{ 0 };
	unsigned int var2{ 0 };
	std::string var3{};
	std::string var4{};
	unsigned char var5{ 0 };
	unsigned int var6{ 0 };
	unsigned int var7{ 0 };
	unsigned int var8{ 0 };
	unsigned int var9{ 0 };
	unsigned int var10{ 0 };
	unsigned short var11{ 0 };
	unsigned short var12{ 0 };
	unsigned short var13{ 0 };
	unsigned short var14{ 0 };
	float var15{ 0.f };
	float var16{ 0.f };
	float var17{ 0.f };
	float var18{ 0.f };
	int var19{ 0 };
	int var21{ 0 };
	int var22{ 0 };
	unsigned char var23[1728];
	unsigned char var24[50];
	unsigned char var25[80];
	unsigned char var26{ 0 };
	unsigned char var27{ 0 };
	unsigned char var28{ 0 };
	unsigned char var29{ 0 };
	unsigned char var30{ 0 };
	unsigned int var31{ 0 };
	unsigned int var32{ 0 };
	unsigned int var33{ 0 };

	// one member-function for save/load
	template<typename Ar>
	void serialize(Ar &ar) {
		ar & YAS_OBJECT_NVP(
			"type"
			, ("var1", var1)
			, ("var2", var2)
			, ("var3", var3)
			, ("var4", var4)
			, ("var5", var5)
			, ("var6", var6)
			, ("var7", var7)
			, ("var8", var8)
			, ("var9", var9)
			, ("var10", var10)
			, ("var11", var11)
			, ("var12", var12)
			, ("var13", var13)
			, ("var14", var14)
			, ("var15", var15)
			, ("var16", var16)
			, ("var17", var17)
			, ("var18", var18)
			, ("var19", var19)
			, ("var21", var21)
			// The Bug starts here
			, ("var22", var22)
			, ("var23", var23)
			, ("var24", var24)
			, ("var25", var25)
			, ("var26", var26)
			, ("var27", var27)
			, ("var28", var28)
			, ("var29", var29)
			, ("var30", var30)
			, ("var31", var31)
			, ("var32", var32)
			, ("var33", var33)
		);
	}
};

/***************************************************************************/

int main() {
	type t1, t2;

	constexpr std::size_t flags = yas::mem | yas::json;

	auto buf = yas::save<flags>(t1);

	// The bug only happens with deserialization
	yas::load<flags>(buf, t2);
}

If you remove this line yas::load<flags>(buf, t2); the bug doesn't happens with 33 variables.

Or, if you comment this lines, the bug doesn't happens with 33 variables.

			//, ("var22", var22)
			//, ("var23", var23)
			//, ("var24", var24)
			//, ("var25", var25)
			//, ("var26", var26)
			//, ("var27", var27)
			//, ("var28", var28)
			//, ("var29", var29)
			//, ("var30", var30)
			//, ("var31", var31)
			//, ("var32", var32)
			//, ("var33", var33)

I´m using Visual Studio 2017.

The compiler error is:

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2243	'type cast': conversion from 'std::tuple<yas::value<const char *,int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,std::string &>,yas::value<const char *,std::string &>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned short &>,yas::value<const char *,unsigned short &>,yas::value<const char *,unsigned short &>,yas::value<const char *,unsigned short &>,yas::value<const char *,float &>,yas::value<const char *,float &>,yas::value<const char *,float &>,yas::value<const char *,float &>,yas::value<const char *,int &>,yas::value<const char *,int &>,yas::value<const char *,int &>,yas::value<const char *,unsigned char (&)[1728]>,yas::value<const char *,unsigned char (&)[50]>,yas::value<const char *,unsigned char (&)[80]>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned char &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>,yas::value<const char *,unsigned int &>> *' to 'std::tuple<> &' exists, but is inaccessible	yas-test	d:\libraries\yas-7.0.1\include\yas\types\utility\object.hpp	136	

Does anyone know if this is a compiler error or some YAS limitation?

Thank you.

zero copy and seek stream

Hello,

I've looked through the docs and examples and this library fits very well with my needs but I'm not finding an important feature which is to be able to do zero copy serialization and seek inside data out stream.

I'd like to use preallocated buffer for serialization, and also be able to seek inside this buffer and serialize without dynamic allocation.

Example:

    yas::mem_ostream                             os(size); <= preallocated memory
    yas::binary_oarchive<yas::mem_ostream, opts> oa(os);
    oa&                                          Packet{ 1, 2, 3, 4 };
    os.seek(offset); <= seek to some offset in memory
    oa&                                          Packet{ 1, 2, 3, 4 };
    os.clear(); <= Reset to zero offset and memset to zeros
    etc...

Is there a way to do this or something to add to the library?

Thanks, Best
Mark

VS2013 error & MinGW error

I was try to use this lib in VS 2013 Community Edition, and for now __cplusplus is 199711 (old c++98)

Validation is check in config.hpp, and throw compilation error.

Another issue is when using MinGW (4.8.3):

C:\...\Desktop\yas-master\yas-master\include\yas\serializers\std_types_serializers.hpp:53: In file included from ..\..\include/yas/serializers/std_types_serializers.hpp:53:0,

C:\..\Desktop\yas-master\yas-master\include\yas\serializers\binary\std\std_optional_serializers.hpp:120: error: expected declaration before '}' token
 } // namespace detail
 ^

Using Qt Creator with both compilers.
Windows 7 x64 :)

error C2760: syntax error on Visual studio 2017

Hello Everyone,
I am using Visual Studio 2017 on Windows. Initially, I have created some examples code and it was working fine.
But when I add them into my code then it gives me many syntax errors.

1>yas\detail\tools\limit.hpp(69): warning C4003: not enough arguments for function-like macro invocation 'max'
1>yas\detail\tools\limit.hpp(69): error C2760: syntax error: unexpected token '(', expected 'id-expression'
1>yas\detail\tools\limit.hpp(83): note: see reference to class template instantiation 'yas::max_limit<T>' being compiled
1>yas\detail\tools\limit.hpp(69): error C2143: syntax error: missing ')' before '>'
1>yas\detail\tools\limit.hpp(69): error C2059: syntax error: '>'
1>yas\detail\tools\limit.hpp(69): error C2059: syntax error: ')'
1>yas\detail\tools\limit.hpp(69): error C2334: unexpected token(s) preceding ':'; skipping apparent function body
1>yas\detail\tools\limit.hpp(83): fatal error C1903: unable to recover from previous error(s); stopping compilation

I have resolved all the above issues, but please give it a look if everything is fine.
I have changed header.hpp
constexpr std::uint8_t d = std::numeric_limits<std::uint8_t>::max();
with
constexpr std::uint8_t d = (std::numeric_limits<std::uint8_t>::max)();

and in limit.hpp FROM

template<typename T>
struct max_limit {
    using type = typename limit_base<T>::type;

    max_limit(const max_limit &) = delete;
    max_limit& operator=(const max_limit &) = delete;

    constexpr max_limit(T &&v, std::uint64_t max)
        :val(std::forward<type>(v))
         ,max(max)
	{}
    constexpr max_limit(max_limit &&r)
        :val(std::forward<type>(r.val))
        ,max(r.max)
	{}

    template<typename L, typename V>
    bool check(const L l, const V &) const {
        return __YAS_SCAST(std::uint64_t, l) <= max;
    }

    type val;
    const std::uint64_t max;
};
template<typename T>
struct minmax_limit {
    using type = typename limit_base<T>::type;

    minmax_limit(const minmax_limit &) = delete;
    minmax_limit& operator=(const minmax_limit &) = delete;

    constexpr minmax_limit(T &&v, std::uint64_t min, std::uint64_t max)
        :val(std::forward<type>(v))
        ,min(min)
        ,max(max)
	{}
    constexpr minmax_limit(minmax_limit &&r)
        :val(std::forward<type>(r.val))
        ,min(r.min)
        ,max(r.max)
	{}

    template<typename L, typename V>
    bool check(const L l, const V &) const {
        return (min <= __YAS_SCAST(std::uint64_t, l) && __YAS_SCAST(std::uint64_t, l) <= max);
    }

    type val;
    const std::uint64_t min;
    const std::uint64_t max;
};

WITH


template<typename T>
struct max_limit {
    using type = typename limit_base<T>::type;

    max_limit(const max_limit &) = delete;
    max_limit& operator=(const max_limit &) = delete;

    constexpr max_limit(T &&v, std::uint64_t max)
        :val(std::forward<type>(v))
	{ this->max = max; }
    constexpr max_limit(max_limit &&r)
        :val(std::forward<type>(r.val))
	{ this->max = r.max; }

    template<typename L, typename V>
    bool check(const L l, const V &) const {
        return __YAS_SCAST(std::uint64_t, l) <= this->max;
    }

    type val;
    const std::uint64_t max;
};
template<typename T>
struct minmax_limit {
    using type = typename limit_base<T>::type;

    minmax_limit(const minmax_limit &) = delete;
    minmax_limit& operator=(const minmax_limit &) = delete;

    constexpr minmax_limit(T &&v, std::uint64_t min, std::uint64_t max)
        :val(std::forward<type>(v))
	{
		this->min = min; this->max = max;
	}
    constexpr minmax_limit(minmax_limit &&r)
        :val(std::forward<type>(r.val))
	{
		this->min = r.min; this.max = r.max;
	}

    template<typename L, typename V>
    bool check(const L l, const V &) const {
        return (this->min <= __YAS_SCAST(std::uint64_t, l) && __YAS_SCAST(std::uint64_t, l) <= this->max);
    }

    type val;
    const std::uint64_t min;
    const std::uint64_t max;
};

If everything is fine, then can you please change it in next release.

Проблема с заголовком бинарного архива на windows+msvc

При попытке десериализации бинарного архива из-под win+msvc вылетает эксепшн YAS_THROW_BAD_ARCHIVE_VERSION, при этом первые семь байт архива выглядят вот так: yas0000.

Environment:
Windows 10 pro
Microsoft Visual Studio Community 2015
Version 14.0.25431.01 Update 3

Полный код:

// YasTest.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"

#include <yas/mem_streams.hpp>
#include <yas/file_streams.hpp>

#include <yas/binary_oarchive.hpp>
#include <yas/binary_iarchive.hpp>

#include <yas/std_types.hpp>

#include <map>
#include <queue>
#include <string>

#include <fstream>
#include <iostream>


using namespace std;

/*****************************************************************************/
/* OpenCorpora format
2
ABDUCE      gVerb,gInfinitive
ABDUCES     gVerb,gPresent gSingular,gPerson3
ABDUCED     gVerb,gPast
ABDUCED     gVerb,gPraedic gParticiple
ABDUCING    gVerb,gGerund
*/

bool replace(std::string& str, const std::string& from, const std::string& to) {
	size_t start_pos = str.find(from);
	if (start_pos == std::string::npos)
		return false;
	str.replace(start_pos, from.length(), to);
	return true;
}

std::string strip_white(const std::string& input)
{
	size_t b = input.find_first_not_of(' ');
	if (b == std::string::npos) b = 0;
	return input.substr(b, input.find_last_not_of(' ') + 1 - b);
}

std::string strip_comments(const std::string& input, const std::string& delimiters)
{
	return strip_white(input.substr(0, input.find_first_of(delimiters)));
}

vector<string> split(string s, string delimiter)
{
	size_t pos = 0;
	std::string token;
	vector<string> res;
	while ((pos = s.find(delimiter)) != std::string::npos) {
		token = s.substr(0, pos);
		res.push_back(token);
		s.erase(0, pos + delimiter.length());
	}
	res.push_back(s);
	return res;
}

vector<string> split(string str, char delimiter) {
	vector<string> internal;
	stringstream ss(str);
	string tok;

	while (getline(ss, tok, delimiter)) {
		internal.push_back(tok);
	}

	return internal;
}

class Gramtab {
public:
	Gramtab(std::string filename) {
		std::ifstream file(filename);
		std::string line;

		while (std::getline(file, line))
		{
			if (line[0] == '/' && line[1] == '/' || line == "")
				continue;

			//Remove spaces
			line.erase(std::remove(line.begin(), line.end(), ' '), line.end());
			//Remove comments
			line = strip_comments(line, "/");

			vector<string> s = split(line, "->");

			grams[s[0]] = convertRightPart(s[1]);

		}

	}

	std::string getTomitaGrams(std::string ancode) {
		return grams[ancode];
	}

private:
	std::map<std::string, std::string> grams;

	std::string convertRightPart(std::string str)
	{
		std::replace(str.begin(), str.end(), '+', ' ');
		replace(str, "NULL", "");
		return str;
	}
};

class PseudoBasis {
public:
	PseudoBasis(std::string b, std::string p)
	{
		basis = b;
		paradigm = p;
	}
	std::string basis;
	std::string paradigm;
};

class Lemma;
class Form {
public:

	Form() {}

	Form(string f, string g_str)
	{
		form = f;
		grams_str = g_str;
	}

	string format()
	{
		return pseudobasis + ": " + form + '\t' + grams_str;
	}
public:
	std::string form;
	std::string grams_str;
	std::string pseudobasis;
	int lemma_id;
	//std::weak_ptr<Lemma> lemma_ptr;
};

/*
ABDUC
%E*va%ES*vb%ED*vc%ED*vd%ING*ve

E*va
ES*vb
ED*vc
ED*vd
ING*ve

Тогда на выходе:
2
ABDUCE      gVerb,gInfinitive
ABDUCES     gVerb,gPresent gSingular,gPerson3
ABDUCED     gVerb,gPast
ABDUCED     gVerb,gPraedic gParticiple
ABDUCING    gVerb,gGerund
*/
class Lemma {
public:

	Lemma() {}

	Lemma(PseudoBasis& b, Gramtab& tab, int lemma_id) {

		id = lemma_id;
		vector<string> raw_forms = split(b.paradigm, '%');
		for (vector<string>::iterator i = raw_forms.begin() + 1; i < raw_forms.end(); i++)
		{
			vector<string> f = split(*i, '*');

			Form form = Form(b.basis + f[0], tab.getTomitaGrams(f[1]));
			form.pseudobasis = b.basis;
			forms.push_back(form);
		}

	}

	string format()
	{
		string res = "";
		for (vector<Form>::iterator i = forms.begin(); i < forms.end(); i++)
		{
			res += (*i).format() + '\n';
		}

		return res;
	}
public:
	std::vector<Form> forms;
	int id;
};


class AotDict {

public:
	AotDict(std::string filename) {
		std::ifstream file(filename);
		std::string line;

		//Reading paradigms
		std::getline(file, line);
		int paradigms_count = std::stoi(line);

		int i = 0;
		while (std::getline(file, line) && i < paradigms_count)
		{
			paradigms.push_back(line);
			i++;
		}

		std::cout << "Found " << paradigms.size() << " paradigms.\n";

		//Skipping accent paradigms
		int accent_paradigms_count = std::stoi(line);
		i = 0;
		while (std::getline(file, line) && i < accent_paradigms_count)
			i++;

		//Skipping changes history
		int changes_count = std::stoi(line);
		i = 0;
		while (std::getline(file, line) && i < changes_count)
			i++;

		//Skipping prefixes
		int prefixes_count = std::stoi(line);
		i = 0;
		while (std::getline(file, line) && i < prefixes_count)
			i++;

		//Reading basises
		int basises_count = std::stoi(line);

		i = 0;
		while (std::getline(file, line) && i < basises_count)
		{
			vector<string> bstr = split(line, ' ');
			if (bstr[0][0] != '#')
				basises.push_back(PseudoBasis(bstr[0], paradigms[std::stoi(bstr[1])]));
			i++;
		}

		std::cout << "Found " << basises.size() << " basises.\n";
	}
	std::vector<PseudoBasis> basises;

private:
	std::vector<std::string> paradigms;
};

class OpenCorporaDict {
public:
	void fromAOT() {
		std::cout << "Starting AOT converter...\n";

		std::cout << "Reading translate grams table...\n";
		Gramtab gramtab = Gramtab("translate_gram.txt");

		std::cout << "Reading AOT dict...\n";
		AotDict aot_dict = AotDict("morphs.mrd");

		std::cout << "Starting convertation...\n";

		float init_size = aot_dict.basises.size();

		for (std::vector<PseudoBasis>::iterator i = aot_dict.basises.begin(); i < aot_dict.basises.end(); i++)
		{
			lemmas.push_back(Lemma(*i, gramtab, i - aot_dict.basises.begin()));

			for (std::vector<Form>::iterator j = lemmas.back().forms.begin(); j < lemmas.back().forms.end(); j++)
			{
				(*j).lemma_id = i - aot_dict.basises.begin();
				forms[(*j).form].push_back(*j);
			}
		}

		std::cout << "AOT converted into OpenCorpora and loaded to memory.\n";
	}

	std::vector<Form> Lookup(const std::string& word)
	{
		return forms[word];
	}

	std::vector<Lemma> lemmas;
	std::map<std::string, std::vector<Form>> forms;
};
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// one free-function serializer/deserializer
template<typename Ar>
void serialize(Ar& ar, Form &o) {
	ar & o.form
		& o.grams_str;
}

// one free-function serializer/deserializer
template<typename Ar>
void serialize(Ar& ar, Lemma &o) {
	ar & o.forms;
}

/*****************************************************************************/
// one free-function serializer/deserializer
template<typename Ar>
void serialize(Ar& ar, OpenCorporaDict& t) {
	ar & t.lemmas
		& t.forms;
}
/*******************************************************************/

int main() {

	OpenCorporaDict dict;
	dict.fromAOT();

	yas::mem_ostream os;
	//yas::binary_oarchive<yas::mem_ostream, yas::binary | yas::endian_little | yas::no_header> oa(os);
	yas::binary_oarchive<yas::mem_ostream> oa(os);
	oa & dict;

	auto ibuf = os.get_intrusive_buffer();
	std::cout.write(ibuf.data, 7) << std::endl;

	yas::mem_istream is(os.get_intrusive_buffer());
	//yas::binary_iarchive<yas::mem_istream, yas::binary | yas::endian_little | yas::no_header> ia(is);
	yas::binary_iarchive<yas::mem_istream> ia(is);
	ia & dict;

	getchar();

	return 0;
}

/*****************************************************************************/

Serialize [unsigned] long error

Hi again :)

I discovered another bug (I suppose)...

Just try to serialize an object that has a "unsigned long" or "long" type.

The bug that will happen is:

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2672	'yas::detail::binary_ostream<OS,257>::write': no matching overloaded function found	yas-test	d:\libraries\yas\include\yas\types\utility\fundamental.hpp	56	
Error	C2893	Failed to specialize function template 'void yas::detail::binary_ostream<OS,257>::write(T,::yas::detail::enable_if_is_any_of<T,char,signed char,unsigned char,bool>::type *)'	yas-test	d:\libraries\yas\include\yas\types\utility\fundamental.hpp	56	
Error	C2780	'void yas::detail::binary_ostream<OS,257>::write(const T *,size_t)': expects 2 arguments - 1 provided	yas-test	d:\libraries\yas\include\yas\types\utility\fundamental.hpp	56	

Code with bug:

struct integer_types
{
	int var1{ 0 };
	unsigned int var2{ 0 };
	unsigned long var3{ 0 }; // BUG
	unsigned long long var4{ 0 };

	YAS_DEFINE_STRUCT_SERIALIZE_NVP(
		"integer_types"
		, ("var1", var1)
		, ("var2", var2)
		, ("var3", var3)
		, ("var4", var4)
	);
};

int main() 
{
	integer_types t0;
	constexpr std::size_t binary_opts = yas::mem | yas::binary;
	yas::mem_ostream mo;
	yas::binary_oarchive<yas::mem_ostream, binary_opts> oa(mo);
	oa & t0;
}

I am using the master branch code, with Visual Studio 2017.

Thank you.

std::vector<bool> specialization

Hi,

I´m playing around with Release 7.01 and I found that serialization/deserialization does not work for std::vector specialization.

Compilation error is
.../types/concepts/array.hpp:66:26: error: lvalue required as unary & operand
and
.../types/concepts/array.hpp:132:25: error: taking address of temporary

how to use difference function between json serialize with binary serialize?

For example, for some types like big integer (uint256), when serializing to binary, I want just to copy the memory, but when serializing to json, I want to convert it to readable format.


template<typename Ar>
void serialize(Ar &ar, uint256_t const& t) {
  std::array<uint8_t, 32> bin = Uint256ToBin(t);
  ar &YAS_OBJECT_NVP("u256", ("", bin));
}

template<typename Ar>
void serialize(Ar &ar, uint256_t const& t) {
  std::string str = Uint256ToStr(t, 10); // base 10
  ar &YAS_OBJECT_NVP("u256", ("", str));
}

A new release?

I've encountered some issues on ARM platform and found I need to add signed char to type traits. Then I found the fixes have already been merged (0622e86). Would you mind release a new patch version (like 7.0.5) for this?

why throw error on save?

yas::save<yas::file | yas::binary>( m_glyphdata_bin.c_str(), *this );
terminate called after throwing an instance of 'yas::io_exception'
  what():  /usr/local/include/yas/file_streams.hpp(67): file already exists

maybe more intuitive to set save overwrite as default
i have to use undocumented feature to solve it

    yas::file_ostream ofs( m_glyphdata_bin.c_str(), 1 );
    yas::save<yas::file | yas::binary>( ofs, *this );

include std_streams.hpp before serialize.hpp

Hi,

on compiling i noticed that serialize.hpp fail compile on gcc 8.1 if 'std_streams.hpp' is not included.
Perhaps adding this:

--- a/include/yas/serialize.hpp
+++ b/include/yas/serialize.hpp
@@ -44,6 +44,7 @@
 #include <yas/json_iarchive.hpp>
 #include <yas/mem_streams.hpp>
 #include <yas/file_streams.hpp>
+#include <yas/std_streams.hpp>

or changing examples ?

error was on

template<std::size_t F, typename ...Types>
typename std::enable_if<
    (F & yas::file) && (F & yas::json)
>::type
save(yas::file_ostream &os, Types &&... args) {
    yas::json_oarchive<yas::file_ostream, (F & (~yas::file))> oa(os);
    oa(std::forward<Types>(args)...);
}
error: 'template<long long unsigned int F, class ... Types> typename std::enable_if<((F & file) && (F & binary))>::type yas::save' conflicts with a previous declaration
save(yas::std_ostream_adapter &os, Types &&... args) {
        ^~~~~~~~~~~~~~~~~~~

perhaps compiler specific ?

OSX catalina GCC macports

Having this issue:

export CC=/opt/local/bin/gcc-mp-9;
export CXX=/opt/local/bin/g++-mp-9

boost is in /opt/local

I added this to CMakeKists.txt

elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra")
include_directories("/opt/local/include")
link_directories("/opt/local/lib")
add_definitions(
-DBOOST_ERROR_CODE_HEADER_ONLY
-DBOOST_CHRONO_HEADER_ONLY
-DYAS_SERIALIZE_BOOST_TYPES
-DYAS_OBJECT_MAX_MEMBERS=50
)


-- The C compiler identification is GNU 9.2.0
-- The CXX compiler identification is GNU 9.2.0
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /opt/local/bin/gcc-mp-9
-- Check for working C compiler: /opt/local/bin/gcc-mp-9 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Check for working CXX compiler: /opt/local/bin/g++-mp-9
-- Check for working CXX compiler: /opt/local/bin/g++-mp-9 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done

vs2017 fatal error

Compiling one_func with VS2017 results in a fatal error. I have not tried the other examples.

yas/object.hpp(203): fatal error C1001: An internal error has occurred in the compiler.

Compiler log attached:
one_func.txt

Build failure with MSVC

I am using YAS 6.0.2 and having trouble with a basic program to serialise a std::pair. The code builds under GCC but not with Visual Studio 2017. Thanks.

yastest.zip

C:\work\deps\yas\include\yas/types/std/pair.hpp(63): warning C4307: '*': integral constant overflow [C:\work\yastest\build\yastest.vcxproj]
C:\work\deps\yas\include\yas/object.hpp(203): fatal error C1001: An internal error has occurred in the compiler. [C:\work\yastest\build\yastest.vcxproj]
  (compiler file 'msc1.cpp', line 1507)
   To work around this problem, try simplifying or changing the program near the locations listed above.
  Please choose the Technical Support command on the Visual C++ 
   Help menu, or open the Technical Support help file for more information
  C:\work\deps\yas\include\yas/object.hpp(323): note: see reference to class template instantiation 'yas::detail::ctmap<KVI>' being compiled
          with
          [
              KVI=std::tuple<std::pair<std::integral_constant<uint32_t,1216469057>,std::integral_constant<uint8_t,0>>,std::pair<std::integral_constant<uint32_t,2885211357>,std::integral_constant<uint8_t,1>>>
          ]
  C:\work\deps\yas\include\yas/types/std/pair.hpp(63): note: see reference to class template instantiation 'yas::object<std::tuple<std::pair<std::integral_constant<uint32_t,1216469057>,std::integral_constant<uint8_t,0>>,std::pair<std::integral_constant<uint32_t,2885211357>,std::integral_constant<uint8_t,1>>>,yas::value<const char *,const _Ty1 &>,yas::value<const char *,const _Ty1 &>>' being compiled
          with
          [
              _Ty1=int
          ]
  Internal Compiler Error in C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\bin\HostX86\x64\CL.exe.  You will be prompted to send an error report to Microsoft later.
  INTERNAL COMPILER ERROR in 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.12.25827\bin\HostX86\x64\CL.exe'
      Please choose the Technical Support command on the Visual C++

std::wstring with utf16 serialization bug

Hello!

I have a little bug serializing std::wstring objects.
The bug happens when I put characters like UTF-16.

I changed your wstring test to reproduce the problem:

#ifndef __yas__tests__base__include__wstring_hpp
#define __yas__tests__base__include__wstring_hpp

/***************************************************************************/

template<typename archive_traits>
bool wstring_test(std::ostream &log, const char *archive_type, const char *test_name) {

	std::vector<std::wstring> wcs_collection = {
		L"A E I O U c C d D p P Á É Í Ó Ú ç Ç d D p P à è ì ò ù ã õ â ê î ô û", // pt-br
		//L"☺ ☻ ♥ ♦ ♣ ♠ • ◘ ○ ◙ ♂ ♀ ♪ ♫ ☼ ► ◄ ↕ ‼ ¶ § ▬ ↨ ↑ ↓ → ← ∟ ↔ ▲ ▼", // symbols
		L"đái dầm chửa hoang bí mật bầy tôi", // vn
		L"我能吞下玻璃而不伤身体。", // zh-cn
		L"私はガラスを食べられます。それは私を傷つけません。", // ja
		L"मैं काँच खा सकता हूँ, मुझे उस से कोई पीडा नहीं होती.", // hi
		L"Я могу есть стекло, оно мне не вредит.", // ru
		L"나는 유리를 먹을 수 있어요. 그래도 아프지 않아요", // ko
		L"I can eat glass and it doesn't hurt me.", // en
		L"ฉันกินกระจกได้ แต่มันไม่ทำให้ฉันเจ็บ", // th
		L"أنا قادر على أكل الزجاج و هذا لا يؤلمني", // ar
		L"לְהַגִּיד בַּבֹּקֶר חַסְדֶּךָ וֶאֱמוּנָתְךָ בַּלֵּילוֹת", // he
		L"Μπορώ να φάω σπασμένα γυαλιά χωρίς να πάθω τίποτα.", // el
		L"Mohu jíst sklo, neublíží mi.", // cs
		L"Мога да ям стъкло, то не ми вреди.", // bg
		L"Cam yiyebilirim, bana zararı dokunmaz.", // tr
		L"Je peux manger du verre, ça ne me fait pas de mal.", // fr
	};

	for (auto& ws : wcs_collection)
	{
		std::wstring wss;

		typename archive_traits::oarchive oa;
		archive_traits::ocreate(oa, archive_type);
		oa& YAS_OBJECT_NVP("obj", ("ws", ws));

		typename archive_traits::iarchive ia;
		archive_traits::icreate(ia, oa, archive_type);
		ia& YAS_OBJECT_NVP("obj", ("ws", wss));

		if (ws != wss) {
			YAS_TEST_REPORT(log, archive_type, test_name);
			return false;
		}
	}

	return true;
}

/***************************************************************************/

#endif // __yas__tests__base__include__wstring_hpp
cmdline: binary

binary: header                   -> passed
binary: endian                   -> passed
binary: version                  -> passed
binary: base64                   -> passed
binary: fundamental              -> passed
binary: compacted_storage_size   -> passed
binary: enum                     -> passed
binary: auto_array               -> passed
binary: std_streams              -> passed
binary: one_function             -> passed
binary: split_functions          -> passed
binary: one_method               -> passed
binary: split_methods            -> passed
binary: serialize                -> passed
binary: serialization            -> passed
binary: yas_object               -> passed
binary: base_object              -> passed
binary: array                    -> passed
binary: bitset                   -> passed
binary: buffer                   -> passed
binary: chrono                   -> passed
binary: complex                  -> passed
binary: string                   -> passed
binary: string_view              -> passed
binary: wstring                  -> [...]yas\tests\base\include\wstring.hpp(77): archive: "binary", test: "wstring" failed!
failed
binary: pair                     -> passed
binary: tuple                    -> passed
binary: vector                   -> passed
binary: list                     -> passed
binary: forward_list             -> passed
binary: deque                    -> passed
binary: map                      -> passed
binary: set                      -> passed
binary: multimap                 -> passed
binary: multiset                 -> passed
binary: unordered_map            -> passed
binary: unordered_set            -> passed
binary: unordered_multimap       -> passed
binary: unordered_multiset       -> passed
binary: optional                 -> passed
binary: variant                  -> passed
binary: boost_fusion_pair        -> passed
binary: boost_fusion_tuple       -> passed
binary: boost_fusion_vector      -> passed
binary: boost_fusion_list        -> passed
binary: boost_fusion_set         -> passed
binary: boost_fusion_map         -> passed
binary: boost_cont_string        -> passed
binary: boost_cont_wstring       -> passed
binary: boost_cont_vector        -> passed
binary: boost_cont_static_vector -> passed
binary: boost_cont_stable_vector -> passed
binary: boost_cont_list          -> passed
binary: boost_cont_slist         -> passed
binary: boost_cont_map           -> passed
binary: boost_cont_multimap      -> passed
binary: boost_cont_set           -> passed
binary: boost_cont_multiset      -> passed
binary: boost_cont_flat_map      -> passed
binary: boost_cont_flat_multimap -> passed
binary: boost_cont_flat_set      -> passed
binary: boost_cont_flat_multiset -> passed
binary: boost_cont_deque         -> passed
binary: boost_tuple              -> passed
binary: boost_variant            -> passed

/***************************************************/
> passed tests: 64
> failed tests: 1
> host endian : little
> host bits   : 64
> YAS version : 7.0.4
/***************************************************/

Test performed with Visual Studio 2019 - ISO C++14 Standard (/std:c++14)

Thanks :)

When I should use _NPV()?

Hi.

I'm trying to understand, but I could not get a clear answer yet by looking in the code ...

What's the difference between using YAS_DEFINE_STRUCT_SERIALIZE () and YAS_DEFINE_STRUCT_SERIALIZE_NVP ()?

Another question, can I serialize an object as binary in an x64 application and deserialize in an x86 application?

Thank you.

Question : when do you plan on supporting msgpack?

Questions:
Do you still plan on supporting msgpack?
Do you plan on supporting XML?
What in the JSON serialiser is not fully compliant?
There are a few header-only libraries that could help: RapidXML, RapidJSON, msgpack-c

Serialization object into Json

How can I do such conversion with YAS? For example:

// input
struct S {
int id;
std::wstring name;
};

// output
{ "id": 12, "name": "test_name" }

how can I serialize template type

template<typename T>
struct A {
 T a;
};

template <typename Ar, typename T>
void serialize(Ar& ar, A<T> const& t) {
  ar& YAS_OBJECT_NVP("a", ("a", t.a));
}

  yas::mem_ostream os;
  yas::binary_oarchive<yas::mem_ostream, YasBinF()> oa(os);
  A<int> a;
  oa.serialize(a);

The compiler complains "use of undefined type"

base64.hpp compile error on big endian (powerpc)

I get this error when compiling on powerpc:

/usr/include/yas/detail/tools/base64.hpp:578:29: error: 'src' was not declared in this scope

And then dozens of similar errors:

/usr/include/yas/detail/tools/base64.hpp:580:10: error: 'i' was not declared in this scope
/usr/include/yas/detail/tools/base64.hpp:584:52: error: there are no arguments to 'YAS_THROW_BASE64_ERROR' that depend on a template parameter, so a declaration of 'YAS_THROW_BASE64_ERROR' must be available

(And so on.)

Seems like big endian hasn't compiled in a while (ever?).

Wrong serialization to json and text of uint64_t

At commit 48533bd (latest on master).
The following fails on my platform (GCC 9.1.0 under Linux).

#include <boost/test/unit_test.hpp>
#include <stdint.h>
#include <yas/serialize.hpp>

BOOST_AUTO_TEST_CASE(SerializeUint64Test) {
  uint64_t x = 25951127024343977;

  auto bufJson = yas::save<yas::mem | yas::json>(x);
  uint64_t xFromJson = 0;
  yas::load<yas::mem | yas::json>(bufJson, xFromJson);
  BOOST_CHECK_EQUAL(x, xFromJson);

  auto bufText = yas::save<yas::mem | yas::text>(x);
  uint64_t xFromText = 0;
  yas::load<yas::mem | yas::text>(bufText, xFromText);
  BOOST_CHECK_EQUAL(x, xFromText);
}

Outputs

error in "SerializeUint64Test": check x == xFromJson failed [25951127024343977 != 5951127024343977]
error in "SerializeUint64Test": check x == xFromText failed [25951127024343977 != 5951127024343977]

Note that serializing to binary is fine. Most other numbers are also fine. Seems like a truncation issue, but other larger numbers are OK.

absl::flat_hash_map

Any plan to support absl::flat_hash_map?

Abseil flat hash map is pretty fast and practical for std::string_view.

Question: is Yas thread-safe?

I know, sounds like a silly question — like, why would one make a non thread-safe serialization lib — but e.g. Cereal turned out to be one. It became less funny, when I looked at Boost::serialization, from which Cereal borrowed the API, and found out neither it is a thread-safe library! Sick!

So, I'm just trying to be on the safe side: is Yas library thread-safe?

How to cross compile for windows with mingw?

I'm using your yas-base-test test case and tried to compile for windows on linux using MinGW but got lot of issues.

The first one is boost not found (it works fine when I compiled for linux but not for windows).

yas/tests/base/../../include/yas/object.hpp:146:13: fatal error: boost/version.hpp: No such file or directory
 #   include <boost/version.hpp>

I just add these lines:

elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
    # Using find package to get boost headers
    find_package(Boost REQUIRED)
    include_directories(${Boost_INCLUDE_DIRS})

And now come with lot of errors (18 557 lines)
Some of these:

In file included from /usr/share/mingw-w64/include/crtdefs.h:10,
                 from /usr/share/mingw-w64/include/stddef.h:7,
                 from /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/include/stddef.h:1,
                 from /usr/include/string.h:33,
                 from /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/include/c++/cstring:42,
                 from /tmp/yas/tests/base/../../include/yas/buffers.hpp:41,
                 from /tmp/yas/tests/base/../../include/yas/mem_streams.hpp:43,
                 from /tmp/yas/tests/base/main.cpp:42:
/usr/share/mingw-w64/include/corecrt.h:80:44: error: conflicting declaration ‘typedef long long unsigned int uintptr_t’
 __MINGW_EXTENSION typedef unsigned __int64 uintptr_t;
                                            ^~~~~~~~~
In file included from /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/include/c++/cstdint:41,
                 from /tmp/yas/tests/base/../../include/yas/detail/type_traits/type_traits.hpp:44,
                 from /tmp/yas/tests/base/../../include/yas/mem_streams.hpp:42,
                 from /tmp/yas/tests/base/main.cpp:42:
/usr/include/stdint.h:90:27: note: previous declaration as ‘typedef long unsigned int uintptr_t’
 typedef unsigned long int uintptr_t;
                           ^~~~~~~~~
In file included from /usr/include/stdint.h:27,
                 from /usr/lib/gcc/x86_64-w64-mingw32/8.3-win32/include/c++/cstdint:41,
                 from /tmp/yas/tests/base/../../include/yas/detail/type_traits/type_traits.hpp:44,
                 from /tmp/yas/tests/base/../../include/yas/mem_streams.hpp:42,
                 from /tmp/yas/tests/base/main.cpp:42:
/usr/share/mingw-w64/include/corecrt.h:128:35: error: conflicting declaration ‘typedef long long int __time_t’
 __MINGW_EXTENSION typedef __int64 __time64_t;
                                   ^~~~~~~~~~
/usr/include/bits/types.h:160:26: note: previous declaration as ‘typedef long int __time_t’
 __STD_TYPE __TIME_T_TYPE __time_t; /* Seconds since the Epoch.  */

Is is due to the import of boost I made?

Simple cmake toolchain I use:

# the name of the target operating system
set(CMAKE_SYSTEM_NAME Windows)
set(CMAKE_EXE_LINKER_FLAGS -static-libstdc++\ -static-libgcc\ -static)

# Which compilers to use for C and C++
set(CMAKE_C_COMPILER /usr/bin/x86_64-w64-mingw32-gcc)
set(CMAKE_CXX_COMPILER /usr/bin/x86_64-w64-mingw32-g++)
set(CMAKE_RC_COMPILER /usr/bin/x86_64-w64-mingw32-windres)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.