clwi/cwpack

An ANSI C encoder/decoder for the MessagePack serialization format / msgpack.org[C / Objective-C / Swift]

CWPack

CWPack is a lightweight and yet complete implementation of the MessagePack serialization format version 5. It also supports the Timestamp extension type.

Excellent Performance

Together with MPack, CWPack is the fastest open-source messagepack implementation. Both totally outperform CMP and msgpack-c

Design

CWPack does no memory allocations and no file handling in its basic setup. All that is done outside of CWPack. Example extensions are included.

CWPack is working against memory buffers. User defined handlers are called when buffers are filled up (packing) or needs refill (unpack).

Containers (arrays, maps) are read/written in parts, first the item containing the size and then the contained items one by one. Exception to this is the cw_skip_items function which skip whole containers.

Example

Pack and unpack example from the MessagePack home page:

void example (void)
{
    cw_pack_context pc;
    char buffer[20];
    cw_pack_context_init (&pc, buffer, 20, 0);

    cw_pack_map_size (&pc, 2);
    cw_pack_str (&pc, "compact", 7);
    cw_pack_boolean (&pc, true);
    cw_pack_str (&pc, "schema", 6);
    cw_pack_unsigned (&pc, 0);

    if (pc.return_code != CWP_RC_OK)  ERROR;
    int length = pc.current - pc.start;
    if (length != 18) ERROR;

    cw_unpack_context uc;
    cw_unpack_context_init (&uc, pc.start, length, 0);

    if (cw_unpack_next_map_size(&uc) != 2) ERROR;
    if (cw_unpack_next_str_lengh(&uc) != 7) ERROR;
    if (strncmp("compact", uc.item.as.str.start, 7)) ERROR;
    if (cw_unpack_next_boolean(&uc) != true) ERROR;
    if (cw_unpack_next_str_lengh(&uc) != 6) ERROR;
    if (strncmp("schema", uc.item.as.str.start, 6)) ERROR;
    if (cw_unpack_next_signed32(&uc) != 0) ERROR;

    if (uc.return_code != CWP_RC_OK)  ERROR;
    cw_unpack_next(&uc);
    if (uc.return_code != CWP_RC_END_OF_INPUT)  ERROR;
}

In the examples folder there are more examples.

Backward compatibility

CWPack may be run in compatibility mode. It affects only packing; EXT & TIMESTAMP is considered illegal, BIN are transformed to STR and generation of STR8 is supressed.

Error handling

When an error is detected in a context, the context is stopped and all future calls to that context are immediatly returned without any actions. Thus it is possible to make some calls and delay error checking until all calls are done.

CWPack does not check for illegal values (e.g. in STR for illegal unicode characters).

Build

CWPack consists of a single src file and three header files. It is written in strict ansi C and the files are together ~ 1.4K lines. No separate build is neccesary, just include the files in your own build.

CWPack has no dependencies to other libraries.

Test

Included in the test folder are a module test and a performance test and shell scripts to run them.

Objective-C

CWPack also contains an Objective-C interface. The MessagePack home page example would look like:

CWPackContext *pc = [CWPackContext newWithContext:my_cw_pack_context];
[pc packObject:@{@"compact":@YES, @"schema":@0}];

CWUnpackContext *uc = [CWUnpackContext newWithContext:my_cw_unpack_context];
NSDictionary *dict = [uc unpackNextObject];

Swift

CWPack also contains a Swift interface. The MessagePack home page example would pack like:

let packer = CWDataPacker()
packer + DictionaryHeader(2) + "compact" + true + "schema" + 0
let data = packer.data

	```

cwpack's Issues

typo in cw_unpack_next_str_lengh() and cw_unpack_next_bin_lengh()

It should be length. 't' is mising.

Feature request: C89 compatible version

It would be quite usefull to have your library in C89 compatible code. For example the struct timespec is C11 and in our environment C11 is not allowed. And your macros (e.g. tryMove4) produces a lot of errors in C89 with "declaration may not appear after executable statement in block".

cwpack.c:788:43: warning: comparison is always false due to limited range of data type [-Wtype-limits]

gcc gives warning against CWP_ITEM_TIMESTAMP decode check.

I believe this is probably a genuine issue... and the correct code is as follows:

Thanks for all your work!

highlight CMP library version

It seems that recent CMP library has changed its API and the performance test sample still works with CMP release 16. It will be nice if we can state that in CWPack document or source code of performance test build script.

Example is not working!

The example described in readme.md does not seem to work with the latest version.
Specifically, the following two functions do not seem to exist in the source of the master branch.

cw_unpack_next_map_size()
cw_unpack_next_str_length()

cw_pack_calculate_size function would be nice

The one problem I have with this library is that I don't know how to calculate in advance how to calculate how much memory I'll need to allocate here:
char buffer[20];
cw_pack_context_init (&pc, buffer, 20, 0);

A really nice function that would maintain the speed of this library but work around this would be a function that allows the user to indicate how many ints they want, how many floats, how many array objects, etc.

I'm thinking:
int cw_pack_calculate_size(int numInts, int numFloats, int numChars, int numBytes, etc...)

And internally it says that an int is of size 4, plus a header of 1(or whatever it is), therefore add numInts * 5 to the size of memory to be allocated, numChars is the number of individual characters across all strings, etc. and at the end it spits out how much memory to allocate before you call cw_pack_context_init.

Thanks for the library and all your work clwi!!

aliasing violations

Code like the following violates aliasing rules, an object shall only be accessed through its declared type or through a char pointer.
I think only the only legal way is to use memcpy (or assembly). Type punning through unions does not help here.

CWPack/src/cwpack_internals.h

Line 54 in d0beec8

#define cw_store32(x) *(uint32_t*)p = (uint32_t)x;

Support for null terminated strings.

Hi Claes.

It would be useful to add this helper function:

static void cw_pack_cstr(cw_pack_context *pc, const char *str) {
cw_pack_str(pc, str, strlen(str));
}

Thanks for CWPack, it is a nice piece of code.

CMake support & version tag

Would you accept a PR which adds a CMake configuration to your project? I have little knowledge in CMake's "packaging" side of things (export, local install, stuff like that) , but I could provide basic support for building CWPack and it's optional expansions from source. The tests could also be integrated to CTest I guess.

And absolutely unrelated... Is it intentional that the version 1.4 tag is missing it's "v" character at the beginning?

Unaligned access with 64-bit data on M4

On ARM Cortex-M4 (STM32F439) I get a hard fault error when CWPack tries to do unaligned access to 64-bit data. M4 does support unaligned access, but only on <=32-bit data (being a 32-bit microcontroller).

The hard fault happens on tmpu64 = *(uint64_t*)ptr; (in cwpack_defines.h) when ptr is anything but a multiple of 4. tmpu32 and tmpu16 are perfectly fine with unaligned access, tmpu64 is not.
Replacing *(uint64_t*)ptr with (uint64_t)(*(uint32_t*)(ptr+4)) << 32u | *(uint32_t*)ptr fixes this problem, but I admit it can be done cleaner.

What are your thoughts on this?

Btw. Enabling FORCE_ALIGNMENT does also solve the issue, but then I do not get all the benefits of fast execution on 32 or 16 bit (most used).

Thanks for this nice little piece of code 🙂.

Writing directly to pack buffer

Hi,

I'm using cwpack to send large binary objects together with some metadata. As the binary objects are quite large I would like to avoid an unnecessary memcpy. Is there a way to just reserve space when packing and then get a pointer that I can pass to my data producer to have it write into the pack buffer directly?