knopp/msgpack_dart

MsgPack implementation for dart / msgpack.org[Dart]

msgpack_dart's Issues

Support Timestamp extension type

It would be great if the timestamp extension type is supported out of the box.

It seems like there are already some functions that test DateTime deserialization, but these aren't wired up to the main test entrypoint. I have tried running these locally and the serialized DateTime currently deserializes to <null>.

$ dart test
Building package executable...
Built test:test.
00:00 +0 -1: test\messagepack_test.dart: unpackSmallDateTime [E]
  Expected: DateTime:<1970-01-01 13:00:00.000>
    Actual: <null>

  package:test_api                 expect
  test\messagepack_test.dart 14:3  unpackSmallDateTime

Publish latest on pub.dev

I want to use this package on the web platform. This is not possible because the latest version on https://pub.dev still have a dependency to dart:io. As the dependency is already removed on branch master, publish a new version to https://pub.dev, please.

Support hooks for external Map and Array builders

@knopp Would you entertain a change that would let a user of the library have more control over how deserialized Arrays and Maps are built?

I'm working on an implementation of Cognitect's transit for Dart and in order to support MessagePack I need some guarantees about map entry orders. If such a hook was available in this API, I would be able to intercept the keys and values as they are being deserialized and be able to preserve their insertion order.

import 'dart:collection';

abstract class MapBuilder<G, M, K, V> {
  // Initialize a new (gestational) map
  G init();
  // Add key and val to the gestational map, return new gestational map, which
  // must be used for the next invocation.
  G add(G m, K key, V val);
  // Convert gestational map into final map and return it.
  M complete(G m);
}

// This is the default builder if the user does not provide a custom one.
class MapBuilderImpl implements MapBuilder<Map, Map, dynamic, dynamic> {
  @override
  init() => Map();

  @override
  add(m, key, val) {
    m[key] = val;
    return m;
  }

  @override
  complete(m) => m;
}

class Deserializer {
  final ExtDecoder? _extDecoder;
  final MapBuilder _mapBuilder;                            //<-- this is new
  final codec = Utf8Codec();
  final Uint8List _list;
  final ByteData _data;
  int _offset = 0;

  Deserializer(
    Uint8List list, {
    ExtDecoder? extDecoder,
    this.copyBinaryData = false,
    MapBuilder? mapBuilder                                 //<-- this is new
  })  : _list = list,
        _data = ByteData.view(list.buffer, list.offsetInBytes),
        _extDecoder = extDecoder,
        _mapBuilder = mapBuilder ?? MapBuilderImpl();      //<-- this is new

  // ... SKIPPING ...

  // Delegates to `mapBuilder` for actual map construction, always passing in
  // the result of the previous call as input to the next call.
  Map _readMap(int length) {
    var mr = _mapBuilder.init();
    while (length > 0) {
      mr = _mapBuilder.add(mr, decode(), decode());
      --length;
    }
    return _mapBuilder.complete(mr);
  }
}

In my case, I would register a MapBuilder that initialized a LinkHashMap in order to preserve insertion order.

If this is something you'd consider, I'll create a PR. If not, I'll just create a fork.

Doesn't support JS runtimes

The library only works on native runtimes, i.e. where it can access dart:io. I'm going to take a look and try to fix it. Will keep this thread up-to-date.

Update Pub version

You recently merged in my PR #1, can we get a new pub release including this fix?

version tags

Thanks for providing this package.

Because the versions tab on pub.dev does not work,
I next checked the tags in the GitHub repo, but they are not maintained.
It's usually a good idea to apply a tag with each release to pub.dev.

https://dev.to/neshaz/a-tutorial-for-tagging-releases-in-git-147e

signal 11 (SIGSEGV), code 2 (SEGV_ACCERR)

OS: Mac Mini M2 13.4.1
Flutter: 3.10.6
Dart: 3.0.6

error message:

I/flutter ( 6213): {c: 53, i: 686085999305953281, k: 1, posts: [{pi: 159309080203182081, cs: [{cy: text, d: POST 2 BY USER 1 RDX 26752 post id: 159309080203182081 roleid 114900035881079, ns: true, as: false, sf: , io: false, ms: false, mw: 0, ss: [], os: [], tgs: []}], eg: [], ri: 114900035881079, ca: 0, su: 108708692099074, y: , b: , ts: 1658763002, mp: 1658763002799, si: {v: 0, l: 0, s: 0, c: 0, k: 0, q: 0, a: 0, g: 0, p: 0, f: 0}}, {pi: 209114773873303553, cs: [{cy: text, d: POST 0 BY USER 1 RDX 26752 post id: 209114773873303553 roleid 114900035881079, ns: true, as: false, sf: , io: false, ms: false, mw: 0, ss: [], os: [], tgs: []}], eg: [], ri: 114900035881079, ca: 0, su: 108903245611010, y: , b: , ts: 1661731653, mp: 1661731653478, si: {v: 0, l: 0, s: 0, c: 0, k: 0, q: 0, a: 0, g: 0, p: 0, f: 0}}, {pi: 211164734316163073, cs: [{cy: text, d: POST 1 BY USER 1 RDX 26752 post id: 211164734316163073 roleid 114900035881079, ns: true, as: false, sf: , io: false, ms: false, mw: 0, ss: [], os: [], tgs: []}], eg: [], ri: 1149000358
The Flutter DevTools debugger and profiler on sdk gphone64 arm64 is available at: http://127.0.0.1:9101?uri=http://127.0.0.1:62300/4cW5emwSI9c=/
W/Parcel  ( 6213): Expecting binder but got null!
I/m.example.mpack( 6213): Compiler allocated 4579KB to compile void android.view.ViewRootImpl.performTraversals()
F/libc    ( 6213): Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb400007a86cb9000 in tid 6234 (1.ui), pid 6213 (m.example.mpack)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/sdk_gphone64_arm64/emu64a:13/TE1A.220922.031/10278734:userdebug/dev-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2023-07-24 10:12:13.897223890+0800
Process uptime: 3s
Cmdline: com.example.mpack
pid: 6213, tid: 6234, name: 1.ui  >>> com.example.mpack <<<
uid: 10173
tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb400007a86cb9000
    x0  00000000fff957e6  x1  0000000000000000  x2  0000000000000000  x3  00000077047dcf85
    x4  b400007a86cb9000  x5  0000000000000000  x6  0000007700847301  x7  000000000000040c
    x8  0000000000000007  x9  00000078ac085c30  x10 b400007a56c5cd80  x11 00000078be736d58
    x12 000000000003d000  x13 0000000000000002  x14 0000000000000002  x15 00000078be736ea8
    x16 00000078be65e000  x17 00000077059e13f1  x18 00000078b4560000  x19 b400007a86c69310
    x20 b400007a56c5cde8  x21 b400007a56c5ce08  x22 0000007700008081  x23 b400007a56c5cf08
    x24 0000007704cd6ec1  x25 00000077047a7974  x26 b400007a86c69310  x27 0000007704cd6ea0
    x28 0000000800000077  x29 00000078be736ec8
    lr  00000078a35309bc  sp  00000078be65d000  pc  00000078a3e95444  pst 00000000a0001000
backtrace:
      #00 pc 0000000000115444  [anon:dart-code]
Lost connection to device.

code:

void test() async {

    ByteData bytes1 = await rootBundle.load('assets/data1');
    var result1 = MPack.deserialize(Uint8List.view(bytes1.buffer));
    print(result1);

    ByteData bytes2 = await rootBundle.load('assets/data2');
    var result2 = MPack.deserialize(Uint8List.view(bytes2.buffer));
    print(result2);

  }

summary:

this crash ONLY happens on combination of following conditions:

1. Flutter app (pure dart code is ok)
2. and in Debug mode, profile mode is ok
3. and in Android emulator, tried android 11, 12, 13, all same, real mobile in debug mode is ok
4. and unpack two data packs, if only unpack one of them, it's ok
5. and two data pack are a little bit big, in my case, about 4k, if one of data packs is small, like 2k, will be ok

very interesting .....

Make offset available publicly in Decoder

I think that if the offset was available publicly I could parse messages that have become concatenated on a stream like:

[1, 2, 3][4, 5, 6]

By using the offset to split the original byte array. Otherwise I can't think of a sensible way of parsing that. If you are ok with that, and the concept is correct, I am happy to make a PR. For now I will just copy-paste and modify the code.