hildjj/node-cbor

Encode and decode CBOR documents, with both easy mode, streaming mode, and SAX-style evented mode.

CBOR

Encode and parse data in the Concise Binary Object Representation (CBOR) data format (RFC8949).

Pointers

This is a monorepo that holds a few related packages:

• cbor: a node-centric CBOR processor
• cbor-web: the cbor package compiled for use on the web, including all of its non-optional dependencies
• cbor-cli: a set of command-line tools for working with the cbor package
• Examples:
  • webpack-demo: bundle cbor using webpack
  • parcel-demo: bundle cbor using parcel
  • browserify-demo: bundle cbor using browserify
  • plain-demo: bundle cbor by just using cbor-web directly

Tooling

• Install with pnpm install -r, see. The important thing (for example) is that the cbor-cli package ends up depending on the local version of cbor.
• monorepo-wide scripts:
  • install-global (make available everywhere) or install-local (use npx if you want to us it outside a package script) install all of the tooling that might be needed locally, but isn't needed on CI
  • deploy: build and deploy cbor-web and all of the actions
  • coverage: run tests and report coverage; look in coverage/lcov-report/index.html.
  • lint: run eslint over all projects

GitHub dependencies

If you really need to get at a specific rev from GitHub, you can no longer do npm install hildjj/node-cbor. Instead you need:

npm install https://gitpkg.now.sh/hildjj/node-cbor/packages/cbor?main

Supported Node.js versions

This project now only supports versions of Node that the Node team is currently supporting. Ava's support statement is what we will be using as well. Currently, that means Node 16+ is required. If you need to support an older version of Node (back to version 6), use cbor version 5.2.x, which will get nothing but security updates from here on out.

node-cbor's Issues

Keep input bytes from Decoder

When reading from a Decoder, add an option not to discard the bytes from the input. The callback would then get a JavaScript object as well as the bytes that were decoded. This would be used in situations where you want to process the JS, then pass the bytes along unmodified.

TypeError: bignumber is not a constructor, nodejs webpack bundle

In a nodejs app bignumber.js throws in initialization.

const MINUS_ONE = new bignumber(-1)
                  ^
TypeError: bignumber is not a constructor
    at Object.<anonymous> (/usr/src/app/server.bundle.js:33351:19)
    at __webpack_require__ (/usr/src/app/server.bundle.js:20:30)
    at Object.<anonymous> (/usr/src/app/server.bundle.js:24786:16)
    at __webpack_require__ (/usr/src/app/server.bundle.js:20:30)
    at Object.module.exports (/usr/src/app/server.bundle.js:85704:17)
    at __webpack_require__ (/usr/src/app/server.bundle.js:20:30)
    at Object.<anonymous> (/usr/src/app/server.bundle.js:50098:21)
    at __webpack_require__ (/usr/src/app/server.bundle.js:20:30)
    at Object.<anonymous> (/usr/src/app/server.bundle.js:48238:12)
    at __webpack_require__ (/usr/src/app/server.bundle.js:20:30)

bignumber version is 6.0.0, required by:

[email protected] /path/build/src
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected]

bignumber is used in a webpack bundle, on node version is LTS 8.12.0, inside a docker container with image node:8.12.0-alpine in an ubuntu server host.

The code referenced by the stack is:

function(module, exports, __webpack_require__) {

"use strict";


const bignumber = __webpack_require__(62)
const utils = __webpack_require__(80)
const url = __webpack_require__(63)

const MINUS_ONE = new bignumber(-1)
const TEN = new bignumber(10)
const TWO = new bignumber(2)

/**
 * A CBOR tagged item, where the tag does not have semantics specified at the
 * moment, or those semantics threw an error during parsing. Typically this will
 * be an extension point you're not yet expecting.
 */
class Tagged {

Any help and how to fix the issue would be greatly appreciated!

Documentation link is broken

The documentation link (http://hildjj.github.io/node-cbor/doc/) gives a 404.

Stray parent?

I'm wondering if I found a bug with zero-length maps where the output includes a very complex object, or if the example I show below is actually the correct behavior and I just don't understand what the parser is returning.

I have an CBOR object that looks like this:

attestationObject
A3
    63
        66 6D 74                    // "fmt"
    64
        6E 6F 6E 65                 // "none"
    67
        61 74 74 53 74 6D 74        // "attStmt"
    A0                              // map(0)
    68
        61 75 74 68 44 61 74 61     // "authData"
    59 01 26
        49 96 0D E5 88 0E 8C 68 74 34 17 0F 64 76 60 5B
        [ ... snip ... ]

What I get out of the parser looks like this:

{
  fmt: 'none',
  attStmt: {
    [Symbol(parent)]: ['fmt',
      'none',
      'attStmt', [Circular],
      'authData', <
      Buffer 49 96 0 d e5 88 0e 8 c 68 74 34 17 0 f 64 76 60 5 b 8 f e4 ae b9 a2 86 32 c7 99 5 c f3 ba 83 1 d 97 63 41 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00... > , 
      [Symbol(parent)]: null, 
      [Symbol(major type)]: 5
    ]
  },
  authData: < Buffer 49 96 0 d e5 88 0e 8 c 68 74 34 17 0 f 64 76 60 5 b 8 f e4 ae b9 a2 86 32 c7 99 5 c f3 ba 83 1 d 97 63 41 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00... >
}

The map key attStmt has a value that is a zero-length map. I don't understand why the object returned by the parser has a parent value. Is there something I'm missing?

Incorrect use of readable.read() in _encodeAll() [BUG]

let cbor = require("cbor");

let str = "";
for(let i = 0; i < 16375; i++)
	str += "a";

cbor.decodeFirstSync(cbor.encode({ str, v: 10 }));

Error: Insufficient data
    at Object.decodeFirstSync (.../node_modules/cbor/lib/decoder.js:158:15)

When is 16375 changed to 16374 then there is no exception. Node version 12.5.0.

Seems that in encoded cbor is missing value "v".

Bug is here:

  _encodeAll(objs) {
    const bs = new NoFilter()
    this.pipe(bs)
    for (const o of objs) {
      if (typeof o === 'undefined') {
        this._pushUndefined()
      } else if (o === null) {
        this._pushNull(null)
      } else {
        this.write(o)
      }
    }
    this.end()
    return bs.read()
  }

Function bs.read() may not always read all data produced by Encoder but just content of node js stream internal buffer (which has default size 16384 bytes). This function must be called multiple times.

Here is example of correct way how to use it:

https://nodejs.org/api/stream.html#stream_readable_read_size

"Note that the while loop is necessary when processing data with readable.read()"

Or better would be to concat buffers from "data" event and return as result and you will not need that "NoFilter".

Similar bug is probably also on other places like decodeFirstSync and other "sync" functions.

Node.js v10 generates DeprecationWarning warnings.

Repro: Run npm test using Node.js version 10 (just released.)

node:46164) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

(node:46164) [DEP0079] DeprecationWarning: Custom inspection function on Objects via .inspect() is deprecated

Additional messages reflecting the same deprecations are listed.

ES2015 ArrayBuffers instead of node.js Buffers

It would be nice if there were a way for the parser to generate ES2015 ArrayBuffers rather than node.js Buffers, for compatibility with APIs that will be consuming ArrayBuffers. Otherwise, it becomes necessary to recursively find and convert Buffers to ArrayBuffers.

UgilfyJS - Unexpected token: name (Simple)

This library is used by autobahn-js.
When adding autobahn to my vue-js app, I have an error when trying to build for production using webpack 2.6 and UglifyJS :

ERROR in static/js/app.bdcd88ac9fe0000fe77f.js from UglifyJs
Unexpected token: name (Simple) [./~/cbor/lib/simple.js:9,0]

node vs. nodejs

Some systems use nodejs instead of node, shall be checked at installation

Encoding circular objects

a = {}
a.a = a
cbor.encode(a) // RangeError: Maximum call stack size exceeded

Mark objects as visited with a symbol, and delete the symbol on a second pass? Throw an error if the symbol exists on encode. Alternately: replace value with NULL or other symbol. See what other libraries do.

Why is an serialized obj unserialized into an array with length = 1?

When serializing a single JS obj (using cbor.encode), it becomes an array when unserialized (using cbor.decode), what is the reason behind this behaviour?

Test more failure modes

Particularly in the .encode and .decode functions, where there's not as much chance to recover.

Add option to filter undefined values

I think there is difference between undefined in javascript and undefined type in cbor (https://tools.ietf.org/html/rfc7049#section-3.8). In cbor it means rather item with encoding problem, but in javascript undefined is more often associated with key or argument which does not exists for example when is function called with less arguments than it has then all "optional" arguments will become undefined.

Seems some cbor deserialisers treats undefined in place of optional value as error instead of ignoring it but accepts null or when is key not encoded in cbor.

Would be nice to have option to not include keys with undefined values.

TD is not a constructor - Node 6.x

I am with the MeshCentral project and have users on Node 6.x. You latest change just broke it with the error below. I did notice in GitHub that Node 6 is no longer supported. Any way this could be reconsidered? Users issues on my side are here. Thanks.

C:\tmp\node_modules\cbor\lib\utils.js:21
const td = new TD('utf8', {fatal: true, ignoreBOM: true})
           ^

TypeError: TD is not a constructor
    at Object.<anonymous> (C:\tmp\node_modules\cbor\lib\utils.js:21:12)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Module.require (module.js:466:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\tmp\node_modules\cbor\lib\commented.js:5:15)
    at Module._compile (module.js:541:32)

    at ChildProcess.exithandler (child_process.js:202:12)
    at emitTwo (events.js:111:20)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:850:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:5)
  killed: false,
  code: 1,
  signal: null,
  cmd: 'xxx' }

Main Example Fails

Hello,
I just downloaded the module (via npm), and tried example from documentation.

It fails:

drasko@Lenin:~/cbor/jstest$ node cborTest.js 

assert.js:90
  throw new assert.AssertionError({
  ^
AssertionError: false == true
    at /home/drasko/cbor/jstest/cborTest.js:8:10
    at Decoder.<anonymous> (/home/drasko/cbor/jstest/node_modules/cbor/lib/decoder.js:119:22)
    at Decoder.g (events.js:273:16)
    at emitNone (events.js:85:20)
    at Decoder.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
    at Function.Module.runMain (module.js:449:11)
    at startup (node.js:148:18)

This is the example I am trying:

drasko@Lenin:~/cbor/jstest$ cat cborTest.js 
var cbor = require('cbor');
var assert = require('assert');

var encoded = cbor.encode(true); // returns <Buffer f5>
cbor.decodeFirst(encoded, function(error, obj) {
  // error != null if there was an error
  // obj is the unpacked object
  assert.ok(obj[0] === true);
});

// Use integers as keys?
var m = new Map();
m.set(1, 2);
encoded = cbor.encode(m); // <Buffer a1 01 02>

Nested Map Issue

I'm trying to decode a structure with nested maps and having a number of issues. Here's my test:

var cbor = require('cbor');

cbor.decodeFirst('a46161a200a161720111a16172016162191388626d6e00626d7819fffe', function(err, val) {
   console.log('val(a): ' + JSON.stringify(val));
});

cbor.diagnose('a46161a200a161720111a16172016162191388626d6e00626d7819fffe', function(err, val) {
   console.log('diag(a): ' + val);
});

cbor.decodeFirst('a26161a200a161720111a16172016162191388', function(err, val) {
   console.log('val(b): ' + JSON.stringify(val));
});

cbor.diagnose('a26161a200a161720111a16172016162191388', function(err, val) {
   console.log('diag(b): ' + val);
});

the output I get is

val(a): {"a":{},"b":5000,"mn":0,"mx":65534}
val(b): {"a":{},"b":5000}
diag(a): {"a": {0: {"r": 1}, 17: {"r": 1}}, "b": 5000, "mn": 0, "mx": 65534}

diag(b): {"a": {0: {"r": 1}, 17: {"r": 1}}, "b": 5000}

the diag outputs are what I expect. The comment function also seems to display the maps. If I add code that prints every time a map is encountered I can see that the event the decode command is actually walking through all levels, they just don't seem to be added to the actual object that is output.

Empty Map Decodes to an empty Object

const a = new Map()
const b = cbor.decode(cbor.encode(a)) 
b // is {} should be Map {}

nested object shown as [Object] with decodeAll

using decodeAllSync on this hex string

Buffer.from([0xbf, 0x6a, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x19, 0x26, 0xfc, 0x64, 0x6c, 0x6f, 0x67, 0x73, 0x9f, 0xbf, 0x64, 0x6e, 0x61, 0x6d, 0x65, 0x6a, 0x72, 0x65, 0x62, 0x6f, 0x6f, 0x74, 0x5f, 0x6c, 0x6f, 0x67, 0x64, 0x74, 0x79, 0x70, 0x65, 0x02, 0x67, 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x9f, 0xbf, 0x63, 0x6d, 0x73, 0x67, 0x78, 0x21, 0x72, 0x73, 0x6e, 0x3a, 0x52, 0x45, 0x51, 0x55, 0x45, 0x53, 0x54, 0x45, 0x44, 0x2c, 0x20, 0x63, 0x6e, 0x74, 0x3a, 0x30, 0x2c, 0x20, 0x69, 0x6d, 0x67, 0x3a, 0x30, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x30, 0x62, 0x74, 0x73, 0x1a, 0x1a, 0xce, 0xa3, 0xc0, 0x65, 0x6c, 0x65, 0x76, 0x65, 0x6c, 0x04, 0x65, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x19, 0x26, 0xfb, 0x66, 0x6d, 0x6f, 0x64, 0x75, 0x6c, 0x65, 0x06, 0xff, 0xff, 0xff, 0xff, 0x62, 0x72, 0x63, 0x00, 0xff)]

is decoding as [ { next_index: 9980, logs: [ [Object] ], rc: 0 } ]

but should decode as

{"next_index": 9980, "logs": [{"name": "reboot_log", "type": 2, "entries": [{"msg": "rsn:REQUESTED, cnt:0, img:0.0.0.0", "ts": 449749952, "level": 4, "index": 9979, "module": 6}]}], "rc": 0}

an online decode I got that from if it helps
http://cbor.me/?bytes=bf.6a.6e.65.78.74.5f.69.6e.64.65.78.19.26.fc.64.6c.6f.67.73.9f.bf.64.6e.61.6d.65.6a.72.65.62.6f.6f.74.5f.6c.6f.67.64.74.79.70.65.02.67.65.6e.74.72.69.65.73.9f.bf.63.6d.73.67.78.21.72.73.6e.3a.52.45.51.55.45.53.54.45.44.2c.20.63.6e.74.3a.30.2c.20.69.6d.67.3a.30.2e.30.2e.30.2e.30.62.74.73.1a.1a.ce.a3.c0.65.6c.65.76.65.6c.04.65.69.6e.64.65.78.19.26.fb.66.6d.6f.64.75.6c.65.06.ff.ff.ff.ff.62.72.63.00.ff

Publish Pre-Compile to ES5

I was trying to npm run build in create-react-app and it failed to minify. create-react-app recommends compiling into ES5. Could you publish pre-compiled to npm?

Rip out coffee-script

OK, I'm done with coffee-script, if nobody is going to maintain the tools.

New coding style: http://standardjs.com/

Approach: rip out tools. hand-edit each cs-generated javascript file. New major release number.

negative zero encoded as zero

Obscure one, but I found this running the official test vectors from Appendix A.

It seems that encoding for the IEEE special number -0.0 gets encoded as hex 00, rather than hex f9 80 00 [see page 41 of RFC 7049]

ndjson

might be nice to have an -n option on the cbor2json and json2cbor tools to use streaming (with ndjson). See https://github.com/jbenet/node-ubjson-stream#streaming

sync decode

It's great that you support streaming decode, but the decoder itself shouldn't need to be async.
If I do happen to have an entire object, you ought to be able to parse it synchronously,
without using a callback or a promise.

This module would be easier to use if had sync {encode, decode}, actually that means it could be a leveldb codec! https://github.com/Level/levelup#custom_encodings That would mean it has the same api as a bunch of other serialization modules, which means it is easy to switch to cbor!

Switch from async to q

I'm using q in a bunch of other places.

Implement stringref extension

The http://cbor.schmorp.de/stringref CBOR extension (tags 25 and 256) permit substantial compression of CBOR encoded objects with repetitive string values. It seems adding this to the CBOR encoder and decoder would introduce relatively little overhead in cases where it isn't used, and could actually run faster when decoding objects where the compression is effective.

This is a topic that interests me, and I'd be willing develop a pull request for it if you are interested.

Finish implementation of value-sharing

It looks like the work for the CBOR value-sharing extension http://cbor.schmorp.de/value-sharing is partially done, but not complete. The tag values 28 and 29 permit a better response to detecting a duplicate object than is currently implemented [throw new Error('Loop detected while CBOR encoding')]

Issue #35 seems like it was intended to track this but perhaps the full idea behind the value-sharing proposal wasn't well expressed. Looking at the code in sandhawke/borc-refs I'm not sure his encoder actually add the tags needed to flesh this out.

Thinking about the implementation of encoding with shared objects, an approach based on WeakMap seems more clean to me than coloring the actual objects using Symbols, though I agree the symbols are unlikely to hurt things. Was this approach considered?

Crazy Xerox-Like number flipping in CBOR-Decoder

Hey there!
I am using tinycbor (c lib) as CBOR encoder, CoAPthon to transmit it and this library to decode it. Unfortunately, I get some very weird number flipping. When I transmit [ 10, 10, 1111 ], all the data is transmitted correctly. When sending [ 10, 10, 1211 ], I will get [ 10, 10, 1277 ] instead! I can not explain how this can happen. I can not be any kind of overflow, since [ 10, 10, 1311 ] is transmitted correctly. And when transmitting [ 10, 10, 1234 ], I also get [ 10, 10, 1277 ].
When I inspect the traffic using wireshark, it tells me, that the problem has to do with node-cbor.
Here is a quick lookup-table for some values:

Sent	Received	Correct
[ 10, 10, 111 ]	[ 10, 10, 111 ]	yes
[ 10, 10, 1111 ]	[ 10, 10, 1111 ]	yes
[ 10, 10, 1211 ]	[ 10, 10, 1277 ]	no
[ 10, 10, 1151 ]	[ 10, 10, 1151 ]	yes
[ 10, 10, 1152 ]	[ 10, 10, 1277 ]	no
[ 10, 10, 1200 ]	[ 10, 10, 1277 ]	no
[ 10, 10, 1279 ]	[ 10, 10, 1277 ]	no
[ 10, 10, 1280 ]	[ 10, 10, 1280 ]	yes
[ 10, 10, 1020 ]	[ 10, 10, 1021 ]	no
[ 10, 10, 1021 ]	[ 10, 10, 1021 ]	yes, probably by accident
[ 10, 10, 1022 ]	[ 10, 10, 1021 ]	no
[ 10, 10, 1023 ]	[ 10, 10, 1021 ]	no
[ 10, 10, 1024 ]	[ 10, 10, 1024 ]	yes
[ 10, 10, 1025 ]	[ 10, 10, 1025 ]	yes
[ 10, 10, 1026 ]	[ 10, 10, 1026 ]	yes
[ 10, 10, 1027 ]	[ 10, 10, 1027 ]	yes
[ 10, 10, 505 ]	[ 10, 10, 509 ]	no
[ 10, 10, 506 ]	[ 10, 10, 509 ]	no
[ 10, 10, 507 ]	[ 10, 10, 509 ]	no
[ 10, 10, 508 ]	[ 10, 10, 509 ]	no
[ 10, 10, 509 ]	[ 10, 10, 509 ]	yes, by accident
[ 10, 10, 510 ]	[ 10, 10, 509 ]	no
[ 10, 10, 511 ]	[ 10, 10, 509 ]	no
[ 10, 10, 512 ]	[ 10, 10, 512 ]	yes
[ 10, 10, 513 ]	[ 10, 10, 513 ]	yes
[ 10, 10, 514 ]	[ 10, 10, 514 ]	yes
[ 10, 10, 515 ]	[ 10, 10, 515 ]	yes
[ 10, 10, 516 ]	[ 10, 10, 516 ]	yes
[ 10, 10, 517 ]	[ 10, 10, 517 ]	yes

One strange thing is, that all numbers between 1151 and 1280 are mapped to 1277, while 1151 and 1280 are transmitted correctly. Also, numbers a little smaller than 1151 and a little bigger than 1280 work fine. This in funny, because 1152 (the first "wrong number") is the sum of 1024 and 128, while 1280 (the first number working again) is the sum of 1024 and 256.
To me, it looks like some certain ranges of numbers are causing these problems.
You will find the same strange pattern around 2048, 4096, 8192 and so on. Its always the same: Numbers smaller than 2048 are causing trouble, numbers above are working fine.
We find these strange patterns at all numbers that are the power of two - but not only at these!
I hope my description helps you to find the cause :)

Yours Sincerely,
Felix

Upgrade to bignumber.js 7.0?

node-cbor/package.json

Line 76 in f14b608

"bignumber.js": "^6.0",

Recently I noticed this deprecation warning
npm WARN deprecated [email protected]: Custom ALPHABET bug fixed in v7.0.2

Dear developer, do you think it worth for an upgrade?

Implement in C

Implement the synchronous interface in C/C++ for node. Compare the performance to the existing javascript implementation, the synchronous-only implementation (https://github.com/dignifiedquire/fast-cbor), and JSON.parse/stringify.

Problem encoding `Uint8Array`

Problem

Code example:

const cbor = require('cbor');

const arr1 = [1, 2, 3, 4];
console.log(cbor.encode(arr1));
console.log(cbor.decode(cbor.encode(arr1)));

const arr2 = new Uint8Array([1, 2, 3, 4]);
console.log(cbor.encode(arr2));
console.log(cbor.decode(cbor.encode(arr2)));

Result:

<Buffer 84 01 02 03 04>
[ 1, 2, 3, 4 ]
<Buffer a4 61 30 01 61 31 02 61 32 03 61 33 04>
{ '0': 1, '1': 2, '2': 3, '3': 4 }

Expected

<Buffer 84 01 02 03 04>
[ 1, 2, 3, 4 ]
<Buffer 68 01 02 03 04>
Uint8Array [ 1, 2, 3, 4 ]

OR

<Buffer 84 01 02 03 04>
[ 1, 2, 3, 4 ]
<Buffer 44 01 02 03 04>
<Buffer 01 02 03 04>

Comments

"BORC" produces second result in node setting and first result in browser setting (after "browserify").

E.g. this is how both libraries look in browser:

explore why highWaterMark doesn't work on node6

for #94, I added a big hack:

node-cbor/test/encoder.ava.js

Line 259 in 474fb2b

if (!process.version.startsWith('v6')) {

in order to get the tests to pass. This does not block #95 but I wanted a link there.

Some binaries not linked at installation

e.g. cbor2comment

npm run dev fails on Windows

using npm run dev generates the following output on Windows from the CMD.exe prompt:

C:\code\node-cbor>npm run dev

> [email protected] predev C:\code\node-cbor
> if [ ! -f coverage/index.html ]; then mkdir coverage; cp .waiting.html coverage/index.html; fi

! was unexpected at this time.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] predev: `if [ ! -f coverage/index.html ]; then mkdir coverage; cp .waiting.html coverage/index.html; fi`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] predev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Burt_\AppData\Roaming\npm-cache\_logs\2018-04-25T02_02_34_180Z-debug.log

Not a big deal, I'll work through it.

remove support for node 6

We have now pinned an old version of ava so I can still test on Node 6. There will be other reasons to move over time.

This is a tracking bug. Please reference this with a bug in your project if you want me to wait for you to stop supporting Node 6.

Allow for both decoding and diagnose outputs on the same input stream

Ideally, you would get one callback with both a JS object and a string.

Detect invalid UTF8

The node UTF8 parser does not error on invalid input.

Support expanded example format from the RFC

Example, from section 2.2.1 of RFC 7049:

0x9f018202039f0405ffff
9F        -- Start indefinite-length array
   01     -- 1
   82     -- Array of length 2
      02  -- 2
      03  -- 3
   9F     -- Start indefinite-length array
      04  -- 4
      05  -- 5
      FF  -- "break" (inner array)
   FF     -- "break" (outer array)

Except that I want the hex to line up vertically.

Ensure maps are sorted

According to https://tools.ietf.org/html/rfc7049#section-3.9

The keys in every map must be sorted lowest value to highest.
Sorting is performed on the bytes of the representation of the key
data items without paying attention to the 3/5 bit splitting for
major types. (Note that this rule allows maps that have keys of
different types, even though that is probably a bad practice that
could lead to errors in some canonicalization implementations.)
The sorting rules are:

  *  If two keys have different lengths, the shorter one sorts
     earlier;

  *  If two keys have the same length, the one with the lower value
     in (byte-wise) lexical order sorts earlier.

all keys in maps need to be sorted. Looking at https://github.com/hildjj/node-cbor/blob/master/src/encoder.coffee#L238-L260 it does not look like this is currently happening.

Ref ipld/js-ipld#19

Add BigInt support

Node 10.7 implements BigInt

Is Decoder.decodeFirstSync returning the first object (or last?)

Based on code review alone, it looks to me like the static decodeFirstSync(input, options) function is looping through all objects in a CBOR stream, then returning the last one, rather than the first one as the function name would imply.

In particular, inclusion of the following code seems an error in the context of this function:

    while (!state.done) {
      const b = s.read(state.value)
      if ((b == null) || (b.length !== state.value)) {
        throw new Error('Insufficient data')
      }
      state = parser.next(b)
    }

I'm wondering if this might be the root cause of Issue #43.

switch from nodeunit to ava

I really like ava.

Async encoding of nodejs Stream

Should be possible to asynchronously encode object with some values of type Stream and return stream which will emit encoded data continuously.

CBOR supports indefinite byte buffers: https://tools.ietf.org/html/rfc7049#section-2.2.2 so size of input streams does not must be known before writing data.

CBOR supports also indefinite arrays and maps so also input object streams are doable.

Diagnostic format for floats, e.g.

RFC7049 says:

An underscore followed by a decimal digit n indicates that the
preceding item (or, for arrays and maps, the item starting with the
preceding bracket or brace) was encoded with an additional
information value of 24+n.  For example, 1.5_1 is a half-precision
floating-point number, while 1.5_3 is encoded as double precision.
This encoding indicator is not shown in Appendix A.  (Note that the
encoding indicator "_" is thus an abbreviation of the full form "_7",
which is not used.)

While you're in there, check this:

As a special case, byte and text strings of indefinite length can be
notated in the form (_ h'0123', h'4567') and (_ "foo", "bar").

Error decoding object with large buffer

The following snippet fails with error at decode:

    let buffer = fs.readFileSync('some_large_file');
    buffer = buffer.slice(0, 16377);

    let object = {
        a: buffer,
        b: 1
    };

    let encoded = cbor.encode(object);
    let decoded = cbor.decode(encoded);

Error: Insufficient data
    at Object.decodeFirstSync (/*****/node_modules/cbor/lib/decoder.js:131:15)
    at /******/index.js:24:24
    at Object.<anonymous> (/*****/index.js:181:1)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:159:18)

The size of buffer for which the error occurs depends on the name of the object properties a and b. That is, if the names are longer then shorter buffers produce the error.

With property names a and b, the error does not occur if I reduce the size of buffer to 16376 bytes, like:

    buffer = buffer.slice(0, 16376);

The error does not occur if I switch the order of a and b in object, like:

    let object = {
        b: 1,
        a: buffer
    };

Reproduced with 2.0.0 and 3.0.0

Perf and binary-parse-stream/binary-parser

@hildjj, @nathan7

node-cbor depends on some very clever work combining generators with readable streams that has cloned into the node-cbor/vendor/binary-parse-stream directory. The README.md is rather sparse, and says it's based on https://github.com/nathan7/binary-parser, but that's a broken link now. However, https://github.com/nathan7/binary-parse-stream still seems present, if a bit stale... (4 years since last commit.

None-the-less, the approach used permits a stream-based parser to be written with sync-looking code, which is very impressive. My gut feeling however is that the performance of node-cbor may be limited by this, in particular yielding once or more for each parsed data item (perhaps as short as one byte of incoming data.) All of these yields seem like a substantial source of overhead for large data sets. A chunk based approach where the parser can remain synchronous during processing of entire Buffer's seem like it could offer substantial performance gains.

ECMAScript async iterators seems like it might be a good forward-looking and chunk-oriented approach. TC 39 declared async iteration "finished" in January 2018, and V8 and Node v10 both seem to have adopted it, at lest experimentally.

I'm considering writing a binary parsing framework similar to binary-parse-stream but based instead on async iterators. Before I dig into alone, I thought it might be valuable to run the idea by you both, to see if what you think, and to see if you know of other implementations doing something like this.

Encoder calls Readable.push

This causes an unneeded copy. Perhaps make BufferStream Readable instead of Writable, and copy the push semantics in? Perhaps look at patching Readable upstream to allow unbuffered access one day.

Add options for serializing undefined

• Error
• null
• undefined

open to others

Switch to jsbi

See dignifiedquire#37

Parsing non-recursive, relatively shallow documents can cause stack overflows

Some CBOR documents that consist of an array of maps can cause stack overflows in your implementation when the document exceeds a certain threshold, even if the document isn't deeply nested.

This happens in the latest version of the "cbor" package.

I found this problem while writing a Ruby JSON-to-CBOR converter. The following is code for the converter and generates a CBOR document, "document.cbor", that demonstrates this problem.

https://gist.github.com/peteroupc/58fd4a29d83fcb7cd0c9

"Push Error" thrown for larger objects

After a certain object size, encoding fails with the error "Push Error" which is meaningless. There is no limit shown in the docs (at least I didn't find it). How can I increase this limit?

const arr = []
for(let i = 0; i < 50000; i++) arr.push(1)
const result = require('cbor').encode(arr) // throws error