yodaos-project/shadownode

Use Node.js in your end devices(QQ: 796448809)

ShadowNode

The Node.js runtime in shadow, enables N-API and vast Node.js packages on edge devices.

The project is another runtime for your Node.js packages, while ShadowNode is designed to be used on memory limited devices. It's inspired and forked from the awesome project Samsung/iotjs.

Quick Start

To get started with ShadowNode, you could download prebuilt binaries on Release Page for following targets:

• Linux x64
• macOS x64

Memory usage and binary footprint are measured at here with real target daily.

Documentation

• Getting Started
• API Reference

Build

Fetch source code

$ git clone https://github.com/yodaos-project/ShadowNode.git
$ cd ShadowNode

Build ShadowNode

$ npm run build

Get available build options

$ tools/build.py --help

Install

$ tools/build.py --install

Run tests

$ npm test

For additional information see Getting Started.

Compared with Node.js

ShadowNode is not designed to be ran identical code that ran on Node.js. While edge environments are experiencing limited resources on runtime, the packages to be ran on ShadowNode shall be rewritten in a resource compact way. However for the very initial thought of sharing the Node.js vast module ecosystem, we would like make ShadowNode compatible with Node.js with our efforts.

• Assert
• Buffer
• C/C++ Addon API
• Child Process
• Crypto
• DNS
• Events
• File System
• HTTP
• Module
• Net
• OS
• Process
• Stream
• Timers
• TLS
• UDP/Datagram
• Zlib

Since the MQTT protocol is commonly used for communication between IoT devices, ShadowNode supports the protocol natively, and keeps the API consistent with the popular library MQTT.js. See MQTT API for details.

The WebSocket is a popular protocol in IoT environment as well, and also supported by ShadowNode natively. See WebSocket API.

For hardware geek, this project benefits from the upstream IoT.js, which has supported the following hardware interfaces, you are able to port ShadowNode to your platforms and start hacking with JavaScript:

• ADC
• BLE
• GPIO
• I2C
• PWM
• SPI
• UART

License

ShadowNode is Open Source software under the Apache 2.0 license. Complete license and copyright information can be found within the code.

shadownode's Issues

tls, crypto: some memory leak detected from CI

The logs:

==8824== 4 bytes in 1 blocks are definitely lost in loss record 1 of 5
==8824==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8824==    by 0x4E859F7: iotjs_buffer_allocate (iotjs_util.c:80)
==8824==    by 0x4E7FD60: iotjs_string_create_with_buffer (iotjs_string.c:55)
==8824==    by 0x4E80AFE: iotjs_jval_as_string (iotjs_binding.c:133)
==8824==    by 0x4E9E948: TlsConstructor (iotjs_module_tls.c:204)
==8824==    by 0x4ED6959: ecma_op_function_call (ecma-function-object.c:465)
==8824==    by 0x4ED6DDA: ecma_op_function_construct_simple_or_external (ecma-function-object.c:594)
==8824==    by 0x4ED6F90: ecma_op_function_construct (ecma-function-object.c:660)
==8824==    by 0x4EF2337: opfunc_construct (vm.c:483)
==8824==    by 0x4EF75D6: vm_execute (vm.c:2866)
==8824==    by 0x4EF7898: vm_run (vm.c:2941)
==8824==    by 0x4ED67D3: ecma_op_function_call (ecma-function-object.c:404)
==8824== 
==8824== 13 bytes in 1 blocks are definitely lost in loss record 2 of 5
==8824==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8824==    by 0x4E859F7: iotjs_buffer_allocate (iotjs_util.c:80)
==8824==    by 0x4E80A67: iotjs_jval_as_string (iotjs_binding.c:125)
==8824==    by 0x4E9E948: TlsConstructor (iotjs_module_tls.c:204)
==8824==    by 0x4ED6959: ecma_op_function_call (ecma-function-object.c:465)
==8824==    by 0x4ED6DDA: ecma_op_function_construct_simple_or_external (ecma-function-object.c:594)
==8824==    by 0x4ED6F90: ecma_op_function_construct (ecma-function-object.c:660)
==8824==    by 0x4EF2337: opfunc_construct (vm.c:483)
==8824==    by 0x4EF75D6: vm_execute (vm.c:2866)
==8824==    by 0x4EF7898: vm_run (vm.c:2941)
==8824==    by 0x4ED67D3: ecma_op_function_call (ecma-function-object.c:404)
==8824==    by 0x4ED6DDA: ecma_op_function_construct_simple_or_external (ecma-function-object.c:594)
==8824== 
==8824== 288 (256 direct, 32 indirect) bytes in 4 blocks are definitely lost in loss record 5 of 5
==8824==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8824==    by 0x4E859F7: iotjs_buffer_allocate (iotjs_util.c:80)
==8824==    by 0x4E8BE9D: iotjs_crypto_hash_create (iotjs_module_crypto_hash.c:15)
==8824==    by 0x4E8BF37: HashConstructor (iotjs_module_crypto_hash.c:27)
==8824==    by 0x4ED6959: ecma_op_function_call (ecma-function-object.c:465)
==8824==    by 0x4ED6DDA: ecma_op_function_construct_simple_or_external (ecma-function-object.c:594)
==8824==    by 0x4ED6F90: ecma_op_function_construct (ecma-function-object.c:660)
==8824==    by 0x4EF2337: opfunc_construct (vm.c:483)
==8824==    by 0x4EF75D6: vm_execute (vm.c:2866)
==8824==    by 0x4EF7898: vm_run (vm.c:2941)
==8824==    by 0x4ED67D3: ecma_op_function_call (ecma-function-object.c:404)
==8824==    by 0x4ED6DDA: ecma_op_function_construct_simple_or_external (ecma-function-object.c:594)
==8824== 
==8824== LEAK SUMMARY:
==8824==    definitely lost: 273 bytes in 6 blocks
==8824==    indirectly lost: 32 bytes in 8 blocks
==8824==      possibly lost: 0 bytes in 0 blocks
==8824==    still reachable: 0 bytes in 0 blocks
==8824==         suppressed: 0 bytes in 0 blocks
==8824== 
==8824== For counts of detected and suppressed errors, rerun with: -v
==8824== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)

child_process: child.kill() does not take effect

child process exit|close|disconnect event is not fire, in fact the child process does not exit

websocket may receive incomplete message

websocket: check packet length when parse payload

we should check packet size is enough when parse frame payload to prevent read/write out of range.

buffer: toString consumes CPU large

When an Buffer object is 51208 Bytes, the buffer.toString() would consumes over 5s to return, consider provide an async way buffer.toString(fn) for this purpose :)

/cc @maxbbn

debug: provide the production way to collection stack trace

http2: support this protocol

util: util.format crashed when execute util.format('port', 1234)

the reason may be string.charAt out of range when invoke util.format

iotjs: unexpected crash in core

The stack is:

#0  0x0000007faca34950 in uv_is_closing () from /usr/lib/libiotjs.so
[Current thread is 1 (Thread 0x7facef3010 (LWP 15097))]
(gdb) bt
#0  0x0000007faca34950 in uv_is_closing () from /usr/lib/libiotjs.so
#1  0x0000007fac9ed750 in iotjs_handlewrap_close () from /usr/lib/libiotjs.so
#2  0x0000007faca33f78 in uv_walk () from /usr/lib/libiotjs.so
#3  0x0000007fac9ec508 in iotjs_entry () from /usr/lib/libiotjs.so
#4  0x0000007fac71a8e8 in __libc_start_main (main=0x0, argc=0, argv=0x0,
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=<optimized out>) at libc-start.c:289
#5  0x0000000000400870 in _start () at ../sysdeps/aarch64/start.S:81
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Performance Tooling Guidance

Currently only supports the following platform:

• aarch64
• armhf

Simpleperf

Download simpleperf, 32-bit version is available at here.

Then run the following commands to get report:

$ simpleperf record -g \
  -p <your process pid> \
  --duration 10 # the sample recording duration, 10 means 10 seconds

recommend set the duration to 30s.

When you gets the above commands finished, you would get the file data.perf under your working directory.

Report

Just run:

simpleperf report --dsos <your libiotjs.so> -n -g

Then you would get the following reports:

Cmdline: /data/simpleperf record -p 25021 --duration 30 -g
Arch: arm64
Event: cpu-cycles (type 0, config 0)
Samples: 408
Error Callchains: 1, 0.245098%
Event count: 132030003

Children  Self    Command  Pid    Tid    Shared Object                                    Symbol
36.82%    36.82%  iotjs    25021  25021  /usr/lib/libiotjs.so                             jmem_heap_alloc_block_internal
36.82%    0.00%   iotjs    25021  25021  /usr/lib/libiotjs.so                             jmem_heap_gc_and_alloc_block
       |
       -- jmem_heap_gc_and_alloc_block
          |
           -- jmem_heap_alloc_block_internal
19.65%    13.81%  iotjs    25021  25021  /usr/lib/libiotjs.so                             ecma_gc_run
       |
       -- ecma_gc_run
          |
          |--41.27%-- ecma_gc_run
          |
          |--12.78%-- jmem_heap_free_block
          |
          |--1.60%-- ecma_gc_get_object_next.isra.4
          |
          |--1.58%-- jmem_decompress_pointer
          |
           --1.50%-- @plt
8.88%     8.88%   iotjs    25021  25021  /usr/lib/libiotjs.so                             @plt
8.24%     2.43%   iotjs    25021  25021  /usr/lib/libiotjs.so                             ecma_gc_mark
       |
       -- ecma_gc_mark
          |
          |--22.78%-- ecma_gc_set_object_visited.isra.2
          |
          |--20.82%-- ecma_gc_mark
          |
          |--17.72%-- ecma_gc_mark_property
          |
          |--7.67%-- @plt
          |
          |--2.57%-- jmem_decompress_pointer
          |
          |--2.56%-- ecma_get_property_list
          |
           --2.56%-- ecma_get_lex_env_type
6.13%     6.13%   iotjs    25021  25021  /usr/lib/libiotjs.so                             jmem_heap_free_block
3.75%     2.21%   iotjs    25021  25021  /usr/lib/libiotjs.so                             vm_loop

support Raspberry Pi

This compiling should move on Travis CI :)

stacktrace: clean dump when current process is exit

If a program in ShadowNode is restarted so many times without restarting OS, the tmp would have too much iotjs.* dump files.

Reported by @lolBig

missing method http.abort

rt

websocket: send invalid utf8 data

send 37K utf8 message every 1s, server will receive an invalid utf8 message after a few moments

var WebSocketClient = require('websocket').client;

var client = new WebSocketClient();

client.on('connect', function (connection) {
  console.log('WebSocket Client Connected');
  connection.on('error', function (error) {
    console.log('Connection Error: ' + error.toString());
  });
  connection.on('close', function () {
    console.log('echo-protocol Connection Closed');
  });
  connection.on('message', function (message) {
    if (message.type === 'utf8') {
      console.log('Received: utf8 ' + message.utf8Data.length);
    } else {
      console.log('Received: binary ' + message.binaryData.byteLength);
      message.binaryData.toString();
    }
  });

  function sendNumber() {
    var number = Math.round(Math.random() * 0xFFFFFF);
    for (var i = 0; i < 5000; ++i) {
      number += Math.round(Math.random() * 0xFFFFFF).toString();
    }
    connection.sendUTF(number);
  }
  setInterval(() => {
    sendNumber()
  }, 1000)
});
setInterval(() => { }, 10000)
client.connect('ws://localhost:8080/', 'echo-protocol')

es2015: implement __proto__ by jerryscript

The current setPrototypeOf implementation are little wrong from standard, and the __proto__ is quite normal property to be used in Node.js modules :)

The following are example to reproduce:

var foo = {};
var bar = {};

Object.setPrototypeOf(bar, foo);
foo.test = 10;
console.log(bar.test); // expect 10 but undefined

built-in module: module `process` behaves anomaly

If change the module process definition in src/modules.json to

{
      "native_files": ["modules/iotjs_module_process.c"],
      "init": "InitProcess",
      "js_file": "js/process.js"
}

and add a file src/js/process.js with contents below

module.exports = native;

rebuild project and run iotjs with script below

require('process') === process // `false`, expecting `true`

build: add linters for JavaScript and clang

We need 2 linters for our codebase:

• ESLint for JavaScript
• clang linter?

If somebody has better option, please leave your suggestion :)

/cc @Rokid/node-js

jerry: add ecma value printer to debug easily

ecma_value_t is an encoded value of primitive value, object, error, etc. There is no direct way to see object detail when debugging.
Helper API to print ecma value should be added.

log: optimize the logging format

This task includes the following 2 parts:

• optimize the console module
• support backtrace in JerryScript

leak: memory is not released in zlib or buffer

Run ShadowNode on TravisCI

This is for common Linux build :)

jerry: support const/let

The keyword const and let are such important features in JavaScript community, let's do support them!

`console.time` and `console.timeEnd`

console.time(label: string): void and console.timeEnd(label: string): void is not supported yet it is expected to be a stable api

websocket: add packet size limit

socket will close if IncomingMessage is large

Roadmap: v0.8.0

Supported Platform

• Raspberry Pi
• RokidOS

Performance

• buffer: optimize up the serialization performance over 20%
• use shared library to cost down the process shared RSS

Development Experience:

• collect the stack (execution offset) and generates the stack trace offline
  • syntax
  • otherwise
• redesign the iotjscode to fit with ShadowNode

API

• install public headers in build script
• support 80% N-API to write a native module
• add WebSocket client implementation, keep consistent with WebSocket-Node
• add console.time() and console.timeEnd() by @legendecas

CoAP: support this protocol

tls: duplicated `close` events

TLSSocket 'close' event would trigger twice when the other end of the socket close initiatively

jerry: record function name in bytecode

JerryScript: use mmap as the allocator for snapshot

With implementation of this feature, we could store line number and function names in snapshot.

https.setTimeout would throw an uncaughtException: TypeError: Expected a function.

rt

mqtt: check broker heartbeat

http: support chunked transfer-encoding

By default, http module cannot not automatically derive content-length from input data since ClientRequest is a duplex stream, client can continuously write data into stream if the stream is not ended, yet headers have been sent at the beginning of the request.
Thus the http client in node.js set header Transfer-Encoding with chunked by default, then encase data with an chunk length at the beginning of each chunk.

url: parse crashed when the specific urlStr includes query string

var url = require('url')

url.parse('https://rokid.com/packages/driver-ssdp/latest?env=rc&sn=0302041802000468', true)

zlib: missing Convenience Methods

tls server support

runtime: Node.js & NPM compatible

ES6 features

• Promise
• TypedArray
• Arrow Function
• Template String
• const/let

Built-in Node.js modules

• http
• https
• path
• url
• crypto
  • createHash()
  • createHmac()
  • randomBytes()
• dns
• events
• process
  • umask()
  • title
• child_process
  • spawn
  • fork
  • exec
• os
• net
• tls
• file
  • fs.createReadStream
  • fs.createWriteStream
• string_decoder
• zlib
  • streams
  • sync/async convenient methods
  • constants

Modules from ecosystem

• debug
• connect
  • connect-router
  • depd
  • body-parser
• iconv-lite
• urllib
• node-fetch
  • option.compress
• mqtt
  • mqtts
• dbus
  • createService
  • getInterface
  • signal
• lowdb
• mz
• sax

http: set content-length by body

attach stack trace to error object

unhandled promise rejection shall not be ignored

An unhandled promise rejection shall be caught globally and print a warning, however it's now been ignored implicitly.

Example code:

asyncFunction()
function asyncFunction () {
  return Promise.reject(new Error('foobar'))
}

websocket: integrate with the WebSocket protocol natively

https: cannot get response

To reproduce:

https.get('https://foo.bar', (res) => {
  // dont get the callback
});

jerry: setInterval/setTimer blocks the GC

Record A at https://github.com/Rokid/ShadowNode/blob/master/src/modules/iotjs_module_tls.c#L107.

And run the following:

var https = require('https');
setInterval(() => {
  https.get('https://www.google.com', () => false);
}, 5000);

And the Record A won't get called anyway.

http/https: support request option `agent`

Options in http.request(options[, callback]) shall support option agent to control how a request behaves by default.

And it is possible to share http codebase with https.request using request agent rather than re-implementing a mostly identical one.

agent <http.Agent> | <boolean> Controls Agent behavior. Possible values:
  - undefined (default): use http.globalAgent for this host and port.
  - Agent object: explicitly use the passed in Agent.
  - false: causes a new Agent with default values to be used.

re-implement the https via the new `tls` module

The TLS module has been released, refactor the https module by TLS, then we could remove the libcurl and openssl dependencies absolutely :)

`process.hrtime([time])` do not return diffs while `time` is given

time is an optional parameter that must be the result of a previous process.hrtime() call to diff with the current time.

demo script:

var NS_PER_SEC = 1e9;
var time = process.hrtime();
// [ 16264, 282919475 ]

setTimeout(() => {
  var diff = process.hrtime(time);
  // expecting [ 1, 2476940 ]
  // actual [ 16265, 285396415 ]

  console.log(`Benchmark took ${diff[0] * NS_PER_SEC + diff[1]} nanoseconds`);
}, 1000);

net: listen on exists port not thrown error

process: process.memoryUsage did not display any useful data

always return { "rss": 0, "heapTotal": 0, "heapUsed": 0, "external": 0 }

Dbus: getInterface crashes when service not exists

当连接一个不存在的interface时，应用会崩溃退出。
应该在回调中使用error参数传递错误信息。

ShadowDebugger: wrap the IoT.js Code with electron

Use Electron to wrap https://samsung.github.io/iotjscode/ to release the first version of debugger :)