thaliproject/jxcore

Evented IO for Chakra, SpiderMonkey & V8 JavaScript

jxcore's Issues

improve scripts to automate the release process

We currently have a set of scripts to build all the components needed to do a jxcore release but is still not a fully automated environment.
We need script(s) for jxcore-cordova too.

SpiderMonkey GC crash.

Running the Thali coordinated test testLocalSeqManager.js in a loop will eventually make the SM's GC crash.
In debug mode the following assert fails:

Assertion failure: (ptrBits & 0x7) == 0, at ../deps/mozjs/incs/js/Value.h:837

Fix the jxcore-cordova test app to create a report log.

Currently the proof of concept to run jxcore tests using a jxcore-cordova app doesn't create a report log on the host machine, it only shows the results in the jxcore-cordova app running on the simulator.

We need a CI environment for JXcore

Can we use Travis for all there desktop platforms? Is there a free service we can use for Mobile for iOS and Android (since we are an open source project)?

Is 'jx install' different than 'jx npm install'?

Marek noticed that if he uses 'jx install' he doesn't have to set various environment variables with our paths and such but with 'jx npm install' he does. Did we set the paths for one but not the other?

externalized paths change not pushed to mainline jxcore

The 'externalized paths' commits in our repo haven't been pushed to mainline jxcore because we use our own URLs and in order to push to mainline jxcore we would need to change them. That, in turn, would cause conflicts when we merge back from mainline jxcore.

These are the three commits:
70b88c9
524499f
a04d16c

Need to clean up skipped tests and investigating known failing ones

In the Node test suite there are tests that were failing with the last release of JXcore from Nubisa (e.g. we didn't make them fail) as well as a bunch of tests that are being skipped. Unfortunately the CI system used by Nubisa is very old and it doesn't output a log and it doesn't know how to count skipped versus passing tests. So we need to go investigate which tests are being skipped and see if we want to actually run them and we also need to investigate the known failures to see if we want to fix them.

SpiderMonkey fails to catch a stack overflow exception.

The investigation of #22 has shown that SM in some cases doesn't properly catch exceptions.
At this stage we don't know if this is a bug in SM or if JXcore doesn't initialize the SM engine properly.
We have a workaround to catch the exception in JXcore, that should be enough to ship New Infra, but this need further investigation for V1.

fix the scripts to build the debug version of iOS and Android libs

ios_compile.sh and android_compile.sh don't accept a --debug flag as input parameter.
Adding the --debug flag in the scripts when configure is called doesn't work.

Why do I need to have a private and public key for a PSK connection?

We'll work around this for now but why if I just want to support PSK do I get an error saying "Missing PFX or certificate + private key."? I only want to support PSK connections, I don't want to support any certs.

Why is jx SM seg faulting on vNext_yarong_417_812 on OS/X?

Repo:

1. Clone and change to this branch https://github.com/thaliproject/Thali_CordovaPlugin/tree/vNext_yarong_417_812
2. Make sure you are on your mac and are using the SpiderMonkey version of JX, latest release
3. Go to thali/install and run 'jx npm run setupDesktop'
4. Go to test/www/jxcore and run 'jx runTests.js'

Expected Result: Happily running tests
Actual Result: Segmentation fault: 11

When I repeat the above but us the latest drop of JX with v8 everything works fine.

What's going on?

JXcore tests do not properly validate presence of right version of OpenSSL on host

Taken from @enricogior's explanation:

Found a problem with the logic used by some tests to verify if the system's OpenSSL version is 1.0.0 or greater. Some tests were skipped erroneously, the fix is simple but it takes time and we have more urgent issues.

The code that doesn't work is like this one:
https://github.com/thaliproject/jxcore/blob/master/test/simple/test-tls-psk-client.js

function checkOpenSSL() {
  exec('openssl version', function(err, data) {;
    if (err) throw err;
    if (/OpenSSL 0\./.test(data)) {
      console.error('Skipping due to old OpenSSL version.');
      return false;
    }
    else {
      return true;
    }
  });
}

if (! checkOpenSSL()){
  console.error("Skipping because OpenSSL version < 1.0.0.");
  process.exit(0)
}

if (! checkOpenSSL()) always succeeds, since checkOpenSSL() is not returning the result of the 'exec' command.
That test fails now that we run it, so beside the fix to verify the OpenSSL version we need to investigate why it fails.

Make sure that Thali's JXcore and mainline JXcore are the same

Did we get all of our changes accepted to JXcore?

Create a test app to run jxcore tests on iOS.

A proof of concept has been tested on the iOS simulator.

Test fail on macOS when run with test.js

test/simple/test-stream-pipe-multi.js fails on macOS when run using the test/test.js script.
It passes when executed directly jx test/simple/test-stream-pipe-multi.js.

Fix failing OpenSSL pummel tests.

On macOS and Linux these two OpenSSL-related tests are failing:
test/pummel/test-https-ci-reneg-attack.js
test/pummel/test-tls-ci-reneg-attack.js

They fail in the current master branch and also in the private branch where we are testing OpenSSL 1.0.1t.

Regenerate our JXcore version of NPM to not put Shawn's path into the npmrc

When we generate npmjx311 (this lives in jxcore\tools\npmjx) we actually have the path /Users/cicoriasmbp13/ all over the place. This means that folks who use JXcore NPM have to create that literal path in their system. If nothing else that is a possible attack on Windows since it requires creating a directory under Users which is just a bad idea.

I believe this is all being set in deps/npm/lib/config/defaults.js as a result of the value of setting the home variable.

Now the obvious question is - o.k. so what path should it be set too? It's pretty obvious that it wants to be set to the user's local home directory. I'm not sure why this isn't being set automatically.

I'm probably sending you on a wild goose chase but this smells like a bug in https://github.com/thaliproject/jxcore/blob/master/tools/npmjx/index.js#L156, that function should figure out where the user's local directory is and update the npmrc to change the paths.

Although the more I think about it the more likely I think it is that putting npmrc into the tar.gz is the actual bug. So maybe we just change the packager logic so that file isn't included?

Do we have a dependency on some random depo for JXC?

I noticed in the logs:

[email protected] (git+https://github.com/ktrzeciaknubisa/jxtools.git#644cf31ea259cb65a97e5c3ed5ea1236dba298a3)

So do we depend in JXC on some random depot in GitHub?

Does JXcore leak memory?

jxcore#915

We need an assessment of the jxcore status.

disable SSLv3 and TLSv1.0

The work consits of:

• disabling SSLv3
• disabling TLSv1.0
• add test to verify SSLv3 is disabled
• add tests to verify TSLv3 is disabled
• modify/disable tests that currently use SSLv3
• modify/disable tests that currently use TLSv3

Fix AppVeyor Build related to custom JxCore binaries

process.sendToMain() bug

The following script shows the bug in the process.sendToMain() method.

jxcore.tasks.setThreadCount(2);

var count = 0;

function runTest() {
  // make sure there are no tasks still running
  jxcore.tasks.killThread(0)
  jxcore.tasks.killThread(1)

  count++;
  if (count > 5) return;

  console.log("---- Count " + count + " -----");

  var subTask = function (param) {
      console.log("sending msg to main");
      process.sendToMain({result: "msg " + param.count});
      return 'ok';
  };

  jxcore.tasks.on("message", function(threadId, obj) {
        console.log("thread " + threadId + ": " + obj.result);
      });

  jxcore.tasks.addTask(subTask, {count: count}, function (err, result) {
        console.log("subTask done");
        runTest();
      });
}

runTest();

The espected result should be:

---- Count 1 -----
sending msg to main
thread 0: msg 1
subTask done
---- Count 2 -----
sending msg to main
thread 0: msg 2
subTask done
---- Count 3 -----
sending msg to main
thread 0: msg 3
subTask done
---- Count 4 -----
sending msg to main
thread 0: msg 4
subTask done

Instead the result is:

---- Count 1 -----
sending msg to main
thread 0: msg 1
subTask done
---- Count 2 -----
sending msg to main
thread 0: msg 2
thread 0: msg 2   <---- unexpected
subTask done
---- Count 3 -----
sending msg to main
thread 0: msg 3
thread 0: msg 3   <---- unexpected
thread 0: msg 3   <---- unexpected
subTask done
---- Count 4 -----
sending msg to main
thread 0: msg 4
thread 0: msg 4   <---- unexpected
thread 0: msg 4   <---- unexpected
thread 0: msg 4   <---- unexpected
subTask done

Create 3.1.4 jxcore and jxcore-cordova release.

Build and upload the jxcore and jxcore-cordova bits.

OSX undefined char16_t

1. Download install script.
2. rm -r ~/.jx && rm -r ~/.npm && rm -r ~/.node-gyp && etc
3. ./jxcore-install.sh --force sm
4. jx install
5. cd temp
6. jx npm install fs-ext

Target arch x64
 CXX(target) Debug/obj.target/fs-ext/fs-ext.o
In file included from ../fs-ext.cc:20:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/node.h:46:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/jx/Proxy/JSEngine.h:51:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/jx/Proxy/Mozilla_340/JXString.h:19:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/jx/Proxy/Mozilla_340/MozJS/MozJS.h:6:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/jx/Proxy/Mozilla_340/MozJS/Isolate.h:17:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/jsapi.h:12:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/mozilla/FloatingPoint.h:12:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/mozilla/Assertions.h:16:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/mozilla/Attributes.h:12:
In file included from /Users/hiqouser/.node-gyp/jxb311/src/mozilla/Compiler.h:12:
/Users/hiqouser/.node-gyp/jxb311/src/mozilla/Char16.h:173:15: error: unknown type name 'char16_t'
typedef const char16_t* char16ptr_t;

I am not familiar with osx. It looks like it is using custom apple clang version. I am not sure but it looks like this clang doesn't want to support char16_t

Android Nougat: bind to a file pipe fails.

11 tests fail on Android Nougat because binding to a file pipe doesn't work.
The code that handles the pipe binding is in libuv:
https://github.com/jxcore/jxcore/blob/166784821f30e62503f04eb267ea26cf7c2c58d7/deps/uv/src/win/pipe.c#L363

Travis should use latest XCode version

This is a sub task of #16

I've tried to update trevis.xml to use the latest XCode version previously, but it was failing because the log size exceeded the max length, the solution maybe to redirect the stderr and/or the stdout to a log file. I need to verify if we can later access the log file for investigation if needed.

Test failing on Linux randomly with TIMEOUT error

The test/simple/test-child-process-fork-net2.js test failed on Travis during a build, we also observed it on some Linux VMs, it occurs randomly.

Should we even build anything for v8?

We run Spidermonkey on both iOS and Android. So does it make any sense for us to build with v8 for any of the desktop builds? Isn't it just going to hide bugs? Maybe we shouldn't build for v8 at all so no one accidentally installs it?

Externalize the paths

Right now we hardcode in paths pointing to our distribution points during the build. That would be what is known as 'bad'. :) We need to submit a PR to JXcore that lets us set the paths in some kind of external configuration file so that different people who download JXcore can choose to point at whatever location they want. I found at least three files in JXcore forexample that point directly at azureedge.net. But that is an endpoint owned by Microsoft. It's not something that should be in the general JXcore repo. Instead those values should point at configuration files that someone building JXcore can set up and then do a compile and produce an output that points where ever they want.

By externalizing this we can keep the JXcore build agonist from Microsoft which is what we want. We are absolutely NOT promising the world to host JXcore! :)

If we had a bunch of money we would make JXcore handle native optional dependencies it can't build better

Right now if we ask JXcore (especially when we are using spidermonkey on desktop) to build an optional dependency it can't handle (think bufferutil that is brought in by socket.io in TestServer) then it fails the whole build. In perfect land it would detect that the node-gyp build failed and it would see that the dependency is optional and realize that the failure really doesn't matter. So it would clean up any entries in node_modules and continue.

Android Nougat: test/simple/test-fs-symlink.js fails

Error: EACCES, link '/data/local/tmp/jxcore/test/fixtures/cycles/root.js' (node.js 138:7)

hTe symbolic link is created correctly and it works if tested in the shell.

We got 'sigfault' on some android devices

Please, see the log from samsung-SM-A300FU and search for Fatal signal 11.

You can also check the logs from some other devices that failed with Fatal signal 11:
samsung-SM-A500FU
samsung-SM-G935F

Android Nougat: test/simple/test-child-process-stdio-inherit.js fails

AssertionError: "" == "asdfasdf" ( 0:0)
    at equal (assert.js:136:11)
    at grandparent/< (/data/local/tmp/jxcore/test/simple/test-child-process-stdio-inherit.js:30:5)
    at emit (events.js

child.stdout.on('data'...) doesn't get called because the child process dies before the event is emitted or captured.

[Design Change Request] npmjx mechanism

How npmjx works is not documented.
There are actually two npmjx modules, one is just a proxy module that points to the real npmjx module.
When you run the "jx npm install" command, jx.exe executes the lib/jx/_jx_install.js script (that is embedded in jx.exe) that downloads the proxy module using a hardcoded url.
The proxy module in turn downloads the real npmjx module (using another hardcoded url). The proxy module contains the tar.gz module that is required to expand the real npmjx module.

There are some problems with the current way this mechanism work:

• code duplication: lib/jx/_jx_install.js and tools/npmjx/index.js do the same thing, they download modules from the cloud.
• both scripts contain hardcoded urls (this problem is already tracked by #8) .
• nothing guarantees that the two modules don't get out of sync pointing to old/wrong urls.
• the current naming and location is confusing: the proxy module lives in tool/npmjx, it should not be named npmjx and since it's not a tool, it should be in the deps/ folder.
• the proxy module seems to be required only in order to download the tar.gz module, therefore why not change the proxy module to just contain the tar.gz code and let the _jx_install.js script do all the heavy lifting of downloading the tar.gz module and the npmjx module.

Find an alternative to Travis for macOS build.

We tried to use Travis with the latest XCode but it didn't work.
Let's find an alternative service and if none will fit our need let's try to setup Jenkins on our own machine.

Referencing an outside variable from a closure in spawn.stderr.on causes jxcore to fail

function childProcessExecPromise(commandString, currentWorkingDirectory) {
  return new Promise(function (resolve, reject) {
    var commandSplit = commandString.split(' ');
    var command = commandSplit.shift();
    var stdErrData = false;
    var theProcess = spawn(command, commandSplit,
                            { cwd: currentWorkingDirectory});
    theProcess.stdout.on('data', function (data) {
      console.log('' + data);
    });
    theProcess.stderr.on('data', function (data) {
      //stdErrData = true;
      console.log('' + data);
    });
    theProcess.on('close', function (code) {
      if (code !== 0) {
        return reject(code);
      }
      if (stdErrData) {
        return reject(-1);
      }
      return resolve();
    });
  });

If I uncomment stdErrData = true; then JXcore faills.

test.js script returns different results compared to test.py

The test/test.js script's main purpose is to run tests on platform that don't have python, such iOS and Android.
On desktop it should also work and return the same results returned by running the tests using the python script tools/test.py, but it currently doesn't.
Several tests fails on macOS when run using test.js.

test-regress-GH-4015 regression on Ubuntu using SpiderMonkey

Test test-regress-GH-4015.js fails on Ubuntu 16.04 using SpiderMonkey.

The test forces a stack overflow and then checks the error signal.
The test passes on Travis running Ubuntu 12.04 and it fails on our dev machines running Ubuntu 16.04.
In our environment the err.signal is set to 'null' instead of the expected 'SIGSEGV'.

=== release test-regress-GH-4015.js ===
Path: _auto_simple_single/test-regress-GH-4015.js
AssertionError: false == true ( 0:0)
    at ok (assert.js:126:5)
    at /home/enrico/github/thaliproject/jxcore/test/_auto_simple_single/test-regress-GH-4015.js:21:5
    at exithandler (child_process.js:

Android Nougat: test/simple/test-child-process-fork-exec-argv.js fails

SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data (/data/local/tmp/jxcore/test/simple/test-child-process-fork-exec-argv.js 28:21)
    at  (/data/local/tmp/jxcore/test/simple/test-child-process-fork-exec-argv.js:28:22)
    at emit (events.js:85:1)
    at $0h/this._handle.onexit (child_process.js:826:1)

The fork process doesn't start, maybe because the child process dies first.

Android Nougat: test/simple/test-timer-match.js is flaky.

test/simple/test-timer-match.js fails randomly when verifying the first timer:

if (time == last_time) {
    throw new Error("Fail!");
  }

Unallocated pointer freed error

We are getting the following error during tests:
jx(1068,0x105751000) malloc: *** error for object 0xa6e00786a2e65: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

To repro:

1. git clone https://github.com/thaliproject/Thali_CordovaPlugin.git
2. git checkout malloc_error
3. cd thali/install/
4. jx npm run setupDesktop
5. cd ../../test/www/jxcore
6. jx npm test

It will core dump with the error above pretty quickly.

some OpenSSL tests are disabled on Windows

Some tests assume that openssl is not installed on Windows and return true (success).
We should change the current behavior and adopt the same behavior of other tests that first check if openssl is installed.

Adding a new method to Array.prototype causes jxcore to fail.

Adding a new method to the system Array type causes unpredictable errors in jxcore.
The error is not occuring immediately and the error message that may appear doesn't provide any indication that the problem is caused by adding a new method to the Array type.
Other system types may also be effected by the same issue.

Figure out what it would take to run tests on a non-jail broken iphone

Investigate failing tests

Investigate the failing tests based on platforms priority and open specific issues based on the investigation results.

modify ios and andoid build scripts to set -j flag as input parameter

Currently ios_compile.sh and android_compile.sh have the hardcoded -j 2 flag, it would be better to pass the flag value as input parameter to the script in order to utilize the full CPU power.

Regression in test.js

A regression was introduced with the latest changes.

Build fails on Travis because of test-tls-psk-client.js

Travis uses Ubuntu 12.04, we tested on macOS and Linux 16.04, so this is something we didn't encounter and we will have to investigate. It may be caused by an old version of OpenSSL on the Travis machine. The test failed should be able to detect it and skip.

=== release test-tls-psk-client.js ===
Path: simple/test-tls-psk-client.js
exists
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT
client connected
secure error: Error: socket hang up
ERROR
47323619255072:error:140F2044:SSL routines:SSL3_SEND_SERVER_HELLO:internal error:s3_srvr.c:1495:
client close
killed
WAIT-ACCEPT
AssertionError: 0 == -1 ( 0:0)
    at equal (assert.js:136:27)
    at /home/travis/build/thaliproject/jxcore/test/simple/test-tls-psk-client.js:212:5
    at emit (events.js
Command: out/Release/jx /home/travis/build/thaliproject/jxcore/test/simple/test-tls-psk-client.js
[02:58|% 100|+ 624|-   1|-  16]: Done
make: *** [test-simple] Error 1

Fix 'jx npm run test-meta' in vNext_yarong_417_812

We have a timer failure. I don't know why and I ran out of time today to investigate. Since this blocks everything please sync to the latest vNext_yarong_417_812, investigate and repair the issue.