nodejs/citgm

Canary in the Gold Mine

citgm's Issues

eslint v2.2.0 broken

Master is working. This issue can be closed when we revert eslint installing from master

#5415 is tracking this over at eslint/eslint

SPDY broken on master

The entire test suite for SPDY is broken on master

verbose:                     | 1) SPDY Client regular h2 ssl mode should send GET
verbose:                     | request:
verbose:                     | Error: timeout of 2000ms exceeded. Ensure the
verbose:                     | done() callback is being called in this test.
verbose: npm-test:           | 2) SPDY Client regular h2 ssl mode "after each"
verbose:                     | hook:
verbose:                     | Uncaught Error: socket hang up
verbose:                     | at TLSSocket.onclose
verbose:                     | (node_modules/spdy-transport/lib/spdy-transport/c…
verbose:                     | onnection.js:186:15)
verbose:                     | at TCP._onclose (net.js:471:12)
verbose:                     |
verbose:                     | 3) SPDY Client regular h2 plain mode should send
verbose:                     | GET request:
verbose:                     | TypeError: Cannot read property 'request' of
verbose:                     | undefined
verbose:                     | at Context.<anonymous> (test/client-test.js:77:26)
verbose:                     |
verbose:                     | 4) SPDY Client regular h2 plain mode should send
verbose:                     | POST request:
verbose:                     | TypeError: Cannot read property 'request' of
verbose:                     | undefined
verbose:                     | at Context.<anonymous> (test/client-test.js:95:26)
verbose:                     |
verbose:                     | 5) SPDY Client regular h2 ssl mode "before each"
verbose:                     | hook for "should receive PUSH_PROMISE":
verbose:                     | TypeError: Cannot read property 'call' of
verbose:                     | undefined
verbose:                     |
verbose:                     |
verbose:                     | 6) SPDY Client regular h2 plain mode should
verbose:                     | receive PUSH_PROMISE:
verbose:                     | TypeError: Cannot read property 'request' of
verbose:                     | undefined
verbose:                     | at Context.<anonymous>
verbose:                     | (test/client-test.js:109:26)
verbose:                     |
verbose:                     | 7)  Uncaught error outside test suite:
verbose:                     | Uncaught TypeError: Cannot read property
verbose:                     | 'currentRetry' of undefined

`npm smoketest`

I'm considering the possibility of adding a support for a npm smoketest convention. Essentially, if module developers want to include a smoketest script in their package.json as an alternative to npm test, then citgm would choose to run that as an alternative. Doing so would allow module developers to create subsets of the tests that make the most sense for smoke testing, or perform additional set up that may be necessary independently of the normal test suite.

Use citgm to test request results in -Error |failure |The canary is dead.

[nodetest@test vmTest]$ citgm -lv request@latest

http over https, tunnel=true

not ok 4 should be equivalent
operator: deepEqual
expected: [ 'https connect to localhost:6767', 'http response', '200 ht tp ok' ]
actual: [ 'err tunneling socket could not be established, cause=certifi cate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...

http over https, tunnel=false

not ok 5 should be equivalent
operator: deepEqual
expected: [ 'https proxy to http', 'http response', '200 http ok' ]
actual: [ 'err certificate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...

http over https, tunnel=default

not ok 6 should be equivalent
operator: deepEqual
expected: [ 'https proxy to http', 'http response', '200 http ok' ]
actual: [ 'err certificate has expired' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...

https over http, tunnel=true

not ok 7 should be equivalent
operator: deepEqual
expected: [ 'http connect to localhost:16167', 'https response', '200 h ttps ok' ]
actual: [ 'http connect to localhost:16167', 'err certificate has expir ed' ]
at: Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
...

https over http, tunnel=false

/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/node_modules/tape/index.js:75
throw err
^
Error: certificate has expired
at Error (native)
at TLSSocket. (_tls_wrap.js:929:36)
at TLSSocket.emit (events.js:104:17)
at TLSSocket._finishInit (_tls_wrap.js:460:8)

Assert: "should be equivalent",
found: [ 'err tunneling socket could not be established, cause=certific ate has expired' ]
wanted: [ 'https connect to localhost:6767', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'err certificate has expired' ]
wanted: [ 'https proxy to http', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'err certificate has expired' ]
wanted: [ 'https proxy to http', 'http response', '200 http ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)
Assert: "should be equivalent",
found: [ 'http connect to localhost:16167', 'err certificate has expire d' ]
wanted: [ 'http connect to localhost:16167', 'https response', '200 http s ok' ]
Request._callback (/tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/tes ts/test-tunnel.js:134:9)

ok test-unix.js ..................... 3/3
total ......................... 1326/1331

not ok

npm ERR! Linux 2.6.32-431.17.1.el6.s390x
npm ERR! argv "/usr/nodetest/T32_12/bin/node" "/usr/nodetest/T32_12/bin/ npm" "run" "test-ci"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! code ELIFECYCLE
npm ERR! [email protected] test-ci: taper tests/test-*.js
npm ERR! Exit status 5
npm ERR!
npm ERR! Failed at the [email protected] test-ci script 'taper tests/test-.js'.
npm ERR! This is most likely a problem with the request package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! taper tests/test-.js
npm ERR! You can get their info via:
npm ERR! npm owner ls request
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /tmp/9eab4220-cc8f-43ca-bb1b-71b791c63a11/request/npm-debug.log
npm ERR! Test failed. See above for more details.
Error |failure |The canary is dead.
Info |rm.tempdir |/tmp/9eab4220-cc8f-43ca-bb1b-71
Info |done |https://github.com/request/requ
| |.0.tar.gz

set npm progress = false

npm/npm#11283 (comment)

We should see a 20% speed up in npm install using this change.

CI: https://ci.nodejs.org/job/thealphanerd-smoker/56/

refactor for consistent output

one of the current shortcomings of the tool is that it takes whatever the modules npm tests generates and just pipes those to stdout, which means in order for us to know if there are any failures, we have to go digging. Which is time consuming. The tool needs to be refactored to generate consistent usable output. /cc @thealphanerd

Allow testing subsystems

Currently we have a single large lookup table.

If we split the look up into subsystems (as we do our commits) we could use a flag to specify only testing a subset of the packages.

checking dependencies recursively

A question was raised about the possibility of having citgm recursively check the dependencies of a module as well as the module itself. We are grabbing the package.json so doing so would be possible but it could potentially be a significant grind to get through everything in one go. Not convinced it's something we should do but wanted to solicit opinions.

@nodejs/smoke-test .. what do you think?

browserify failing

Looks like v13.0.0 was published 2 days ago. It was tagged with a proceeding v. Past released have had no proceeding character.

Rather than updating the lookup I have modified citgm-all to consider modules the don't fetch from npm as flaky, so our CI runs will at least pass.

I've followed up with the person who did the release and I'm going to wait to see what happens before I make changes to the lookup table.

Call stack size exceeded in CI

https://ci.nodejs.org/job/thealphanerd-smoker/nodes=ubuntu1504-64/11/console

TAP output for citgm-all

I'm currently using citgm-all with a lookup.json file as part of Jenkins CI, and it'd be useful to have citgm-all output a TAP file in the same way node tools/test.py does with its --progress=tap option. That way, with the Jenkins TAP plugin it'd be really easy to redirect the citgm-all output to a .tap file for a range of tests ( avoiding having to dig through the console log on every failure).

The individual npm modules often have their own TAP outputs for their tests, but an overall TAP file with one TAP result for each module would be useful.

I made a draft of what I imagined the tap output would look like, let me know if it seems feasible. I'm not sure how much work it would take to add this.

1..2
ok 1 express
  ---
  duration_ms: 100.137
  ...
not ok 2 lodash
   # not ok 20 lodash-test-connection
   # ECONNRESET
   # 
  ---
  duration_ms: 60.88  
  ...

Feedback/criticism welcome!

Dropping support for centos in CI due to many failures

many modules were failing in centOS

https://ci.nodejs.org/job/thealphanerd-smoker/14/

Turning off support for now, to be investigated

torrent-stream failing due to npm weirdness

npm is not resolving all the dependencies causing the module to fail

issue on torrent-stream tracker
issue on npm tracker

bluebird test suite blows up in child_process

Turns out that process.stdout in a child_process is created using net, whereas in node proper it is made using tty. If a call to any of the methods tty exposes that net does not have, everything will explode

I have submitted a PR to fix this nodejs/node#5061

ppcbe fedora20 hangs on ```npm install -g citgm```

Not sure exactly what is happening, but we can use this issue to track failures due to npm install hanging.

Deprecate dockerify

Currently it does not look like we are going to be using citgm-dockerify in production.

The binary does not currently have tests.

I'd like to suggest deprecating the binary for now so we can focus on core requirements

/cc @jasnell

Evaluate Node Checker

http://nodechecker.com/

Lodash v4.0.0 fails

perhaps @jdalton can chime in

verbose:                     |
verbose:                     | Error: Cannot find module '../lib/fp/convert.js'
verbose:                     | at Function.Module._resolveFilename
verbose:                     | (module.js:325:15)
verbose:                     | at Function.Module._load (module.js:276:25)
verbose:                     | at Module.require (module.js:353:17)
verbose:                     | at require (internal/module.js:12:17)
verbose:                     | at Object.<anonymous>
verbose:                     | (/private/var/folders/ty/q7q6b07j5r3c7nvnpzm6hkq4…
verbose:                     | 0000gn/T/1287562e-5561-4d58-8c21-eb82a940002f/lod…
verbose:                     | ash/test/test-fp.js:47:28)
verbose:                     | at Object.<anonymous>
verbose:                     | (/private/var/folders/ty/q7q6b07j5r3c7nvnpzm6hkq4…
verbose:                     | 0000gn/T/1287562e-5561-4d58-8c21-eb82a940002f/lod…
verbose:                     | ash/test/test-fp.js:931:3)
verbose:                     | at Module._compile (module.js:397:26)
verbose:                     | at Object.Module._extensions..js
verbose:                     | (module.js:404:10)
verbose:                     | at Module.load (module.js:343:32)
verbose:                     | at Function.Module._load (module.js:300:12)

mkdirp

v5 and up throw an error.

throw new Error('Unsupported Node version: ' + nodeVersion);

seems to be coming from mock-fs

CI integration

The tool needs to be integrated into the CI environment so that fully automated runs are possible. The current design of the tool does not make that easy. /cc @thealphanerd

VM Automation

citgm-dockerify is great for testing on linux, not so great for automating against other platforms.
It would be ideal to have the ability to automate launching VM images, pushing a node install, then running a (or multiple) citgm test on the launched image. Using something like https://www.npmjs.com/package/pkgcloud and cloud-init ought to do the trick. Just need to play around with it to get it working.

Gyp fails when running on prerelease version

"pre" versions of node cannot be installed

What is the best practice to avoid this?

/cc @jasnell @rvagg

Request Being flakey

Across different systems and versions we are having inconsistent results testing request

on centos 5 / 6

verbose: npm-test:           | not ok test-localAddress.js ......... 5/5

on ubuntu 1404

verbose:                     | Assert: "Test file timed out:                     
verbose:                     | test-localAddress.js",                            
verbose:                     | found: true                                       
verbose:                     | wanted: false                                     
verbose:                     | test-localAddress.js:0  

On OSX 10.10 there seems to be some sort of rate at which it randomly fails during the phantomJS stages

Add Caching layer to improve performance on multiple runs

One of the largest bottlenecks right now is downloading tar balls and running npm install.

If we could smartly cache we could likely get away with only calling npm rebuild

This could likely be accomplished naively by not cleaning up the temp folders, but there might be a more elegant way.

npm engines support

One additional check we may need to perform is a quick check against engines in the package.json to make sure that the version of the module being tested will be compatible with the version of node used. Leaving this here as a remind to get that added in.

Add modules that test debugger and instrumentation

Currently we are not testing this... probably a good idea to do so

examples in readme don't work

currently the examples in the readme include -v but do not include a loglevel, and as such they fail 😢

• should there be a sane default?
• should citg inform individuals when specific options are used incorrectly?
  • is this a feature offered by commander?

When a package does not exist the current failure message is not descriptive enough

the current error message is

error:   failure             | No project downloaded

It would be nice to have more information, like perhaps that the package doesn't exist on npm, or the link provided returned a 404, etc...

Stylus broken on master

verbose: npm-test:           | 1 failing
verbose: npm-test:           | 1) sourcemap basic inline:
verbose:                     | TypeError: test/sourcemap/basic.inline.styl:10:17
verbose:                     | 6|   width: 500px
verbose:                     | 7|   margin: 0 auto
verbose:                     | 8|
verbose:                     | 9|   h2
verbose:                     | 10|     color: green
verbose:                     | -----------------------^

Will dig in more soon

Todo Items

• Improve Reporting (may factor to use a test runner like mocha) this is going to be difficult until we have more consistency among module test output
• If the npm module does not include tests, can we detect that and automatically "fail-over" to github
• Make sure we're running properly on Windows
• Make sure we're running properly on Linux
• Can we simplify choices for known modules (e.g. lodash)
• Verify that the -k|--hmac option is working correctly

current react release does not support v6 engine

React has been moved to flakey due to this. The engine field has been bumped in master. Once a new version is cut react can be moved from flakey and this issue closed

ESLint broken on master

1) CLIEngine executeOnText() should report one
verbose:                     | message when using local cwd .eslintrc:
verbose:                     |      TypeError: Path must be a string. Received

Looks related to #80

moment requires grunt-cli globally installed

PR sent to fix this

moment/moment#2830

Can citgm provide more options to ignore things?

Some node-modules only support 64 bit node.js and not 32 bit node.js
Some do not support certain platforms, is there a way to skip running tests based on different conditions?
arch, os, platforms,.. node.js bits.?

Improve how Flaky modules are handled

Currently all modules that have a problem have been put in the blanket category of flaky allowing it to fail on all versions and platforms. It would be nice to be able to be more granular about flakiness.

Reasons a module may be flaky (thanks @mhdawson for coming up with this list)

• platform specific failures
• broken for all versions
• broken on specific versions
• flaky in that they pass most of the time but intermittently fail

Do people have ideas for other potential cases of flakiness?

socket.io failing in CI

  1 failing

  1) socket.io socket should handle very large binary data:
     Error: timeout of 2000ms exceeded

CITGM - eslint test only failure on PPC

The CITGM testing shows a PPC specific failure for the npm tests on eslint. This is not a problem running the module (ie eslint itself does not have a problem) but because of dependencies in the test.

https://ci.nodejs.org/job/thealphanerd-smoker/21/nodes=ppcle-ubuntu1404/console

Myles and I have identified that the issue is a missing binary in the phantomjs module so we'll see what we can do to have that added. Looks like phantomjs chooses its platforms based at least partially on what Node needs so hopefully they will be receptive to adding the PPC binaries now that Node builds for those platoforms.

jade repo is incorrect in published version

patched in pugjs/pug#2238 (comment)

waiting for it to be released

Add option to format output in markdown for github

Would be nice to have a flag that would print formatted output ready to be pasted in a github issue

Add hapi support

Moving discussion from https://github.com/rvagg/iojs-smoke-tests/issues/1

I'm +1 on adding Hapi given its heavy test burden and enterprise popularity. Does it need anything more than an entry on the README?

Better output when tests fail

Originally mentioned in #5

Currently if tests fail we only get the message The canary has died

it would be nice to have more info without having to rerun the suite. Not 100% this is easily doable but we can use this issue to track progress

Feedback from doing a release smoke test

When I did the io.js 2.5.0 release, I used CITGM to do some smoke testing. The experience was generally positive. Some complaints / things that would be nice to have:

1. Some way to detect/filter/message about modules that try to download gyp headers, at least during the pre-release smoke test.
2. Some type of consistent output from the modules. This might be a pipe dream, and would probably require working with module authors. But it would be nice to just see X out of Y tests passed. It's also not obvious what breaks because of the platform and what is just broken in the module. I treated it as "oh there are a few failures, but the module still works."
3. A pre-release suite that runs a carefully selected subset of modules with some expected output might be a start.

Deal with flakey modules currently in lookup

There are currently a number of modules in lookup.json that are flakey for a number of reasons.

Does not run nicely in child process

• bluebird
• yeoman-generator

Needs color output on stdout

• chalk
• colors
• yosay

published version broken, fixed on master

• jquery
• express

requires special setup or env

• handlebars
• mongodb
• redis
• mime
• mongodb

What would we like to do with these modules?

We can easily implement a "flakey" check for modules that we know are broken but can be fixed

Should we do research on getting modules that require color output or do not run nicely in child processes?

Should we drop support for modules that require specific environments or setup outside of npm install?

Enable custom npm debug level

This might be useful when chasing down really odd bugs on specific systems.

body-praser broken on node.js master

It is fine on v5

  176 passing (494ms)
  1 failing

  1) bodyParser.urlencoded() with parameterLimit option with extended: false should work with Infinity limit:

      AssertionError: 0 == 10000
      + expected - actual

      -0
      +10000

      at test/urlencoded.js:666:12
      at Test._assertFunction (node_modules/supertest/lib/test.js:247:11)
      at Test.assert (node_modules/supertest/lib/test.js:148:18)
      at Server.assert (node_modules/supertest/lib/test.js:127:12)
      at emitCloseNT (net.js:1525:8)

chalk cannot be tested as expected

Chalk is failing within citgm. I think the styling is currently being stripped.

Allow module flakyness changes without code review

Modules may become flaky / unflaky at a very fast rate. Would people be open to changes to module flakyness to be merged without code review? (As long as it is documented)

argument to skip specific module in citgm-all

it may be easier to add known failures to the arguments rather than constantly modifying the lookup

Allow alternative npm test command

In some node-modules like request,

The npm test runs different tests and we only want to run test-ci .
How will we do it via citgm?
Thanks

"scripts": {
"test": "npm run lint && npm run test-ci && npm run test-browser",
"test-ci": "taper tests/test-.js",
"test-cov": "istanbul cover tape tests/test-.js",
"test-browser": "node tests/browser/start.js",
"lint": "eslint lib/ *.js tests/ && echo Lint passed."
},

Jenkins setup

I've set up a Jenkins job to start playing around with this: https://jenkins-iojs.nodesource.com/job/smoker, currently only hooked up to a fast Linux slave while I test this out.

Successful run here: https://jenkins-iojs.nodesource.com/job/smoker/6/nodes=debian8-64/console (well, "successful" in the "it runs" sense, I'm only running 3 modules and hapi is failing 4 tests).

Jenkins config has the following for Linux after cloning the repo you give it (default nodejs/node) at the commit provided:

First compile and install Node:

./configure --prefix=$(pwd)/smoker
make -j $(getconf _NPROCESSORS_ONLN)
make install

then set up citgm:

export PATH=$(pwd)/smoker/bin:$PATH
cd smoker
node -v
npm -v
npm install --prefix=$(pwd) --global --loglevel=error citgm

then tests

export PATH=$(pwd)/smoker/bin:$PATH
export npm_config_nodedir=$(pwd)

cd smoker

citgm --nodedir=$npm_config_nodedir --lookup --verbose through2@latest
citgm --nodedir=$npm_config_nodedir --lookup --verbose bignum@latest
citgm --nodedir=$npm_config_nodedir --lookup --verbose hapi@latest

Some notes:

1. Can't run without --verbose, is this intended? Should we have a less verbose option?
2. --nodedir doesn't pass down properly if the module isn't set up quite right, e.g. https://github.com/justmoon/node-bignum/blob/5f668a1c44324fd2ec074bafb1bd215883d8e856/package.json#L37 (although I can fix that one by removing that line, but it's a useful case), hence the use of export npm_config_nodedir=$(pwd) to achieve the same thing.
3. I noticed when I some combination of commandline options wrong and it borked that it exited with code 0 indicating a pass, it's probably worth figuring out that path in bin/citgm (it's not immediately apparent to me just glancing through it).

Is this roughly how you envisioned it being used @jasnell, perhaps you can give some feedback?