Node/install times out causing build fail

[circleci-public]

Node/install times out causing build fail

[gmemstr]

I wonder if it's worth switching to https://github.com/nvm-sh/nvm over our own install script.

[KyleTryon]

@halilb Transient network errors are to be somewhat expected. While we offer the ability to install node through this orb, it would not be considered best practice to rely on downloading and installing node into your CI process for each run.

For that we would highly suggest creating a custom docker image if none fits your exact needs.

https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic/

You can start with either our ruby or node image and install the other. We will be offering -node variants of all of our new images in the near future but if you are looking for a specific version combination of Node and Ruby it would be best to build an image based on either of these above.

The reason to base your custom docker image of these (and why to use a docker image rather than install) is because the layers of the docker images will be highly cached on the CircleCI service meaning your spin up time will continue to be fast. There will be no chance of issues downloading or installing node on each run as it would be backed into the image, and it would be faster to download any uncached docker layers than download and install Node.

If the newest version of Node is acceptable, the node variant of the Ruby image will be available shortly.

[lfalkner]

@KyleTryon this is problematic for users who are using MacOS images though (and thus for react-native developers, since react-native does not support the latest version of node either).

[gmemstr]

v4 of this orb moves to the nvm tool - we've opted for a major tag due to some of the parameters changing to accomadate nvm arguments.

[iynere]

@Gowiem hi there, I ran into this issue many times when developing the orb—i do believe it tends to be unavoidable. in the instances where the command hangs for so long, more time doesn't seem to help—a rerun is typically required.

we can revisit the release key import code in the near future, however—perhaps Node has added/removed some keys or keyservers & we can update our logic accordingly. thanks!

[Gowiem]

@iynere Got it and understandable -- Thanks for the info.

[iynere]

@Gowiem apologies, i misspoke here:

we can revisit the release key import code in the near future, however—perhaps Node has added/removed some keys or keyservers & we can update our logic accordingly. thanks!

the release keys are actually pulled anew on each run of the orb's install command:

https://github.com/CircleCI-Public/node-orb/blob/master/src/commands/install-node.yml#L159-L165

so that's not the issue.

next time you run into this problem, could you provide any additional info around particular keys or keyservers that seem to be hanging? that would help us debug further

[Gowiem]

@iynere Be happy to provide that info -- How can I find that out from the CircleCI failure logs? Is that information outputted through some debug logging that I can turn on?

[iynere]

@Gowiem ah yes, sorry—add a debug: true parameter to the orb command !

[Gowiem]

@iynere I've got some debug output for you from a reproduced gpg timeout:

Selected version of Node.js is 10.17.0
Importing Node.js trusted release keys...
gpg: directory '/home/circleci/.gnupg' created
gpg: keybox '/home/circleci/.gnupg/pubring.kbx' created
gpg: key C273792F7D83545D: 7 signatures not checked due to missing keys
gpg: /home/circleci/.gnupg/trustdb.gpg: trustdb created
gpg: key C273792F7D83545D: public key "Rod Vagg <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported DD8F2338BAE7501E3DD5AC78C273792F7D83545D from hkp://p80.pool.sks-keyservers.net:80
gpg: key D7062848A1AB005C: 1 signature not checked due to a missing key
gpg: key D7062848A1AB005C: public key "Beth Griggs <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 4ED778F539E3634C779C87C6D7062848A1AB005C from hkp://p80.pool.sks-keyservers.net:80
gpg: key 7434390BDBE9B9C5: 4 signatures not checked due to missing keys
gpg: key 7434390BDBE9B9C5: public key "Colin Ihrig <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 94AE36675C464D64BAFA68DD7434390BDBE9B9C5 from hkp://p80.pool.sks-keyservers.net:80
gpg: key B63B535A4C206CA9: 8 signatures not checked due to missing keys
gpg: key B63B535A4C206CA9: public key "Evan Lucas <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported B9AE9905FFD7803F25714661B63B535A4C206CA9 from hkp://p80.pool.sks-keyservers.net:80
gpg: key B01FBB92821C587A: 1 signature not checked due to a missing key
gpg: key B01FBB92821C587A: public key "Gibson Fahnestock <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 77984A986EBC2AA786BC0F66B01FBB92821C587A from hkp://p80.pool.sks-keyservers.net:80
gpg: key C97EC7A07EDE3FC1: 22 signatures not checked due to missing keys
gpg: key C97EC7A07EDE3FC1: public key "keybase.io/jasnell <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 from hkp://p80.pool.sks-keyservers.net:80
gpg: key 09FE44734EB7990E: 4 signatures not checked due to missing keys
gpg: key 09FE44734EB7990E: public key "Jeremiah Senkpiel <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported FD3A5288F042B6850C66B31F09FE44734EB7990E from hkp://p80.pool.sks-keyservers.net:80
gpg: key 770F7A9A5AE15600: 1 signature not checked due to a missing key
gpg: key 770F7A9A5AE15600: public key "Michaël Zasso (Targos) <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 from hkp://p80.pool.sks-keyservers.net:80
gpg: key E73BC641CC11F4C8: 15 signatures not checked due to missing keys
gpg: key E73BC641CC11F4C8: public key "Myles Borins <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 from hkp://p80.pool.sks-keyservers.net:80
gpg: key C273792F7D83545D: 4 signatures not checked due to missing keys
gpg: key C273792F7D83545D: "Rod Vagg <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported DD8F2338BAE7501E3DD5AC78C273792F7D83545D from hkp://p80.pool.sks-keyservers.net:80
gpg: key F07496B3EB3C1762: 1 signature not checked due to a missing key
gpg: key F07496B3EB3C1762: public key "Ruben Bridgewater <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported A48C2BEE680E841632CD4E44F07496B3EB3C1762 from hkp://p80.pool.sks-keyservers.net:80
gpg: key F13993A75599653C: public key "Shelley Vohr (security is major key) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported B9E2F5981AA6E0CD28160D9FF13993A75599653C from hkp://p80.pool.sks-keyservers.net:80
gpg: key D7062848A1AB005C: "Beth Griggs <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported 4ED778F539E3634C779C87C6D7062848A1AB005C from hkp://p80.pool.sks-keyservers.net:80
gpg: key F13993A75599653C: "Shelley Vohr (security is major key) <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported B9E2F5981AA6E0CD28160D9FF13993A75599653C from hkp://p80.pool.sks-keyservers.net:80
gpg: key 7434390BDBE9B9C5: 3 signatures not checked due to missing keys
gpg: key 7434390BDBE9B9C5: "Colin Ihrig <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported 94AE36675C464D64BAFA68DD7434390BDBE9B9C5 from hkp://p80.pool.sks-keyservers.net:80
gpg: key B63B535A4C206CA9: 7 signatures not checked due to missing keys
gpg: key B63B535A4C206CA9: "Evan Lucas <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported B9AE9905FFD7803F25714661B63B535A4C206CA9 from hkp://p80.pool.sks-keyservers.net:80
gpg: key B01FBB92821C587A: "Gibson Fahnestock <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported 77984A986EBC2AA786BC0F66B01FBB92821C587A from hkp://p80.pool.sks-keyservers.net:80
gpg: key C97EC7A07EDE3FC1: 19 signatures not checked due to missing keys
gpg: key C97EC7A07EDE3FC1: "keybase.io/jasnell <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported 71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 from hkp://p80.pool.sks-keyservers.net:80
gpg: key 09FE44734EB7990E: 2 signatures not checked due to missing keys
gpg: key 09FE44734EB7990E: "Jeremiah Senkpiel <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported FD3A5288F042B6850C66B31F09FE44734EB7990E from hkp://p80.pool.sks-keyservers.net:80
gpg: key 770F7A9A5AE15600: "Michaël Zasso (Targos) <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported 8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 from hkp://p80.pool.sks-keyservers.net:80
gpg: key E73BC641CC11F4C8: 15 signatures not checked due to missing keys
gpg: key E73BC641CC11F4C8: "Myles Borins <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 from hkp://p80.pool.sks-keyservers.net:80
gpg: key C273792F7D83545D: 4 signatures not checked due to missing keys
gpg: key C273792F7D83545D: "Rod Vagg <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported DD8F2338BAE7501E3DD5AC78C273792F7D83545D from hkp://p80.pool.sks-keyservers.net:80
gpg: key F07496B3EB3C1762: 1 signature not checked due to a missing key
gpg: key F07496B3EB3C1762: "Ruben Bridgewater <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Success! Imported A48C2BEE680E841632CD4E44F07496B3EB3C1762 from hkp://p80.pool.sks-keyservers.net:80
gpg: key 6D5A82AC7E37093B: 1 signature not checked due to a missing key
gpg: key 6D5A82AC7E37093B: public key "Christopher Dickinson <[email protected]>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 9554F04D7259F04124DE6B476D5A82AC7E37093B from hkp://p80.pool.sks-keyservers.net:80
gpg: key B0A78B0A6C481CF6: public key "isaacs (http://blog.izs.me/) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
Success! Imported 93C7E9E91B49E432C2F75674B0A78B0A6C481CF6 from hkp://p80.pool.sks-keyservers.net:80

gpg: signal Hangup caught ... exiting
Too long with no output (exceeded 10m0s): context deadline exceeded

It seemed to contact the last server in between the 1-2 minute mark and then the log output just hung for the following 8-9 minutes before Circle killed the job.

Let me know if I can provide any more information! Thanks for looking into this!

[iynere]

thank you !! @Gowiem

[halilb]

We're experiencing a similar problem on our builds and nearly half of our builds are failing because of this.

This is our configuration:

version: 2.1
orbs:
  codecov: codecov/[email protected]
  node: circleci/[email protected]

executors:
  node_executor:
    docker:
      - image: circleci/ruby:2.6.3-node
    resource_class: large

commands:
  install_node_version:
    description: Installs the right version of node
    steps:
      - node/install:
          install-npm: true
          install-yarn: false
          node-version: '12.13.0'

jobs:
  setup:
    executor: node_executor
    steps:
      - checkout
      - install_node_version

And this is the output when it fails:

Selected version of Node.js is 12.13.0
A different version of Node.js is installed (v10.16.3); removing it
Importing Node.js trusted release keys...
Too long with no output (exceeded 10m0s): context deadline exceeded

[KyleTryon]

@Gowiem @lfalkner @halilb

Apologies for the delay. The node orb version https://github.com/CircleCI-Public/node-orb/releases/tag/v2.1.1 has been released. Please view the orb on the registry here: https://circleci.com/orbs/registry/orb/circleci/node?version=2.1.1

If you continue to experience any continued issue please let us know.

[SamuelM333]

I'm using orb version v3.0.0 and I'm experiencing long install times. The following output was after 15+ minutes. Killed it before it ended.

Latest LTS version of Node.js is 12.18.2
Importing Node.js trusted release keys...
Success! Imported DD8F2338BAE7501E3DD5AC78C273792F7D83545D from hkp://keyserver.ubuntu.com:80
Success! Imported 4ED778F539E3634C779C87C6D7062848A1AB005C from hkp://pgp.mit.edu:80
Success! Imported 94AE36675C464D64BAFA68DD7434390BDBE9B9C5 from hkp://ipv4.pool.sks-keyservers.net
Success! Imported 71DCFD284A79C3B38668286BC97EC7A07EDE3FC1 from hkp://ipv4.pool.sks-keyservers.net

[luis02lopez]

Sometimes Orb circleci/[email protected] is REALLY slow getting the trusted keys! it's possible to re-open this issue? @KyleTryon

[KyleTryon]

I think we may remove using the trusted keys. They are simply slow and unreliable. We dont enforce this behavior in all orbs and it has caused some issue.

Would love to hear yalls opinion from a security trade off view.

Realistically I believe it would be fine.

[KyleTryon]

I wonder if it's worth switching to https://github.com/nvm-sh/nvm over our own install script.

This is likely more in line with what users would need who are looking specifically to install NodeJs. Its a good option, especially if perhaps the node orb is being used to add NVM to other images.

My vote is yes.

[luis02lopez]

Hey, @KyleTryon I think is important to verify keys and fingerprints when downloading, if by any chance the source get corrupted we prevent downloading a malicious binary.

What I have seen is that sometimes is slow and sometimes is not, so could be a problem with the entities in which we are trying to get the trusted keys. I'm not sure if it's a good idea to just narrow down the number of sources from where we're trying to get the trusted keys and remove those that are really slow or maybe are even corrupted/broken.

[kevinhaas]

This is happening very often now, is there any plan to fix this yet? Just curious because I have to remove this orb if not, as it's just not allowing CI to complete.

[KyleTryon]

@kevinhaas Yes. We would love feedback from you. We have a PR out now to change the install command to use NVM (via a MAJOR version change). #48

We'd like to know if this would be an acceptable solution for most folks, and we are very close to pushing this live 👍 . This however (in accordance to the nvm install docs) does not utilize any gpg keys to install.

CC: @luis02lopez

[kevinhaas]

Yes, that looks great. That's how I install nvm/node personally.

Look forward to seeing this fixed. Thanks!

[tripper54]

+1 for nvm. The node orb is basically unusable as is.

Bonus for you guys is this will be much easier to maintain.
Everyone's a winner!

[luis02lopez]

Let's go for NVM then @KyleTryon 😉 is better to have a usable orb than the most secure orb but not working as expected.