Error during npm install

[thenickdude]

Error during npm install

[thenickdude]

Do you have git installed? (I thought it was installed by default, but maybe not)

[sslupsky]

Hi, Yes, I performed the step "sudo yum install git".

Though I think I recall that I did the installation using sudo (and not the interactive sudo session). I tried to "reinstall" it using an interactive sudo session and it said it was already installed.

[thenickdude]

Hm, if you run this command on its own, do you get any error output?

/usr/bin/git ls-remote -h -t ssh://[email protected]/thenickdude/node-progress.git

[sslupsky]

I should have tried that. I will fire up the EC2 instance again and try that and let you know.

[sslupsky]

Fired up the instance again and installed lz4 and git while in sudo interactive mode. Received the same error mentioned earlier. When I tried the /usr/bin/git there was no error message but nothing appeared to happen ... the command hung. I had to ctrl-c to get back to the command prompt:

[root@ip-10-0-0-63 ~]# /usr/bin/git ls-remote -h -t ssh://[email protected]/thenickdude/node-progress.git
^C

[sslupsky]

Give me a sec ... there may be some sort of network issue with the instance. I can ssh into my instance fine but for some reason the instance cannot ping google.com ... I'll try to figure out what is going and get back to you.

[thenickdude]

Ah yep, that was my next question, heh. I think the instance either needs to be assigned a public IP address, or there needs to be a NAT instance in your VPC to be able to access the internet.

…

On 9 Oct. 2017 8:24 am, "sslupsky" ***@***.***> wrote: Give me a sec ... there may be some sort of network issue with the instance. I can ssh into my instance fine but for some reason the instance cannot ping google.com ... I'll try to figure out what is going and get back to you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1Rg6Qe--pk9zjcdWlMUS1jJohaKfbXks5sqSFygaJpZM4PxzYr> .

[thenickdude]

Alternatively you can install snap-to-s3 locally on your machine and upload that entire folder to your instance (it's a ton of tiny files so it's better to tar it up first). EDIT: Although in this case you would also have to set up a VPC endpoint for S3, which I've not tried before myself.

…

On 9 Oct. 2017 8:31 am, "Nicholas Sherlock" ***@***.***> wrote: Ah yep, that was my next question, heh. I think the instance either needs to be assigned a public IP address, or there needs to be a NAT instance in your VPC to be able to access the internet. On 9 Oct. 2017 8:24 am, "sslupsky" ***@***.***> wrote: > Give me a sec ... there may be some sort of network issue with the > instance. I can ssh into my instance fine but for some reason the instance > cannot ping google.com ... I'll try to figure out what is going and get > back to you. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#3 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AB1Rg6Qe--pk9zjcdWlMUS1jJohaKfbXks5sqSFygaJpZM4PxzYr> > . >

[sslupsky]

Yeah, there were some weird ACL's on the VPC. I created a new VPC and launched an instance again. I have outgoing network working now. I am now seeing some sort of permissions error. I tried it on the EC2 instance and on my local machine and get the same error:

git ls-remote -h -t ssh://[email protected]/thenickdude/node-progress.git
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

If I use the GitHub HTTPS URL, I see the following:

[root@ip-10-0-0-111 ~]# /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git
f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master
2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format
684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2
e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3
d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4
aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5
2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{}
8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0
5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{}
783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0
0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{}
1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1
8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{}
5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0
66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{}
aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1
26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2
496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4
44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{}
66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5
fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8
6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{}
d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0

[thenickdude]

Okay yeah I thought it was weird that npm picked the SSH connection method instead of HTTPS too (SSH seems like it requires a key stored at GitHub). I'll poke around with it here and see if I can get it to switch.

…

On 9 Oct. 2017 9:20 am, "sslupsky" ***@***.***> wrote: Yeah, there were some weird ACL's on the VPC. I created a new VPC and launched an instance again. I have outgoing network working now. I am now seeing some sort of permissions error. I tried it on the EC2 instance and on my local machine and get the same error: git ls-remote -h -t ***@***.***/thenickdude/node-progress.git Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. If I use the GitHub HTTPS URL, I see the following: ***@***.*** ~]# /usr/bin/git ls-remote -h -t https://github.com/ thenickdude/node-progress.git f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master 2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format 684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2 e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3 d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4 aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5 2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{} 8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0 5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{} 783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0 0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{} 1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1 8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{} 5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0 66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{} aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1 26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2 496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4 44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{} 66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5 fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8 6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{} d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1Rg_Ew4w9LyjOUJP-mEckW0HcE7zc7ks5sqS6ngaJpZM4PxzYr> .

[thenickdude]

I've now pushed a new version to NPM, 0.3.1, which I think should solve this issue by explicitly using HTTPS URLs for the git dependencies, please give it a go.

[sslupsky]

Thank you.

The update has cleared up the ssh public key issue. However, when I tried with the new dependency URL's I am seeing the same error at the command line:

[root@ip-10-0-0-111 ~]# npm install -g snap-to-s3
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git
npm ERR!
npm ERR! fatal: Cannot come back to cwd: Permission denied
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2017-10-09T14_38_00_114Z-debug.log

The error's reported in the log are similar as reported earlier as well:

38 silly fetchPackageMetaData error for progress@git+https://github.com/thenickdude/node-progress.git#f901750478a76057b9271bda333dd1dcdd5406dd Error while executing:
38 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git
38 silly fetchPackageMetaData
38 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied
38 silly fetchPackageMetaData
38 silly fetchPackageMetaData exited with error code: 128
39 silly fetchPackageMetaData error for tar-stream@git+https://github.com/thenickdude/tar-stream.git#3160e7d60fe142f04e126b8b261248d023200e1b Error while executing:
39 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t https://github.com/thenickdude/tar-stream.git
39 silly fetchPackageMetaData
39 silly fetchPackageMetaData fatal: Cannot come back to cwd: Permission denied
39 silly fetchPackageMetaData
39 silly fetchPackageMetaData exited with error code: 128

When I run the git command from the command line I get the following output:

[root@ip-10-0-0-111 ~]# /usr/bin/git ls-remote -h -t https://github.com/thenickdude/node-progress.git
f901750478a76057b9271bda333dd1dcdd5406dd refs/heads/master
2d87576b9dd831de221e2024e640ee727dff583f refs/heads/time-format
684f627ab1cee46ae0ae09095dd8550f3788e6b6 refs/tags/0.0.2
e6194546b6f48fd4ecd37a63238c04a3281a5e45 refs/tags/0.0.3
d83fa84f73b4dadd86d03dfb6c1bf832eb713dbd refs/tags/0.0.4
aa38082c4194bcadddf2df6b260fba2f5623d2f9 refs/tags/0.0.5
2b48b1d6cb42fe9352d2a1a87f8aad4a2e8fc726 refs/tags/0.0.5^{}
8628a2c919928181804b5ffd0557095df6103a54 refs/tags/0.1.0
5a12cafdee006ce4dd8624783d483b8371d32ed8 refs/tags/0.1.0^{}
783ca37c06f0f49a49b6efbbb0957806d849f5be refs/tags/1.0.0
0c0066de85818485257b1c20476f485928c97037 refs/tags/1.0.0^{}
1ac9fec5866152e4f51806a009d1761269326295 refs/tags/1.0.1
8cf6883d06e419caec653e10f0e5ef253c5033f0 refs/tags/1.0.1^{}
5641744a66ce4e024d495ae7fbfc8dda56eca9ab refs/tags/1.1.0
66b7f73c93d5bb89ed96709937656a0b05e4c2a8 refs/tags/1.1.0^{}
aeac72488eac33b0572928a65d248ef6e764e78a refs/tags/1.1.1
26510351b88188c6e5273fb86cd558ac1cdaafb8 refs/tags/1.1.2
496e7cf984dc039df74bf72d8e7aa8fad3d94cce refs/tags/1.1.4
44e98342224e0f77bfa8c8105f428df36bdda3e1 refs/tags/1.1.4^{}
66495c58fa48cf0f35a894a791c9baad3ad7d1b2 refs/tags/1.1.5
fe178d7f9cff9903b93eae660eefa7e3ed2641ff refs/tags/1.1.8
6b9524c0d07df9555d20ae95c65918020c50e3e2 refs/tags/1.1.8^{}
d84326ed9ab7720592b6bbc9c108849cd2a79908 refs/tags/2.0.0

All the other dependencies appear to be fine. The problem appears to be associated with only the two dependencies that are specified to your GitHub repos for some reason.

I think the issue has something to do with the user permissions. If I install nvm and node from my user account (instead of using the interactive sudo mode), the installation of snap-to-s3 goes smoothly and I can run snap-to-s3 from the command line. However, I cannot execute snap-to-s3 as sudo as you discussed during the installation steps.

So, what I did was edited the .bashrc for the root user and changed the NVM_DIR to point to the user account:

export NVM_DIR="/home/ec2-user/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && . "$NVM_DIR/bash_completion" # This loads nvm bash_completion

To get sudo to execute snap-to-s3, you need to also run it with the "simulate as initial login" (-i) option"

sudo -i snap-to-s3 --help

This gets me to the point of being able to run snap-to-s3 as sudo. I have not actually tested to see if it works yet ... I will do that now and report back.

[sslupsky]

snap-to-s3 is now executing the migrations. The output has confirmed that two migrations have been successful so far and I confirmed that the tar files are in S3. When it is complete I will do the validation and confirm the migration.

Thank you for your patience. This is an amazing tool. Thank you for making it public.