Users are reporting some issues with cache.nixos.org - gather information here

[rbvermaa]

Users are reporting some issues with cache.nixos.org - gather information here

[grahamc]

I'm going to close this issue. I know not every error is addressed here, but also it is impossible to address every issue permanently. If you experience a similar issue, please open a ticket for the specific error you find, so we can triage and diagnose per-error. Please also link back to this one, so it can help route people to a more specific issue. Thank you for the help and reports, sorry it took a while to address it.

[terlar]

You can disable ipv6. I am currently using networking.enableIPv6 = false; to avoid this issue.

[rbvermaa]

@matthewbauer We have changed cache.nixos.org to return a 404. There might be some URL's that still have the 403 cached, but these should disappear in the next few hours, and return the 404. This should make nix-index work again.

[rbvermaa]

Also, are you using IPv6? Just found similar issue here: https://community.fastly.com/t/i-often-cant-access-fastly-servers-using-https-ipv6-rst-packets-received/1317/4 . Perhaps it is related.

[picnoir]

Yup, looks related, I have the same router.

I'm gonna look for an option to force IPv4 on nix calls.

Thank you for your help!

[grahamc]

Here are two graphs of 503s.

Before:

After:

I've gone ahead and increased the timeout connect from 2000ms to 5000ms.

[matthewbauer]

I'm getting it every time I try to use nix-index:

$ nix-index
+ querying available packages

error: fetching the references of store path '/nix/store/f03qiw1qz47qx685f2wfb76crxsp0ymg-node-grunt-cli-1.3.1' failed
caused by: request GET 'http://cache.nixos.org/f03qiw1qz47qx685f2wfb76crxsp0ymg.narinfo' failed with HTTP error 403 Forbidden

It looks like it's a different url failed each time.

[rbvermaa]

@matthewbauer The 403 is expected, as that is what S3 returns when an object does not exist. However, it is weird that nix would fail on that. @edolstra any idea how that could happen?

Could you add some information about the nix version you are using when you get this error?

[rbvermaa]

It might be that the error doesn't come from nix, but from nix-index, as well.

[rbvermaa]

Looked at the nix-index code, and indeed error seems to come from nix-index, specifically https://github.com/bennofs/nix-index/blob/master/src/hydra.rs#L173 .

This is probably triggered by cache.nixos.org returning the 403 error that S3 returns on objects that do not exist. The Cloudfront setup returned a 404. Will see if we can change this to a 404.

[picnoir]

Hi! I'm the one reported the issue on the nixos discourse. I'm gonna answer @rbvermaa questions here.

Can you give some information about the host system, the nix version used and how you installed nix?

The host system is a nixos 18.09 with nix 2.1.1.

Also, how easy is it for you to reproduce?

Really easy, it fails ~80% of the times.

Once it downloaded one file, I have no problem downloading the following ones in the same CLI session (ie nix-channel, nix-build or nixos-rebuild call).

For instance, if I try to update my channels, I get

~ » nix --version                                                                         ninjatrappeur@thinkpad-nix
nix (Nix) 2.1.1
------------------------------------------------------------
~ » sudo nix-channel --update                                                             ninjatrappeur@thinkpad-nix
[sudo] Mot de passe de ninjatrappeur : 
unpacking channels...
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 266 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 582 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 1189 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 2297 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 4008 ms
^Cerror: interrupted by the user
------------------------------------------------------------
~ » sudo nix-channel --update                                                             ninjatrappeur@thinkpad-nix
unpacking channels...
created 2 symlinks in user environment
----------------------------------------

(I interrupt the first call to reset the exponential retry delay).

My internet provider is OVH telecom if it's any help.

I can privately share with you my IP address if it's any help for the debug process.

[rbvermaa]

@NinjaTrappeur Thanks for the info. Does a curl call work without issues? e.g. could you post output of:

curl -v https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo

[picnoir]

~ » curl -v https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo              ninjatrappeur@thinkpad-nix
*   Trying 2a04:4e42:1d::729...
* TCP_NODELAY set
* Connected to cache.nixos.org (2a04:4e42:1d::729) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cache.nixos.org:443 
* stopped the pause stream!
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cache.nixos.org:443 
------------------------------------------------------------
~ » curl -v -4 https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo           ninjatrappeur@thinkpad-nix
*   Trying 151.101.38.217...
* TCP_NODELAY set
* Connected to cache.nixos.org (151.101.38.217) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=v2.shared.global.fastly.net
*  start date: Oct 15 11:17:24 2018 GMT
*  expire date: Mar 20 20:22:20 2019 GMT
*  subjectAltName: host "cache.nixos.org" matched cert's "cache.nixos.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x13bc2b0)
> GET /hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo HTTP/2
> Host: cache.nixos.org
> User-Agent: curl/7.61.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 404 
< server: Varnish
< retry-after: 0
< content-type: text/html
< accept-ranges: bytes
< accept-ranges: bytes
< date: Tue, 16 Oct 2018 10:11:44 GMT
< via: 1.1 varnish
< x-served-by: cache-ams4431-AMS
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1539684705.547020,VS0,VE88
< content-length: 3
< 
* Connection #0 to host cache.nixos.org left intact
404%                                               

Looks like an IPv6 error.

[hamishmack]

Any update on this. I'm not sure if this is the same issue, but we are seeing HTTP error 503 followed by error 7 while decompressing xz file here (Wellington, NZ):

copying path '/nix/store/vkfs0i8j9jk7b0y1n49ykraf49w0fqb4-python2.7-pycrypto-3.6.6' from 'https://cache.nixos.org'...
copying path '/nix/store/h54y4zm7pzckn67y1ixdbz6ga8v7gmbj-python2.7-libcloud-1.2.1' from 'https://cache.nixos.org'...
warning: unable to download 'https://cache.nixos.org/nar/1kigq2qc4d7pf9dpfna21p5r2shifkfbpdda0bzpw2p8hav6plfp.nar.xz': HTTP error 503; retrying in 264 ms
warning: unable to download 'https://cache.nixos.org/nar/1kigq2qc4d7pf9dpfna21p5r2shifkfbpdda0bzpw2p8hav6plfp.nar.xz': HTTP error 503; retrying in 593 ms
warning: unable to download 'https://cache.nixos.org/nar/1npnb3jcfqhyw816ncsscjl7wpwh06pygcw8cgv4jiix9q9bcrx4.nar.xz': HTTP error 503; retrying in 292 ms
error 7 while decompressing xz file
warning: unable to download 'https://cache.nixos.org/nar/1npnb3jcfqhyw816ncsscjl7wpwh06pygcw8cgv4jiix9q9bcrx4.nar.xz': HTTP error 503; retrying in 576 ms
error: build of '/nix/store/kczj7517hjs2l5j9kvy3s76lxn89la8l-nixops-1.6.drv' failed

Rerunning the nix command always seems to get a bit further and eventually works.

[rbvermaa]

@hamishmack Sorry, I missed the notification for this issue. I have contacted Fastly support, to see if they can help diagnose this issue, will update here when I hear back.

[rbvermaa]

We've changed some settings to be able to debug this issue better, based on suggestions by Fastly. Hopefully this gives us some more information about the 503 errors.

If you experience this again on your machine, can you let us know and go to https://www.fastly-debug.com/ and post the information here?

[terlar]

I experience this issue and just disabled IPv6 to be able to upgrade my NixOS, going to that page it just infinitely spinns sayingCollecting data please wait..., I waited for 30 minutes, wasn't sure if it was supposed to return something by then.

[rbvermaa]

@terlar Does the page spin after you switched to IPv4, or when you were still on IPv6?

[terlar]

Both as far as I can remember. I am currently on vacation, but I will double check when I have access to my computer.

[bmillwood]

I'm also having the 503 and then error 7 while decompressing xz file thing. And again, it seems to work if I rerun it enough times.

From fastly:

Please submit text block below with your ticket to Fastly
ewogICJnZW9pcCI6IHsKICAgICJjaSI6ICJob25nIGtvbmciLAogICAgInN0IjogIk5PIFJFR0lPTiIsCiAgICAiY3QiOiAiaG9uZyBrb25nIiwKICAgICJjbyI6ICJBUyIsCiAgICAiY19hc24iOiAiNDc2MCIsCiAgICAiY19hc25fbmFtZSI6ICJwY2N3IGxpbWl0ZWQiLAogICAgInJfaXAiOiAiMjE4LjEwMi4xMS4xMDYiLAogICAgInJfYXNuIjogIjQ3NjAiLAogICAgInJfYXNuX25hbWUiOiAicGNjdyBsaW1pdGVkIiwKICAgICJyX2NpIjogImhvbmcga29uZyIsCiAgICAicl9zdCI6ICJOTyBSRUdJT04iLAogICAgInJfY3QiOiAiaG9uZyBrb25nIiwKICAgICJyX2NvIjogIkFTIgogIH0sCiAgInBvcExhdGVuY3kiOiB7CiAgICAibnJ0IjogNDksCiAgICAiaXRtIjogNjAsCiAgICAidHlvIjogNTQsCiAgICAiaG5kIjogNTUsCiAgICAiaGtnIjogMiwKICAgICJzaW4iOiAzOCwKICAgICJmanIiOiAxOTYsCiAgICAibGF4IjogMjA1LAogICAgImZyYSI6IDE5NwogIH0sCiAgInBvcEFzc2lnbm1lbnRzIjogewogICAgImFjIjogImhrZyIsCiAgICAiYXMiOiAiaGtnIgogIH0sCiAgInJlcXVlc3QiOiB7CiAgICAicmVzb2x2ZXJfaXAiOiAiMjE4LjEwMi4xMS45NyIsCiAgICAicmVzb2x2ZXJfYXNfbmFtZSI6ICJIS1RJTVMtQVAgSEtUIExpbWl0ZWQsIEhLIiwKICAgICJyZXNvbHZlcl9hc19udW1iZXIiOiAiNDc2MCIsCiAgICAicmVzb2x2ZXJfY291bnRyeV9jb2RlIjogIkhLIiwKICAgICJjbGllbnRfaXAiOiAiMjE5Ljc5LjEzMC4xMzUiLAogICAgImNsaWVudF9hc19uYW1lIjogIkhLVElNUy1BUCBIS1QgTGltaXRlZCwgSEsiLAogICAgImNsaWVudF9hc19udW1iZXIiOiAiNDc2MCIsCiAgICAidGltZSI6ICIyMDE4LTEyLTIxVDE3OjE4OjIwLjAwMFoiLAogICAgImhvc3QiOiAid3d3LmZhc3RseS1kZWJ1Zy5jb20iLAogICAgImFjY2VwdCI6ICJ0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htbDtxPTAuOSwqLyo7cT0wLjgiLAogICAgInVzZXJhZ2VudCI6ICJNb3ppbGxhLzUuMCAoWDExOyBMaW51eCB4ODZfNjQ7IHJ2OjYzLjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvNjMuMCIsCiAgICAiYWNjZXB0bGFuZ3VhZ2UiOiAiZW4tVVMsZW47cT0wLjUiLAogICAgImFjY2VwdGVuY29kaW5nIjogImd6aXAiLAogICAgImZhc3RseXNlcnZlcmlwIjogIjE1MS4xMDEuNzYuNjQiLAogICAgInhmZiI6ICIyMTkuNzkuMTMwLjEzNSIsCiAgICAiZGF0YWNlbnRlciI6ICJIS0ciLAogICAgImJhbmR3aWR0aF9tYnBzIjogIjY5Ljc1IiwKICAgICJjd25kIjogMTAwLAogICAgIm5leHRob3AiOiAiMTcyLjIwLjEwMC4xIiwKICAgICJydHQiOiAxMS42MzcsCiAgICAiZGVsdGFfcmV0cmFucyI6IDAsCiAgICAidG90YWxfcmV0cmFucyI6IDAKICB9Cn0=

Client IP Info
IP	219.79.130.135
AS Name	HKTIMS-AP HKT Limited, HK
AS Number	4760
City	hong kong
Continent	AS
Country	hong kong
State	NO REGION
Resolver IP Info
IP	218.102.11.97
AS Name	HKTIMS-AP HKT Limited, HK
AS Number	4760
Country Code	HK
Server Connection Info
IP	151.101.76.64
Datacenter	HKG
BW to server	69.75mbps
Congestion Window	100
Next Hop	172.20.100.1
RTT	11.637ms
Delta Retransmits	0
Total Retransmits	0
POP Latency (ms)
NRT	49
ITM	60
TYO	54
HND	55
HKG	2
SIN	38
FJR	196
LAX	205
FRA	197
Request Info
Time	Sat Dec 22 2018 01:18:20 GMT+0800 (HKT)
Host	www.fastly-debug.com
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept-Language	en-US,en;q=0.5
Accept-Encoding	gzip
X-Forwarded-For	219.79.130.135

[splack]

I'm getting 499, 503, 504 and just hung connections with no response. Is there a way to see the upstream status documented somewhere?

[samueldr]

IRC report:

• https://logs.nix.samueldr.com/nixos/2019-02-20#1550675671-1550675476;

[millettjon]

I started getting lots of these a few hours ago:
warning: unable to download 'https://cache.nixos.org/ax3igj2aglvv46vkcpmyklr6lcqlwz7z.narinfo': Couldn't connect to server (7); retrying in 253 ms
warning: unable to download 'https://cache.nixos.org/im74kvbg0swj3akq4gcbwnlw8pj6lz1a.narinfo': Couldn't connect to server (7); retrying in 267 ms
w
Using wget returns 404 for the same urls.

Here is the fastly report. Note that is doesn't seem to complete. I waited more than 5 minutes and it is still working. The partial information is below.
| Debug
Collecting data please wait.

Client IP Info
IP 181.226.182.157
AS Name
AS Number
City
Continent
Country
State
Resolver IP Info
IP
AS Name
AS Number
Country Code
Server Connection Info
IP 151.101.0.64
Datacenter SCL
BW to server
Congestion Window
Next Hop
RTT
Delta Retransmits
Total Retransmits
Request Info
Time Wed Feb 20 2019 15:42:24 GMT-0500 (EST)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip
X-Forwarded-For 181.226.182.157

[millettjon]

Still having this issue. Curl now works with ipv4 (-4) option but not with (-6). Any way to force nix to use ipv4?

[grahamc]

I've been chatting with Fastly support. They've told me a lot of the 503 errors are due to a low connect timeout between Fastly and S3. I've changed this value from 1000ms to 2000ms. We'll start with this change and see how the number of reports over time changes.