Comments (7)
zimbatm
commented on May 29, 2024
It should take ~24h for the DNS to switch to the Route53-provided ones
from infra.
grahamc
commented on May 29, 2024
@zimbatm I propose we set the TTLs quite low, then do the switch, then bring them back up to normal levels. Then if there is a major mistake of some sort which we can't find now, it won't take a long time to repair them.
from infra.
zimbatm
commented on May 29, 2024
@grahamc I don't think that this TTL is configurable. It's not served by the registrar DNS, but by the "org" TLD.
from infra.
grahamc
commented on May 29, 2024
Right. I mean record TTLs.
from infra.
grahamc
commented on May 29, 2024
Basically I would suggest setting all of them to 60, and then a week or so after everything is known to be good, reverting the commit and going back up to current levels.
from infra.
grahamc
commented on May 29, 2024
The switch happened at 19:25Z, and cache.nixos.org and releases.nixos.org had some bad records. @edolstra repaired them in 8d7b0b6...45b25e6. Because of the 1hr TTL on both of those records, there will be a downtime for some users for the next or so while the TTL is replaced. Something to learn from and to handle differently next time :)
from infra.
grahamc
commented on May 29, 2024
More information: https://discourse.nixos.org/t/the-cache-is-down-with-workaround/1720/3
from infra.
Related Issues (20)
- Rate limiting on hydra.nixos.org
- Clarify existing licensing and apply MIT license HOT 17
- Setup an SSO service for admin interfaces HOT 4
- Set up zrepl alerting
- Public GitHub webhooks re-broadcaster / message queue HOT 2
- Move survey.nixos.org to a less ridiculous hosting on non-critical-infra HOT 5
- Decommission eris.nixos.org HOT 1
- Monitoring data is not backed up
- Onboard ofborg onto non-critical-infra
- Fastly reads from >= two locations HOT 4
- Meeting infrastructure
- ICS links for Discourse calendars HOT 2
- Alert on continued failure of hydra-scale-equinix-metal.service
- Put hydra advances query result in a public s3 bucket somewhere on a cron job HOT 1
- Document non-critical-infra team HOT 4
- Self-hosted releases.nixos.org HOT 6
- Alert on nixos.org Matrix federation issues HOT 2
- Deprecate netboot.gsc.io ("netboot-foundation" EQM host) HOT 4
- vaultwarden: smtp credentials invalid HOT 1
- Nix manual returns 404 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from infra.