nixos/nixos-channel-scripts

Building

$ nix-build

Running

$ ./result/bin/mirror-nixos-branch nixos-16.03-small https://hydra.nixos.org/job/nixos/release-16.03-small/tested/latest-finished

$ ./result/bin/mirror-nixos-branch nixos-unstable-small https://hydra.nixos.org/job/nixos/unstable-small/tested/latest-finished

$ ./result/bin/generate-programs-index /data/releases/nixos-files.sqlite ./programs.sqlite http://nix-cache.s3.amazonaws.com/ /data/releases/nixos/unstable-small/nixos-16.09pre89017.9db1990-tmp/store-paths /data/releases/nixos/unstable-small/nixos-16.09pre89017.9db1990-tmp/unpack/nixos-16.09pre89017.9db1990/nixpkgs

nixos-channel-scripts's Issues

generate-programs-index.cc broken due to `.ls.xz` to `.ls` conversion

With commit NixOS/nix@042975e, Hydra no longer generates .ls.xz files. It now only generates .ls files. This breaks the generate-programs-index.cc script, which is used to generate the command-not-found hook for NixOS.

Bring back command-not-found support

It was disabled for the second time in 7f2bbc8

It took a few hours extra on full rebuild to update the channel due to this feature.

Hydra now publishes .ls.xz for each narfile, so we just need to change the index generation to unpack the file, parse it and index it.

We need to change https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc to download files like https://cache.nixos.org/sb3r5grph8lyv9zvam2my3vwwag64fi7.ls.xz

Make channel scripts fetch artifacts from S3 directly, not cache.nixos.org

The channel scripts hit into NixOS/infra#212 sometimes when fetching ISOs from cache.nixos.org to copy to the channels bucket. This makes runs take hours when they should take minutes.

There is no benefit gained from going through cache.nixos.org for this use case. As noticed by @K900, the .iso nar.xz files on the cache are unlikely to be touched by anyone other than the channel scripts, so we don't even end up pre-warming the cache or anything along those lines - if anything, we just waste cache bytes that could be used by other paths.

hydra jobsetevals is sorted assending

@edolstra Was trying to figure out why a script I wrote based on mirror-nixos-branch wasn't working. Finally tracked it down to the fact that that the jobsetevals attribute is sorted in ascending order.

This means that this code

my $evalId = $releaseInfo->{jobsetevals}->[0] or die;

references the oldest jobseteval and not the newest.

You can see this here via the following curl request to our hydra server. The official nixos one seems to rarely have more than one evaluation on tap, so this is harder to see (and catch) on it.

curl -LH "Accept: application/json" https://hydra.sharcnet.ca/job/ccpkgs/ccpkgs-17.09/tested/latest-finished

{"jobset":"ccpkgs-17.09"
,"id":125368
,"buildmetrics":{}
,"timestamp":1529286374
,"buildstatus":0
,"system":"x86_64-linux"
,"stoptime":1529532645
,"nixname":"ccpkgs-17.09"
,"releasename":null
,"buildoutputs":
  {"out":{"path":"/nix/store/xbkrm8jvd92w96skjjfs7140ib17yrib-ccpkgs-17.09"}}
,"starttime":1529532645
,"finished":1
,"project":"ccpkgs"
,"drvpath":"/nix/store/0lb0qv0wagxb7w3a9dbnq7qmjlwll93b-ccpkgs-17.09.drv"
,"jobsetevals":[101,107]
,"job":"tested"
,"priority":100
,"buildproducts":{}
}

command-not-found and nixos with flakes

Hi,

I have a nixos setup using flakes and enabling
programs.command-not-found.enable = true;
does not lead to having suggestions which package to install.
I think the reason for this is that the programs.sqlite is missing.

NixOS channels: copy also PXE boot files

Once NixOS/nixpkgs#14740 is merged, it would be nice to also copy nix-build -A netboot nixos/release.nix result to the channel so NixOS can be booted using (i)PXE.

Note that once this is implemented, netboot should be part of the tested job.

generate-programs-index finds wrong attrPath for "dig" binary

sqlite> select * from Programs where name = "dig";
dig|x86_64-linux|bind

Which is technically correct: dig is part of the bind derivation, but not in its default output. Instead, it's part of the dnsutils output which does not get installed by a simple nix-shell -p bind (it needs either nix-shell -p bind.dnsutils or nix-shell -p dnsutils since there is an alias in top-level).

Something somewhere in generate-programs-index should probably be more output-aware, but I lack the ability to easily test changes so this is just an issue report and not a PR :-)

gnome iso needs to be added to the channel

see NixOS/nixpkgs#99648 (comment)

Use manifest for channels copy script

See:

• #57 (comment)
• #25 (comment)

Instead of adding boundless complexity in the channels copy script, make it so that Nixpkgs self-describes what outputs are exposed.

Revive command-not-found support

We no longer generate programs.sqlite because that would require downloading all packages from cache.nixos.org. Need a solution. E.g. Hydra could upload a content listing of every NAR to the cache, which generate-programs-index.pl could download (with some local caching).

Publish channels to S3

I see three good reasons to do this:

• S3+cloudfront have been extremely stable (not that nixos.org was down for any long time) and they're decentralized, leaving last single point of failure on the side when using NixOS
• nix-channel --update would not redownload channel on each invocations since it would respect the etag header
• we won't need to recycle channels to free disk space, so declarative $NIX_PATH wouldn't fail anymore since upstream nixpkgs.tar.xz is missing

Would this add much to the costs?

cc @edolstra

adb missing from command-not-found

I'm not sure how exactly the programs.sqlite database is generated or if this is even a bug. But it was surprising to me, so I thought I'd bring it up.

command-not-found adb doesn't find a package for adb, even though android-platform-tools has a file bin/adb which is a symlink to platform-tools/adb.

"older releases are also available" link is broken in nixos page

https://nixos.org/download.html#nixos-more has a link https://nixos.org/download.html#nixos-more pointing to https://releases.nixos.org/?prefix=nixos which only contains the text "Error: [object Object]"

Protect pushes to NixOS/nixpkgs channel branches with a special commit status.

IT is possible for admins to push and merge branches in to the protected channel branches. This is a mistake I made.

To prevent this, we should have the channel scripts set a special status on the commit and then require pushes to those branches to have that special status.

Generate parseable JSON file with a history of channel updates

For the sake of tracking NixOS release process I think we should provide a way to introspect the old releases. This used to be visible before S3 as we just had folders.

Given the current pipeline using these scripts we could simply append a JSON file http://nixos.org/channels/history.json with the following format:

{
  version: 1
  channels: [
     {
       name = "nixos-16.09-small",
       date = "2016-11-16 06:08:34";
       commit = "https://github.com/NixOS/nixpkgs-channels/commits/6270733155c381090fc5c7de6bddc26fbf35f47f",
       url = "https://d3g5gsiof5omrk.cloudfront.net/nixos/16.09/nixos-16.09.1032.6270733",
       hydraUrl = "https://hydra.nixos.org/eval/1304255",
     },
     ...
   ]
}

cc @edolstra @rbvermaa

Open questions:

• how to deal with the file once it gets bigger

nixos programs.sqlite missing large numbers of packages

I'm writing a custom bash command_not_found_handle that does a lookup into /nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite to find out which nix package to install for a given program. This is a sqlite database that's generated on the nixos cache servers, and is distributed via the nixos channel.

I'm noticing that a lot of very commonly used programs are missing from both the unstable nixos and the 20.09 nixos versions. For example, the "vim" binary is not found. Looking at the 20.03 version, I can find vim, but the record count in the programs table seems suspiciously low, around ~40,000 program records for all recent versions of programs.sqlite. The nix package database at search.nixos.org claims to have over ~80,000 packages so I would expect at least 80,000 records or more in the programs table.

I noticed a similar issue cropping up with nix-index: nix-community/nix-index#163

In the case of nix-index, it looks like there are mal-formed JSON records being generated at cache.nixos.org, which is causing nix-index to skip the creation of records for many popular packages, such as vim and emacs. I'm not sure if this is the same problem afflicting programs.sqlite, but it may be a possibility.

Is this the right issue tracker for this problem, or is there a more appropriate place to put this? It looks like the cache.nixos.org server scripts that generate JSON files need to be fixed.

Thanks!

get programs.sqlite with flakes

Now that we're moving towards flakes instead of channels, the injected programs.sqlite is no longer available.

What would be a good way to provide this information?

It could be injected in nixpkgs as a commit on the just-built branch, but that seems rather crude. Perhaps in a different repo that has the same branch names as the ones that hydra makes on nixpkgs?

[question] Running `generate-programs-index` on `nixos-rebuild`?

Just wondered if it would make sense to (optionally) run generate-programs-index on a nixos-rebuild.

A benefit of this would be that I can use the command-not-found-handler from NixOS without using channels (I pin nixpkgs etc. on my own locally and generate a NIX_PATH using nix.nixPath with the corresponding paths).

I guess something like this would become necessary in the future anyways when flakes replace channels, right?

nixpkgs-channels was archived

I noticed this issue in the logs:

Sep 08 18:53:34 bastion update-nixpkgs-20.03-darwin-start[6768]: remote: This repository was archived so it is read-only.
Sep 08 18:53:34 bastion update-nixpkgs-20.03-darwin-start[6768]: fatal: unable to access 'https://github.com/NixOS/nixpkgs-channels.git/': The requested URL returned error: 403

generate-programs-index crashes sometimes

Fortunately it doesn't seem to happen too often, so no channel has been blocked for more than a day by this so far.

Example:

           PID: 1117916 (generate-progra)
           UID: 497 (hydra-mirror)
           GID: 65534 (nobody)
        Signal: 11 (SEGV)
     Timestamp: Mon 2021-11-29 13:42:37 CET (2h 10min ago)
  Command Line: generate-programs-index /scratch/hydra-mirror/nixos-files.sqlite /scratch/hydra-mirror/release-nixos-21.11/nixos-21.11beta333771.8e6b3914626/unpack/nixos-21.11beta333771.8e6b3914626/programs.sqlite http://nix-cache.s3.amazonaws.com/ /scratch/hydra-mirror/release-nixos-21.11/nixos-21.11beta333771.8e6b3914626/store-paths /scratch/hydra-mirror/release-nixos-21.11/nixos-21.11beta333771.8e6b3914626/unpack/nixos-21.11beta333771.8e6b3914626/nixpkgs
    Executable: /nix/store/n0mg1bi82a8rndk6nq1r1dqgblkavxbr-nixos-channel-native-programs/bin/generate-programs-index
 Control Group: /system.slice/update-nixos-21.11.service
          Unit: update-nixos-21.11.service
         Slice: system.slice
       Boot ID: 81eee5b9ce8b408c810c24c5fa1cb38b
    Machine ID: d10f095896ae4cb0be9598b874b8ba07
      Hostname: bastion
       Storage: /var/lib/systemd/coredump/core.generate-progra.497.81eee5b9ce8b408c810c24c5fa1cb38b.1117916.1638189757000000.lz4 (inaccessible, truncated)
       Message: Process 1117916 (generate-progra) of user 497 dumped core.

Unfortunately the coredumps don't seem usable, most likely due to truncation to 2G (only ?? lines in backtrace).

mirror-channel.pl: Upload binary cache in topologically sorted order

Otherwise, if we upload a path before all its dependencies are uploaded, users can get a "don't know how to build these paths" warning from Nix.

NixOS/nix#765

Stable URLs for the ISO images

To support netbooting NixOS from https://netboot.xyz/, we need a stable URL for the ISO images per channel.

Adding a redirect to the htaccess file

nixos-channel-scripts/mirror-nixos-branch.pl

Lines 213 to 215 in c3a9ba3

	write_file($htaccess,
	"Redirect /channels/$channelName $target\n" .
	"Redirect /releases/nixos/channels/$channelName $target\n");

should be should be enough.

stable channel

For sake of those just wanting to always be on latest stable, it would be great if there was nixpkgs-stable that would point to nixos-22-5 channel for example.

Due to flakes, upgrading is a manual step of updating the lock file.

This change would make writing documentation and snippets a lot easier, while making sure the users upgrade to the latest stable.

Run nixos-channel-scripts on a CI

The main benefit here would be more transparency into the channel update process so that anyone can see if it failed and what went wrong by looking at the logs.

We briefly discussed with @grahamc and @edolstra and it's only a matter of figuring out how to pull it off.

ISOs are redirecting using a 301, should be 302

See the ISO links at https://nixos.org/download.html

Steps to reproduce:
curl -v https://channels.nixos.org/nixos-20.03/latest-nixos-plasma5-x86_64-linux.iso
Pointing to
https://releases.nixos.org/nixos/20.03/nixos-20.03.3087.b4db68ff563/nixos-plasma5-20.03.3087.b4db68ff563-x86_64-linux.iso using a 301.

Expected:
Should use a 302.

Link also the manual in a channel

Each channel should also have a link to the manual at https://d3g5gsiof5omrk.cloudfront.net/nixpkgs/nixpkgs-17.03pre94051.cbec6a3

channels going back in time

@BlessJah reported on IRC yesterday that 17.09 went back in time with a channel update. He wrote a script he runs daily to check this now: http://dpaste.com/3DYQTAD

@grahamc and I confirmed that our channels had downgraded to. @grahamc explained "the channel is based on the newest build of hydra which has completely finished building. if you restart jobs, the
newest to completely finish goes back to an older build. the channel update tool has code to guard against going back in time, but it seems to be broken"

@romildo on IRC reported the same thing happening for 18.03pre (unstable) today.