chaotic-aur/toolbox

Unified kit with all the scripts required for maintaining the repository 🧰

Chaotic AUR

CLI

• chaotic pr{,epare} ${INPUTDIR} $@

  It generates a building script to be later run in a containerized environment. $INPUTDIR is the name of the directory in "$PWD" which contains a PKGBUILD.

• chaotic {lw,lowerstrap}

  It generates a lowerdir for later chrooting.

• chaotic {mk,makepkg} ${INPUTDIR} $@

  Builds the package in a container using systemd-nspawn. $INPUTDIR is the result of a prepare

• chaotic {mkd,makepwd} [${PACKAGES[@]}]

  Prepare and build all packages in the current directory. If PACKAGES are not provided then it will try to build all sub-directories.

• chaotic {si,interfere-sync}

  Sync packages' interference repo.

• chaotic {bump,interfere-bump} [${PACKAGES[@]}]

  Generate and push a PKGREL_BUMPS file for use in interfere. Old entries will automatically be removed. Uses repoctl.

• chaotic {sp,package-lists-sync}

  Sync package list repo.

• chaotic {dp,deploy} ${INPUTDIR}

  Sign the package and send to primary node.

• chaotic {dbb,db-bump}

  Add recently deployed packages to the database, while moving replaced packages to the archive. Uses repoctl.

• chaotic db-rebuild Completely rebuild the database and files database from scratch, keeping the original DB around until the rebuild is finished.

• chaotic {rm,remove} ${PACKAGES[@]}

  Remove and archive all listed packages. Uses repoctl.

• chaotic {get,aur-download} [-r] ${PACKAGES[@]}

  Download listed packages' sources from AUR. Uses repoctl.

• chaotic cl{,eanup} ${INPUTDIR}

  Safely deletes old package sources.

• chaotic help {syncthing,rsync}

  Instructions to the mirroring services. RSync is one-way (primary->cluster) only, and Syncthing is both ways.

• chaotic routine {hourly,morning,afternoon,nightly,midnight}

  Run the specified routine.

• chaotic routine clean-archive

  When on a primary node, clean up the archive folder.

• chaotic {clean-duplicates,dedup}

  Moves packages which have been replaced by newer (modify time) packages to the archive folder.

• chaotic {clg,clean-logs}

  After a chaotic makepwd, remove successfull and "already built" logs.

• chaotic {cls,clean-srccache} ${PACKAGE}

  Removes cached sources from a specific package.

• chaotic clean-sigs {,-q}

  Moves package files or archive files without their corresponding signature or archive file to the package archive. Files have to be at least 1 hour old to be considered.

• chaotic {srt,sort-logs}

  Unclutters and sorts the raw log directory into easy to read subcategories

• chaotic find-discarded

  Find and print discarded packages. All packages that are in the database, but do not exist in the package list repo, are considered discarded, but have to be deleted manually.

Involved directories

• /var/cache/chaotic/sources/${PACKAGETAG}

  Per-package SRCDEST.

• /var/cache/chaotic/lower/{latest,$DATESTAMP}

  Lowerdirs.

• /var/cache/chaotic/cc/{PACKAGETAG}

  Per-package ~/.ccache.

• /var/cache/chaotic/packages

  Container-shared pacman's cache.

• /var/lib/chaotic/interfere

  Cloned version of interfere repository

Dependencies

pacman -S --needed base-devel git arch-install-scripts repoctl fuse-overlayfs rsync python-telegram-send openssh

One needs an active mirror or a setting (in /etc/chaotic.conf) like this:

export CAUR_URL='https://builds.garudalinux.org/repos/chaotic-aur/x86_64'
export REPOCTL_CONFIG='/etc/chaotic/repoctl.conf'
export CAUR_REPOCTL_DB_URL="${CAUR_URL}/chaotic-aur.db.tar.zst"
export CAUR_REPOCTL_DB_FILE="/tmp/chaotic/db.tar.zst"

To create a gpg key for the root user refer to this ArchWiki article for more information. If you find problems when using "sudo", read the "su" subsection. Then generate a ssh keypair for the root user.

sudo ssh-keygen

The ssh public key (cat /root/.ssh/id_rsa.pub) then needs to be added to the primary servers' root authorized keys (/root/.ssh/authorized_keys). After that follow these instructions to export the gpg public key. This key will have to be uploaded to keyserver.ubuntu.com in order for the key to be verified. Then, configure it as follows in /etc/chaotic.conf, like this:

export CAUR_DEPLOY_PKGS="/var/www/chaotic-aur/x86_64"
export CAUR_URL="http://localhost:8080/chaotic-aur/x86_64"
export CAUR_SIGN_KEY='8A9E14A07010F7E3'
export CAUR_TYPE='cluster'
export REPOCTL_CONFIG='/etc/chaotic/repoctl.toml'

You'll find more options in src/chaotic first lines.

Supported type values are primary, cluster, and dev.

Furthermore, a valid .gitconfig needs to be supplied in /root/.gitconfig to allow bumping packages via chaotic bump with a meaningful email and name supplied:

[user]
email = "[email protected]"
name = "Chaotic Dragon (Node) ☭"

To allow pushing to the interfere repo, a fitting ssh key also needs to be added to authenticate the node.

To have clean logs & less bandwidth usage /etc/pacman.conf settings need to be adjusted:

• Enable NoProgressBar

• Use Server = file:///path-to-local-repo as repo link if a local mirror is available

• Don't use ILoveCandy

To deploy faster replace openssh with openssh-hpn on all nodes (adds performance-related patches). It should be noted that openssh-hpn recently changed its configuration and behavior and therefore doesn't provide a 1:1 replacement for openssh anymore.

Installation

Install dependencies, then:

sudo groupadd chaotic_op
sudo usermod -aG chaotic_op $(whoami)

make build && sudo make install

Lint

pacman -S --needed yarn shellcheck
yarn install
yarn run lint

toolbox's Issues

[Request] Log repo and interfere

Duplicate of #81.

[Request] Interfere bypass and testing

Would be nice if there were a way to bypass (broken) interferes or to test a (not yet merged) interfere. When testing interferes, the package directory shouldn't be cleaned because the build is likely to be run multiple times. (This could also be useful for checking artifacts when debugging the PKGBUILD.)

chaotic mkd -n package # bypass interfere
chaotic mkd -i package # test interfere stored someplace like package.int

Use PNPM instead of Yarn

PNPM is faster than yarn and has a lot more customizability than yarn. Would be an excellent choice to speedup the toolchain.

[Request] Add info to logs: pkgbuild origin and interfere

If possible, would be nice if logs indicated pkgbuild source and whether an interfere was used. This would make finding problems easier.

chaotic aur Synchronizing package databases repeatedly

Sometimes I need to run pacman -Syu several times to get all update, because of slow internet connection.

For other repos， if I ran “pacman -Syu”several times consecutively. The data base is only need to down load once. Subsequent invocation will show something like:

 core is up to date
 extra is up to date
 community is up to date

But this is not the case for chaotic, it need to update the database each time. There must be something wrong.

ERROR: Remote key not fetched correctly from keyserver.

# pacman-key --recv-key FBA220DFC880C036 --keyserver keyserver.ubuntu.com
gpg: keyserver receive failed: Server indicated a failure
==> ERROR: Remote key not fetched correctly from keyserver.

First step of instructions on https://aur.chaotic.cx/ fails

Import PGP key 349BC7808577C592? [Y/n] error: required key missing from keyrin"

I get error message Import PGP key 349BC7808577C592? [Y/n] error: required key missing from keyrin even though website says pacman-key --recv-key 3056513887B78AEB --keyserver keyserver.ubuntu.com.

The keys differ. I tried different mirrors.

[Request] universal interfere

Add an interfere that applies to all packages.

Can be used to do some clean-up that should be applied to all packages, like:

• Remove groups.
• Remove replaces.
• Add new-line to the end of PKGBUILD to prevent problems like this.

[Request] Add markers to track modified packages

Currently, manually interfered packages are marked with an optdep on chaotic-interfere. Interfered packages can be listed with: pactree -rsd1 -o1 chaotic-interfere

Automatic corrections and custom pkgbuilds are not marked. Marking them would make finding package sources easier when debugging, as well as improve transparency around packages that differ from AUR.

Suggestions (open to renaming or revision).

• chaotic-autocorrect to mark packages that were automatically modified by Toolbox.

  • Some changes are made blindly. Detecting whether they had any effect may not be worthwhile.

• chaotic-custom to mark packages that use an alternate/custom git repo.

  • Marking by alternate repo allows the change to be made in toolbox, rather than by editing each custom pkgbuild.
  • While split packages are not necessarily "modified". Marking them could make them easier to manage.
  • Maybe the git url could be included in the optdep description.

    chaotic-custom: https://example.com/example/package.git
    

Related: chaotic-aur/packages#2938

[Discussion] Base Singularity on base-devel instead of base-master

I propose some change on the Singularity file for the toolbox, i don't know why it was setup that way (and i think there was a very good reason behind it, so clear me up if i'm wrong).
But since it in a clean docker anyway, there shouldn't be any issues to go directly grab the base-devel docker container. And with that we could remove the pacman line to go grab it.

From: gitlab.archlinux.org:5050/archlinux/archlinux-docker:base-master
pacman -Syyu --noconfirm --noprogressbar --quiet
pacman -S --noconfirm --noprogressbar --quiet base base-devel

to

From: gitlab.archlinux.org:5050/archlinux/archlinux-docker:base-devel
pacman -Syyu --noconfirm --noprogressbar --quiet

aarch64 builds?

Hi, would you consider aarch64 builds? I think it makes sense because those machines are usually quite under-powered and compilation takes forever.

Appstream Stores support for chaotic-aur packages

Is it possible to support appstream for chaotic-aur packages so that user could see the app infomation on software centre which uses appstream backend (e.g KDE Discover, Gnome Software etc.) and installs the package from it.

[Request] Update PKGBUILD version info of -git packages match last build

Would also be nice for packages with pkgver() to set pkgver, pkgrel, epoch to match the last build. That should allow bumps to work for -git packages.

The presence of pkgver() would be checked only. It does not have to be run. The problem with -git packages is pkgver and pkgrel often contain some old version that is updated by pkgver(), which prevents bumps from working.

Delta update support (pacman)

Hey guys,

I did some testing on how efficient delta updates (if implement) would be in pacman. And since you're having one of the biggest packages around, I thought you might be interesting to take a look at the findings and take part in the discussion:

https://lists.archlinux.org/pipermail/pacman-dev/2022-May/025568.html

The numbers are super promising, with an average saving of 40% on "source code heavy" packages and sometimes above 99% for data heavy packages with applying times below 1 second on modern computers.

Best regards,

Ruben