quattor/template-library-standard

template-library-standard's Issues

Change value of 'model' field to real model name in CPU templates

Hi Folks,

I propose that we change the value of the model field in the CPU templates to the real model name of the CPU and assign the current value of model to the name key. This would bring the CPU templates into line with the NIC templates and allow us to more easily use conditional logic on matching a value of CPU model to provide extra config if required. So for example, what I propose is:

Change from:

structure template hardware/cpu/intel/xeon_e5_2650v3;

"manufacturer" = "Intel";
"model" = "Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30 GHz";
"speed" = 2300; # MHz
"arch" = "x86_64";
"cores" = 10;
"max_threads" = 20;
"type" = "haswell"; # Intel codename
"power" = 105; # TDP in watts

to

structure template hardware/cpu/intel/xeon_e5_2650v3;

"manufacturer" = "Intel";
"model" = "xeon_e5_2650v3";
"name" = "Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30 GHz";
"speed" = 2300; # MHz
"arch" = "x86_64";
"cores" = 10;
"max_threads" = 20;
"type" = "haswell"; # Intel codename
"power" = 105; # TDP in watts

For extra consistency, I also propose changing the case of the manufacturer value to all lower case.

Comments please.

features/cvmfs/client does not compare versions correctly

Stumbled across this the other day, currently the feature compares two strings with < which is somewhat non-nonsensical even when the strings are simple dotted decimals let alone values such as 2.1.19-1.

 if (CVMFS_CLIENT_VERSION < '2.1' && ! is_defined(SELF['cvmfs'])) {

This looks like another place where pkg_compare_version could be used (as discussed in quattor/template-library-core#90), especially as these are indeed package versions.

kernel_version_arch: remove use of kernel magic version

In 14.8 ncm-grub will not support the magic version YUM-managed (quattor/configuration-modules-core#237). Instead /system/kernel/version should be left undefined.
This requires quattor/template-library-core#53 first.

Generation of DISK_VOLUME_PARAMS will not work with panc 10.3

e.g. in function filesystem_layout_mod

See quattor/pan#105.

filesystem/config: add esp flag to a UEFI boot partition

As stated in quattor/template-library-core#152, a UEFI boot partition should have both boot and esp flags to be label-type independent. Template library should be updated to set these flags instead of the GPT-specific bios_grub after quattor/template-library-core#153 is merged.

Clean-up os/kernel_version_arch to remove all the errata management

Clean-up os/kernel_version_arch to remove all the errata management and other things that predate the use of YUM.

As discussed in the review of #140.

core machine type does not compile

Hi,

Compilation of the core machine type fails if you try to use it with vanilla-scdb script supplied clean templates or anything else I tried so far. I had to make some changes for the compilation to work, more are probably needed but here is the first one:

--- cfg/standard/machine-types/core.pan 2014-07-15 11:48:25.000000000 +0100
+++ /tmp/scdb-vanilla.bak/cfg/standard/machine-types/core.pan   2014-07-15 11:40:57.000000000 +0100
@@ -144,7 +144,7 @@
 #
 # Add site specific configuration if any
 #
-include { return(GLITE_BASE_CONFIG_SITE) };
+include { if (is_defined(GLITE_BASE_CONFIG_SITE)) if_exists(GLITE_BASE_CONFIG_SITE) ; };

 # Default repository configuration template
 variable PKG_REPOSITORY_CONFIG ?= 'repository/config';

Move hardware/nic/legacy contents to appropriate hardware/nic/vendor

As discussed in #69, templates originally in hardware/nic and moved to hardware/nic/legacy during #69 should have been moved to the appropriate hardware/nic/vendor namespace. The vendor for these templates is mentioned in the name attribute so it should be pretty straightforward to do...

EL8: NetworkManager rebuild resolv.conf at each reboot

In recent updates of EL8 (CentOS Stream), NetworkManager rebuild resolv.conf at each reboot with an inappropriate content, breaking name resolution (and thus ability to fix the problem with Quattor). It doesn't happen on all machines, what triggers it is not clear but it is expected that by default NetworkManager may take control of resolv.conf. dns=none should be set by Quattor in the configuration file to prevent this to happen, as it manages the resolv.conf content.

Several articles can be found about this, most of them on Ubuntu but it is the same NetworkManager...

cvmfs: use metaconfig and make the code more generic

CVMFS configuration currently relies on ncm-filecopy to produce file that would be better managed by ncm-metaconfig (key/value pairs).

Also there is currently different hashes to describe the configuration of different CVMFS servers. We should try to have something more generic, allowing a site to add its specific servers, if any, without changing the code.

Pakiti config

In the template features/pakiti/client/config :

variable PAKITI_RPMS ?= if_exists('config/pakiti/client/config');
variable PAKITI_RPMS ?= 'features/pakiti/client/rpms';
include { PAKITI_RPMS };

should be replaced by something like :

include {if_exists('config/pakiti/client/config');};
variable PAKITI_RPMS ?= 'features/pakiti/client/rpms';
include { PAKITI_RPMS };

Use metaconfig for Pakiti 3 support

As discussed in #50.

frontier file name 'customized.sh'

The configuration of a frontier-squid instance creates "customized.sh" (ending with d), instead of the one expected by frontier-squid, which is "customize.sh"

In line 29 of template-library-standard/features/frontier/config.pan
'/software/components/filecopy/services/{/etc/squid/customized.sh}' = dict(

Is there any reason for this?
The command 'service frontier-squid reload' does not include the configuration from the file "customized.sh" file

way to set lvm raid type when vol type = lvm

I can’t seem to find a way to set raid type for when vol type is lvm. I like be to able to set raid type for lvm vol so I can see something like this appear in profile.

/system/blockdevices/logical_volumes/data1/type = raid1

and get following appear in ks, (the --type raid1)

<snip>
lvm lvcreate -W y --yes -n data1 \
       --type raid1 \
       vg1 \
       -l 99%FREE \
</snip>

Noticed there is lvs_add function, which sets the size, but can’t see a way to set type for lv, therefore I think we need a way to set type similar to way size is being set for vol type lvm. I see md/ approach when type=raid but nothing to set raid type when vol type is lvm.

I think we need something like this to be added to, variable DISK_LV_BY_VG = {

  foreach (l; sz; lv) {
    SELF[l]["volume_group"] = vg;
-    if (sz != -1) {
-        SELF[l]["size"] = sz;
+    if (is_defined(sz['size']) && sz['size'] != -1) {
+        SELF[l]["size"] = sz['size'];
+    };
+    if (is_defined(sz['raid_type'])) {
+       

And lvs_add function modified to accept list of vol groups with options as dict.
i.e modfy lvs_add as

function lvs_add = {
  function_name = 'lvs_add';
@@ -47,8 +47,11 @@ function lvs_add = {

  foreach (l; sz; lv) {
    SELF[l]["volume_group"] = vg;
-    if (sz != -1) {
-        SELF[l]["size"] = sz;
+    if (is_defined(sz['size']) && sz['size'] != -1) {
+        SELF[l]["size"] = sz['size'];
+    };
+    if (is_defined(sz['raid_type'])) {
+        SELF[l]["type"] = sz['raid_type'];

Or have I not looked at the right place?

Rename features/ into feature/

For the sake of consistency with what is done in other repos and with the fact that we don't use plural in directory name (with the exception of machine-types!), it'd be good to rename features/ in this repo into feature. Hopefully impact of the change is mostly internal to the template library... Sites could always add template wrappers to renamed templates in features/ to help with the transition (to be documented).

cvmfs 2.1.20 rpms

The packages defined in features/cvmfs/rpms are failing for a fresh installation of cvmfs version 2.1.20.

The documentation says:

take care of replacing the cvmfs-keys package by the cvmfs-config-default package

The package cvmfs-config-default is installed as a dependency when installing cvmfs. Therefore, I think it should be enough to remove the line

 '/software/packages/{cvmfs-keys}' ?= nlist();

from the template features/cvmfs/rpms.

No LICENSE

Rework maven configuration using YUM

@stdweird reported a successful configuration of Maven with YUM using the repo https://github.com/quattor/release/blob/master/src/scripts/build_all_repos.sh#L122. Should be used to provide a YUM-based Maven installation to replace the SPMA one (cfg/standard/features/maven).

When done, Maven should be optionally configurable in rpms/quattor-development.pan (template-library-os).

frontier feature outdated?

I'm looking for a component to build a Squid proxy (for CVMFS).

I gave a look at frontier feature, but it seems to be a bit outdated:

• uses PKG_ARCH_GLITE while not in the template-library-grid
• uses explicit frontier-squid rpm version number (not yum based spma)

These are easy to fix, but is this feature still maintained? Would we better use something else?

Thanks n advance,

Pierre

cvmfs client is not configured correctly

Last updates in the cvmfs client configuration has added a space after the semi-colon in the definition of the CVMFS_SERVER_URL variable.
For example, in our log we can see:
(download) escaped
http://cernvmfs.gridpp.rl.ac.uk:8000/opt/cms/.cvmfspublished to
%20http://cernvmfs.gridpp.rl.ac.uk:8000/opt/cms/.cvmfspublished
[01-23-2018 16:44:16 CET]
A '%20' string has been added to the url and cvmfs is failing.

hardware/cpu/intel add embedded cpus

Would it be possible to also add embedded type cpus to this template library such as the Atom c2558 and the Celeron 1047UE?

This is because these CPUs are used in the Mellanox SN2100 and SN2700 switches respectively and we would like to be able to manage these with Quattor.

cvmfs: specify which client version to be installed

The current template
template-library-standard/features/cvmfs/rpms.pan
doesn't allow a staged rollout of a new client release across of a batch farm. Once the latest package is added to the repository, any 'aq make' on a worker node for a separate and different fix will install the cvmfs-client package as well.

I propose to replace
'/software/packages/{cvmfs}' ?= dict();
with something else i.e. pkg_repl where one can make use of CVMFS_CLIENT_VERSION

Thanks,
Catalin

nic/by_driver/bnx2* templates: improperly designated as Qlogic drivers

I saw that in the new nic/by_driver/bnx2*templates, the name starts with QLogic.... For me (I think I am the original author of the template), bnx was standing for Broadcam NetExtreme, thus the original description in the name attribute. It is very minor but does anybody has a strong idea on this point?

16.8.0-rc1: wrong namespace for some network cards and CPUs

After the changes introduced to move templates for network cards to by-driver, some have been forgotten and still have legacy namespace. Also some CPUs have a namespace casing mismatch with the template name...

[panc-check-syntax] syntax error [/tmp/scdb-vanilla/cfg/standard/hardware/cpu/intel/core_duo_e8400.pan:1.1-1.53]
[panc-check-syntax] misplaced or misnamed template; relative directory and name must match 'hardware/cpu/intel/core_duo_E8400'
[panc-check-syntax] 
[panc-check-syntax] syntax error [/tmp/scdb-vanilla/cfg/standard/hardware/cpu/intel/xeon_e5140.pan:1.1-1.49]
[panc-check-syntax] misplaced or misnamed template; relative directory and name must match 'hardware/cpu/intel/xeon_E5140'
[panc-check-syntax] 
[panc-check-syntax] syntax error [/tmp/scdb-vanilla/cfg/standard/hardware/cpu/intel/xeon_e5150.pan:1.1-1.49]
[panc-check-syntax] misplaced or misnamed template; relative directory and name must match 'hardware/cpu/intel/xeon_E5150'
[panc-check-syntax] 
[panc-check-syntax] syntax error [/tmp/scdb-vanilla/cfg/standard/hardware/nic/by_driver/e100.pan:1.1-1.44]
[panc-check-syntax] misplaced or misnamed template; relative directory and name must match 'hardware/nic/legacy/e100'

This is a showstopper for 16.8.0 release.

cvmfs/client feature can set /software/components/profile/env to undefined in certain circumstances

When neither of the two variables in this if statement (features/cvmfs/client.pan#L317) exist the following occurs on build:

Buildfile: /opt/aquilon/etc/build.xml
-verify.object.profile:
delete.object.dep:
compile.object.profile:
     [panc] 1/1 template(s) being processed
     [panc] validation error [/var/quattor/cfg/domains/workerplay/profiles/vm47.nubes.stfc.ac.uk.tpl]
     [panc] element at /{ components, { accounts, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, groups, { nagios, { gid, 201 } }, kept_groups, { adm, , amanda, , amandabackup, , apache, , apacheds, , bin, , cimsrvr, , daemon, , dhcp, , dhcpd, , dip, , disk, , exim, , floppy, , ftp, , games, , gdm, , gopher, , haldaemon, , ident, , kmem, , ldap, , lemon, , lock, , lp, , mail, , mailnull, , man, , mem, , mysql, , nagios, , news, , nfsnobody, , nginx, , nobody, , nscd, , ntp, , oprofile, , pcap, , pegasus, , plugdev, , postdrop, , postfix, , postgres, , quagga, , radiusd, , radvd, , root, , rpc, , rpcuser, , rpm, , saslauth, , services, , sindes, , slocate, , smmsp, , squid, , sshd, , stap-server, , stapdev, , stapusr, , sys, , tcpdump, , tomcat, , tty, , usb, , users, , utmp, , uucp, , uuidd, , vboxusers, , vcsa, , webalizer, , wheel, , wine, , xfs,  }, kept_users, { abrt, , adm, , amanda, , amandabackup, , apache, , apacheds, , avahi, , avahi-autoipd, , bin, , daemon, , dbus, , dhcp, , dhcpd, , dialout, , distcache, , exim, , ftp, , games, , gdm, , gopher, , haldaemon, , halt, , hpglview, , ident, , ldap, , lemon, , lp, , mail, , mailnull, , man, , mysql, , nagios, , named, , news, , nfsnobody, , nginx, , nobody, , nscd, , nslcd, , ntp, , operator, , oprofile, , pcap, , pegasus, , postfix, , postgres, , quagga, , radiusd, , radvd, , root, , rpc, , rpcuser, , rpm, , saslauth, , screen, , services, , shutdown, , sindes, , smmsp, , squid, , sshd, , stap-server, , sync, , tcpdump, , tomcat, , tss, , utempter, , uucp, , uuidd, , vcsa, , webalizer, , xfs,  }, preserved_accounts, dyn_user_group, remove_unknown, false, rootpwd, $6$PDQDq/SO$UIclajVXXXLYlYWywVIfRxaRPvCRUfKTf7FsYOrzOZY3pQuWWISxITZf6EOCET3QjPazO0CpVyFVZVFcK7OiG0, shadowpwd, true, users, { nagios, { comment, nagios, groups, [ nagios ], homeDir, /home/tier1/nagios, shell, /bin/sh, uid, 201 } }, version, 14.10.0 }, altlogrotate, { active, true, configDir, /etc/logrotate.d, configFile, /etc/logrotate.conf, dependencies, { pre, [ spma ] }, dispatch, true, entries, { cernvmfs-fsck, { compress, true, create, true, frequency, weekly, ifempty, true, missingok, true, pattern, /var/log/cvmfs-fsck.log, rotate, 2 }, fetch-crl-cron, { compress, true, create, true, frequency, monthly, ifempty, true, missingok, true, pattern, /var/log/fetch-crl-cron.ncm-cron.log, rotate, 12 }, global, { compress, true, create, true, frequency, weekly, global, true, include, /etc/logrotate.d, rotate, 4 }, wtmp, { create, true, createparams, { group, utmp, mode, 0664, owner, root }, frequency, monthly, global, true, pattern, /var/log/wtmp, rotate, 1 } }, version, 14.10.0 }, authconfig, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, enableforcelegacy, false, method, { files, { enable, true }, nis, { domain, csf, enable, true, servers, [ nis0.gridpp.rl.ac.uk, nis1.gridpp.rl.ac.uk, nis2.gridpp.rl.ac.uk ] } }, passalgorithm, sha512, safemode, false, startstop, true, usecache, true, usemd5, true, useshadow, true }, autofs, { active, true, dispatch, true, maps, { cvmfs, { enabled, true, mapname, /etc/auto.cvmfs, mountpoint, /cvmfs, preserve, true, type, program }, home-tier1, { enabled, true, entries, { _2a, { location, nfs1.gridpp.rl.ac.uk:/home/tier1/&, options, -rw,intr,rsize=8192,wsize=8192,actimeo=60,addr=130.246.183.1 } }, mapname, /etc/auto.home-tier1, mountpoint, /home/tier1, options, , preserve, false, type, file }, misc, { enabled, true, entries, { kickstart, { location, install.gridpp.rl.ac.uk:/kickstart, options, -ro,soft,rsize=8192,wsize=8192 } }, mapname, /etc/auto.misc, mountpoint, /misc, options, , preserve, false, type, file }, rutherford, { enabled, true, entries, { _2a, { location, &.data.rl.ac.uk:/rutherford/&, options, -intr,retrans=5,timeo=20,rw,hard,nosuid,rsize=8192,wsize=8192 } }, mapname, /etc/auto.rutherford, mountpoint, /rutherford, options, , preserve, false, type, file }, stage, { enabled, true, entries, { _2a, { location, &.stage.rl.ac.uk:/stage/&, options, -intr,retrans=5,timeo=20,rw,hard,nosuid,rsize=8192,wsize=8192,acregmin=30,acregmax=180,acdirmin=30,acdirmax=180 } }, mapname, /etc/auto.stage, mountpoint, /stage, options, , preserve, false, type, file } }, preserveMaster, false }, ccm, { active, true, cache_root, /var/lib/ccm, configFile, /etc/ccm.conf, debug, 0, dependencies, { pre, [ spma ] }, dispatch, true, force, 0, get_timeout, 30, lock_retries, 3, lock_wait, 30, profile, http://aquilon.gridpp.rl.ac.uk/profiles/vm47.nubes.stfc.ac.uk.json, retrieve_retries, 3, retrieve_wait, 30, version, 14.10.0, world_readable, 0 }, cdp, { active, true, configFile, /etc/cdp-listend.conf, dependencies, { pre, [ spma ] }, dispatch, true, fetch, /usr/sbin/ccm-fetch, fetch_smear, 30, version, 14.10.0 }, chkconfig, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, service, { acpid, { on, , startstop, true }, atd, { on, , startstop, true }, auditd, { on, , startstop, true }, autofs, { on, , startstop, true }, cdp-listend, { on, , startstop, true }, crond, { on, , startstop, true }, cups, { off,  }, cvmfs, { on, , startstop, false }, edac, { on, , startstop, true }, fetch_2dcrl_2dcron, { on, , startstop, true }, irqbalance, { on, , startstop, true }, kudzu, { off,  }, lldpd, { on, , startstop, true }, ncm-cdispd, { on, , startstop, true }, network, { on, , startstop, true }, nrpe, { on, , startstop, true }, ntpd, { on,  }, psacct, { on,  }, rpcidmapd, { on, , startstop, true }, rsyslog, { on, , startstop, true }, sshd, { on, , startstop, true }, yum, { off, 2345 } } }, cron, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, entries, [ { command, /usr/local/bin/healthcheck_systemtemp, frequency, * * * * *, name, healthcheck_systemtemp, user, root }, { command, sleep 340 && /usr/sbin/pakiti2-client, frequency, @reboot, name, pakiti2-startup, user, root }, { command, /usr/sbin/magdb-discover, name, magdb-discover, timing, { hour, 13, minute, 30, smear, 30 }, user, root }, { command, /usr/sbin/fetch-crl  --no-check-certificate --loc /etc/grid-security/certificates -out /etc/grid-security/certificates -a 24 --quiet, frequency, AUTO 3,9,15,21 * * *, name, fetch-crl-cron, user, root }, { command, /usr/local/bin/cvmfs_fsck, comment, Cron to check and fix cvmfs cache integrity, env, {  }, frequency, 00 09 * * *, name, cvmfs-fsck, user, root } ], securitypath, /etc, version, 14.10.0 }, dirperm, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, paths, [ { owner, atlasprg:atlas, path, /pool/atlas/recovery/, perm, 0775, type, d }, { owner, atlasprg:atlas, path, /etc/atlas/, perm, 0775, type, d }, { owner, root:root, path, /etc/cms/PhEDEx, perm, 0755, type, d }, { owner, root:root, path, /etc/cms/JobConfig, perm, 0755, type, d }, { owner, cvmfs:cvmfs, path, /pool/cache/cvmfs2/, perm, 0755, type, d } ], version, 14.10.0 }, filecopy, { active, true, dependencies, { pre, [ spma ] }, dispatch, true, forceRestart, false, services, { _2fetc_2fatlas_2fsetup_2esh_2elocal, { backup, false, config, # This file is for local ATLAS overrides, please modify this via quattor
     [panc]
     [panc] , forceRestart, false, owner, atlasprg:atlas, perms, 0755 }, _2fetc_2fbash_5fcompletion_2ed_2fquattor, { backup, true, config, # Bash completion for unified quattor command wrapper

New frontier feature needed

As reported in #47, the frontier feature we were shipping was no-longer working.
This feature was removed in #51 (f3fb7e4), we need a new feature to replace it (possibly based on the removed one).

Repository is a fork of guillaume's

This should be the other way around!

16.10.0-rc1 fetch-crl/config: function matches used instead of match

In last changes, a typo led to use matches() instead of match() in a test which doesn't return a boolean...

Showstopper for 16.10/0-rc1.

Passing mount and mkfs options to filesystem/config

We use the filesystem/config from the standard library.
We would like to create a filesystem using this code in a FILESYSTEM_LAYOUT_CONFIG_SITE template :

variable DISK_VOLUME_PARAMS = {
    t = dict();
    t['osd'] = dict(
        'size',         -1,
        'mountpoint',   '/var/lib/ceph/osd/vdb',
        'fstype',       CEPH_FS,
        'type',         'partition',
        'device',       'vdb1',
        'preserve',     true,
        'mkfsopts',     CEPH_DISK_OPTIONS[CEPH_FS]['mkfsopts'],
        'mountopts',    CEPH_DISK_OPTIONS[CEPH_FS]['mountopts'],
    );
    t;
};

The first problem is that the attributes 'mkfsopts' and 'mountopts' are not taken into account. I've tried to workaround this problem by adding the necessary code in filesystem/config, but then I fall on a second problem : the mount options are not taken into account in the generation of the /etc/fstab.