dchirikov/luna

Provisioning tool for clusters

Disclaimer

Luna is a baremetal provisioning tool that uses an image based approach. It delivers full images of operating systems and not a 'recipe' on how to configure one. It also dramatically speeds up installation time, and reduces administrative efforts.

Overview

Luna uses the BitTorrent protocol to provision nodes. As such, every booting node helps the others to boot.

Once a node is fully booted it stops being a torrent seeder and other nodes can no longer use it to download the image. The torrent client only acts in the initrd environment.

Luna does not require any additional services to run on a node. By default it changes very a limited number of files on provisioned nodes. It us usually limited to /etc/hostname and /etc/sysconfig/network-scripts/ifcfg-* files.

Number of nodes	Time for cold boot, min	xCAT cold boot, min
1	3	9
36	4	26
72	4	53

Image size is 1GB. Provisioning node is equiped with a 1Gb ethernet interface.

In a cluster of 300 nodes. Boot time using luna has been measured to be aproximately 5 minutes. This includes BIOS POST procedures and all starting systemd services.

Getting started

Let's assume you have a server using the IP address 10.30.255.254 to provision the cluster

Server preparation

Build RPM

yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum -y install wget python-docutils gcc-c++ rb_libtorrent-devel boost-devel make rpm-build redhat-rpm-config
git clone https://github.com/clustervision/luna
cd luna
make rpm

Note. Instead of building luna from scratch you can easily configure repository and install everything from it:

curl https://updates.clustervision.com/luna/1.2/centos/luna-1.2.repo > /etc/yum.repos.d/luna-1.2.repo
yum install luna

Install hostlist

Source code is available here

wget https://www.nsc.liu.se/~kent/python-hostlist/python-hostlist-1.17.tar.gz
rpmbuild -ta python-hostlist-1.17.tar.gz
yum -y install python-hostlist-1.17-1.noarch.rpm

Install Luna

yum -y install rpm/RPMS/x86_64/luna-[0-9]*rpm

Configure DB credentials

vim /etc/luna.conf

Setup environment

[ -f /root/.ssh/id_rsa ] || ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ''

# Disable SELINUX

sed -i -e 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0

You can also choose not to disable selinux but install luna-selinux package

# Configure xinetd

mkdir /tftpboot
sed -e 's/^\(\W\+disable\W\+\=\W\)yes/\1no/g' -i /etc/xinetd.d/tftp
sed -e 's|^\(\W\+server_args\W\+\=\W-s\W\)/var/lib/tftpboot|\1/tftpboot|g' -i /etc/xinetd.d/tftp
cp /usr/share/ipxe/undionly.kpxe /tftpboot/luna_undionly.kpxe

# Configure nginx and named

cp /usr/share/luna/nginx-luna.conf /etc/nginx/conf.d/luna.conf

echo 'include "/etc/named.luna.zones";' >> /etc/named.conf
touch /etc/named.luna.zones

# Enable and start services

systemctl enable nginx
systemctl enable mongod
systemctl enable named
systemctl enable xinetd

systemctl restart xinetd
systemctl restart mongod
systemctl restart nginx
systemctl restart named

Generate a CentOS image

export OSIMAGE_PATH=/opt/luna/os/compute
mkdir -p ${OSIMAGE_PATH}/var/lib/rpm
rpm --root ${OSIMAGE_PATH} --initdb
yum -y install yum-utils
yumdownloader centos-release
rpm --root ${OSIMAGE_PATH} -ivh centos-release\*.rpm
yum --installroot=${OSIMAGE_PATH} -y groupinstall Base
yum --installroot=${OSIMAGE_PATH} -y install kernel
yum --installroot=${OSIMAGE_PATH} -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum --installroot=${OSIMAGE_PATH} -y install luna-client*.rpm

Setup sshd, paswordless access and password for the root user in osimage

mkdir ${OSIMAGE_PATH}/root/.ssh
chmod 700 ${OSIMAGE_PATH}/root/.ssh

mount -t devtmpfs devtmpfs ${OSIMAGE_PATH}/dev/
chroot ${OSIMAGE_PATH}

ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
abrt-auto-reporting enabled
passwd
exit
umount ${OSIMAGE_PATH}/dev/

cat /root/.ssh/id_rsa.pub >> ${OSIMAGE_PATH}/root/.ssh/authorized_keys
chmod 600 ${OSIMAGE_PATH}/root/.ssh/authorized_keys

Configure a new luna cluster

luna cluster init --frontend_address 10.30.255.254
luna network add -n cluster -N 10.30.0.0 -P 16
luna cluster makedhcp --network cluster --start_ip 10.30.128.1 --end_ip 10.30.140.255
systemctl start lweb ltorrent
systemctl enable lweb ltorrent
luna osimage add -n compute -p ${OSIMAGE_PATH}
luna osimage pack compute
luna bmcsetup add -n base
luna network add -n ipmi -N 10.31.0.0 -P 16
luna switch add -n switch01 --oid .1.3.6.1.2.1.17.7.1.2.2.1.2 --network ipmi --ip 10.31.253.21
luna group add -n compute -o compute -N cluster
luna group change compute -b base
luna group change -n compute --bmcnetwork --setnet ipmi
luna group change compute -i BMC -A
luna group change compute -i BMC --setnet ipmi
luna node add -g compute
luna cluster makedns

Please note that group and corresponding node has interface named BOOTIF. This is special placeholder for interface connected to provision network. If interface is know it can be renamed or recreated. Another placeholde is BMC. It is used in ipmitool commands to set up BMC interface.

In service mode you can perform an inventory of the interfaces, local disks, BMC features

luna node change -n node001 --service y

(Optional) Configure storage partitioning

You can boot the nodes in diskless mode, or write your own partitioning script using:

luna group change -n compute --partscript -e

Sample partitioning script for a device called /dev/sda:

parted /dev/sda -s 'mklabel msdos'
parted /dev/sda -s 'rm 1; rm 2'
parted /dev/sda -s 'mkpart p ext2 1 256m'
parted /dev/sda -s 'mkpart p ext3 256m 100%'
parted /dev/sda -s 'set 1 boot on'

mkfs.ext2 /dev/sda1
mkfs.ext4 /dev/sda2

mount /dev/sda2 /sysroot
mkdir /sysroot/boot
mount /dev/sda1 /sysroot/boot

(Optional) Install a bootloader on the nodes using a postscript

cat << EOF | luna group change -n compute --post -e

mount -o bind /proc /sysroot/proc
mount -o bind /dev /sysroot/dev
chroot /sysroot /bin/bash -c "/usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg; /usr/sbin/grub2-install /dev/sda"
umount /sysroot/dev
umount /sysroot/proc
EOF

Add a node to the cluster

luna node add -g compute

A node name will be automatically generated using the default nodeXXX format

luna node change -n node001 -s switch01
luna node change -n node001 -p 1

Start luna's services

systemctl ltorrent start
systemctl lweb start

Check if is working properly

curl "http://10.30.255.254:7050/luna?step=boot"
wget "http://10.30.255.254:7050/boot/compute-vmlinuz-3.10.0-327.10.1.el7.x86_64"
curl "http://10.30.255.254:7050/luna?step=install&node=node001"

Also it is possible to fetch install and boot scripts for the node usin luna CLI:

luna node show node001 --script boot
luna node show node001 --script install

Update DHCP and DNS configurations

luna cluster makedhcp -N cluster -s 10.30.128.1 -e 10.30.140.255
luna cluster makedns

Boot a node

Luna supports multiple modes of booting a node:

• Booting from localdisk:

luna node change -n node001 --localboot y

• Booting into service mode for diagnostics:

luna node change -n node001 --service y

• Configure the BMC when booting:

luna node change -n node001 --setupbmc y

luna's Issues

Remarks on installation of RHEL6

We have a large cluster (>800 nodes) that still use a RHEL6 based distribution (yeah, it's old). I have been trying to set up luna for the last week to get rid of the old deploy software (which is very slow and unreliable) and now it works. These notes are just in case my work is useful for other people (or just for posterity). I had make the following changes:

• luna/osimage.py
  The dracut (from dracut-004-388 rpm) syntax is different, for example it doesn't know the --kver option. Instead, the kernel version is the last argument:
  # dracut_cmd = (['/usr/sbin/dracut', '--force', '--kver', kernver] +
# modules_add + modules_remove + drivers_add +
# drivers_remove + [tmp_path + '/' + initrdfile])
dracut_cmd = (['/usr/sbin/dracut', '--force'] +
['--add', " ".join(modules_add) ] + ['--omit', " ".join(modules_remove)] + drivers_add + drivers_remove + [tmp_path + '/' + initrdfile] + [kernver])

Also had to tweak the argument to --add and --omit there a bit.
I do not know how stable the dracut syntax is, maybe it would be an idea to define a template of this in the database, so this was changeable without touching the code.
Since this dracut is so old, I also had to create "check" and "install" files from the contrib/dracut/95luna/module-setup.sh file, but this is straightforward.

• Compatibility symlinks:
  In the scripts luna-start.sh and templ_install.sh, external programs are called with full path, i.e.:
  luna-start:109: /usr/bin/ps ...
  Since ps is in /bin/ on this old Linux, I created a symlink. Also for tar and dracut.
  Personally, I prefer to set the PATH at the start of the script properly and then use command -p to save the full path in a variable. Then I can also check if this executable is actually present.

• contrib/dracut/95luna/luna-start.sh and templates/templ_install.cfg:
  At first, dracut only installed dash, not bash, into the initrd. dash wouldn't understand the function definitions, e.g.,
  luna-start.sh:25: function are_macs_equal () {
  It did not understand the "function" keyword and I think it also didn't like the space between the function name and the (empty) argument parentheses.
  For compatibility, I think omitting the "function" and deleting the whitespace would be a good idea. By the way, the functions in templ_install.cfg don't have parentheses:
  templ_install.cfg:6: function update_status {
  Maybe it would also be a good idea to use /bin/bash instead of /bin/sh in luna-start.sh when calling the install script:
  luna-start.sh:162: /bin/sh /luna/install.sh && RES="success"
  since bash is also used explicitly in other places, e.g. in the shebang of luna-start.sh (and also in the shebang of templ_install.cfg, but this is obviously not used when running the install script)

• Compiling a recent rb_libtorrent (1.0.9) with the old gcc 4.4.7 was also not directly successful, but it worked eventually. Considerably older version did not work well together with ltorrent (performance was very low).

• Lastly, I build these images on a Centos 7 node, which is means yum and rpm are much more recent, so I have to rebuild the RPM db (rm -f /var/lib/rpm/Name && rpm --rebuilddb) once a node is installed or the old rpm will not be able to read the db.

• One feature I added was to have the option to change the localboot parameter in the same way the status is updated using curl. Our nodes are set to boot from the network first. To deploy, localboot must be off, but not every reboot should be a deploy. I put one function in node.py (update_localboot) and a branch in get() in manager.py which sets localboot to 'yes'. This is triggered from a function update_localboot (which calls curl) in templ_install.cfg, which is called from the postscript script after the boot loader is installed:
  curl -s [...]/luna?step=configure&node={{ p['name'] }}&localboot=$1
  But maybe this is superseded by the REST API being developed?

I don't expect any changes to your upstream for such an old system (which is hopefully scrapped next year). But if any of the above sound like they a good idea, I'll be happy to send a patch.

kernel cmdline different for some nodes

I am using luna to install nodes in a cluster made of heterogeneous nodes (completely different vendors). Some nodes need kernel option console=ttyS0 some console=ttyS1. I tried changing options in the Console Redirection settings in the BIOS, but to no avail. And you cannot specify both console=tty[0,1] at the same time, due to a limitation of the Linux kernel.

Otherwise, all nodes use the same image (most configuration is done on first by boot anyway).
Since the kernel options are tied to the osimage, I clone the image now and then change the kernopts parameter. The nodes are put in different groups and then each group gets a different (cloned) image.
Is this the way to do it?

It would be easier if nodes could add/override kernel options. Or that images could be cloned "lightly", where no files are copied, just reused.

Add support for diskless nodes

Please, add support for diskless nodes, like stateless/statelite in xCAT.

luna2 API work

Hello team,
I just wanted to express interest in luna2 branch where your team seem to work on an API for the Luna baremetal provisioning tool. Looking forward to it becoming available in master/stable branch.

The project I'm involved in, Airship, has a component Airship-Drydock. It is a small orchestrator, which takes a YAML document as an input, and follows it to deploy baremetal nodes with a help of baremetal node deployment tool and do post-install configuration on them.

At the moment Airship-Drydock supports only Ubuntu MAAS as a node driver. There is an ongoing work on blueprint with plans to support OpenStack Ironic as a driver for the same purpose. Drydock talks to the API of the deployment tool, that is why I'm particularly interested in the API development you are working on.

Keeping an eye on Luna development, I would like to thank you for your great project, and hope to work on inclusion of Luna with it's unique features (esp. rapid deployment speed) as a possible node driver for the Drydock.

Thank you.