This Ansible playbook automates the process of updating SlurmDB, Controller, and Client nodes. The playbook performs tasks such as creating backup directories, verifying configurations, downloading and extracting new releases from GitHub, and restarting Slurm services. https://confluence.dug.com/pages/viewpage.action?spaceKey=DUGIT&title=20240122+Update+London%27s+SLURM+to+slurm-23-02-7-1-DUG-5
- Ansible installed on the control machine.
- SSH access to the Slurm nodes (SlurmDB, Controller, and Client).
- GitHub personal access token for downloading releases.
- SQL key for database access.
get_url_args
: Dictionary containing the destination path for downloaded files.asset_name
: The name of the GitHub release asset.github_pat
: GitHub personal access token.sql_key
: SQL key for database access.backup_dir
: Directory path for backups.gh_release_api
: GitHub API URL for the latest release.
lslurmdb
: Hosts related to SlurmDB.lslurmcontroller
: Hosts related to the Slurm Controller.lslurmclient
: Hosts related to the Slurm Client.
- Ensure the backup directory exists
- Creates backup directories on the SlurmDB and SlurmController nodes.
- name: Ensure the backup directory exists
ansible.builtin.file:
path: "{{ backup_dir }}"
mode: '0755'
state: directory
- Run the sanity check script
- Runs the
sanity_slurm_conf.sh
script to verify the Slurm configuration on the Controller.
- Runs the
ansible.builtin.shell: DEBUG=1 /d/sw/slurm/etc/sanity_slurm_conf.sh
- Extract and compare configuration files
- Compares the current Slurm configuration file and failed if it doesn't match (ensure all dynamic changes have been written)
failed_when: diff_result.rc != 0
when: inventory_hostname in groups['lslurmcontroller']
- Obtain GitHub release details
- Fetches the latest release details from the GitHub API
ansible.builtin.set_fact:
gh_release_asset_url: "{{ gh_release.json.assets | selectattr('name', 'contains', asset_name) | map(attribute='url') | first }}"
asset_dir_name: "{{ gh_release.json.assets | selectattr('name', 'contains', asset_name) | map(attribute='name') | first | regex_replace('.tar.bz2', '') }}"
- Download and extract the release
- Downloads and extracts the specified release asset from GitHub.
ansible.builtin.unarchive:
src: "{{ download_dest.dest }}"
dest: "{{ get_url_args.dest }}"
remote_src: yes
- Stop Slurm services
- Stops the SlurmDB, SlurmController, and SlurmClient services and failed if it still on active state
failed_when: "'restarted' in slurmdbd_status.state"
- Backup the configuration
- Copies the existing configuration to a backup directory.
ansible.builtin.copy:
src: "/var/pool/slurm"
dest: "{{ get_url_args.dest }}/backup_var_spool_slurm_{{ lookup('pipe', 'date +%Y%m%d') }}"
- Create relative symlinks
- Creates symbolic links for the new Slurm version.
ansible.builtin.file:
src: "{{ asset_dir_name }}"
dest: "{{ get_url_args.dest }}/d/sw/slurm/latest"
state: link
- Restart Slurm services
- Restarts the Slurm services on the SlurmDB, SlurmController, and SlurmClient nodes and failed if it stopped
failed_when: "'stopped' in slurmdbd_status.state"
-
Prepare the environment
- Ensure you have the required access tokens and keys in
~/.ssh/github_token.txt
and~/.ssh/sql_token.txt
.
- Ensure you have the required access tokens and keys in
-
Run the playbook
ansible-playbook -i inventory/hosts.yaml playbooks/slurm_rollout.yaml
- Run the playbook
ansible-playbook -i inventory/hosts.yaml playbooks/slurm_rollout.yaml -vv
- This is being done in the tested environment. Please change the
FastX3
services to the relevant services, and ensure the paths are correct as per the configuration. - The playbook assumes that the GitHub token and SQL key files are stored in the
~/.ssh
directory. - Adjust file paths and variables according to your specific setup.
- to move into /d/sw/slurm from its localData
- update services to relevant services
- update
host.yaml
to relevant host - test the key for dumping sql database into backup dir
- sanity check the size of dumped sql database