nrel/hpc

A collection of various resources, examples, and executables for the general NREL HPC user community's benefit. Use the following website for accessing documentation.

This repository serves as a collection of walkthroughs, utilities, and other resources to improve the NREL HPC user's quality of life, both novice and veteran.

We are here to help: If you need help with a specific issue or would like to see a topic covered please open an issue. If you have materials that could be useful for the NREL community, please see our contributing guidelines, and open a pull request.

How to Use This Repository

1. Login to an NREL HPC system
2. From your login shell, run git clone https://github.com/NREL/HPC
3. View the README.md within any directories that interest you, or see the table of contents below.

Table of Contents

🎓 General Tutorials

• Markdown
• Bash
• Git
• Transferring Files
• File Systems
• JupyterHub
• GPU
• Software Environments
• Building MPI Applications
• Building Packages
• Optimized Tensorflow
• HPC Benchmarking: STREAM

🐦 Kestrel Examples

📈 Applications

• Gaussian
• Lammps
• Paraview
• Plexos
• Plexos Quick-start
• VASP

ℹ️ Programming Languages

• C
• Fortran
• Julia
• Python

🔧 Utilities and Tools

• Sample Slurm Batch Scripts

Contributing

Please see our contribution guidelines for a rundown on how we'd like the contents of this repository to be formatted.

hpc's Issues

Link doesn't exist

HPC/general/beginner/bash/ has a link to https://nrel.github.io/HPC/bash that doesn't exist

Globus docs differ

The docs at https://nrel.github.io/HPC/Documentation/Data-and-File-Systems/Transferring-Data/globus/ and https://www.nrel.gov/hpc/globus-file-transfer.html differ in particular with using the CLI.

Also Globus does not appear to be installed on Eagle like described in nrel.gov docs?

Review maintainer's guide

• How to structure working groups meetings
• Guidelines for reviewing pull requests/issues

HDF5

Add info on working with HDF5 files, including Lustre striping optimizations.

Update Globus documentation

Test and update Globus docs under Managing Data -> Transferring Files.

Broken link

There's a broken link in HPC/general/beginner/intro-to-linux/ labeled 'HPC University - Introduction to Linux'

FastX docs

Add how to reattach to a FastX session to the docs.

Update favicon

Use favicon from the archive branch

Jupyterhub

Just starting the conversation to get JH docs up on website, and flesh out in repo.

First thing, have the admins seen this? https://github.com/jupyterhub/the-littlest-jupyterhub
I hadn't seen this before; since we already have an instance, this would only be interesting if it lowered management overhead dramatically or provided options more easily than the current instance. We're well within the 100-user limit.

Second, who is part of the "Jupyterhub" team now? We probably need >= 1 ACO person who can make changes to the instance, and 1 HPC Apps person in charge of usage/documentation.

Pull NREL branding down from gh-pages

Broken Links

Broken links in HPC/general/intermediate/building-mpi-applications/

Visible merge conflicts in Swift documentation

https://nrel.github.io/HPC/Documentation/Systems/Swift/running/

Several places in this document have indications of merge conflicts, especially in the code blocks.
e.g.:

ml intel-oneapi-compilers
ml intel-oneapi-mpi
export I_MPI_PMI_LIBRARY=/nopt/nrel/apps/210728a/level01/gcc-9.4.0/slurm-20-11-5-1/lib/libpmi2.so
<<<<<<< HEAD
=======
export UCX_TLS=all
>>>>>>> auxsys

<<<<<<< HEAD
srun --mpi=pmi2  ./a.out -F
=======
srun   ./a.out -F
>>>>>>> auxsys

etc. There are several places with this error that need to be corrected to the right version.

blog post: git-lfs on eagle

So far, User Ops has been advising a conda-forge install. We can either write that up, or look for a better way. also, hidden in the git-lfs docs is the charging model... might be worth mentioning.

Error installing tensorflow from wheel

Hi,

I'm following the steps (https://github.com/NREL/HPC/tree/master/workshops/Optimized_TF) for creating an environment similar to py38tf24.yml but when I reach the point to install the precompiled TensorFlow installation from a wheel, I get the following error:

ERROR: tensorflow-2.4.0-cp38-cp38-linux_x86_64.whl is not a supported wheel on this platform.

Any ideas?

Update README for gh-pages

Update the README in the gh-pages branch to include the docker for running the mk-docs locally

Recommendation for Anaconda

The description of installing Anaconda environments given in (https://nrel.github.io/HPC/languages/python/conda.html is quite thorough. My only suggestion would be to add a small subsection for cloning an already existing environment.

It can be used in case there is an existing environment somewhere on Eagle that works without issues and one wants to use it as a basis for their own environment. By cloning it, one can then add other packages as well and use it for their own projects. This is useful e.g. for cases when one wants to utilize GPUs for reinforcement learning policy training and do not want to create an environment from scratch, which can be proved to be cumbersome (CuDNN drivers, TF-gpu, etc.)

You can clone an environment as:

conda create --clone <env_name> --name <env_name_copy>

Fix building packages navigation

add navigation links to the building packages content...

Broken link

Broken link in HPC/general/intermediate/Jupyterhub/

Git content

pull Git content from gh-pages-archive to the current gh-pages

Design logo

We need a logo for gh-pages site to replace the NREL branding that was taken down.

gh-pages redesign: missing lustre 'best practices' link

https://nrel.github.io/HPC/Documentation/Data-and-File-Systems/File-Systems/filesystemsio/

"Lustre parallel file system: Accessiblle across all nodes. When using this file system please familiarize yourself with the [best practices] section"

[best practices] section link is non-functional; should probably link to:
https://github.com/NREL/HPC/blob/gh-pages/Documentation/Data-and-File-Systems/File-Systems/Lustre/lustrebestpractices/index.html

Node packing

Based on discussion with @diskwarrior document or have a workshop on strategies for packing multiple jobs into a node.

Python 2 legacy code on Eagle

We want to encourage Python 3, but there are cases where Py2 environments or conversions are useful though, particularly for academic offerings written in 2 that will never get ported to 3.

Maybe the user story could be "I need to use an awful Python 2 module that will never be updated to Python 3, how can I do it?" and options (1--find an alternative so you can use up-to-date auxiliary tools; 2--convert to 3 so you can use up-to-date auxiliary tools, include link to https://docs.python.org/3/library/2to3.html; 3--set up a Python 2 custom environment).

Originally posted by @chchang6 in #26 (comment)

Harbor blog post

HPC/applications/plexos-hpc-walkthrough/ is out of date

HPC/applications/plexos-hpc-walkthrough/ directory is all written for a workshop that was held on Peregrine. The content is pretty out of date but not a bad resource. It might not be worth updating but maybe archive it and reference it

Ways of running MPI jobs blog post

mpirun vs srun vs mpiexec

Bash content

pull the bash content from the archive branch

Lustre stripe blog

Summarize Lustre striping benchmarks on Eagle in a blog post - @timkphd

ML content missing from gh-pages

It looks like the ML content was lost when migrating to the new layout.

JupyterLab - How to

We have received recent community requests for JupyterLab. Ideally, we can write up a blog or module about using jupyterlab on europa, or how to setup jupyterlab using conda.

HPC/applications/plexos-quick-start/ internal link

There's an internal link in HPC/applications/plexos-quick-start/ to confluence (https://tickets.hpc.nrel.gov/collab/display/CSCT/Eagle+Plexos+Examples). Can we post this somewhere so that both our internal and external users can access the content?

References to Peregine

References to Peregrine on HPC/general/intermediate/building-packages/

Indexing for announcements and blog

Make the announcements and blog posts searchable

References to Peregrine

There are references to Peregrine in:

HPC/general/intermediate/building-packages/

and

HPC/general/intermediate/software-environment-basics/conda-how-to.md

Update Contrib guide

Processes and layouts have changed. The contributing guidelines should reflect the current state as well as current expectations.

Design changes

Issue for catching inspiration or suggested designs for the github pages site:

• https://support.nesi.org.nz/hc/en-gb
• https://docs.nersc.gov/

Update workshop main readme with latest content

Broken Links

Both links in HPC/general/intermediate/building-mpi-applications/ appear to go nowhere

Broken Links

There are links in HPC/general/intermediate/software-environment-basics/conda-how-to.md that don't point to an active page. I'm not sure if the links 'conda create', 'conda env create' and 'conda clean' were meant to point to anything or if that was a mistake

New MSS info in gh-pages

Transfer this CMS content to mss.md and update with current info: https://www.nrel.gov/hpc/mass-storage-system.html

The policy page (https://www.nrel.gov/hpc/mass-storage-system-policies.html) will stay on the CMS.

Monthly Announcement limitations

_includes/announcement.html seems to be limiting to 1 (good idea for the landing page) but may not be sorting as we expect. After adding more recent announcements, I still only see November.

Also, maybe we add a link to previous announcements?

Pages layout improvements

Asses the layout and navigation of the GH pages site.

How to transfer files needs updated

HPC/general/beginner/how-to-transfer-files the readme needs updated from Peregrine and how to transfer files. Could use another markdown referencing something like FileZilla

Swift content

• Slurm partitions table as currently the only partition is test
• Info regarding GPU nodes not currently available
• Swift File Systems
• Add info on the ssh keys + github
• Mention --cpus-per-task for numpy etc

Search box

Update the search box to say nrel.github.io so it is site specific

Add "Eagle" to systems gh-pages section

Blog post: Dangers of `conda init` when using NREL HPC conda module

Maintainers Guide Additions

Define clear process/expectations for reviewing and merging pull requests

https://www.prestashop-project.org/maintainers-guide/reviewing-pull-requests/
https://www.prestashop-project.org/maintainers-guide/summary-github-lifecycle/

Singularity update

Singularity page still references Peregrine. Also additional suggested changes:

• Add GPU section
• Add a comparison of Docker / Singularity
• Add recommendation of making Docker images which can be pulled as Singularity containers for portability
• Check status of singularity.automounts = True for read/write in new version of Singularity on Eagle
• Steps to create Singularity container from Docker daemon directly
• Stub for NREL's Harbor registry.

Broken Link

There's a broken link in HPC/general/intermediate/software-environment-basics/README.md labeled 'detailed guide'