epcced/epcc-reframe

Repository for Reframe configuration and tests for EPCC systems

EPCC ReFrame

Repository for ReFrame configuration and tests for EPCC systems.

Structure

• configuration/: configuration files for different EPCC systems
• tests/: test library. All tests should, wherever possible, be configured so that they can run on all systems

Executing the test suite

Run all tests

To run the full suite of tests on ARCHER2, run the following commands:

module load reframe
module load epcc-reframe

epcc-reframe -r

To run the full suite of tests on Cirrus, run the following commands:

module load epcc-reframe

epcc-reframe -r

The last command is just a wrapper around ReFrame to call all the tests and use the ARCHER2 configuration. The -r is the run flag in ReFrame.

Run specific tests

Alternatively you may only wish to run a subset of tests. To run a specific test

reframe -C ${EPCC_REFRAME_CONFIG} -c *path-to-a-test* -r

To run all tests in a directory:

reframe -C ${EPCC_REFRAME_CONFIG} -c *path-to-a-test-directory* -R -r

Where EPCC_REFRAME_CONFIG is set by the epcc-reframe module and is the path for the ARCHER2 configuration.

Contributing and style guide

Please see detailed instructions here.

epcc-reframe's Issues

Create a style guide

Having a consistent style guide helps when creating new, or upgrading older tests.
Furthermore, including linter configuration files in the repository would ensure everyone can easily follow the rules.

I am partial to the philosophy behind Black - I think it makes for readable code.
Regarding PEP8, I don't like the 79 character limit (E501) and W503, as I feel the first pressures people into writing more cryptic variable names, and the latter is supposed to be reversed soon. I would suggest ignoring W503 and setting a line-length of 120 characters.

Add module tests

Add simple tests to ensure that all modulepaths that should be visible, are visible (on login and compute nodes)

Add Singularity tests

Add tests that Singularity works as expected:

• Pull image from DockerHub and run on frontend
• Run pre-existing image on compute nodes
• Run prexisting parallel image across multiple compute nodes (and check performance)

Update CP2K LiH HFX Benchmark

Update cp2k benchmark in reframe to run at desired scale for continuous monitoring.

Update existing tests to use new parameterisation setup

ReFrame has changed how parameterised tests are specified in more recent versions and existing tests need to be updated to use the new syntax. Documentation on the new syntax can be found at:

https://reframe-hpc.readthedocs.io/en/stable/tutorial_advanced.html

Add runtime option on Lustre caching to benchio tests

At the moment it is up to the system if Lustre client caching is used or not which can lead to inconsistencies with test performance. We should update the benchio source code to allow the test to switch caching off at runtime for all the different tests.

For MPIIO:

  call MPI_File_sync(fh,ierr)
  call MPI_File_close(fh, ierr)

For HDF5:

  ! Close the file.
  CALL h5fflush_f( file_id , H5F_SCOPE_GLOBAL_F,ierr )
  CALL h5fclose_f(file_id, ierr)

For file-per-process:

! Declare the interface for POSIX fsync function
  interface
    function fsync (fd) bind(c,name="fsync")
    use iso_c_binding, only: c_int
      integer(c_int), value :: fd
      integer(c_int) :: fsync
    end function fsync
  end interface

...file writing...

  ! Flush and sync
  flush(10)
  ret = fsync(fnum(10))

  ! Handle possible error
  if (ret /= 0) stop "Error calling FSYNC"

Add distributed STREAM test

Add a test that runs distributed STREAM on compute nodes: https://github.com/adrianjhpc/DistributedStream

Add placement tests

Add tests to verify that Slurm process/thread placement on compute nodes is behaving as expected.

Test MPMD job submission

Test that MPMD jobs run as expected. How these are implemented depends on the scheduler, for example, Slurm would typically use hetjob to run this type of job.

Configuration of directories for IO

The paths of the filesystem are currently configured in each test doing I/O. At the moment, they all write to a shared z19 directory on each of the archer2 filesystem. That means the tests will break when run from another project account or when run on a machine with different file systems , such as on the TDS.

It would be good for the base directories to be configured at run time. One would add an option to the reframe command or modify a config file and all the tests would perform output within those directories.

CP2K 2GHz test needs lower bound for performance test

epcc-reframe/tests/apps/cp2k/cp2k_check.py

Line 91 in b5eb137

'archer2:compute': {'perf': (335, 12, 'seconds'),

Small error but it should read:

        # Performance test reference values
        self.reference = {
                'archer2:compute': {'perf': (335, 12, -12, 'seconds'),
                }
            }

Add per OST Lustre IO tests

Suggestion from HPC Systems team

Thinking ahead to reframe tests we might want to run - on the spinning-rust lustre, there's one OST per server (and if the site E-1000 we have is anything to go by, this will also be the case for the NVMe) - it might be worth doing some kind of I/O tests on every OST in individually to look for slowdowns

IO performance tests

Add a test that runs the IO500 benchmark (at 10 nodes initially)

Run synthetic MPI tests

We should setup synthetic MPI tests to check that performance and functionality is working as expected. We can use OSU MPI benchmarks for this and build on top of the ReFrame example at:

https://reframe-hpc.readthedocs.io/en/stable/tutorial_deps.html

Decide on missing core functionality tests

Which core functionality testing is missing from the EPCC reframe tests?