epcced/eidf-docs

EIDF Services Documentation

EIDF Documentation

The Edinburgh International Data Facility (EIDF) is built and operated by EPCC at the University of Edinburgh. EIDF is a place to store, find and work with data of all kinds. You can find more information on the service and the research it supports on the EIDF website.

This repository contains the documentation for the service and is linked to a rendered version currently hosted on Github pages.

Rendered documentation

• EIDF Documentation (HTML)

How to contribute

We welcome contributions from the EIDF community and beyond. Contributions can take many different forms, some examples are:

• Raising Issues if you spot a mistake or something that could be improved
• Adding/updating material via a Pull Request
• Adding your thoughts and ideas to any open issues

To find out how to contribute, please read CONTRIBUTING.

eidf-docs's Issues

[Documentation]: TOTP now enabled for Ultra2 (and thus CS-2), but not reflected in docs.

What documentation issue are you reporting

Incorrect Documentation

Issue

Issue Reported

Lack of TOTP/MFA docs.

DSC UserDoc should say that the storage is not backed up

The DSC is not backed up and that should be in the UserDoc. In case it helps, here is how Cirrus does it:

https://cirrus.readthedocs.io/en/main/user-guide/resource_management.html?highlight=back-up#backup-policies

[Documentation]: Missing docs for SDF-CS1

What documentation issue are you reporting

Missing Documentation

Issue

As the Cerebras system is already available to people, it would be good to have some documentation as soon as possible that covers:

• How to request an account on SDF-CS1 via EPCC SAFE (may be useful to point to SAFE docs at: https://epcced.github.io/safe-docs/)
• How to log on and transfer data to SDF-CS1 via SSH
• How to setup models for use with CS-1
• How to use Slurm to submit jobs to use CS-1

[Documentation]: CPU Architecture of the GPU Service

What documentation issue are you reporting

No response

Issue

A prospect is keen on knowing the CPU Architecture of the CPUs one gets with the GPU Service. Worth documenting on https://epcced.github.io/eidf-docs/services/gpuservice/ ?

[Documentation]: EIDFGPU Service K8s Lessons

What documentation issue are you reporting

Incomplete Documentation

Issue

Overarching issues to ensure documentation is broadly self-contained and accessible.

• Some overview of what is K8s and the justification behind it being used in the EIDFGPU service.
• Ensure standard K8s terms are defined before use:
  • YAML manifest file
  • Service/deployment/job
  • Helm Chart
• Add that Docker image caching is node dependent and that there is no cluster-wide image repo leading to varying PodInitializing times

[Documentation]: The first time you login on to the VM you MUST go through the vdi

What documentation issue are you reporting

Incomplete Documentation

Issue

If you login directly using:

$ ssh -J [SAFE-username]@eidf-gateway.epcc.ed.ac.uk [VM-username]@[VM/IP]

it will fail with a slightly cryptic:

Enter passphrase for key '/Users/mario/.ssh/id_rsa':
[email protected]'s password:
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Even though the correct pass phrase and password have been used. In order for the ssh access to succeed you MUST apparently go through the vdi first where your password will be reset BEFORE you can login to the VM directly. The documentation should state this.

[Documentation]: Missing MATLAB docs

What documentation issue are you reporting

Missing Documentation

Issue

No MATLAB docs. It's not always clear when users should opt for a high tasks-per-node with a single cpus-per-task or the opposite for their jobs.

[Documentation]: Missing VM sizes for DSC

What documentation issue are you reporting

Missing Documentation

Issue

We should tell users the stock sizes for VMs to make it easier for them to request changes etc.

[Documentation]: URL structures

What documentation issue are you reporting

Formatting

Issue

Issue Reported

This may be a result of the templating system used, but several directory level URLs (which might be useful) result in 404 - e.g.

https://epcced.github.io/eidf-docs/services/virtualmachines/
https://epcced.github.io/eidf-docs/services/datacatalogue/

[Documentation]: SLURM time examples missing

What documentation issue are you reporting

Incomplete Documentation

Issue

Lack of time-based examples with Ultra2 and CS-1 codes. Default is hour which can be a bit short for some large/complex CS-1 codes.

Requires adding a note about it and putting the code in the 3 or 4 example snippets of SLURM batch.

[Documentation]: https://epcced.github.io/eidf-docs/services/jhub/quickstart/ Managed Jupyter Hub docs are blank

What documentation issue are you reporting

No response

Issue

https://epcced.github.io/eidf-docs/services/jhub/quickstart/ Managed Jupyter Hub docs are blank

These pages should at least say "Coming soon" or similar

[Documentation]: Suggestion - enable https://pre-commit.ci/

What documentation issue are you reporting

Formatting

Issue

We already have a .pre-commit-config.yaml in the repo, but this is not run if contributors don't have pre-commit installed locally or if they submit a PR directly in GitHub.

We could add https://pre-commit.ci to the repo, which will ensure the checks are run on every PR. Any formatting changes will also be automatically committed back to the branch by the tool, which is handy.

[Documentation]: Short Video Examples

What documentation issue are you reporting

Missing Documentation

Issue

It would be good to have a video walkthrough of registering, logging in and changing password.

[Documentation]: Portal (application process) doc improvements

What documentation issue are you reporting

Incomplete Documentation

Issue

I drafted some simple, quick changes to the Portal page; the area of interest is VM Resources getting conflated with GPU Service.

https://uoe-my.sharepoint.com/:w:/r/personal/kavousan_ed_ac_uk/Documents/Projects/EIDF/Portal%20userdoc.docx?d=w39c114295108405a84fd4a12ff5c8091&csf=1&web=1&e=UW2bWQ

[Documentation]: Broken Links

What documentation issue are you reporting

Missing Documentation

Issue

Broken links at the bottom of https://epcced.github.io/eidf-docs/access/

[Documentation]: Virtual Desktop Overview page

What documentation issue are you reporting

Missing Documentation

Issue

Is it worth having a userdoc page like https://epcced.github.io/eidf-docs/services/gpuservice/ that describes the service and the hardware at a high level? I am guessing the Cloud may be a bit more heterogeneous and opaque, but thought to consult.

[Documentation]: Lack of clarity about uniqueness of usernames across projects

What documentation issue are you reporting

Missing Documentation

Issue

Users expect to have a single EIDF username across services which can be true IFF their project spans those services (or the services do not share an authentication system ie LDAP vs IPA).

This should be added to the FAQs so we can point users to it.

[Documentation]: Add more detail about first time login and password change

What documentation issue are you reporting

No response

Issue

The documentation should be updated to make it more explicit that the initial password can only be changed using the VDI. Access via direct ssh using the jumphost will not work until this is done.

[Documentation]: Update VD Policy to refer to patching

What documentation issue are you reporting

Incomplete Documentation

Issue

In response to SPACe EPCC PL (Mark Sawyer) comment, can we please update the "EIDF Data Science Cloud Policies", with (something like) the following section at the top:

Patching of user VMs

While the EIDF updates and patches the hypervisors and the Cloud manager software as part of the EIDF Maintenance sessions, it is the responsibility of project PIs to keep the VMs in their projects up to date. To help with this endeavour, the Ubuntu operating system patches for security and alerts users on log-on to the VMs to reboot them as necessary for the changes to take effect. It also encourages users to update packages.

[Documentation]: ipujobs tutorial 1 yaml is misaligned

What documentation issue are you reporting

Formatting

Issue

The YAML for the first ipujob is misaligned (probably a tabs/spaces thing in an editor) such that it fails the YAML parser when run.

[Documentation]: Missing Namespace Documentation

What documentation issue are you reporting

Incomplete Documentation

Issue

Issue Reported about the lack on information about namespaces on new projects - causing an error on operations trying to target the wrong namespace

[Documentation]: Missing information on how to ask for support

What documentation issue are you reporting

Missing Documentation

Issue

People seem to be struggling to find the [email protected] support address. It would be good to add a documentation section that details how to request support and documents this address.

[Documentation]: missing info about ProxyCommand

What documentation issue are you reporting

Missing Documentation

Issue

The docs don't highlight that the -i flag isn't honoured by the jump host. It would be useful for the docs to explain how to get around this, both for SSH and SCP

[Documentation]:

What documentation issue are you reporting

Incorrect Documentation

Issue

"Known Issues" section is incorrect:

1. Users can be given sudo rights by PIs if requested
2. There is a known issue with Firefox on current Ubuntu builds due to an issue with snapd not working with non-standard home directories (though this may be fixed soon)