fmarrabal / qe-gpu Goto Github PK

QE-GPU: GPU-Accelerated Quantum ESPRESSO
----------------------------------------

Quantum ESPRESSO is an integrated suite of Open-Source computer codes for 
electronic-structure calculations and materials modeling at the nanoscale. 

The aim of QE-GPU is to create a "plugin-like" component for the standard 
Quantum ESPRESSO package that allows to exploit the capabilities of NVIDIA 
GPU graphics cards in order to allow materials scientists to do better and 
fast science. GPU acceleration is currently available for the Plane-Wave 
Self-Consistent Field (PWscf) code and the energy barriers and reaction 
pathways through the Nudged Elastic Band method (NEB) package.

QE-GPU is provided "as is" and with no warranty. This software is distributed 
under the GNU General Public License, please see the files LICENSE and 
DISCLAIMER for details. This README presents an introduction to compiling, 
installing, and using QE-GPU.


                               * * * * *

   !!! THIS VERSION IS COMPATIBLE ONLY WITH QE 5.3 AND FORWARD !!!
   
     !!! QE-GPU NOW FOLLOWS QUANTUM ESPRESSO RELEASE ROADMAP !!!

                               * * * * *

How to compile
--------------

1. Copy QE-GPU in espresso directory

Move to the espresso root directory, uncompress the archive
$ tar zxvf QE-GPU-<TAG-NAME>.tar.gz 

and create a symbolic link with the name GPU
$ ln -s QE-GPU-<TAG-NAME> GPU

Replace <TAG-GPU> with the corred TAG name/id


2. Run QE-GPU configure

The QE-GPU configure is located in the GPU directory. An example of serial
configuration is the following:

$ cd GPU
$ ./configure --disable-parallel --enable-openmp \
  --enable-cuda --with-gpu-arch=sm_35 \
  --with-cuda-dir=<full-path-where-CUDA-is-installed> \
  --with-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu


An example for parallel execution:
$ cd GPU
$ ./configure --enable-parallel --enable-openmp --with-scalapack \
  --enable-cuda --with-gpu-arch=sm_35 \
  --with-cuda-dir=<full-path-where-CUDA-is-installed> \
  --without-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu


For additional options for QE-GPU see "./configure --help". Here a summary
of all GPU-related options

  --enable-cuda           enable CUDA (default: no)

  --enable-debug-cuda     enable CUDA debug (default: no)

  --with-cuda-dir=<path>  specify CUDA installation directory (default is
                          /usr/local/cuda/ - MANDATORY)

  --with-gpu-arch=<arch>  (sm_13|sm_20|sm_21|sm_30|sm_35) Specify the GPU
                          target architecture (default: all)

  --with-magma            (yes|no|<path>) Use MAGMA. Self-compile or a <path>
                          can be specified (default: no)

  --with-phigemm          (yes|no|<path>|profile) Use PHIGEMM. Self-compile or
                          a <path> can be specified (default: yes)

  --with-pinned-mem       (yes|no) Use pinned memory for accelerated BLAS
                          (default: no )

IMPORTANT NOTES:
- specify always the right GPU compute capability or leave the make to 
  compile for all possible compute capabilities which support Double Precision
- specify the correct PATH of CUDA (where CUDA is installed). Usually HPC 
  systems have the env variable $CUDA_HOME or $CUDA_ROOT
- "pw-gpu.x" will be placed under "bin/" (and also under "GPU/PW/")


                               * * * * *


Trouble-makers, inconsistencies, Known bugs, etc. 
-------------------------------------------------

- if you are using Intel MPI, please add to DFLAGS "-DMPICH_SKIP_MPICXX" to 
  ignore MPI C++ bindings.
  
- if the calculation fail due to "S non definite positive" or other errors
  related to MAGMA then recompile by disabling MAGMA support.

   
                               * * * * *


In case of CUDA/GPU code errors...
----------------------------------
        
It may happen that some ported routines fail. If this happen you can easily
decide to run the code skipping that specific "GPU" section that may produce 
some issues. To do that it is possible to compile the code with few additional
flags that exclude the GPU code related to ADDUSDENS, NEWD or VLOC_PSI. 

These flags are: 
-D__DISABLE_CUDA_ADDUSDENS 
-D__DISABLE_CUDA_VLOCPSI 
-D__DISABLE_CUDA_NEWD

Without editing the make.sys, you can run make in this way:
$ make -f Makefile.gpu MANUAL_DFLAGS="-D__DISABLE_CUDA_ADDUSDENS" pw-gpu

You can also specify more of these flags together.