gaspi-forum/gaspi-standard

The acual GASPI specification as defined by the GASPI-Forum

GASPI-Standard

The actual GASPI specification as defined by the GASPI-Forum.

gaspi-standard's Issues

LaTeX improvements

Hi,

The GASPI specification provided in http://www.gaspi.de/, currently:

https://raw.githubusercontent.com/GASPI-Forum/GASPI-Forum.github.io/master/standards/GASPI-17.1.pdf

has two major typesetting issues:

1. The document is using non-vectorial fonts, which are very ugly for use in computer screen. Here is what I see in okular (same in zathura and mupdf):
   
   While, if I compile the document from the repository using pdflatex GASPI_Standard_Draft.tex it shows ok: I'm not sure what causes the difference. This is what I found on the PDF metadata; maybe is related to the pdfTeX version:

# The original from the GASPI-forum website
% pdfinfo GASPI-17.1.pdf | egrep '(Creator:|Producer:)'  
Creator:         TeX
Producer:        pdfTeX-1.40.16
# The one I compiled:
% pdfinfo GASPI_Standard_Draft.pdf  | egrep '(Creator:|Producer:)'  
Creator:         LaTeX with hyperref
Producer:        pdfTeX-1.40.20

2. The PDF document doesn't have a PDF index enabled, which I find very useful to quickly jump between sections, specially when I'm not familiar with the complete structure of the document. I was able to enable the index with the hyperref package, but there is a conflict with one of the macros that cause the document to fail. I disabled the macro, as I don't know how to fix it. Here is the patch:

diff --git a/gaspi-standard/tex/GASPI_Standard_Draft.tex b/gaspi-standard/tex/GASPI_Standard_Draft.tex
index 6847d29..18cd600 100644
--- a/gaspi-standard/tex/GASPI_Standard_Draft.tex
+++ b/gaspi-standard/tex/GASPI_Standard_Draft.tex
@@ -57,8 +57,8 @@
 \newcommand{\zerowsep}{\hskip 0pt plus 0.1pt minus 0.1pt}
 
 \makeatletter
-\newcommand{\ZSEP}[1]{\ifx#1\@@@EOZ@@@\let\next\relax\else\ifx#1\_#1\zerowsep\else#1\fi\let\next\ZSEP\fi\next}
-\newcommand{\zsep}[1]{\ZSEP{}#1\@@@EOZ@@@}
+%\newcommand{\ZSEP}[1]{\ifx#1\@@@EOZ@@@\let\next\relax\else\ifx#1\_#1\zerowsep\else#1\fi\let\next\ZSEP\fi\next}
+\newcommand{\zsep}[1]{#1}
 \makeatother
 
 \newcommand{\sol}[1]{\emph{\zsep{#1}}}
@@ -262,6 +262,10 @@ basicstyle=\ttfamily
 , classoffset=0
 }\lstinputlisting{#1}}
 
+\usepackage{hyperref}
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%                              Document                                 %%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

After enabling the index generation, I can see the sections and quickly jump between them:

Some clarifications of global reduction operations needed

In sec. 11.3 there is the following sentence:

Starting a reduction operation for the same group in a separate thread before previously invoked operation is finished on all processes of the group is not allowed and yields undedined behavior.

1. "in a separate thread" is not needed here, since it also undefined to do it like this:

gaspi_allreduce(..., myGroup, GASPI_TEST);
gaspi_allreduce(..., myGroup, GASPI_BLOCK);

Here both reduction operations are started in the same thread and potentially overlap.

2. I don't think, that "is finished on all processes of the group" is intended, since then you cannot write

gaspi_allreduce(..., myGroup, GASPI_BLOCK);
gaspi_allreduce(..., myGroup, GASPI_BLOCK);

since the local process doesn't know, whether all other processes have already left the first gaspi_allreduce. You would have to place a barrier inbetween the two calls.

Another issue: it should be guaranteed, that the results of the reduction are bitwise equal across all ranks of the group.

Dynamic allocation of GASPI segments

I have a library, where I can't precalculate the memory footpring in a GASPI segment. Thus the idea is to create segments on demand. Does the following code work in principle:

void* GetMemory(size_t size)   // collective called by all members of group
{
  void* p = Allocate(size, seg_id);
  if (p == 0)
  {
    gaspi_segment_avail_local(&seg_id);  // from the extension
    gaspi_segment_alloc(seg_id, size, GASPI_MEM_UNINITIALIZED);
    gaspi_segment_ptr(seg_id, &p);
    for (auto i : group)
      gaspi_segment_register(seg_id, i, GASPI_BLOCK);
  }
  std::vector<unsigned int> ids(group.size(), 0);
  remote_ids[my_rank] = seg_id;
  gaspi_allreduce(ids.data(), ids.data(), 1, GASPI_OP_SUM, GASPI_TYPE_UINT, group, GASPI_BLOCK)); 
}

What it does: it tries to allocate the requested size on the already existing segment seg_id. If there is not enough memory in that old segment, a new seg_id is locally retrieved, bind to newly allocated memory and then registered at some remote ranks (stored in group). Afterwards all members of the group enter the gaspi_allreduce to exchange their segment ids, so that the ids can be used for communcation. The allreduce serves also as a barrier here (see gaspi_segment_create).

Question: works gaspi_segment_register as written here? That is, is it a one-sided call, which does either nothing on the remote rank or the remote rank receives the register request somehow magically?

Wish: If I'd use gaspi_segment_bind (with some memory from the outside), I have difficulties to release the segment. I cannot use segment_delete, since the memory is not owned by the segment. I could of course just release the memory at the outside, which renders the segment id invalid. But then I cannot reuse the segment id, since something like gaspi_segment_unbind is missing, isn't it?

Define rank mapping of MPI Interoperability

Currently, the standard is very slim regarding MPI interoperability.
I propose to add a definition, that a GASPI rank is equal to the MPI rank in MPI_COMM_WORLD.

In addition we could think about an additional gaspi_proc_init, which takes an MPI communicator and only initializes GASPI for the group of that MPI communicator (also taking over the ranks of that communicator).

Missing parameters in gaspi_reduce_operation (C interface)

The GASPI specification for gaspi_reduce_operation is not complete. the number of elements and the element size are missing. The GPI2 implementation uses both parameters.

Local-only use of segments

Sec. 7.2.1 (gaspi_segment_alloc) reads:

After successful procedure completion, i. e. return value GASPI_SUCCESS, the
segment can be accessed locally. In case that there is a connection established to
a remote Gaspi process, it can also be used for passive communication between
the two Gaspi processes.

This sounds as if it cannot be used as the local segment in gaspi_wirte and gaspi_read. However, sec. 8.2.1 (gaspi_write) reads differently:

A valid gaspi_write communication request requires that the local and the
remote segment are allocated, that there is a connection between the local and
the remote Gaspi process and that the remote segment has been registered on
the local Gaspi process.

I guess the latter is intended. We should harmonize 7.2.1 accordingly.

To put this issue to an extreme: is it actually necessary to have local segments at all? Couldn't gaspi_write and the like just take a pointer to the local memory?