Shared face operations on parallel NC mesh

[termi-official]

Shared face operations on parallel NC mesh

[termi-official]

I am actually not sure if this is technically a bug or some kind of underspecification issue. From what I understand here is that the returned index from GetSharedFace returns an index to the face set of the parallel NC mesh and that the gaps are due to the faces of the ghost elements (=ghost faces?). However, GetSharedFace is user-facing and the doc string tells that it returns the local face index.

Here I see 3 issues, which I am not sure how to resolve and which I could not find in the mesh source code.

1. How exactly is local face index defined? I expect that the local face index set is a continuous integer range, which seems to be true after inspecting the internals of ParNCMesh, but the bounds are not clear.
2. Is there some way to query the upper bound for the local face index range?
3. Is there some mapping from the local face index to the "local shared face index" (\in [0, mesh->GetNSharedFaces()) )

Edit: Furthermore, should we add this information to https://mfem.org/howto/ncmesh/ or does it belong somewhere else?

[dylan-copeland]

Hi @termi-official, this is my initial guess for the explanation, but I may be wrong and would appreciate any corrections from experienced NC mesh developers.

First, ParMesh::GetSharedFaceTransformationsByLocalIndex uses as its input FaceNo the result of GetSharedFace(sf). The first line of GetSharedFaceTransformationsByLocalIndex uses faces_info[FaceNo]. As you mentioned, the documentation describes the returned FaceNo as a "local index":

   /// Return the local face index for the given shared face.
   int GetSharedFace(int sface) const;

The concept of "local index" could be interpreted in many ways, but the fact is that FaceNo is an index in faces_info, an array in the serial Mesh class.

My understanding is that a parallel NC mesh maintains a global serial Mesh, see Cerveny 2019, e.g. Figure 6.3. In order to keep data mostly distributed in parallel, there is an assumption that the parallel mesh is obtained by refinement of an initially small serial Mesh stored on all MPI ranks, and each rank performs refinements only in its own elements and ghost elements. Thus every rank has a global serial Mesh, but it is local in the sense that it is updated only by local refinements. I believe this is the sense in which "local index" is local.

I am not sure what you are trying to accomplish, but the code in GetSharedFaceTransformationsByLocalIndex may be useful. It seems that mesh->GetNSharedFaces() is the right way to get the local number of shared faces, and GetNumFacesWithGhost returns the size of faces_info, which contains all "local faces" in this context. Considering this line in GetSharedFaceTransformationsByLocalIndex, the ghost faces are at the end of faces_info:
bool is_ghost = Nonconforming() && FaceNo >= GetNumFaces();

If this is right, the simple description of "local index" without further explanation is certainly a good way to confuse users. I think we need a lot of new documentation to make the overall implementation clear.

[termi-official]

Thanks for the detailed answer here @dylan-copeland !

Actually I was also tracing down ParMesh::GetSharedFaceTransformationsByLocalIndex which started the confusion.

The concept of "local index" could be interpreted in many ways, but the fact is that FaceNo is an index in faces_info, an array in the serial Mesh class.

My understanding is that a parallel NC mesh maintains a global serial Mesh, see Cerveny 2019, e.g. Figure 6.3. In order to keep data mostly distributed in parallel, there is an assumption that the parallel mesh is obtained by refinement of an initially small serial Mesh stored on all MPI ranks, and each rank performs refinements only in its own elements and ghost elements. Thus every rank has a global serial Mesh, but it is local in the sense that it is updated only by local refinements. I believe this is the sense in which "local index" is local.

[...] It seems that mesh->GetNSharedFaces() is the right way to get the local number of shared faces, and GetNumFacesWithGhost returns the size of faces_info, which contains all "local faces" in this context. Considering this line in GetSharedFaceTransformationsByLocalIndex, the ghost faces are at the end of faces_info:
bool is_ghost = Nonconforming() && FaceNo >= GetNumFaces();

This is part of my struggle. The local element indices are associated to the leaf element array in NCMesh, which also defines the ghost elements. However, there is no face pendant and I could not find a full definition for the difference between ghost and shared faces. From my understanding a shared face would be a face which is attached to a local element and a ghost element. This seems to be true for conforming faces, but for nonconforming faces things are different. For example, face 12 in figure 1 above is attached to the faces 8 and 10 in global numbering. The number of shared faces here is 2, so only the small faces are counted and interestingly these indices are interleaved with the ghost face indices.

I am not sure what you are trying to accomplish [...]

Short term: Implementing a prototype for a distributed memory parallel implementation of local time stepping (which is quite urgent for me). Long term: Understanding the data structures and algorithms associted to the implemented NC meshes to possibly optimize them further (AMR timings are between 50% and 80% of my total runtime right now).

GetNumFacesWithGhost

Thanks for this pointer here, I am not sure how I missed this function.

If this is right, the simple description of "local index" without further explanation is certainly a good way to confuse users. I think we need a lot of new documentation to make the overall implementation clear.

Indeed. I can give this one a try once I have a better understanding and finished my ongoing PRs.