5.2 Setting up the ghost layer

The framework's ghost handling is element-centric. You specify which kinds of elements should be ghosts and how they connect by listing their faces before partitioning.

• 
  
  void FEM_Add_ghost_layer(int nodesPerFace,int doAddNodes);
  SUBROUTINE FEM_Add_ghost_layer(nodesPerFace,doAddNodes)
  INTEGER, INTENT(IN) :: nodesPerFace,doAddNodes
  This routine creates a new layer of ghosts around each FEM chunk. nodesPerFace is the number of shared nodes that together form a ``face''. doAddNodes specifies that you want ghost nodes around your ghost elements. If doAddNodes is 0, ghost elements will have invalid -1 (in C) or 0 (in Fortran) connectivity entries where there is no corresponding local node.

  A face is an unordered ``tuple'' of nodes, and is an abstract way to describe which ghosts your application needs--an element will be added to your chunk if it connects to at least one of your elements' faces. For example, if you have a 3D, tetrahedral element that require ghosts on all 4 of its sides, this is equivalent to requiring ghosts of every element that shares all 3 nodes of one of your triangular faces, so for you a face is a 3-node triangle. If you have a 2D shape and want edge-adjacency, for you a face is a 2-node edge. If you want node-adjacent ghosts, a face is a single node.

  Calling this routine several times creates several layers of ghost elements, and the different layers need not have the same parameters.

• 
  
  void FEM_Add_ghost_elem(int elType,int facesPerElem,const int *elem2face);
  SUBROUTINE FEM_Add_ghost_elem(elType,facesPerElem,elem2face)
  INTEGER, INTENT(IN) :: elType,facesPerElem
  INTEGER, INTENT(IN) :: elem2face(nodesPerFace,facesPerElem)
  

  This call is used to specify which type of element is to be added to the current ghost layer. facesPerElem and elem2face specify a mapping between each element and the surrounding faces. The elem2face table lists, for each face, the nodes of this element which form the face, specified as element-local numbers--indices into this element's connectivity entry. The elem2face table should have nodesPerFace*facesPerElem entries, and no entry should be greater than nodePerEl for that element type.

  Because all faces must take up the same space in the array, elem2face can include special indices-- -1 for C, 0 for Fortran--that indicate the corresponding face is actually shorter than usual. For example, if nodesPerFace for this layer is 4, for 4-node quadrilateral faces, you could set one entry in elem2face to -1 to specify this is a 3-node triangular face. Faces of different lengths will never match, so this is just a simple way to add ghosts from two kinds of faces at once.

The above two routines are always used together. For example, if your elements are 3-node triangles and you only require one shared node for inclusion in a single ghost layer, you would use:

   FEM_Add_ghost_layer(1,1); /* 1 node per face: node adjacency */
   const static int tri2node[]={0,1,2};
   FEM_Add_ghost_elem(0,3,tri2node); /* triangles are surrounded by 3 nodes */

If you require two shared nodes (a shared edge), the code will look like:

   FEM_Add_ghost_layer(2,1); /* 2 nodes per face: edge adjacency */
   const static int tri2edge[]={0,1,  1,2,  2,0};
   FEM_Add_ghost_elem(0,3,tri2edge); /*triangles are surrounded by 3 edges */