A ghost entity is a local, read-only copy of a real entity on another chunk. Ghosts are typically added to the boundary of a chunk to allow the real (non-ghost) elements at the boundary to access values across the processor boundary. This makes a chunk ``feel'' as if it was part of a complete unpartitioned mesh; and can be useful with cell-centered methods, and in mesh modification.
|
In Figure 4, we begin with a small mesh partitioned into pieces on the left and right. In Figure 5, we have added ghost elements (dark hashing) that share an edge with adjacent real elements (light hatching). In Figure 6, we add ghost elements that share at least one node with adjacent real elements.
For real elements, the element connectivity always consists of real nodes.
But for ghost elements, the adjacent nodes may be missing, or may themselves
be ghosts.
Thus ghost element connectivity lists may include the invalid
value -1 (in C) or 0 (in Fortran) to indicate that the corresponding
node is not present; or may include values
less than this, which indicate the corresponding node is a ghost.
In C, ghost node
is indicated by the value
, while
in Fortran, ghost node
is indicated by the value
.
This node indexing system is illustrated in Figure 7,
This indexing system is bizarre, but it allows us to keep
the real and ghost nodes clearly separate, while still
allowing real and ghost nodes to be added in increasing order
at both ends.
Since the C tests are complicated, in C we recommend using these macros:
For example, a quadrilateral ghost element that is adjacent to, respectively, two real nodes 23 and 17, the tenth local ghost node, and one not-present node might have a connectivity entry of 23,17,-11,-1 (in C) or 23,17,-10,0 (in Fortran).
Applications may wish to use some other numbering, such as by storing all the ghost nodes after all the real nodes. The code to extract and renumber the connectivity of some 3-node triangles stored in FEM_ELEM+2 would be:
void FEM_Add_ghost_layer(int nodesPerFace,int doAddNodes);
SUBROUTINE FEM_Add_ghost_layer(nodesPerFace,doAddNodes)
INTEGER, INTENT(IN) :: nodesPerFace,doAddNodes
This routine creates a new layer of ghosts around each FEM chunk. nodesPerFace is the number of shared nodes that together form a ``face''. doAddNodes specifies that you want ghost nodes around your ghost elements. If doAddNodes is 0, ghost elements will have invalid -1 (in C) or 0 (in Fortran) connectivity entries where there is no corresponding local node.
A face is an unordered ``tuple'' of nodes, and is an abstract way to describe which ghosts your application needs--an element will be added to your chunk if it connects to at least one of your elements' faces. For example, if you have a 3D, tetrahedral element that require ghosts on all 4 of its sides, this is equivalent to requiring ghosts of every element that shares all 3 nodes of one of your triangular faces, so for you a face is a 3-node triangle. If you have a 2D shape and want edge-adjacency, for you a face is a 2-node edge. If you want node-adjacent ghosts, a face is a single node.
Calling this routine several times creates several layers of ghost elements, and the different layers need not have the same parameters.
This call is used to specify which type of element is to be added to the current ghost layer. facesPerElem and elem2face specify a mapping between each element and the surrounding faces. The elem2face table lists, for each face, the nodes of this element which form the face, specified as element-local numbers--indices into this element's connectivity entry. The elem2face table should have nodesPerFace*facesPerElem entries, and no entry should be greater than nodePerEl for that element type.
Because all faces must take up the same space in the array, elem2face can include special indices-- -1 for C, 0 for Fortran--that indicate the corresponding face is actually shorter than usual. For example, if nodesPerFace for this layer is 4, for 4-node quadrilateral faces, you could set one entry in elem2face to -1 to specify this is a 3-node triangular face. Faces of different lengths will never match, so this is just a simple way to add ghosts from two kinds of faces at once.
The above two routines are always used together. For example, if your elements are 3-node triangles and you only require one shared node for inclusion in a single ghost layer, you would use:
If you require two shared nodes (a shared edge), the code will look like:
The FEM framework can create ghosts not only of things that are on other processors, but also for various problem symmetries, like mirror reflection, and various types of periodicities. The interface for these ghosts is simple--you ask for the symmetries to be created, then you will get extra ghosts along each symmetry boundary. The symmetry ghosts are updated properly during any communication, even if the symmetry ghosts are ghosts of real local elements from the same chunk.
Figure 8 shows a chunk of a mesh for a rectangular domain with horizontal linear translational periodicity--that is, the domain repeats horizontally. Symmetry ghosts lie along the left and right sides; ordinary cross-processor parallel ghosts lie along the top edge where this chunk joins up with the rest of the domain; and the external boundary along the bottom of the chunk has no ghosts.
void FEM_Add_linear_periodicity(
int nFaces,int nPer,
const int *facesA,const int *facesB,
int nNodes,const double *nodeLocs
);
SUBROUTINE FEM_Add_linear_periodicity(nFaces,nPer,facesA,facesB,
nNodes,nodeLocs)
INTEGER, INTENT(IN) :: nFaces, nPer, nNodes
INTEGER, INTENT(IN) :: facesA(nPer,nFaces), facesB(nPer,nFaces)
double precision, INTENT(IN) :: nodeLocs(3,nNodes)
Make facesA and facesB match up under linear translation. Each face of facesA must match up with exactly one face of facesB, but both the faces and the nodes within a face can be permuted in any order--the order is recovered by matching 3d locations in the nodeLocs array.
This call can be repeated, for example if the domain is periodic along several directions. This call can only be issued from init().
void FEM_Sym_coordinates(int elTypeOrMinusOne,double *locs);
SUBROUTINE FEM_Sym_coordinates(elTypeOrZero,locs)
INTEGER, INTENT(IN) :: elTypeOrZero
double precision, intent(inout) :: locs(3,<number of items>)
This call adjusts the 3d locations listed in locs so they respect the symmetries of their corresponding item. If elTypeOrZero is an element type, the locations are adjusted to match with the corresponding element; if elTypeOrZero is zero, the locations are adjusted to match up with the corresponding node.
This call is needed because symmetry ghost nodes and elements initially have their original locations, which must be adjusted to respect the symmetry boundaries. Thus this call is needed both for initial location data (e.g., from FEM_Get_node_data) as well as any communicated location data (e.g., from FEM_Update_ghost_field).
This call can only be issued from driver().
The geometric symmetry layer in the preceeding section is actually a thin wrapper around this lower, more difficult to use layer.
void FEM_Set_sym_nodes(const int *canon,const int *sym);
SUBROUTINE FEM_Set_sym_nodes(canon,sym)
INTEGER, INTENT(IN) :: canon(nNodes)
INTEGER, INTENT(IN) :: sym(nNodes)
This call describes all possible symmetries in an extremely terse format.
It can only be called from init().
The ``canonicalization array'' canon maps nodes to their canonical
representative--if canon(
)=canon(
), nodes
and
are
images of each other under some symmetry. The sym array has bits set
for each symmetry boundary passing through a node.
For example, a 2d domain with 6 elements A, B, C, D, E, and F and 12 nodes numbered 1-12 that is mirror-symmetric on the horizontal boundaries but periodic in the vertical boundaries would look like:
If we mark the left border with 1, the top with 2, the right with 4, and the bottom with 8, this situation is indicated by topologically pasting the top row to the bottom row by setting their canon entries equal, and marking each node with its symmetries.
void FEM_Get_sym(int elTypeOrMinusOne,int *destSym);
void FEM_Get_sym(elTypeOrZero,destSym);
INTEGER, INTENT(IN) :: elTypeOrMinusOne
INTEGER, INTENT(OUT) :: destSym(nItems)
This call extracts the list of symmetry conditions that apply to an item type. If elType is an element type, it returns the symmetry conditions that apply to that element type; if elType is -1 (zero for Fortran), it returns the symmetry conditions that apply to the nodes. Symmetry conditions are normally only nonzero for ghost nodes and elements.
Mirror symmetry conditions are not yet supported, nor are multiple layers of symmetry ghosts, but both should be easy to add without changing this interface.
February 12, 2012
FEM Homepage
Charm Homepage