FEM Framework Manual
| Previous: 7 Mesh Modification | Up: CHARM++ Finite Element Method | Next: 9 Old Communication Routines |
| Previous: 7 Mesh Modification | Up: CHARM++ Finite Element Method | Next: 9 Old Communication Routines |
The FEM framework's communication layer is called IDXL. This small library handles sending and receiving data to and from a sparse subset of 1D indices into a user array. The sparse index subset is called an "Index List", hence the name of the library.
An Index List, as you might expect, is a list of indices that need to be sent and received. An Index List includes both the indices that need to be sent, as well as the indices to be received, from each chunk.
Consider two chunks and where needs some information has, such as if has ghosts of real elements on . 's Index List thus has a send portion with the -local indices for the elements sends; and 's Index List contains a receive portion with the -local indices for the elements receives. Thus across processors, the corresponding send and receive portions of and 's Index Lists match, as shown in Figure 10.
You refer to an Index List via an opaque handle--in C, the integer typedef IDXL_t; in Fortran, a bare INTEGER.
IDXL_t FEM_Comm_shared(int mesh,int entity);
INTEGER function FEM_Comm_shared(mesh,entity)
INTEGER, INTENT(IN) :: mesh,entity
Return a read-only copy of the Index List of shared nodes. The send and receive portions of this list are identical, because each shared node is both sent and received. Shared nodes are most often used with the send/sum communication pattern.
Must be called from driver. mesh must be a reading mesh. entity must be FEM_NODE. You may not call IDXL_Destroy on the returned list.
IDXL_t FEM_Comm_ghost(int mesh,int entity);
INTEGER function FEM_Comm_ghost(mesh,entity)
INTEGER, INTENT(IN) :: mesh,entity
Return a read-only copy of the Index List of ghost entities. The send portion of this list contains real, interior entities, which are sent away; the receive portion of the list contains the ghost entites, which are received. Ghosts are most often used with the send/recv communication pattern.
Elements to be sent out are listed starting at zero (one in Fortran); but ghost elements to be received are also listed starting at zero (one in Fortran). If real and ghost elements are kept in separate arrays, this is usable as-is; but if ghosts and real elements are kept together, you will need to shift the ghost indices using IDXL_Combine or IDXL_Shift.
This routine must be called from driver. mesh must be a reading mesh. entity must not include FEM_GHOST-ghosts are already included. You may not call IDXL_Destroy on the returned list.
IDXL_t IDXL_Create(void);
INTEGER function IDXL_Create()
Create a new, empty Index List. This list can then be filled up using IDXL_Copy or IDXL_Combine.
Must be called from driver. You must eventually call IDXL_Destroy on the returned list.
void IDXL_Combine(IDXL_t dest,IDXL_t src,int startSend,int startRecv);
SUBROUTINE IDXL_Combine(dest,src,startSend,startRecv)
INTEGER, INTENT(IN) :: dest,src,startSend,startRecv
Add the shifted contents of the src Index List to dest. The send portion of src is shifted so the first index sent will be startSend; for a ghost index list this is the index of the first sent real entity. The receive portion of src is similarly shifted so the first index received will be startRecv; for a ghost index list this is the index of the first received ghost entity.
This routine does not check for duplicates--if an index originally appears in dest and the also in the shifted src, it will be listed twice.
void IDXL_Destroy(IDXL_t l);
SUBROUTINE IDXL_Destroy(l)
INTEGER, INTENT(IN) :: l
Destroy this Index List, and free the list storage allocated by the framework. Only call this routine with lists you created using IDXL_Create; not lists obtained directly from the FEM framework.
void IDXL_Print(IDXL_t l);
SUBROUTINE IDXL_Print(l)
INTEGER, INTENT(IN) :: l
Print out the contents of this Index List. This routine shows both the send and receive indices on the list, for each chunk we communicate with.
void IDXL_Copy(IDXL_t dest,IDXL_t src);
SUBROUTINE IDXL_Print(dest,src)
INTEGER, INTENT(IN) :: dest,src
Copy the contents of the source Index List into the destination Index List, which should be empty.
void IDXL_Shift(IDXL_t l,int startSend,int startRecv);
SUBROUTINE IDXL_Shift(l,startSend,startRecv)
INTEGER, INTENT(IN) :: l,startSend,startRecv
Like IDXL_Combine, but only shifts the indices within a single list.
void IDXL_Add_entity(int newIdx,int nBetween,int *between);
SUBROUTINE IDXL_Add_node(newIdx,nBetween,between)
INTEGER, INTENT(IN) :: newIdx,nBetween
INTEGER, INTENT(IN) :: between(nBetween)
This call adds a new entity, with local index newIdx, to this Index List. The new entity is sent or received by each chunk that sends or receives all the entites listed in the between array. For example, when adding a new node along an edge, nBetween is 2 and between lists the endpoints of the edge; this way if the edge is shared with some chunk, the new node will be shared with that chunk.
This routine only affects the current chunk- no other chunks are affected. To ensure the communication lists match, IDXL_Add_entity must be called on all the chunks that send or receive the entity, to create the local copies of the entity.
IDXL_Add_entity adds the new entity to the end of the communication list, and so must be called in the same order on all the chunks that share the new entity. For example, if two new nodes and are added between chunks and , if chunk calls IDXL_Add_entity with its local number for before it calls IDXL_Add_entity with its local number for , chunk must also add its copy of node before adding .
IDXL Layouts are normally used to describe arrays of data associated with nodes or elements. The layout abstraction allows you to use IDXL routines to communicate any sort of data, stored in a variety of formats.
Like Index Lists, Layouts are referred to via an opaque handle--in a C program via the integer typedef IDXL_Layout_t, and in Fortran via a bare integer.
In most programs, the data to be communicated is a dense array of data of one type. In this case, there is only one layout routine you need to know:
IDXL_Layout_t IDXL_Layout_create(int type,int width);
INTEGER function IDXL_Layout_create(type,width)
INTEGER, INTENT(IN) :: type,width
The simplest data layout to describe--a dense array of this IDXL datatype, indexed by entity number, with width pieces of data per entity. Note that the number of entities is not stored with the layout-the number of entities to be communicated depends on the communication routine.
The IDXL datatypes are:
| IDXL Datatype | C Datatypes | Fortran Datatypes |
| IDXL_BYTE | unsigned char | INTEGER*1 |
| char | LOGICAL*1 | |
| IDXL_INT | int | INTEGER |
| IDXL_REAL | float | SINGLE PRECISION |
| REAL*4 | ||
| IDXL_DOUBLE | double | DOUBLE PRECISION |
| REAL*8 |
For example, if you keep a dense array with 3 doubles of force per node, you'd call this routine as:
This routine was once called FEM_Create_simple_field.
These advanced routines are only needed if you want to exchange data stored in an array of user-defined types. Most programs only need IDXL_Layout_create.
IDXL_Layout_t IDXL_Layout_offset(int type, int width, int offsetBytes, int distanceBytes,int skewBytes);
INTEGER function IDXL_Layout_offset(type,width,offsetBytes,distanceBytes,skewBytes)
INTEGER, INTENT(IN) :: type,width,offsetBytes,distanceBytes,skewBytes
The most general data layout-an array indexed by entity, containing width pieces of data per entity. This routine expands on IDXL_Layout_create by adding support for user-defined types or other unusual data layouts. You describe your layout by giving various in-memory byte offsets that describe the data is stored. Again, the number of entities is not stored with the layout-the number of entities to be communicated depends on the communication routine.
For example, if your node data is all stored in a struct (in fortran, a named TYPE), offsetBytes gives the distance between the start of the struct and the force; and distanceBytes gives the size in bytes of the struct.
In C, the offsetof and sizeof keywords are useful for finding these values. In Fortran, we provide a special routine called foffsetof that returns the distance, in bytes, between its two arguments.
void IDXL_Layout_destroy(IDXL_Layout_t layout);
SUBROUTINE IDXL_Layout_destroy(layout)
INTEGER, INTENT(IN) :: layout
Destroy this Layout. You only need call this routine if you repeatedly create layouts.
int IDXL_Get_layout_type(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_type(layout)
Return the IDXL datatype for this layout.
int IDXL_Get_layout_width(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_width(layout)
Return the layout width--the number of data items that are communicated per entity.
int IDXL_Get_layout_distance(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_distance(layout)
Return the layout distance--the number of bytes between successive entity's data items.
Before IDXL was made a separate library, FEM included these routines, which are still preserved for backward compatability.
IDXL_Layout_t FEM_Create_simple_field(int type,int width);
INTEGER function FEM_Create_simple_field(type,width)
INTEGER, INTENT(IN) :: type,width
This routine is completely interchangable to IDXL_Layout_create.
int FEM_Create_field(int type,int width,int offset,int distance);
INTEGER function FEM_Create_field(type, width, offset, distance)
INTEGER, INTENT(IN) :: type, width, offset, distance
This routine is like a call to IDXL_Layout_offset with the rarely used skewBytes set to zero.
void IDXL_Comm_sendsum(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sendsum(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data
Sum these indices of shared entites across all chunks that share them. The user data array is interpreted according to the given layout.
If comm is zero, this routine is blocking and finishes the communication immediately. If comm is not zero, this routine is non-blocking and equivalent to a call to IDXL_Comm_send followed by a call to IDXL_Comm_sum.
This routine is typically used to sum up partial values on shared nodes. It is a more general version of the old FEM routine FEM_Update_field. For example, to sum up the shared-node values in a 3d force vector indexed by node, you would use:
void IDXL_Comm_sendrecv(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sendrecv(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data
Send these (typically real) send indices and copy in these (typically ghost) receive indices. The user data array is interpreted according to the given layout.
If comm is zero, this routine is blocking and finishes the communication immediately. If comm is not zero, this routine is non-blocking and equivalent to a call to IDXL_Comm_send followed by a call to IDXL_Comm_sum.
This routine is typically used to obtain the values of ghost entities. It is a more general version of the old FEM routine FEM_Update_ghost_field. For example, to obtain 7 solution values per ghost element, storing gElem ghosts in the array just after the nElem regular elements, we could:
IDXL_Comm_t IDXL_Comm_begin(int tag,int context);
INTEGER function IDXL_Comm_begin(tag,context)
INTEGER, INTENT(IN) :: tag,context
Start a non-blocking communication operation with this (user-defined) tag and communication context (0, or an AMPI communicator).
Every call to this routine must eventually be matched by a call to IDXL_Comm_wait. Warning: for now, tag and context are ignored, and there can be only one outstanding communication operation.
void IDXL_Comm_send(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,const void *data);
SUBROUTINE IDXL_Comm_send(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(IN) :: data
When comm is flushed, send these send indices, with this layout, from this data array.
This routine is always non-blocking; as the data array passed in will not be copied out until the call to IDXL_Comm_flush.
void IDXL_Comm_recv(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_recv(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(OUT) :: data
When comm is finished, copy in these receive indices, with this layout, into this data array.
This routine is always non-blocking; as the data array passed in will not be copied into until the call to IDXL_Comm_wait.
void IDXL_Comm_sum(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sum(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data
When comm is finished, add in the values for these receive indices, with this layout, into this data array.
This routine is always non-blocking; as the data array passed in will not be added to until the call to IDXL_Comm_wait.
void IDXL_Comm_flush(IDXL_Comm_t comm);
SUBROUTINE IDXL_Comm_flush(comm)
INTEGER, INTENT(IN) :: comm
Send all outgoing data listed on this comm. This routine exists because there may be many calls to IDXL_Comm_send, and sending one large message is more efficient than sending many small messages.
This routine is typically non-blocking, and may only be issued at most once per IDXL_Comm_begin.
void IDXL_Comm_wait(IDXL_Comm_t comm);
SUBROUTINE IDXL_Comm_wait(comm)
INTEGER, INTENT(IN) :: comm
Finish this communication operation. This call must be issued exactly once per IDXL_Comm_begin. This call inclues IDXL_Comm_flush if it has not yet been called.
This routine always blocks until all incoming data is received, and is the last call that can be made on this comm.
November 23, 2009
FEM Homepage
Charm Homepage
