8.3 IDXL Communication

This section brings together all the pieces of IDXL: Index Lists are used to determine what to send and what to receive and Layouts are used to determine where to get and put the communicated data.

8.3.1 Communication Routines

void IDXL_Comm_sendsum(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sendsum(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data

Sum these indices of shared entites across all chunks that share them. The user data array is interpreted according to the given layout.

If comm is zero, this routine is blocking and finishes the communication immediately. If comm is not zero, this routine is non-blocking and equivalent to a call to IDXL_Comm_send followed by a call to IDXL_Comm_sum.

This routine is typically used to sum up partial values on shared nodes. It is a more general version of the old FEM routine FEM_Update_field. For example, to sum up the shared-node values in a 3d force vector indexed by node, you would use:

// C++ version:
     double *force=new double[3*nNodes];
     IDXL_Layout_t force_layout=IDXL_Layout_create(IDXL_DOUBLE,3);
     IDXL_t shared_indices=FEM_Comm_shared(mesh,FEM_NODE);
     
     ... in the time loop ...
         IDXL_Comm_sendsum(0,shared_indices,force_layout,force);

! F90 Version
     double precision, allocatable :: force(:,:)
     integer :: force_layout, shared_indices
     ALLOCATE(force(3,nNodes)) ! (could equivalently use force(3*nNodes) )
     force_layout=IDXL_Layout_create(IDXL_DOUBLE,3)
     shared_indices=FEM_Comm_shared(mesh,FEM_NODE)
     
     ... in the time loop ...
         CALL IDXL_Comm_sendsum(0,shared_indices,force_layout,force)

void IDXL_Comm_sendrecv(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sendrecv(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data

Send these (typically real) send indices and copy in these (typically ghost) receive indices. The user data array is interpreted according to the given layout.

If comm is zero, this routine is blocking and finishes the communication immediately. If comm is not zero, this routine is non-blocking and equivalent to a call to IDXL_Comm_send followed by a call to IDXL_Comm_sum.

This routine is typically used to obtain the values of ghost entities. It is a more general version of the old FEM routine FEM_Update_ghost_field. For example, to obtain 7 solution values per ghost element, storing gElem ghosts in the array just after the nElem regular elements, we could:

// C++ version:
     double *elem=new double[7*(nElem+gElem)];
     IDXL_Layout_t elem_layout=IDXL_Layout_create(IDXL_DOUBLE,7);
     IDXL_t ghost_original=FEM_Comm_ghost(mesh,FEM_ELEM+1);
     IDXL_t ghost_shifted=IDXL_Create(); // ghosts start at nElem
     IDXL_Combine(ghost_shifted,ghost_original,0,nElem);
     
     ... in the time loop ...
         IDXL_Comm_sendrecv(0,ghost_shifted,elem_layout,elem);

! F90 Version
     double precision, allocatable :: elem(:,:)
     integer :: elem_layout, ghost_original,ghost_shifted
     ALLOCATE(elem(7,nElem+gElem))
     elem_layout=IDXL_Layout_create(IDXL_DOUBLE,7)
     ghost_original=FEM_Comm_ghost(mesh,FEM_ELEM+1)
     ghost_shifted=IDXL_Create() ! ghosts start at nElem+1
     CALL IDXL_Combine(ghost_shifted,ghost_original,1,nElem+1)
     
     ... in the time loop ...
         CALL IDXL_Comm_sendrecv(0,ghost_shifted,elem_layout,elem)

8.3.2 Advanced Communication Routines

IDXL_Comm_t IDXL_Comm_begin(int tag,int context);
INTEGER function IDXL_Comm_begin(tag,context)
INTEGER, INTENT(IN) :: tag,context

Start a non-blocking communication operation with this (user-defined) tag and communication context (0, or an AMPI communicator).

Every call to this routine must eventually be matched by a call to IDXL_Comm_wait. Warning: for now, tag and context are ignored, and there can be only one outstanding communication operation.

void IDXL_Comm_send(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,const void *data);
SUBROUTINE IDXL_Comm_send(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(IN) :: data

When comm is flushed, send these send indices, with this layout, from this data array.

This routine is always non-blocking; as the data array passed in will not be copied out until the call to IDXL_Comm_flush.

void IDXL_Comm_recv(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_recv(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(OUT) :: data

When comm is finished, copy in these receive indices, with this layout, into this data array.

This routine is always non-blocking; as the data array passed in will not be copied into until the call to IDXL_Comm_wait.

void IDXL_Comm_sum(IDXL_Comm_t comm,IDXL_t indices,IDXL_Layout_t layout,void *data);
SUBROUTINE IDXL_Comm_sum(comm,indices,layout,data)
INTEGER, INTENT(IN) :: comm,indices,layout
varies, INTENT(INOUT) :: data

When comm is finished, add in the values for these receive indices, with this layout, into this data array.

This routine is always non-blocking; as the data array passed in will not be added to until the call to IDXL_Comm_wait.

void IDXL_Comm_flush(IDXL_Comm_t comm);
SUBROUTINE IDXL_Comm_flush(comm)
INTEGER, INTENT(IN) :: comm

Send all outgoing data listed on this comm. This routine exists because there may be many calls to IDXL_Comm_send, and sending one large message is more efficient than sending many small messages.

This routine is typically non-blocking, and may only be issued at most once per IDXL_Comm_begin.

void IDXL_Comm_wait(IDXL_Comm_t comm);
SUBROUTINE IDXL_Comm_wait(comm)
INTEGER, INTENT(IN) :: comm

Finish this communication operation. This call must be issued exactly once per IDXL_Comm_begin. This call inclues IDXL_Comm_flush if it has not yet been called.

This routine always blocks until all incoming data is received, and is the last call that can be made on this comm.