2. iRecv Library

The iRecv library provides asynchronous communication mode for use in chare arrays in the message-passing style. The control flow of a message-passing program is broken into two parts (to provide the efficient "split phase" structure): nonblocking communication operations and iwait with callback functions as continuations. It thus provides a style that MPI programmers may find intuitive. This library aids in porting existing MPI codes to CHARM++ without using expensive context-switching of threads. In this approach, a chare array element is used to represent a virtual processor.

There are three functions in iRecv library.

void send(buf, size, dest, tag, refno)
void *buf;
int size, dest, tag, refno;
Sends message which is pointed by buf to another array element whose index is specified by dest with tag and refno. buf is a message buffer containing the data to be sent; size is the total size of the message in bytes. Like in MPI_send, the tag is used for matching message on destination array element. The integer refno is a reference number, usually the iteration number.

void irecv(buf, size, source, tag, refno)
void *buf;
int size, source, tag, refno;
This function registers tag and buf with the library. When the desired message arrives, it copies the matching message into the location given by the buf.

void iwaitAll(f, data, refno)
recvCallBack f;
void *data;
int refno;
This function registers a callback function f with the library. This function is invoked with data as its argument when all the previously issued irecvs with refno as reference number complete.

To use the iRecv library, first one has to create a chare array, which is inherited from class receiver. The sender entry methor of the chare array element prepares the message buffer and calls send function to send message to another array element; The receiver specifies the matching tags and buffer to get the message. After irecv, the receiver needs to call iwaitAll function to wait for all the irecv function calls to complete. However, iwaitAll is a nonblocking function. The callback function will be called after the relevant irecv calls complete.

Here is an example:

     int size = 100;
     for (int i=0; i<size; i++) buf[i] = data[i];
     send(buf, size*sizeof(double), neighbor, tag, iter);
     irecv(buf, size*sizeof(double), neighbor, tag, iter);
     iwaitAll(callfunc, this, iter);

and callback function can be declared as:

     void callfunc(void *obj)
     {
       ... do something with obj
     }