3.3 Parameter Marshalling

In CHARM++, chares, groups and nodegroups communicate by invoking each others methods. The methods may either take several parameters, described here; or take a special message object as described in the next section. Since parameters get marshalled into a message before being sent across the network, in this manual we use ``message'' to mean either a literal message object or a set of marshalled parameters.

For example, a chare could have this entry method declaration in the interface (.ci) file:

  entry void foo(int i,int k);

Then invoking foo(2,3) on the chare proxy will eventually invoke foo(2,3) on the remote chare.

Since CHARM++ runs on distributed memory machines, we cannot pass an array via a pointer in the usual C++ way. Instead, we must specify the length of the array in the interface file, as:

  entry void bar(int n,double arr[n]);

Since C++ does not recognize this syntax, the array data must be passed to the chare proxy as a simple pointer. The array data will be copied and sent to the destination processor, where the chare will receive the copy via a simple pointer again. The remote copy of the data will be kept until the remote method returns, when it will be freed. This means any modifications made locally after the call will not be seen by the remote chare; and the remote chare's modifications will be lost after the remote method returns- CHARM++ always uses call-by-value, even for arrays and structures.

This also means the data must be copied on the sending side, and to be kept must be copied again at the receive side. Especially for large arrays, this is less efficient than messages, as described in the next section.

Array parameters and other parameters can be combined in arbitrary ways, as:

  entry void doLine(float data[n],int n);
  entry void doPlane(float data[n*n],int n);
  entry void doSpace(int n,int m,int o,float data[n*m*o]);
  entry void doGeneral(int nd,int dims[nd],float data[product(dims,nd)]);

The array length expression between the square brackets can be any valid C++ expression, including a fixed constant, and may depend in any manner on any of the passed parameters or even on global functions or global data. The array length expression is evaluated exactly once per invocation, on the sending side only. Thus executing the doGeneral method above will invoke the (user-defined) product function exactly once on the sending processor.

3.3.1 Marshalling User-Defined Structures and Classes

The marshalling system uses the pup framework to copy data, meaning every user class that is marshalled needs either a pup routine, a ``PUPbytes'' declaration, or a working operator|. See the PUP description in Section 3.16 for more details on these routines.

//Declarations:
class point3d {
public:
    double x,y,z;    
    void pup(PUP::er &p) {
      p|x; p|y; p|z;
    }
};

typedef struct {
    int refCount;
    char data[17];
} refChars;
PUPbytes(date);

class date {
public:
    char month,day;
    int year;
    //...non-virtual manipulation routines...
};
inline void operator|(PUP::er &p,date &d) {
    p|d.month; p|d.day;
    p|d.year;
}

//In the .ci file:
    entry void pointRefOnDate(point3d &p,refChars r[d.year],date &d);

Any user-defined types in the argument list must be declared before including the ``.decl.h'' file. As usual in C++, it is often dramatically more efficient to pass a large structure by reference (as shown) than by value.

For efficiency, arrays (like refChars above) are always copied as blocks of bytes and passed via pointers. This means classes that need their pup routines to be called, such as those with dynamically allocated data or virtual methods cannot be passed as arrays-use CkVec or STL vectors to pass lists of complicated user-defined classes. For historical reasons, pointer-accessible structures cannot appear alone in the parameter list (because they are confused with messages).

The order of marshalling operations on the send side is:

• Call ``p|a'' on each marshalled parameter with a sizing PUP::er.
• Compute the lengths of each array.
• Call ``p|a'' on each marshalled parameter with a packing PUP::er.
• memcpy each arrays' data.

The order of marshalling operations on the receive side is:

• Create an instance of each marshalled parameter using its default constructor.
• Call ``p|a'' on each marshalled parameter using an unpacking PUP::er.
• Compute pointers into the message for each array.

Finally, very large structures are most efficiently passed via messages, because messages are an efficient, low-level construct that minimizes copying and overhead; but very complicated structures are easiest to pass via marshalling, because marshalling uses the high-level pup framework.