6.3 Inventing New Types of CpmDestinations

It is possible for the user to create new types of CpmDestinations, and to write functions that return these new destinations. In order to do this, one must have a mental model of the steps performed when a Cpm message is sent. This knowledge is only necessary to those wishing to invent new kinds of destinations. Others can skip this section.

The basic steps taken when sending a CPM message are:

1. The destination-structure is created. The first argument to the launcher is a CpmDestination. Therefore, before the launcher is invoked, one typically calls a function (like CpmSend) to build the destination-structure.

2. The launcher allocates a message-buffer. The buffer contains space to hold a function-pointer and the function's arguments. It also contains space for an ``envelope'', the size of which is determined by a field in the destination-structure.

3. The launcher stores the function-arguments in the message buffer. In doing so, the launcher converts the arguments to a contiguous sequence of bytes.

4. The launcher sets the message's handler. For every launcher, there is a matching function called an invoker. The launcher's job is to put the argument data in the message and send the message. The invoker's job is to extract the argument data from the message and call the user's function. The launcher uses CmiSetHandler to tell CONVERSE to handle the message by calling the appropriate invoker.

5. The message is sent, received, and handled. The destination-structure contains a pointer to a send-function. The send-function is responsible for choosing the message's destination and making sure that it gets there and gets handled. The send-function has complete freedom to implement this in any manner it wishes. Eventually, though, the message should arrive at a destination and its handler should be called.

6. The user's function is invoked. The invoker extracts the function arguments from the message buffer and calls the user's function.

The send-function varies because messages take different routes to get to their final destinations. Compare, for example, CpmSend to CpmEnqueueFIFO. When CpmSend is used, the message goes straight to the target processor and gets handled. When CpmEnqueueFIFO is used, the message goes to the target processor, goes into the queue, comes out of the queue, and then gets handled. The send-function must implement not only the transmission of the message, but also the possible ``detouring'' of the message through queues or into threads.

We now show an example CPM command, and describe the steps that are taken when the command is executed. The command we will consider is this one:

Cpm_print_integer(CpmEnqueueFIFO(3), 12);

Which sends a message to processor 3, ordering it to call print_integer(12).

The first step is taken by CpmEnqueueFIFO, which builds the CpmDestination. The following is the code for CpmEnqueueFIFO:

typedef struct CpmDestinationSend
{
  void *(*sendfn)();
  int envsize;
  int pe;
}
*CpmDestinationSend;

CpmDestination CpmEnqueueFIFO(int pe)
{
  static struct CpmDestinationSend ctrl;
  ctrl.envsize = sizeof(int);
  ctrl.sendfn  = CpmEnqueueFIFO1;
  ctrl.pe = pe;
  return (CpmDestination)&ctrl;
}

Notice that the CpmDestination structure varies, depending upon which kind of destination is being used. In this case, the destination structure contains a pointer to the send-function CpmEnqueueFIFO1, a field that controls the size of the envelope, and the destination-processor. In a CpmDestination, the sendfn and envsize fields are required, additional fields are optional.

After CpmEnqueueFIFO builds the destination-structure, the launcher Cpm_print_integer is invoked. Cpm_print_integer performs all the steps normally taken by a launcher:

1. It allocates the message buffer. In this case, it sets aside just enough room for one int as an envelope, as dictated by the destination-structure's envsize field.

2. It stores the function-arguments in the message-buffer. In this case, the function-arguments are just the integer 12.

3. It sets the message's handler. In this case, the message's handler is set to a function that will extract the arguments and call print_integer.

4. It calls the send-function to send the message.

The code for the send-function is here:

void *CpmEnqueueFIFO1(CpmDestinationSend dest, int len, void *msg)
{
  int *env = (int *)CpmEnv(msg);
  env[0] = CmiGetHandler(msg);
  CmiSetHandler(msg, CpvAccess(CpmEnqueueFIFO2_Index));
  CmiSyncSendAndFree(dest->pe,len,msg);
}

The send-function CpmEnqueueFIFO1 starts by switching the handler. The original handler is removed using using CmiGetHandler. It is set aside in the message buffer in the ``envelope'' space described earlier -- notice the use of CpmEnv to obtain the envelope. This is the purpose of the envelope in the message -- it is a place where the send-function can store information. The destination-function must anticipate how much space the send-function will need, and it must specify that amount of space in the destination-structure field envsize. In this case, the envelope is used to store the original handler, and the message's handler is set to an internal function called CpmEnqueueFIFO2.

After switching the handler, CpmEnqueueFIFO1 sends the message. Eventually, the message will be received by CsdScheduler, and its handler will be called. The result will be that CpmEnqueueFIFO2 will be called on the destination processor. Here is the code for CpmEnqueueFIFO2:

void CpmEnqueueFIFO2(void *msg)
{
  int *env;
  CmiGrabBuffer(&msg);
  env = (int *)CpmEnv(msg);
  CmiSetHandler(msg, env[0]);
  CsdEnqueueFIFO(msg);
}

This function takes ownership of the message-buffer from CONVERSE using CmiGrabBuffer. It extracts the original handler from the envelope (the handler that calls print_integer), and restores it using CmiSetHandler. Having done so, it enqueues the message with the FIFO queueing policy. Eventually, the scheduler picks the message from the queue, and print_integer is invoked.

In summary, the procedure for implementing new kinds of destinations is to write one send-function, one function returning a CpmDestination (which contains a reference to the send-function), and one or more CONVERSE handlers to manipulate the message.

The destination-function must return a pointer to a ``destination-structure'', which can in fact be any structure matching the following specifications:

• The first field must be a pointer to a send-function,
• The second field must the an integer, the envelope-size.

This pointer must be coerced to type CpmDestination.

The send-function must have the following prototype:

    void sendfunction(CpmDestination dest, int msglen, void *msgptr)

It can access the envelope of the message using CpmEnv:

    int *CpmEnv(void *msg);

It can also access the data stored in the destination-structure by the destination-function.