Generalized Message Scheduling

In order to unify the scheduling of all concurrent entities, such as message-driven objects and threads, we generalize the notion of a message. A generalized message is an arbitrary block of memory, with the first few bytes specifying a function that will handle the message. The scheduler dispatches a generalized message by invoking its handler with the message pointer as a parameter. Any function that is used for handling messages must first be registered with the scheduler. When a handler is registered, the scheduler returns a handle to the function. When a message is sent, this handle must be stored in the message using macros provided by the scheduler.

There are two kinds of messages in the system waiting to be scheduled -- messages that have come from the network, and those that are locally generated. The scheduler's job is to repeatedly deliver these messages to their respective handlers. Since network utilization issues demand timely processing of messages from the network interface, the scheduler first extracts as many messages as it can from the network, calling the handler for each of them. The handler for a particular message may be a user-written function, or a function in the runtime of a particular language. These handlers may enqueue the messages for scheduling (with an optional priority) if they desire such functionality, or may process them immediately. After emptying messages from the network, the scheduler dequeues one message from its queue and delivers it to its handler. This process continues until the Converse function for terminating the scheduler is called by the user program.

Converse supplies two additional variants of the scheduler for flexibility. The first allows the programmer to specify the number of messages to be delivered. The second, called the ``poll mode'', runs the scheduler loop until there are no messages left in either the network's queue or the scheduler's queue. For modules written in the explicit control regime, control stays within the user code all the time. However, for modules in the implicit control regime, control must shift back and forth between a system scheduler and user code. For these apparently incompatible regimes to coexist, it is necessary to expose the scheduler to the module developer, rather than keeping it buried inside the run-time system. A single-threaded module can explicitly relinquish control to the scheduler to allow execution of multi-threaded and message-driven components. This control transfer need not be exposed to the application, however. For example, an MPI implementation on Converse may internally call the scheduler in poll mode when any of the MPI functions are called, allowing other computations to complete while the MPI process is waiting for a message.

The pseudo-code for these variants of the scheduler is shown in figure 2.3. CsdScheduler implements the implicit control regime, CmiGetSpecificMsg implements programming paradigms with explicit control regime.

{SchedCode}
void CsdScheduler()
{ 
  while ( 1 ) { 
    CmiDeliverMsgs() ;
    if (exit called by handler)
      return;
    get a message from the scheduler queue ;
    call the message's handler ; 
    if (exit called by handler)
      return;
  }
}

void CmiDeliverMsgs()
{ 
  while ( there are messages in the network ) { 
    receive a message from the network ;
    call the message's handler ; 
    if (exit called by handler)
      return;
  }
}

void CmiGetSpecificMsg(int spechdlr)
{
  while ( 1 ) {
    if ( there is message in the network ) { 
      receive a message from the network ;
      call the message's handler ; 
      if ( handler == spechdlr )
        return;
    }
    if ( scheduler queue not empty ) {
      get a message from the scheduler queue ;
      call the message's handler ; 
      if (handler == spechdlr)
        return;
    }
  }
}

Figure 2.3: Pseudo-code for scheduler variants