3.17 Callbacks

A callback is a generic way to transfer control back to a client after a CHARM++ library has finished. For example, after finishing a reduction, you might want the results passed to some chare's entry method. To do this, you create an object of type CkCallback with the chare's CkChareID and entry method index, then pass the callback object to the reduction library.

3.17.1 Client Interface

You can create a CkCallback object in a number of ways, depending on what you want to have happen when the callback is finally invoked. The callback will be invoked with a CHARM++ message; but the message type will depend on the CHARM++ library that actually invokes the callback. Check the library documentation to see what kind of message the library will send to your callback. In any case, you are required to free the message passed to you via the callback.

The callbacks that go to chares require an ``entry method index'', an integer that identifies which entry method will be called. You can get an entry method index using the syntax:

myIdx=CkIndex_ChareName::EntryMethod(parameters);

Here, ChareName is the name of the chare (group, or array) containing the desired entry method, EntryMethod is the name of that entry method, and parameters are the parameters taken by the method. These parameters are only used to resolve the proper EntryMethod; they are otherwise ignored. An entry method index is the CHARM++ version of a function pointer.

There are a number of ways to build callbacks, depending on what you want to have happen when the callback is invoked:

1. CkCallback(CkCallbackFn fn,void *param) When invoked, the callback will pass param and the result message to the given C function, which should have a prototype like:
   void myCallbackFn(void *param,void *message)
   

   This function will be called on the processor where the callback was created, so param is allowed to point to heap-allocated data. Of course, you are required to free any storage referenced by param.

2. CkCallback(CkCallback::ignore) When invoked, the callback will do nothing. This can be useful if the CHARM++ library requires a callback, but you don't care when it finishes, or will find out some other way.

3. CkCallback(CkCallback::ckExit) When invoked, the callback will call CkExit(), ending the Charm++ program.

4. CkCallback(int ep,const CkChareID &id) When invoked, the callback will send its message to the given entry method of the given Chare. Note that a chare proxy will also work in place of a chare id:
    CkCallback myCB(CkIndex_myChare::myEntry(NULL),myChareProxy);
   

5. CkCallback(int ep,const CkArrayID &id) When invoked, the callback will broadcast its message to the given entry method of the given array. An array proxy will work in the place of an array id.

6. CkCallback(int ep,const CkArrayIndex &idx,const CkArrayID &id) When invoked, the callback will send its message to the given entry method of the given array element.

7. CkCallback(int ep,const CkGroupID &id) When invoked, the callback will broadcast its message to the given entry method of the given group.

8. CkCallback(int ep,int onPE,const CkGroupID &id) When invoked, the callback will send its message to the given entry method of the given group member.

One final type of callback, a CkCallback(CkCallback::resumeThread), can only be used from within threaded entry methods. This type of callback is typically hidden within a thread-capable library, so is discussed further in the library section.

3.17.2 Library Interface

Here, a ``library'' is simply any CHARM++ code which can be called from several different places. From the point of view of a library, a CkCallback is a destination for the library's result. CkCallback objects can be freely copied, marshalled, or even sent in messages.

Postponing the discussion on threads for a moment, the only thing you can do with a CkCallback is to move it around or send a message to it:

//Main library entry point, called by asynchronous users:
void myLibrary(...library parameters...,const CkCallback &cb)
{
  ..start some parallel computation, dragging cb along...
}

//Internal library routine, called when computation is done
void myLibraryDone(...parameters...,const CkCallback &cb)
{
  ...prepare a return message...
  cb.send(msg);
}

A CkCallback will accept any message type, or even NULL. The message is immediately sent to the user's client function or entry point, so you do need to document the type of message you will send to the callback so the user knows what to expect.

As an alternative to ``send'', the callback can be used in a contribute collective operation. This will internally invoke the ``send'' method on the callback when the contribute operation has finished.

Threaded entry methods can be suspended and resumed through the CkCallbackResumeThread class. CkCallbackResumeThread is derived from CkCallback and has specific functionality for threads. This class automatically suspends the thread when the destructor of the callback is called. A suspended threaded client will resume when the ``send'' method is invoked on the associated callback. It can be used in situations when the return value is not needed, and only the synchronization is important. For example:

// Call the "doWork" method and wait until it has completed
void mainControlFlow() {
  ...perform some work...
  // call a library
  doWork(...,CkCallbackResumeThread());
  // or send a broadcast to a chare collection
  myProxy.doWork(...,CkCallbackResumeThread());
  // The thread is suspended until doWork calls 'send' on the callback
  
  ...some more work...
}

Alternatively, if doWork returns a value of interest, this can be retrieved by passing a pointer to CkCallbackResumeThread. This pointer will be modified by CkCallbackResumeThread to point to the incoming message. Notice that the input pointer has to be cast to (void*&):

// Call the "doWork" method and wait until it has completed
void mainControlFlow() {
  ...perform some work...
  MyMessage *mymsg;
  myProxy.doWork(...,CkCallbackResumeThread((void*&)mymsg));
  // The thread is suspended until doWork calls send on the callback

  ...some more work using "mymsg"...
}

Notice that the instance of CkCallbackResumeThread is constructed as an anonymous parameter to the ``doWork'' call. This insures that the callback is destroyed as soon as the function returns, thereby suspending the thread.

It is also possible to allocate a CkCallbackResumeThread on the heap or on the stack. We suggest that programmers avoid such usage, and favor the anonymous instance construction shown above. For completeness, we still present the code for heap and stack allocation of CkCallbackResumeThread callbacks below.

For heap allocation, the user must explicitly ``delete'' the callback in order to suspend the thread.

// Call the "doWork" method and wait until it has completed
void mainControlFlow() {
  ...perform some work...
  CkCallbackResumeThread cb = new CkCallbackResumeThread();
  myProxy.doWork(...,cb);
  ...do not suspend yet, continue some more work...
  delete cb;
  // The thread suspends now

  ...some more work after the thread resumes...
}

For a callback that is allocated on the stack, its destructor will be called only when the callback variable goes out of scope. In this situation, the function ``thread_delay'' can be invoked on the callback to force the thread to suspend. This also works for heap allocated callbacks.

// Call the "doWork" method and wait until it has completed
void mainControlFlow() {
  ...perform some work...
  CkCallbackResumeThread cb;
  myProxy.doWork(...,cb);
  ...do not suspend yet, continue some more work...
  cb.thread_delay();
  // The thread suspends now

  ...some more work after the thread is resumed...
}

N.B.: a CkCallbackResumeThread can be used to suspend a thread only once.

Deprecated usage: in the past, ``thread_delay'' was used to retrieve the incoming message from the callback. While that is still allowed for backward compatibility, its usage is deprecated. The old usage is subject to memory leaks and dangling pointers.