3.15 Quiescence Detection

In CHARM++, quiescence is defined as the state in which no processor is executing an entry point, and no messages are awaiting processing.

CHARM++ provides two facilities for detecting quiescence: CkStartQD and CkWaitQD.

CkStartQD registers with the system a callback that should be made the next time quiescence is detected. CkStartQD has two variants which expect the following arguments:

1. An index corresponding to the entry function that is to be called, and a handle to the chare on which that entry function should be called. The syntax of this call looks like this:
    CkStartQD(int Index,const CkChareID* chareID);
   

   To retrieve the corresponding index of a particular entry method, you must use a static method contained within the CkIndex object corresponding to the chare containing that entry method. The syntax of this call is as follows:

   myIdx=CkIndex_ChareClass::EntryMethod(parameters);
   

   where ChareClass is the C++ class of the chare containing the desired entry method, EntryMethod is the name of that entry method, and parameters are the parameters taken by the method. These parameters are only used to resolve the proper EntryMethod; they are otherwise ignored.

2. A CkCallback object. The syntax of this call looks like:
     CkStartQD(const CkCallback& cb);
   

   Upon quiescence detection, specified callback is called with no parameters.

CkWaitQD, by contrast, does not register a callback. Rather, CkWaitQD blocks and does not return until quiescence is detected. It takes no parameters and returns no value. A call to CkWaitQD simply looks like this:

  CkWaitQD();

Keep in mind that CkWaitQD should only be called from threaded entry methods because a call to CkWaitQD suspends the current thread of execution, and if it were called outside of a threaded entry method it would suspend the main thread of execution of the processor from which CkWaitQD was called and the entire program would come to a grinding halt on that processor.

3.15.1 Completion Detection

Completion detection is a method for automatically detecting completion of a distributed process within an application. It is a module, and therefore is only included when ``-module completion'' is specified when linking your application.

Completion is reached within a distributed process when the participating objects have produced and consumed an equal number of events globally. The number of global events that will be produced and consumed does not need to be known, just the number of producers is required.

First, the detector should be constructed. This call would typically belong in application startup code (it initializes the group that keeps track of completion):

CProxy_CompletionDetector detector = CProxy_CompletionDetector::ckNew();

When it is time to start completion detection, make the following call to the library:

void start_detection(int num_producers, CkCallback start, CkCallback finish,
int prio_)

detector.start_detection(10, CkCallback(CkIndex_chare1::start_test(0),
                                       thisProxy),
                             CkCallback(CkIndex_chare2::finish_test(0),
                                       thisProxy), 0);

The num_producers parameter is the number of objects (chares) that will produce elements. So if every array element will produce one event, then it would be the size of the array.

The start callback notifies your program that it is safe to begin producing and consuming (this state is reached when the module has finished its internal initialization).

The finish callback is instigated when completion has been detected (all objects participating have produced and consumed an equal number of elements globally).

The prio parameter is the priority that the detector will run at. This is still under development, but it should be set below the application's priority if possible.

Once initialization is complete (the ``start'' callback is triggered), make the following call to the library:

void CompletionDetector::produce(int events_produced)
void CompletionDetector::produce() // 1 by default

detector.ckLocalBranch()->produce(4);

Once all the ``events'' that this chare is going to produce have been sent out, make the following call:

void CompletionDetector::done(int producers_done)
void CompletionDetector::done() // 1 by default

detector.ckLocalBranch()->done();

The application can now start consuming elements, using the following calls:

void CompletionDetector::consume(int events_consumed)
void CompletionDetector::consume() // 1 by default

detector.ckLocalBranch()->consume();

At some point, when everyone is consuming elements, completion will be reached. The system will detect that state and will trigger the finish callback. At that point, start_detection can be called again to restart the process.