3 CHARM++ Programming

3.0.1 What's the basic programming model for Charm++?

Parallel objects using "Asynchronous Remote Method Invocation":

Asynchronous

in that you do not block until the method returns-the caller continues immediately.

Remote

in that the two objects may be separated by a network.

Method Invocation

in that it's just C++ classes calling each other's methods.

3.0.2 What is an ``entry method''?

Entry methods are all the methods of a chare where messages can be sent by other chares. They are declared in the .ci files, and they must be defined as public methods of the C++ object representing the chare.

3.0.3 When I invoke a remote method, do I block until that method returns?

No! This is one of the biggest differences between Charm++ and most other ``remote procedure call'' systems like CORBA, Java RMI, or RPC. ``Invoke an asynchronous method'' and ``send a message'' have exactly the same semantics and implementation. Since the invoking method does now wait for the remote method to terminate, it normally cannot receive any return value. (see later for a way to return values)

3.0.4 Why does Charm++ use asynchronous methods?

Asynchronous method invocation is more efficient because it can be implemented as a single message send. Unlike with synchronous methods, thread blocking and unblocking and a return message are not needed.

Another big advantage of asynchronous methods is that it's easy to make things run in parallel. If I execute:

a->foo();
b->bar();

Now foo and bar can run at the same time; there's no reason bar has to wait for foo.

3.0.5 Can I make a method synchronous? Can I then return a value?

Yes. If you want synchronous methods, so the caller will block, use the [sync] keyword before the method in the .ci file. This requires the sender to be a threaded entry method, as it will be suspended until the callee finishes. Sync entry methods are allowed to return values to the caller.

3.0.6 What is a threaded entry method? How does one make an entry method threaded?

A threaded entry method is an entry method for a chare that executes in a separate user-level thread. It is useful when the entry method wants to suspend itself (for example, to wait for more data). Note that threaded entry methods have nothing to do with kernel-level threads or pthreads; they run in user-level threads that are scheduled by Charm++ itself.

In order to make an entry method threaded, one should add the keyword threaded withing square brackets after the entry keyword in the interface file:

module M {
  chare X {
    entry [threaded] E1(void);
  };
};

3.0.7 If I don't want to use threads, how can an asynchronous method return a value?

The usual way to get data back to your caller is via another invocation in the opposite direction:

void A::start(void) {
  b->giveMeSomeData();
}
void B::giveMeSomeData(void) {
  a->hereIsTheData(data);
}
void A::hereIsTheData(myclass_t data) {
  ...use data somehow...
}

This is contorted, but it exactly matches what the machine has to do. The difficulty of accessing remote data encourages programmers to use local data, bundle outgoing requests, and develop higher-level abstractions, which leads to good performance and good code.

3.0.8 Isn't there a better way to send data back to whoever called me?

The above example is very non-modular, because b has to know that a called it, and what method to call a back on. For this kind of request/response code, you can abstract away the ``where to return the data'' with a CkCallback object:

void A::start(void) {
  b->giveMeSomeData(CkCallback(CkIndex_A::hereIsTheData,thisProxy));
}
void B::giveMeSomeData(CkCallback returnDataHere) {
  returnDataHere.send(data);
}
void A::hereIsTheData(myclass_t data) {
  ...use data somehow...
}

Now b can be called from several different places in a, or from several different modules.

3.0.9 Why should I prefer the callback way to return data rather than using [sync] entry methods?

There are a few reasons for that:

• The caller needs to be threaded, which implies some overhead in creating the thread. Moreover the threaded entry method will suspend waiting for the data, preventing any code after the remote method invocation to proceed in parallel.

• Threaded entry methods are still methods of an object. While they are suspended other entry methods for the same object (or even the same threaded entry method) can be called. This allows for potential problems if the suspending method does leave some objects in an inconsistent state.

• Finally, and probably most important, [sync] entry methods can only be used to return a value that can be computed by a single chare. When more flexibility is needed, such in cases where the resulting value needs to the contribution of multiple objects, the callback methodology is the only one available. The caller could for example send a broadcast to a chare array, which will use a reduction to collect back the results after they have been computed.

3.0.10 How does the initializazion in Charm work?

Each processor executes the following operations strictly in order:

1. All methods registered as initnode;
2. All methods registered as initproc;
3. On processor zero, all mainchares constructor method is invoked (the ones taking a CkArgMsg*);
4. The read-onlies are propagated from processor zero to all other processors;
5. The nodegroups are created;
6. The groups are created. During this phase, for all the chare arrays have been created with a block allocation, the corresponding array elements are instantiated;
7. Initialization terminated and all messages are available for processing, including the messages responsible for the instantiation of array elements manually inserted.

This implies that you can assume that the previous steps has completely finished before the next one starts, and any side effect from all the previous steps are committed (and can therefore be used).

Inside a single step there is no order guarantee. This implies that, for example, two groups allocated from mainchare can be instantiated in any order. The only exception to this is processor zero, where chare objects are instantiated immediately when allocated in the mainchare, i.e if two groups are allocated, their order is fixed by the allocation order in the mainchare constructing them. Again, this is only valid for processor zero, and in no other processor this assumption should be made.

To notice that if array elements are allocated in block (by specifying the number of elements at the end of the ckNew function), they are all instantiated before normal execution is resumed; if manual insertion is used, each element can be constructed at any time on its home processor, and not necessarily before other regular communication messages have been delivered to other chares (including other array elements part of the same array).

3.0.11 Does Charm++ support C and Fortran?

3.0.12 What is a proxy?

A proxy is a local C++ class that represents a remote C++ class. When you invoke a method on a proxy, it sends the request across the network to the real object it represents. In Charm++, all communication is done using proxies.

A proxy class for each of your classes is generated based on the methods you list in the .ci file.

3.0.13 What are the different ways one can can create proxies?

Proxies can be:

• Created using ckNew. This is the only method that actually creates a new parallel object. ``CProxy_A::ckNew(...)'' returns a proxy, as described in the manual.

• Copied from an existing proxy. This happens when you assign two proxies or send a proxy in a message.

• Created from a ``handle''. This happens when you say ``CProxy_A p=thishandle;''

• Created uninitialized. This is the default when you say ``CProxy_A p;''. You'll get a runtime error ``proxy has not been initialized'' if you try to use an uninitialized proxy.

3.0.14 What is wrong if I do A *ap = new CProxy_A(handle)?

This will not compile, because a CProxy_A is not an A. What you want is CProxy_A *ap = new CProxy_A(handle).

3.0.15 Why is the def.h usually included at the end? Is it necessary or can I just include it at the beginning?

You can include the def.h file once you've actually declared everything it will reference- all your chares and readonly variables. If your chares and readonlies are in your own header files, it is legal to include the def.h right away.

However, if the class declaration for a chare isn't visible when you include the def.h file, you'll get a confusing compiler error. This is why we recommend including the def.h file at the end.

3.0.16 How can I use a global variable across different processors?

Make the global variable ``readonly'' by declaring it in the .ci file. Remember also that read-onlies can be safely set only in che mainchare constructor. Any change after the mainchare constructor has finished will be local to the processor that made the change. To change a global variable later in the program, every processor must modify it accordingly (e.g by using a chare group. Note that chare arrays are not guaranteed to cover all processors)

3.0.17 Can I have a class static read-only variable?

One can have class-static variables as read-onlies. Inside a chare, group or array declaration in the .ci file, one can have a readonly variable declaration. Thus:

chare someChare {
  ...
  readonly CkGroupID someGroup;
  ...
};

is fine. In the .h declaration for class someChare, you will have have to put someGroup as a public static variable, and you are done.

You then refer to the variable in your program as someChare::someGroup.

3.0.18 How do I measure the time taken by a program or operation?

You can use CkWallTimer() to determine the time on some particular processor. To time some parallel computation, you need to call CkWallTimer on some processor, do the parallel computation, then call CkWallTimer again on the same processor and subtract.

3.0.19 What do CmiAssert and CkAssert do?

These are just like the standard C++ assert calls in <assert.h>- they call abort if the condition passed to them is false.

We use our own version rather than the standard version because we have to call CkAbort, and because we can turn our asserts off when CMK_OPTIMIZE is defined.

3.0.20 Can I know how many messages are being sent to a chare?

No.

There is no nice library to solve this problem, as some messages might be queued on the receiving processor, some on the sender, and some on the network. You can still:

• Send a return receipt message to the sender, and wait until all the receipts for the messages sent have arrived, then go to a barrier;
• Do all the sends, then wait for quiescence.

3.0.21 What is "quiescence"? How does it work?

Quiescence is When nothing is happening anywhere on the parallel machine.

A low-level background task counts sent and received messages. When, across the machine, all the messages that have been sent have been received, and nothing is being processed, quiescence is triggered.

3.0.22 Should I use quiescence detection?

Probably not.

In some ways, quiescence is a very strong property (it guarentees nothing is happening anywhere) so if some other library is doing something, you won't reach quiescence. In other ways, quiescence is a very weak property, since it doesn't guarentee anything about the state of your application like a reduction does, only that nothing is happening. Because quiescence detection is on the one hand so strong it breaks modularity, and on the other hand is too weak to guarentee anything useful, it's often better to use something else.

Often global properties can be replaced by much easier-to-compute local properties. For example, my object could wait until all its neighbors have sent it messages (a local property my object can easily detect by counting message arrivals), rather than waiting until all neighbor messages across the whole machine have been sent (a global property that's difficult to determine). Sometimes a simple reduction is needed instead of quiescence, which has the benefits of being activated explicitly (each element of a chare array or chare group has to call contribute) and allows some data to be collected at the same time. A reduction is also a few times faster than quiescence detection. Finally, there are a few situations, such as some tree-search problems, where quiescence detection is actually the most sensible, efficient solution.