2.1 PROJECTIONS API for CHARM++ Applications

2.1.1 Selective Tracing

CHARM++ allows user to start/stop tracing the execution at certain points in time on the local processor. Users are advised to make these calls on all processors and at well-defined points in the application.

Users may choose to have instrumentation turned off at first (by command line option +traceoff - see section 2.3) if some period of time in middle of the applications execution is of interest to the user.

Alternatively, users may start the application with instrumentation turned on (default) and turn off tracing for specific sections of the application.

Again, users are advised to be consistent as the +traceoff runtime option applies to all processors in the application.

• void traceBegin()

  Enables the runtime to trace events (including all user events) on the local processor where traceBegin is called.

• void traceEnd()

  Prevents the runtime from tracing events (including all user events) on the local processor where traceEnd is called.

2.1.2 User Events

PROJECTIONS has the ability to visualize traceable user specified events. User events are usually displayed in the Timeline view as vertical bars above the entry methods. Alternatively the user event can be displayed as a vertical bar that vertically spans the timelines for all processors. Follow these following basic steps for creating user events in a charm++ program:

1. Register an event with an identifying string and either specify or acquire a globally unique event identifier. All user events that are not registered will be displayed in white.

2. Use the event identifier to specify trace points in your code of interest to you.

The functions available are as follows:

• int traceRegisterUserEvent(char* EventDesc, int EventNum=-1)

  This function registers a user event by associating EventNum to EventDesc. If EventNum is not specified, a globally unique event identifier is obtained from the runtime and returned. The string EventDesc must either be a constant string, or it can be a dynamically allocated string that is NOT freed by the program. If the EventDesc contains a substring ``***'' then the Projections Timeline tool will draw the event vertically spanning all PE timelines.

  EventNum has to be the same on all processors. Therefore use one of the following methods to ensure the same value for any PEs generating the user events:

  1. Call traceRegisterUserEvent on PE 0 in main::main without specifying an event number, and store returned event number into a readonly variable.
  2. Call traceRegisterUserEvent and specify the event number on processor 0. Doing this on other processors would have no effect. Afterwards, the event number can be used in the following user event calls.

  Eg. traceRegisterUserEvent("Time Step Begin", 10);

  Eg. eventID = traceRegisterUserEvent(``Time Step Begin'');

There are two main types of user events, bracketed and non bracketed. Non-bracketed user events mark a specific point in time. Bracketed user events span an arbitrary contiguous time range. Additionally, the user can supply a short user supplied text string that is recorded with the event in the log file. These strings should not contain newline characters, but they may contain simple html formatting tags such as <br>, <b>, <i>, <font color=#ff00ff>, etc.

The calls for recording user events are the following:

• void traceUserEvent(int EventNum)

  This function creates a user event that marks a specific point in time.

  Eg. traceUserEvent(10);

• void traceUserBracketEvent(int EventNum, double StartTime, double EndTime)

  This function records a user event spanning a time interval from StartTime to EndTime. Both StartTime and EndTime should be obtained from a call to CmiWallTimer() at the appropriate point in the program.

  Eg.

     traceRegisterUserEvent("Critical Code", 20); // on PE 0
     double critStart = CmiWallTimer();;  // start time
     // do the critical code
     traceUserBracketEvent(20, critStart,CmiWallTimer());
  

• void traceUserSuppliedNote(char * note)

  This function records a user specified text string at the current time.

• void traceUserSuppliedBracketedNote(char *note, int EventNum, double StartTime, double EndTime)

  This function records a user event spanning a time interval from StartTime to EndTime. Both StartTime and EndTime should be obtained from a call to CmiWallTimer() at the appropriate point in the program.

  Additionally, a user supplied text string is recorded, and the EventNum is recorded. These events are therefore displayed with colors determined by the EventNum, just as those generated with traceUserBracketEvent are.

2.1.3 Function-level Tracing for Adaptive MPI Applications

Adaptive MPI (AMPI) is an implementation of the MPI interface on top of CHARM++. As with standard MPI programs, the appropriate semantic context for performance analysis is captured through the observation of MPI calls within C/C++/Fortran functions. Unfortunately, AMPI's implementation does not grant the runtime access to information about user function calls. As a result, the tracing framework must provide an explicit API for capturing this piece of performance information in addition to MPI calls (which are known to the runtime).

The functions, similar to those used to capture user events, are as follows:

• int _TRACE_REGISTER_FUNCTION_NAME(const char *name);
  

  This function registers an AMPI function name. The tracing framework assigns to function name a unique id and returns it. It is the user's responsibility to ensure that name is unique and consistent.

  This registration function should be called near the start of the application, just after MPI_Init.

• int _TRACE_REGISTER_FUNCTION_ID(const char *name, int idx);
  

  This function registers an AMPI function name to be associated explicitly to the id idx. It is the user's responsibility to ensure that name as well as idx are unique and consistent. The tracing framework returns idx.

  This is an alternative registration function and should be called near the start of the application just after MPI_Init.

• void _TRACE_BEGIN_FUNCTION_NAME(const char *name);
  

  This function tells the tracing framework to record a begin event associated with the registered function name. If this were called in a C/C++ environment with pre-processor support, the line number of the function call will also be recorded.

• void _TRACE_BEGIN_FUNCTION_ID(int idx);
  

  This function tells the tracing framework to record a begin event associated with the registered function indexed by idx. If this were called in a C/C++ environment with pre-processor support, the line number of the function call will also be recorded.

• void _TRACE_END_FUNCTION_NAME(const char *name);
  

  This function tells the tracing framework to record a end event associated with the registered function name.

• void _TRACE_END_FUNCTION_ID(int idx);
  

  This function tells the tracing framework to record a end event associated with the registered function indexed by idx.

AMPI function events captured by the use of this API are recognized by the visualization system and used for special AMPI-specific views in addition to standard CHARM++ entry methods.