Conversion to AMPI

In order to convert existing MPI application components to AMPI, one has to make sure that variables global in scope (such as common blocks, global variables etc) are not defined before and used after a blocking call, such as MPI_Recv. The reason for this restriction is straightforward. If a global variable is defined before a blocking call, it may be modified by another user-level thread when the defining thread blocks and another thread is scheduled. Thus, after returning from a blocked MPI call, the original thread will see a different value of that variable. In order to use AMPI, such variables should be localized (copied to variables local to the subroutines, which are on stack) or privatized (made accessible only through thread-private variables).

One typically finds three kind of global variables in MPI programs. The first type are variables that are initialized at startup (say by reading them from a configuration file), and never modified. This type of variables need not be privatized for each thread, since each thread sets and expects the same values. The second type are temporary variables that, though they have global storage scope, are defined and used only within the scope of a small set of subroutines. If there are no blocking MPI calls made while the variables are live, this kind of variable need not be privatized because AMPI threads are non-preemptive. Finally, there are truly global variables, which have different values for each thread but long lifetimes. These global variables must be privatized. Careful inspection of the program may reveal such variables.

However, sometimes such careful inspection may not be possible. In that case, we have devised a method to systematically put all the global variables in a private area allocated dynamically or on thread's stack. The idea is to make a user-defined type, and make all the global variables members of that type. In the main program, we allocate a variable of that type, and then pass a pointer to that variable to every subroutine that makes use of global variables. Access to these global variables in such subroutines should be made through this pointer. This is equivalent to converting the original procedural component an object-based component, where the object state encapsulates the previously global data, and all the procedures that operated on these global data become that object's instance methods.

{CodeOne}
MODULE shareddata
  INTEGER :: myrank
  DOUBLE PRECISION :: xyz(100)
END MODULE

PROGRAM MAIN
  USE shareddata
  include 'mpif.h'
  INTEGER :: i, ierr
  CALL MPI_Init(ierr)
  CALL MPI_Comm_rank(MPI_COMM_WORLD, myrank, ierr)
  DO i = 1, 100
    xyz(i) =  i + myrank
  END DO
  CALL subA
  CALL MPI_Finalize(ierr)
END PROGRAM

SUBROUTINE subA
  USE shareddata
  INTEGER :: i
  DO i = 1, 100
    xyz(i) = xyz(i) + 1.0
  END DO
  CALL MPI_Send(....)
c blocking call: potential context-switch
  CALL MPI_Recv(....)
END SUBROUTINE

Figure 5.7: Original MPI program

This conversion process is illustrated in figures 5.7, 5.8, and 5.9. Figure 5.7 shows the original MPI code. The main program and subroutine subA share an array xyz and a variable myrank as global variables. We first aggregate the shared global variables into a user-defined type called chunk as shown in figure 5.8. We change the main program to be a subroutine MPI_Main. Then we modify the MPI_Main subroutine to dynamically allocate a thread-private variable of the chunk type and change the references to them. Subroutine subA is then modified to take this variable as argument. Code in figure 5.9 shows the converted AMPI program.

{CodeTwo}
MODULE shareddata
  TYPE chunk
    INTEGER :: myrank
    DOUBLE PRECISION :: xyz(100)
  END TYPE
END MODULE

Figure 5.8: Conversion to AMPI: Shared data is aggregated in a user-defined type.

{CodeThree}
SUBROUTINE MPI_Main
  USE shareddata
  USE AMPI
  INTEGER :: i, ierr
  TYPE(chunk), pointer :: c
  CALL MPI_Init(ierr)
  ALLOCATE(c)
  CALL MPI_Comm_rank(MPI_COMM_WORLD, c%myrank, ierr)
  DO i = 1, 100
    c%xyz(i) =  i + c%myrank
  END DO
  CALL subA(c)
  CALL MPI_Finalize(ierr)
END SUBROUTINE

SUBROUTINE subA(c)
  USE shareddata
  TYPE(chunk) :: c
  INTEGER :: i
  DO i = 1, 100
    c%xyz(i) = c%xyz(i) + 1.0
  END DO
END SUBROUTINE

Figure 5.9: Conversion to AMPI: References to shared data are made through thread-private variables.

This conversion process, though mechanical, is cumbersome, and can indeed be automated. We have developed AMPIzer^5.2 [56], a simple prototype source-to-source translator, based on the Polaris [13] compiler front end. AMPIzer can recognize all global variables in FORTRAN90 or FORTRAN77 programs and automatically converts the program to be a threaded AMPI component.

Large scientific applications are typically built from combining multiple MPI modules together. These MPI modules themselves are typically derived from complete MPI applications. Interaction between these modules occurs using function calls into each other, and exhibit the same limitations as interfaces derived from the object models as described in chapter 3. AMPI can, in principle, use the Charisma binding for Charm++ by providing parallel Charm++ wrappers for the AMPI threaded component. However, we need to provide an interface model for AMPI that is consistent with MPI's paradigm, so that adapting legacy MPI codes to Charisma becomes a less-daunting task. AMPI implementation of the Charisma interface model is described in the next section.