3.1 Writing a Pup Routine

The runtime system can automatically move your thread stack to the new processor, but unless you use isomalloc, you must write a pup routine to move any global or heap-allocated data to the new processor. A pup (Pack/UnPack) routine can perform both packing (converting your data into a network message) and unpacking (converting the message back into your data). A pup routine is passed a pointer to your data block and a special handle called a ``pupper'', which contains the network message.

In a pup routine, you pass all your heap data to routines named pup_type or fpup_type, where type is either a basic type (such as int, char, float, or double) or an array type (as before, but with a ``s'' suffix). Depending on the direction of packing, the pupper will either read from or write to the values you pass- normally, you shouldn't even know which. The only time you need to know the direction is when you are leaving a processor, or just arriving. Correspondingly, the pupper passed to you may be deleting (indicating that you are leaving the processor, and should delete your heap storage after packing), unpacking (indicating you've just arrived on a processor, and should allocate your heap storage before unpacking), or neither (indicating the system is merely sizing a buffer, or checkpointing your values).

pup functions are much easier to write than explain- a simple C heap block and the corresponding pup function is:

     typedef struct {
       int n1;/*Length of first array below*/
       int n2;/*Length of second array below*/
       double *arr1; /*Some doubles, allocated on the heap*/
       int *arr2; /*Some ints, allocated on the heap*/
     } my_block;
 
     void pup_my_block(pup_er p,my_block *m)
     {
       if (pup_isUnpacking(p)) { /*Arriving on new processor*/
         m->arr1=malloc(m->n1*sizeof(double));
         m->arr2=malloc(m->n2*sizeof(int));
       }
       pup_doubles(p,m->arr1,m->n1);
       pup_ints(p,m->arr2,m->n2);
       if (pup_isDeleting(p)) { /*Leaving old processor*/
         free(m->arr1);
         free(m->arr2);
       }
     }

This single pup function can be used to copy the my_block data into a message buffer and free the old heap storage (deleting pupper); allocate storage on the new processor and copy the message data back (unpacking pupper); or save the heap data for debugging or checkpointing.

A Fortran block TYPE and corresponding pup routine is as follows:

     MODULE my_block_mod
       TYPE my_block
         INTEGER :: n1,n2x,n2y
         DOUBLE PRECISION, ALLOCATABLE, DIMENSION(:) :: arr1
         INTEGER, ALLOCATABLE, DIMENSION(:,:) :: arr2
       END TYPE
     END MODULE
 
     SUBROUTINE pup_my_block(p,m)
       IMPLICIT NONE
       USE my_block_mod
       USE pupmod
       INTEGER :: p
       TYPE(my_block) :: m
       IF (fpup_isUnpacking(p)) THEN
         ALLOCATE(m%arr1(m%n1))
         ALLOCATE(m%arr2(m%n2x,m%n2y))
       END IF
       call fpup_doubles(p,m%arr1,m%n1)
       call fpup_ints(p,m%arr2,m%n2x*m%n2y)
       IF (fpup_isDeleting(p)) THEN
         DEALLOCATE(m%arr1)
         DEALLOCATE(m%arr2)
       END IF
     END SUBROUTINE

You indicate to TCHARM that you want a pup routine called using the routine below. An arbitrary number of blocks can be registered in this fashion.

void TCHARM_Register(void *block, TCharmPupFn pup_fn)
SUBROUTINE TCHARM_Register(block,pup_fn)
TYPE(varies), POINTER :: block
SUBROUTINE :: pup_fn

Associate the given data block and pup function. Can only be called from the parallel context. For the declarations above, you call TCHARM_Register as:

          /*In C/C++ driver() function*/
          my_block m;
          TCHARM_Register(m,(TCharmPupFn)pup_my_block);
 
          !- In Fortran driver subroutine
          use my_block_mod
          interface
            subroutine pup_my_block(p,m)
              use my_block_mod
              INTEGER :: p
              TYPE(my_block) :: m
            end subroutine
          end interface
          TYPE(my_block), TARGET :: m
          call TCHARM_Register(m,pup_my_block)

Note that the data block must be allocated on the stack. Also, in Fortran, the "TARGET" attribute must be used on the block (as above) or else the compiler may not update values during a migration, because it believes only it can access the block.

void TCHARM_Migrate()
subroutine TCHARM_Migrate()

Informs the load balancing system that you are ready to be migrated, if needed. If the system decides to migrate you, the pup function passed to TCHARM_Register will first be called with a sizing pupper, then a packing, deleting pupper. Your stack and pupped data will then be sent to the destination machine, where your pup function will be called with an unpacking pupper. TCHARM_Migrate will then return. Can only be called from in the parallel context.