4 Combining Frameworks

This section describes how to combine multiple frameworks in a single application. You might want to do this, for example, to use AMPI communication inside a finite element method solver.

You specify how you want the frameworks to be combined by writing a special setup routine that runs when the program starts. The setup routine must be named TCHARM_User_setup. If you declare a user setup routine, the standard framework setup routines (such as the FEM framework's init routine) are bypassed, and you do all the setup in the user setup routine.

The setup routine creates a set of threads and then attaches frameworks to the threads. Several different frameworks can be attached to one thread set, and there can be several sets of threads; however, the most frameworks cannot be attached more than once to single set of threads. That is, a single thread cannot have two attached AMPI frameworks, since the MPI_COMM_WORLD for such a thread would be indeterminate.

void TCHARM_Create(int nThreads, TCharmThreadStartFn thread_fn)
SUBROUTINE TCHARM_Create(nThreads,thread_fn)
INTEGER, INTENT(in) :: nThreads
SUBROUTINE :: thread_fn

Create a new set of TCHARM threads of the given size. The threads will execute the given function, which is normally your user code. You should call TCHARM_Get_num_chunks() to get the number of threads from the command line. This routine can only be called from your TCHARM_User_setup routine.

You then attach frameworks to the new threads. The order in which frameworks are attached is irrelevant, but attach commands always apply to the current set of threads.

The commands to attach the various frameworks are:

AMPI

You attach a new MPI_COMM_WORLD to the current threads using the MPI_Attach routine, which takes a string, the name of the communicator. These threads will then be able to use AMPI calls.

FEM Framework

You attach the current FEM mesh to the current threads using the FEM_Attach routine, which takes an integer which is normally zero. These threads will then be able to use FEM calls. Be sure to set up an FEM mesh using the FEM_Set calls before your FEM_Attach.

Multiblock Framework

You attach the current multiblock mesh to the current threads using the MBLK_Attach routine, which takes no parameters. These threads will then be able to use Multiblock calls. Be sure to set up the multiblock mesh using the MBLK_Set calls before your MBLK_Attach.

For example, the C code to start an application having one set of threads that performs both AMPI communication and FEM computations inside a routine called myDriver would be:

     #include "tcharmc.h"
     #include "ampi.h"
     #include "fem.h"
     
     /* Called to start the program */
     void TCHARM_User_setup(void)
     {
         TCHARM_Create(TCHARM_Get_num_chunks(),myDriver);
         MPI_Attach("myAMPIFEM");        
         ... Usual FEM_Set calls, as normally made from init() ...
         FEM_Attach(0);
     }

The Fortran code to start an application consisting of two sets of threads, both of which perform AMPI communication, would be:

     SUBROUTINE tcharm_user_setup()
       IMPLICIT NONE
       include 'tcharmf.h'
       include 'ampif.h'
       
    ! First set of threads, running "compute1"
       call tcharm_create(tcharm_get_num_chunks(),compute1)
       call ampi_attach('part1')
       
    ! Second set of threads, running "compute2"
       call tcharm_create(tcharm_get_num_chunks(),compute2)
       call ampi_attach('part2')
     END SUBROUTINE