1.4 Structure of an AMPI FEM Framework Program

In addition to the classic init/driver structure above, you can write an FEM framework program using the MPI style. This is a more general, more flexible method of running the program, but it is more complicated than the classic mode. All FEM framework calls are available in either mode.

   main program
      MPI_Init
      FEM_Init(MPI_COMM_WORLD)
      if (I am master processor)
         read mesh
      partition mesh
      time loop
          FEM computations
          communicate boundary conditions
          more FEM computations
      end time loop
   end main program

In this mode, the FEM framework does not set a default reading or writing mesh, and does no partitioning; so you must use the FEM_Mesh routines to create and partition your mesh. See the AMPI manual for details on how to declare the main routine.

The driver() portion of a classic FEM program strongly resembles an MPI mode main routine--in fact, a classic FEM program can even make MPI calls from its driver() routine, because the FEM framework is implemented directly on top of MPI.

There is even a special shell script for collecting up the FEM framework source code to build a non-Charm, MPI-only version of the FEM framework. To build FEM in this manner, you first build Charm++ normally, then run a script to collect up the neccessary source files (the FEM framework, a small number of Charm configuration and utility files, and the METIS library), and finally build the library using the usual MPI compiler commands:

 > cd charm/
 > ./src/libs/ck-libs/fem/make_fem_alone.sh
 > cd fem_alone/
 > mpicc -I. -DFEM_ALONE=1 -c *.c *.C
 > ar cr libfem_alone.a *.o

You will then have to build your application with the MPI compilers, and manually point to this ``fem_alone'' directory to find include files and the new FEM library. A typical compiler invocation would be:

 > mpif90 -I$HOME/charm/fem_alone -L$HOME/charm/fem_alone foo.f90 -lfem_alone -o foo

This ``standalone'', non-Charm++ method of building the FEM framework prevents the use of load balancing or the other features of Charm++, so we do not recommend it for normal use.