\documentclass[11pt]{article}

\newif\ifpdf
\ifx\pdfoutput\undefined
  \pdffalse
\else
  \pdfoutput=1
  \pdftrue
\fi

\ifpdf
  \pdfcompresslevel=9
  \usepackage[pdftex,colorlinks=true,plainpages=false]{hyperref}
\else
\fi

\pagestyle{headings}

\title{Fortran90 Bindings for Charm++\footnote{last modified 3/3/2001 by Gengbin Zheng}}
\author{Gengbin Zheng, Jayant Desouza}

\begin{document}

\maketitle

To interface Fortran90 to Charm and thus obtain a parallel version of
your program you need to do the following things:
\begin{enumerate}
\item Write a Charm Interface file (extension .ci)
\item Write your F90 program with some constraints
\item Compile and Link with Charm's Fortran library
\item Run it !
\end{enumerate}

\section{Overview}

Here I suppose you've already known most concepts in charm++ and done some 
charm++ programming.  \\
Unlike in C++, we don't have class here in Fortran90. Thus, Chare is 
represented as a fortran type structure. Here is an example:

\begin{verbatim}

      ## Just replace Hello throughout with your chare's name. ##
      ## and add your chare's personal data below where indicated ##
      ## Everything else remains the same ##
      MODULE HelloMod

      TYPE Hello
      ## your chare's data goes here, the integer below is an example ##
      integer data
      END TYPE

      TYPE HelloPtr
      TYPE (Hello), POINTER ::  obj
      integer*8 aid
      END TYPE

      END MODULE
\end{verbatim}
You can think of this module as a chare declaration. Type [Hello] defines 
arbitary user private data and HelloPtr defines the Chare pointer which 
fortran program will use later to communicate with the f90charm runtime 
library. \\
As same in C++ charm program, you need to write a .ci interface file
so that charm translator will generate helper functions. However, for
Fortran90 charm, the syntax is a little different. First, it only support
Chare array(1D currently), you cannot define Chare and Group types. Second, 
for the message declaration, you must list all the data fields explicitly in
.ci files.  By this, the helper functions generated by translator know the 
contents of the messages and thus can do the message marshalling and 
unmarshalling for the fortran program. 

For each Chare defined in the .ci file, user must write these functions
for charm++ f90 runtime:

  \verb+SUBROUTINE <ChareName>_allocate(objPtr, aid, index)+

  You can think of this function as a constructor for each array element 
with array index [index]. In this function user must allocate memory for 
the Chare's user data. User can also initialize the data.	\\
  For each chare entry method, you should write the fortran90 subroutine
for it:

  \verb+SUBROUTINE entry(charePtr, myIndex, data1, data2 ... )+

  Note, the first argument is the Chare pointer as you declared previously, the second argument is the array index which will be passed from charm runtime. And you need to enumerate all the message fields declared in .ci file in the 
subroutine.

On the other side, for each Chare declared in .ci files, these subroutines are 
available for user to call in Fortran90 program:

  \verb+<ChareName>_CkNew(integer n, integer*8 aid)+

  This subroutine creates the chare array.

And for each entry method, this function is available for use:

  \verb+SendTo_<ChareName>_<Entry>(charePtr, myIndex, data1, data2 ... )+

  This subroutine will send a message to the array element with the index
as myIndex. Fortran90 charm runtime will pack the parameters into a message
and sent to the chare array element.

There are several others things you need to know.

First, as same in C++ Charm, each .ci file will generate two header files:
.decl.h and .def.h. However, in Fortran90 charm, you are not able to include 
these C++ files in Fortran90 code. Thus, currently, it is user's 
responsibility to write a simple C++ code to include these two headers files. 
You should also declare readonly variables in this file. This is as simple as 
this:

\begin{verbatim}
#include "hello.decl.h"
int chunkSize; 	// readonly variables define here
#include "hello.def.h"
\end{verbatim}

In future, this file can be generated automatically by translator.
Second, you can still use readonly variables as in Charm++. However, since
fortran90 lacks the concepts of global variables as in C, you have to use it
explicitly. Here are the two helper functions that translator generates:
take the readonly variable chunkSize as an example,
\begin{verbatim}
Set_Chunksize(chunkSize);
Get_Chunksize(chunkSize);
\end{verbatim}
These two functions can be used in user's fortran program to set and get 
readonly variables.

Third, for user's convenience, several charm++ runtime library functions
have their Fortran interface defined in f90charm library. These currently
include:
\begin{verbatim}
CkExit()
CkMyPe(integer mype)
CkNumPes(integer pes)
CkPrintf(...)    // note, the format string must terminated with '$$'
\end{verbatim}

Here is a summary of current constraints to write f90 binding charm++ programs:
\begin{enumerate}
\item in .ci files, only 1D Chare array is supported.
\item messages defined in .ci files must be basic types or fixed size array.
\item readonly variables must be basic types.
\item instead of program main, your main program starts from subroutine 
f90charmmain.
\end{enumerate}

All these are best explained with an example: the hello program.  It is a
simple ring program.  When executed, an array of several parallel
CHAREs is created.  Each chare "says" hello when it receives a
message, and then sends a message to the next chare.  The Fortran main() 
subroutine starts off the events.  And the SayHi() subroutine does the 
say-hello and forward.

\section{Writing F90 Program}
To start, you need to create a Fortran Module to represent a chare,
e.g. \{ChareName\}Mod.

\begin{verbatim}

      ## Just replace Hello throughout with your chare's name. ##
      ## and add your chare's personal data below where indicated ##
      ## Everything else remains the same ##
      MODULE HelloMod

      TYPE Hello
      integer data
      ## your chare's data goes here, the above integer is an example ##
      END TYPE

      TYPE HelloPtr
      TYPE (Hello), POINTER ::  obj
      integer*8 aid
      END TYPE

      END MODULE
\end{verbatim}

In the Fortran file you must write an allocate funtion for this chare
with the name: \{ChareName\}\_allocate.

\begin{verbatim}
      ## Just replace Hello throughout with your chare's name. ##
      ## Everything else remains the same ##
      SUBROUTINE Hello_allocate(objPtr, aid, index)
      USE HelloMod
      TYPE(HelloPtr) objPtr 
      integer*8 aid
      integer index

      allocate(objPtr%obj)
      objPtr%aid = aid;
      ## you can initialize the Chare user data here
      objPtr%obj%data = index
      END SUBROUTINE
\end{verbatim}

If you have written code for a chare array "Hello", as above, you will
have the following subroutine available to you:

  \verb+<ChareName>_CkNew(integer n, integer*8 aid)+
     This function creates a chare array at runtime
     where n is the number of items in the array, and
           aid is the array ID which is returned to you.

We will see an example of using this function below.

\section{Writing Charm++ Interface File}

Now that you have the chare, you need to write one or more ENTRY
POINTs for it.  This is nothing but a Fortran subroutine that uses the
above module.
\begin{verbatim}
      ## p1, p2, etc represent user parameters
      ## the "objPtr, myIndex" stuff is required in every Entry Point.
      ## CkExit() must be called by the chare to terminate.
      SUBROUTINE SayHi(objPtr, myIndex, p1)
      USE HelloMod
      IMPLICIT NONE

      TYPE(HelloPtr) objPtr
      integer myIndex
      integer p1

      objPtr%obj%data = 20
      if (myIndex < 4) then
          call SendTo_Hello_SayHi(objPtr%aid, myIndex+1, 1);
      else 
          call CkExit()
      endif
\end{verbatim}
Once you have written code for an Entry Point, you now have available
to you the following subroutine:
\begin{verbatim}
   SendTo_<ChareName>_<SubroutineName>(integer*8 aid, integer myIndex,
                          other parameters ....)
     where aid is the target chare array ID
           myIndex is the specific within the array
                   we wish to send a message to
           other parameters are the parameters in the above function.
\end{verbatim}

\section{Compile and Link}
You must have a Fortran f90charmmain() subroutine where execution starts.
Typically this main function will create a chare array and send a
message to one of its members to get the computation started.
\begin{verbatim}
      SUBROUTINE f90charmmain()
      USE HelloMod
      integer i
      integer*8 aid

      call CkPrintf("Hello\n $$")

      call Hello_CkNew(5, aid)

      call SendTo_Hello_SayHi(aid, 0, 1);

      END
\end{verbatim}
\section{Writing Charm++ Interface File}
Now we move on to Step 2, the Charm Interface file.  This file must
have a different name from the Fortran file, and must have an
extension of .ci, e.g. if the above Fortran file is hellof.f90, this
file could be called hello.ci.  Basically, its purpose is to tell
Charm what you've done in the Fortran file, since Charm cannot read
Fortran as of now.
\begin{verbatim}
      ## Just replace Hello throughout with your chare's name. ##
      ## and add your chare's entry points below where indicated ##
      ## Everything else remains the same ##
      mainmodule hello {
      
        // Create some arbitrary message to hold your parameters
        // The message must have a unique name for each Entry Point.
        message HiMsg { int } ;
      
        array Hello {
          entry Hello();

          // Note how your Fortran function takes the above defined
          // message instead of a list of parameters.
          entry void SayHi(HiMsg *);

          // Other entry points go here

        };              
      };
\end{verbatim}
If the function has more parameters: e.g. A Fortran function FF(int,
int, long) in module H, would be described in the .ci file as:
\begin{verbatim}
  message MarshalParams { int, int, long };
  array H {
    entry H();
    entry void FF(MarshalParams *);
  }
\end{verbatim}
As mentioned above, the message must have a unique name.

\section{Compile and Link}
Lastly, you need to compile and link the Fortran program with the
Charm program as follows: (Let's say you have written hellof.f90, 
hello.ci and hello.C)
\begin{verbatim}
  charmc hello.ci -language f90charm
\end{verbatim}
    will create hello.decl.h, hello.def.h

\begin{verbatim}
  charmc -c hello.C
\end{verbatim}
    will compile the hello.C with hello.decl.h, hello.def.h.

\begin{verbatim}
  charmc -c hellof.f90
\end{verbatim}
    charmc will invoke fotran compiler;

\begin{verbatim}
  charmc -o hello hello.o hellof.o -language f90charm
\end{verbatim}
    will link hellof.o, hello.o against Charm's Fortran90 library
    to create your executable program 'hello'.

\section{Run Program}

Finally, to run the program, type:

./charmrun +p2 hello

which will run 'hello' on two virtual processors.


\end{document}
