6 Writing a library using TCharm

Until now, things were presented from the perspective of a user--one who writes a program for a library written on TCharm. This section gives an overview of how to go about writing a library in Charm++ that uses TCharm.

• Compared to using plain MPI, TCharm provides the ability to access all of Charm++, including arrays and groups.

• Compared to using plain Charm++, using TCharm with your library automatically provides your users with a clean C/F90 API (described in the preceeding chapters) for basic thread memory managment, I/O, and migration. It also allows you to use a convenient "thread->suspend()" and "thread->resume()" API for blocking a thread, and works properly with the load balancer, unlike CthSuspend/CthAwaken.

The overall scheme for writing a TCharm-based library "Foo" is:

1. You must provide a FOO_Init routine that creates anything you'll need, which normally includes a Chare Array of your own objects. The user will call your FOO_Init routine from their main work routine; and normally FOO_Init routines are collective.

2. In your FOO_Init routine, create your array bound it to the running TCharm threads, by creating it using the CkArrayOptions returned by TCHARM_Attach_start. Be sure to only create the array once, by checking if you're the master before creating the array.

   One simple way to make the non-master threads block until the corresponding local array element is created is to use TCharm semaphores. These are simply a one-pointer slot you can assign using TCharm::semaPut and read with TCharm::semaGet. They're useful in this context because a TCharm::semaGet blocks if a local TCharm::semaGet hasn't yet executed.

   //This is either called by FooFallbackSetuo mentioned above, or by the user
   //directly from TCHARM_User_setup (for multi-module programs)
   void FOO_Init(void)
   {
     if (TCHARM_Element()==0) {
       CkArrayID threadsAID; int nchunks;
       CkArrayOptions opts=TCHARM_Attach_start(&threadsAID,&nchunks);
     
     //actually create your library array here (FooChunk in this case)
       CkArrayID aid = CProxy_FooChunk::ckNew(opt);
     }
     FooChunk *arr=(FooChunk *)TCharm::semaGet(FOO_TCHARM_SEMAID);
   }
   

3. Depending on your library API, you may have to set up a thread-private variable(Ctv) to point to your library object. This is needed to regain context when you are called by the user. A better design is to avoid the Ctv, and instead hand the user an opaque handle that includes your array proxy.
   //_fooptr is the Ctv that points to the current chunk FooChunk and is only valid in
   //routines called from fooDriver()
   CtvStaticDeclare(FooChunk *, _fooptr);
   
   /* The following routine is listed as an initcall in the .ci file */
   /*initcall*/ void fooNodeInit(void)
   {
     CtvInitialize(FooChunk*, _fooptr);
   }
   

4. Define the array used by the library
   class FooChunk: public TCharmClient1D {
      CProxy_FooChunk thisProxy;
   protected:
      //called by TCharmClient1D when thread changes
      virtual void setupThreadPrivate(CthThread forThread)
      {
         CtvAccessOther(forThread, _fooptr) = this;
      }
      
      FooChunk(CkArrayID aid):TCharmClient1D(aid)
      {
         thisProxy = this;
         tCharmClientInit();
         TCharm::semaPut(FOO_TCHARM_SEMAID,this);
         //add any other initialization here
      }
   
      virtual void pup(PUP::er &p) {
        TCharmClient1D::pup(p);
        //usual pup calls
      }
      
      // ...any other calls you need...
      int doCommunicate(...);
      void recvReply(someReplyMsg *m);
      ........
   }
   

5. Block a thread for communication using thread->suspend and thread->resume
   int FooChunk::doCommunicate(...)
   {
      replyGoesHere = NULL;
      thisProxy[destChunk].sendRequest(...);
      thread->suspend(); //wait for reply to come back
      return replyGoesHere->data;
   }
   
   void FooChunk::recvReply(someReplyMsg *m)
   {
     if(replyGoesHere!=NULL) CkAbort("FooChunk: unexpected reply
   n");
     replyGoesHere = m;
     thread->resume(); //Got the reply - start client again
   }
   

6. Add API calls. This is how user code running in the thread interacts with the newly created library. Calls to TCHARM_API_TRACE macro must be added to the start of every user-callable method. In addition to tracing, these disable isomalloc allocation.

   The charm-api.h macros CDECL, FDECL and FTN_NAME should be used to provide both C and FORTRAN versions of each API call. You should use the "MPI capitalization standard", where the library name is all caps, followed by a capitalized first word, with all subsequent words lowercase, separated by underscores. This capitalization system is consistent, and works well with case-insensitive languages like Fortran.

   Fortran parameter passing is a bit of an art, but basically for simple types like int (INTEGER in fortran), float (SINGLE PRECISION or REAL*4), and double (DOUBLE PRECISION or REAL*8), things work well. Single parameters are always passed via pointer in Fortran, as are arrays. Even though Fortran indexes arrays based at 1, it will pass you a pointer to the first element, so you can use the regular C indexing. The only time Fortran indexing need be considered is when the user passes you an index-the int index will need to be decremented before use, or incremented before a return.

   CDECL void FOO_Communicate(int x, double y, int * arr) {
      TCHARM_API_TRACE("FOO_Communicate", "foo"); //2nd parameter is the name of the library
      FooChunk *f = CtvAccess(_fooptr);
      f->doCommunicate(x, y, arr);
   }
   
   //In fortran, everything is passed via pointers
   FDECL void FTN_NAME(FOO_COMMUNICATE, foo_communicate)
        (int *x, double *y, int *arr)
   {
      TCHARM_API_TRACE("FOO_COMMUNICATE", "foo");
      FooChunk *f = CtvAccess(_fooptr);
      f->doCommunicate(*x, *y, arr);
   }