8.4.1 Charm++ interface

The Charm++ interface is the raw interface of the library and slightly more difficult to use but gives more flexibility. To use the charm++ based library, user has to create their own charm arrays which derive from predefined arrays in library. By overiding default methods, user can add in additional functions.

For the plane-based library, there are several relevant member functions:'doFFT', 'doIFFT', 'doneFFT' and 'doneIFFT'. 'doFFT' and 'doIFFT' need to be called to start the computation. 'doneFFT' and 'doneIFFT' are callback functions, and they need to be inheritated.

The sample codes below should shed more light on this. For complete sample programs, refer to file under your-charm-dir/pgms/charm++/fftdemo/.

In the sample code below, we will illustrate how to use the plane-based library in 4 steps: initializing the data struct; creating array element; starting the computation and finally ending the computation.

For initializing, a NormalFFTinfo struct will be used. Keep in mind that data storage needs to be allocated and initalized by the user. Since in-place FFT will occur, user should also make duplicate copies of data when needed.

    main::main(CkArgMsg *m)
    {
         ...
         /* Assume FFT of size  nx*ny*nz */
         int srcDim[] = ny, nx, destDim[] = nx, nz;

         complex *plane =  new complex[nx*ny];
         ... // Initialize FFT data here

         NormalFFTinfo src_fftinfo(srcArray, destArray,
                                   srcDim, destDim, true, plane, 1, 1);

         ...
     }

Next step is to create the charm array:

     main::main(CkArgMsg *m)
     {
          ...
          /* Assume FFT of size  nx*ny*nz */
          int srcDim[] = ny, nx, destDim[] = nx, nz;

          /* create the source array */
          srcArray = CProxy_SrcArray::ckNew();
          for (z = 0; z < dim; z+=1) {
              complex *plane =  new complex[nx*ny];
              ... // Initialize FFT data here: data needs to be in x-major order

              NormalFFTinfo src_fftinfo(srcArray, destArray,
                                       srcDim, destDim, true, plane, 1, 1);

              // insert one plane object: this contains data of x-y plane at z coordinate       
              srcArray(z).insert(src_fftinfo);  
           }

          /* destination array will be created in similar fashion */
          ...
      }

Following we will start the FFT computation by making a call to 'doFFT()'. 'doFFT(int id1, int id2)' takes two inputs: id1 defines the ID number of the source FFT, while id2 defines the ID number of the destination FFT. There is a similar method called 'doFFT()' to be used to invoke inverse FFTs. In this example, 3 FFT's are done simultaneously by invoking a 'doAllFFT()' method. And 'doAllFFT()' is defined as:

    void SrcArray::doAllFFT() {
        doFFT(0, 0);
        doFFT(1, 1);
        doFFT(2, 2);
    }

The last step is to get data at destination side. For this purpose, inheritance of method 'doneFFT()' is defined below. 'doneFFT(int id)' takes the FFT ID number as input. For inverse FFTs, relevant member function is 'doneIFFT()'.

    void destArray::doneFFT(int id) {
        count ++;
        if(count==3) {
            count = 0;
            /* A reduction is induced:  this will call the predefined reduction client when all array elements finish their computation */
            contribute(sizeof(int), &count, CkReduction::sum_int);
       }
    }

Next we will demonstrate the usage of pencil-based library in similar steps.

First is the initialization of data struct LineFFTinfo:

     main::main()  
    {
         ...
         /* Assume FFT of size  nx*ny*nz */
         int size[] = nx, ny, nz;

         complex *pencil =  new complex[nx];
         ... /* Initialize FFT data here */

         LineFFTinfo fftinfo(xlinesProxy, ylinesProxy, zlinesProxy, size, true, pencil);
         ...
     }

Second is the creation of array:

     main::main()    
    {
         ...
         /* Assume FFT of size  nx*ny*nz */
         int size[] = nx, ny, nz;

         xlinesProxy = CProxy_myXLines::ckNew();
         for (z = 0; z < sizeZ; z++)
             for (y = 0; y < sizeY; y+=THICKNESS)
                 complex *pencil =  new complex[nx];
                 ... /* Initialize FFT data here */

                 LineFFTinfo fftinfo(xlinesProxy, ylinesProxy, zlinesProxy,
                                     size, true, pencil);
                 xlinesProxy(y, z).insert(fftinfo);
             
         
         xlinesProxy.doneInserting();

         /* ylinesProxy /zlinesProxy are created in similar fashion */
         ...
     }

Next is the starting of the computation. A method called doFirstFFT() needs to be called. doFirstFFT(int id, int direction) takes two parameters: id specifies the ID number of the target FFT, direction tells whether FFTs is to be done in forward(direction=1) or backward(direction=0) direction.

    void myXLines::doAllFFT() {
        doFirstFFT(0, 1);
        doFirstFFT(1, 1);
        doFirstFFT(2, 1);
    }

Finally, it's the step to finish the FFT at receiver side. In this case, we call the array of destination myZLines. Similarly as in the plane-based version, doneFFT() is inherited. doneFFT(int id, int direction) takes two inputs, which are explained the same as in doFirstFFT(int id, int direction).

    void myZLines::doneFFT(int id, int direction) {
        count ++;
        if(count==3) {
            count = 0;
            contribute(sizeof(int), &count, CkReduction::sum_int);
        }
    }