8.2 Data Layout

IDXL is designed to send and receive data directly out of your arrays, with no intermediate copying. This means IDXL needs a completely general method for specifying how you store your data in your arrays. Since you probably don't change your storage layout at runtime, you can create a ``data layout'' once at the beginning of your program, then use it repeatedly for communication.

IDXL Layouts are normally used to describe arrays of data associated with nodes or elements. The layout abstraction allows you to use IDXL routines to communicate any sort of data, stored in a variety of formats.

Like Index Lists, Layouts are referred to via an opaque handle--in a C program via the integer typedef IDXL_Layout_t, and in Fortran via a bare integer.

8.2.1 Layout Routines

In most programs, the data to be communicated is a dense array of data of one type. In this case, there is only one layout routine you need to know:

IDXL_Layout_t IDXL_Layout_create(int type,int width);
INTEGER function IDXL_Layout_create(type,width)
INTEGER, INTENT(IN) :: type,width

The simplest data layout to describe--a dense array of this IDXL datatype, indexed by entity number, with width pieces of data per entity. Note that the number of entities is not stored with the layout-the number of entities to be communicated depends on the communication routine.

The IDXL datatypes are:

IDXL Datatype	C Datatypes	Fortran Datatypes
IDXL_BYTE	unsigned char	INTEGER*1
	char	LOGICAL*1
IDXL_INT	int	INTEGER
IDXL_REAL	float	SINGLE PRECISION
		REAL*4
IDXL_DOUBLE	double	DOUBLE PRECISION
		REAL*8

For example, if you keep a dense array with 3 doubles of force per node, you'd call this routine as:

// C++ version:
     double *force=new double[3*n];
     IDXL_Layout_t force_layout=IDXL_Layout_create(IDXL_DOUBLE,3);

! F90 Version
     double precision, allocatable :: force(:,:)
     integer :: force_layout
     ALLOCATE(force(3,n)) ! (could equivalently use force(3*n) )
     force_layout=IDXL_Layout_create(IDXL_DOUBLE,3)

This routine was once called FEM_Create_simple_field.

8.2.2 Advanced Layout Routines

These advanced routines are only needed if you want to exchange data stored in an array of user-defined types. Most programs only need IDXL_Layout_create.

IDXL_Layout_t IDXL_Layout_offset(int type, int width, int offsetBytes, int distanceBytes,int skewBytes);
INTEGER function IDXL_Layout_offset(type,width,offsetBytes,distanceBytes,skewBytes)
INTEGER, INTENT(IN) :: type,width,offsetBytes,distanceBytes,skewBytes

The most general data layout-an array indexed by entity, containing width pieces of data per entity. This routine expands on IDXL_Layout_create by adding support for user-defined types or other unusual data layouts. You describe your layout by giving various in-memory byte offsets that describe the data is stored. Again, the number of entities is not stored with the layout-the number of entities to be communicated depends on the communication routine.

• offsetBytes The number of bytes from the start of the array to the start of the data.
• distanceBytes The number of bytes taken by one entity.
• skewBytes The number of bytes between each piece of data. Since this can almost always be determined from the size of the base data type, this parameter can be left as zero.

Figure 11: Describing a complex data layout.

For example, if your node data is all stored in a struct (in fortran, a named TYPE), offsetBytes gives the distance between the start of the struct and the force; and distanceBytes gives the size in bytes of the struct.

In C, the offsetof and sizeof keywords are useful for finding these values. In Fortran, we provide a special routine called foffsetof that returns the distance, in bytes, between its two arguments.

// C++ version:
     typedef struct {
        double d[3], v[3], force[3], a[3];
        double m;
     } node;
     node *nodes=new node[n];
     IDXL_Layout_t force_layout=IDXL_Layout_offset(IDXL_DOUBLE,3,
              offsetof(node,force),sizeof(node),0);

! F90 Version
     TYPE node
        DOUBLE PRECISION :: d(3), v(3), force(3), a(3)
        DOUBLE PRECISION :: m
     END TYPE
     integer :: force_layout
     ALLOCATE(nodes(n))
     force_layout=IDXL_Layout_create(IDXL_DOUBLE,3,
   &          foffsetof(nodes(1),nodes(1)%force),
   &          foffsetof(nodes(1),nodes(2)),0)

void IDXL_Layout_destroy(IDXL_Layout_t layout);
SUBROUTINE IDXL_Layout_destroy(layout)
INTEGER, INTENT(IN) :: layout

Destroy this Layout. You only need call this routine if you repeatedly create layouts.

int IDXL_Get_layout_type(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_type(layout)

Return the IDXL datatype for this layout.

int IDXL_Get_layout_width(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_width(layout)

Return the layout width--the number of data items that are communicated per entity.

int IDXL_Get_layout_distance(IDXL_Layout_t layout);
INTEGER function IDXL_Get_layout_distance(layout)

Return the layout distance--the number of bytes between successive entity's data items.

8.2.3 Layout Compatability Routines

Before IDXL was made a separate library, FEM included these routines, which are still preserved for backward compatability.

IDXL_Layout_t FEM_Create_simple_field(int type,int width);
INTEGER function FEM_Create_simple_field(type,width)
INTEGER, INTENT(IN) :: type,width

This routine is completely interchangable to IDXL_Layout_create.

int FEM_Create_field(int type,int width,int offset,int distance);
INTEGER function FEM_Create_field(type, width, offset, distance)
INTEGER, INTENT(IN) :: type, width, offset, distance

This routine is like a call to IDXL_Layout_offset with the rarely used skewBytes set to zero.