3.1 BigSim Programming Environment

The basic philosophy of the BigSim Emulator is to hide intricate details of the simulated machine from the application developer. Thus, the application developer needs to provide initialization details and handler functions only and gets the result as though running on a real machine. Communication, Thread creation, Time Stamping, etc are done by the emulator.

3.1.1 BigSim API: Level 0

void addBgNodeInbuffer(bgMsg *msgPtr, int nodeID)
low-level primitive invoked by Blue Gene emulator to put the message to the inbuffer queue of a node.

msgPtr - pointer to the message to be sent to target node;

nodeID - node ID of the target node, it is the serial number of a bluegene node in the emulator's physical node.

void addBgThreadMessage(bgMsg *msgPtr, int threadID)
add a message to a thread's affinity queue, these messages can be only executed by a specific thread indicated by threadID.

void addBgNodeMessage(bgMsg *msgPtr)
add a message to a node's non-affinity queue, these messages can be executed by any thread in the node.

boolean checkReady()
invoked by communication thread to see if there is any unattended message in inBuffer.

bgMsg * getFullBuffer()
invoked by communication thread to retrieve the unattended message in inBuffer.

CmiHandler msgHandlerFunc(char *msg)
Handler function type that user can register to handle the message.

void sendPacket(int x, int y, int z, int msgSize,bgMsg *msg)
chip-to-chip communication function. It send a message to Node[x][y][z].

bgMsg is the message type with message envelop used internally.

3.1.2 Initialization API: Level 1a

All the functions defined in API Level 0 are used internally for the implementation of bluegene node communication and worker threads.

From this level, the functions defined are exposed to users to write bluegene programs on the emulator.

Considering that the emulator machine will emulate several Bluegene nodes on each physical node, the emulator program defines this function BgEmulatorInit(int argc, char **argv)
to initialize each emulator node. In this function, user program can define the Bluegene machine size, number of communication/worker threads, and check the command line arguments.

The size of the simulated machine being emulated and the number of thread per node is determined either by the command line arguments or calling following functions:

void BgSetSize(int sx, int sy, int sz)
set Blue Gene Machine size;

void BgSetNumWorkThread(int num)
set number of worker threads per node;

void BgSetNumCommThread(int num)
set number of communication threads per node;

int BgRegisterHandler(BgHandler h)
register user message handler functions;

For each simulated node, the execution starts at BgNodeStart(int argc, char **argv)
called by the emulator, where application handlers can be registered and computation is triggered by creating a task at required nodes.

Similar to pthread's thread specific data, each bluegene node has its own node specific data associated with it. To do this, the user needs to define its own node-specific variables encapsulated in a struct definition and register the pointer to the data with the emulator by following function:

void BgSetNodeData(char *data)

To retrieve the node specific data, call:

char *BgGetNodeData();

After completion of execution, user program invokes a function:

void BgShutdown()

to terminate the emulator.

3.1.3 Handler Function API: Level 1a

The following functions can be called in user's application program to retrieve the simulated machine information, get thread execution time, and perform the communication.

void BgGetSize(int *sx, int *sy, int *sz);

int BgGetNumWorkThread();

int BgGetNumCommThread();

int BgGetThreadID();

double BgGetTime();

void BgSendPacket(int x, int y, int z, int threadID, int handlerID, WorkType type, int numbytes, char* data);
This sends a trunk of data to Node[x, y, z] and also specifies the handler function to be used for this message i.e. the handlerID; threadID specifies the desired thread to handle the message, ANYTHREAD means no preference.

To specify the thread category:

1:

a small piece of work that can be done by communication thread itself, so NO scheduling overhead.

0:

a large piece of work, so communication thread schedules it for a worker thread