ConverseScheduler
[Converse]

Collaboration diagram for ConverseScheduler:

The portion of Converse responsible for scheduling the execution of incoming messages. More...


Files
file	convcore.c
	converse main core
file	converse.h
	Main Converse header file.
file	conv-lists.C
	Declarations of CdsFifo* routines.
file	conv-lists.h
	Definitions of CdsFifo routines.
Typedefs
typedef CkQ< void * >	_Fifo
typedef void *	CdsFifo
Functions
void	CsdBeginIdle (void)
void	CsdStillIdle (void)
void	CsdEndIdle (void)
void	CmiHandleMessage (void *msg)
	Takes a message and calls its corresponding handler.
void	CmiDeliversInit ()
int	CmiDeliverMsgs (int maxmsgs)
	CpvDeclare (void *, CsdObjQueue)
void	CsdSchedulerState_new (CsdSchedulerState_t *s)
void *	CsdNextMessage (CsdSchedulerState_t *s)
	Dequeue and return the next message from the unprocessed message queues.
void *	CsdNextLocalNodeMessage (CsdSchedulerState_t *s)
int	CsdScheduler (int maxmsgs)
void	CkClearAllArrayElements ()
void	machine_OffloadAPIProgress ()
void	CsdScheduleForever (void)
	The main scheduler loop that repeatedly executes messages from a queue, forever.
int	CsdScheduleCount (int maxmsgs)
void	CsdSchedulePoll (void)
void	CsdScheduleNodePoll (void)
void	CmiDeliverSpecificMsg (int handler)
	CpvStaticDeclare (CthThread, CthMainThread)
	CpvStaticDeclare (CthThread, CthSchedulingThread)
	CpvStaticDeclare (CthThread, CthSleepingStandins)
	CpvDeclare (int, CthResumeNormalThreadIdx)
	CpvStaticDeclare (int, CthResumeSchedulingThreadIdx)
void	CthStandinCode ()
static CthThread	CthSuspendNormalThread ()
void	CthEnqueueSchedulingThread (CthThreadToken token, int, int, unsigned int )
CthThread	CthSuspendSchedulingThread ()
void	CthResumeNormalThread (CthThreadToken *token)
void	CthResumeSchedulingThread (CthThreadToken *token)
void	CthEnqueueNormalThread (CthThreadToken token, int s, int pb, unsigned int prio)
void	CthSetStrategyDefault (CthThread t)
void	CthSchedInit ()
void	CsdInit (char **argv)
CdsFifo	CdsFifo_Create (void)
CdsFifo	CdsFifo_Create_len (int len)
void	CdsFifo_Enqueue (CdsFifo q, void *elt)
void *	CdsFifo_Dequeue (CdsFifo q)
void	CdsFifo_Push (CdsFifo q, void *elt)
void *	CdsFifo_Pop (CdsFifo q)
void	CdsFifo_Destroy (CdsFifo q)
void **	CdsFifo_Enumerate (CdsFifo q)
int	CdsFifo_Empty (CdsFifo q)
void *	CdsFifo_Peek (CdsFifo q)
int	CdsFifo_Length (CdsFifo q)
Variables
int	_exitHandlerIdx

Detailed Description

The portion of Converse responsible for scheduling the execution of incoming messages.

Converse provides a scheduler for message delivery: methods can be registered to the scheduler, and then messages allocated through CmiAlloc can be sent to the correspondent method in a remote processor. This is done through the converse header (which has few common fields, but is architecture dependent).

In converse the CsdScheduleForever() routine will run an infinite while loop that looks for available messages to process from the unprocessed message queues. The details of the queues and the order in which they are emptied is hidden behind CsdNextMessage(), which is used to dequeue the next message for processing by the converse scheduler. When a message is taken from the queue it is then passed into CmiHandleMessage() which calls the handler associated with the message.

Incoming messages that are destined for Charm++ will be passed to the charm scheduling routines.

Typedef Documentation

typedef CkQ<void*> _Fifo

Definition at line 13 of file conv-lists.C.

typedef void* CdsFifo

Definition at line 16 of file conv-lists.h.

Function Documentation

void CsdBeginIdle ( void )

Definition at line 1556 of file convcore.c.

References CcdCallBacks(), and CcdRaiseCondition().

void CsdStillIdle ( void )

Definition at line 1568 of file convcore.c.

References CcdRaiseCondition().

void CsdEndIdle ( void )

Definition at line 1573 of file convcore.c.

References CcdRaiseCondition().

Referenced by CsdScheduleCount(), and CsdScheduleForever().

void CmiHandleMessage ( void * msg )

Takes a message and calls its corresponding handler.

Definition at line 1587 of file convcore.c.

References CmiHandlerInfo::hdlr, and CmiHandlerInfo::userPtr.

Referenced by CldHopHandler(), CldReadytoExec(), CldTokenHandler(), CmiDeliverSpecificMsg(), CmiImmediateMsgHandler(), ComlibNotifyMigrationDone(), CpdDeliverMessageInt(), CpdDeliverSingleMessage(), CpdFreezeModeScheduler(), and handleDebugMessage().

void CmiDeliversInit ( )

Definition at line 1618 of file convcore.c.

Referenced by ConverseCommonInit().

int CmiDeliverMsgs ( int maxmsgs )

Definition at line 1622 of file convcore.c.

References CsdScheduler().

CpvDeclare	(	void *	,
		CsdObjQueue
	)

void CsdSchedulerState_new ( CsdSchedulerState_t * s )

Definition at line 1631 of file convcore.c.

References CsdSchedulerState_t::gridQ, CsdSchedulerState_t::localCounter, CsdSchedulerState_t::localQ, CsdSchedulerState_t::nodeLock, CsdSchedulerState_t::nodeQ, CsdSchedulerState_t::objQ, and CsdSchedulerState_t::schedQ.

Referenced by CpdFreezeModeScheduler().

void* CsdNextMessage ( CsdSchedulerState_t * s )

Dequeue and return the next message from the unprocessed message queues.

This function encapsulates the multiple queues that exist for holding unprocessed messages and the rules for the order in which to check them. There are five (5) different Qs that converse uses to store and retrieve unprocessed messages. These are: Q Purpose Type internal DeQ logic -----------------------------------------------------------

PE offnode pcQ FIFO
PE onnode CkQ FIFO
Node offnode pcQ FIFO
Node onnode prioQ prio-based
Scheduler prioQ prio-based

The PE queues hold messages that are destined for a specific PE. There is one such queue for every PE within a charm node. The node queues hold messages that are destined to that node. There is only one of each node queue within a charm node. Finally there is also a charm++ message queue for each PE.

The offnode queues are meant for holding messages that arrive from outside the node. The onnode queues hold messages that are generated within the same charm node.

The PE and node level offnode queues are accessed via functions CmiGetNonLocal() and CmiGetNonLocalNodeQ(). These are implemented separately by each machine layer and hide the implementation specifics for each layer.

The PE onnode queue is implemented as a FIFO CkQ and is initialized via a call to CdsFifo_Create(). The node local queue and the scheduler queue are both priority queues. They are initialized via calls to CqsCreate() which gives each of them three separate internal queues for different priority ranges (-ve, 0 and +ve). Access to these queues is via pointers stored in the struct CsdSchedulerState that is passed into this function.

The order in which these queues are checked is described below. The function proceeds to the next queue in the list only if it does not find any messages in the current queue. The first message that is found is returned, terminating the call. (1) offnode queue for this PE (2) onnode queue for this PE (3) offnode queue for this node (4) highest priority msg from onnode queue or scheduler queue

Note:: : Across most (all?) machine layers, the two GetNonLocal functions simply access (after observing adequate locking rigor) structs representing the scheduler state, to dequeue from the queues stored within them. The structs (CmiStateStruct and CmiNodeStateStruct) implement these queues as pc (producer-consumer) queues". The functions also perform other necessary actions like PumpMsgs() etc.

Definition at line 1699 of file convcore.c.

References CdsFifo_Dequeue(), CmiGetNonLocal(), CmiGetNonLocalNodeQ(), CmiLock(), CmiUnlock(), CqsDequeue(), CqsEmpty(), CqsGetPriority(), CqsPrioGT(), CsdSchedulerState_t::gridQ, CsdSchedulerState_t::localCounter, CsdSchedulerState_t::localQ, msg, CsdSchedulerState_t::nodeLock, CsdSchedulerState_t::nodeQ, CsdSchedulerState_t::objQ, and CsdSchedulerState_t::schedQ.