ConverseScheduler
[Converse]

Collaboration diagram for ConverseScheduler:

The portion of Converse responsible for scheduling the execution of incoming messages. More...

Files
file	convcore.C
	converse main core
file	converse.h
	Main Converse header file.
file	conv-lists.C
	Declarations of CdsFifo* routines.
file	conv-lists.h
	Definitions of CdsFifo routines.
Typedefs
typedef CkQ< void * >	_Fifo
typedef void *	CdsFifo
Functions
void	CsdBeginIdle (void)
void	CsdStillIdle (void)
void	CsdEndIdle (void)
void	CmiHandleMessage (void *msg)
	Takes a message and calls its corresponding handler.
void	CmiDeliversInit (void)
int	CmiDeliverMsgs (int maxmsgs)
	CpvDeclare (void *, CsdObjQueue)
void	CsdSchedulerState_new (CsdSchedulerState_t *s)
void *	CsdNextMessage (CsdSchedulerState_t *s)
	Dequeue and return the next message from the unprocessed message queues.
void *	CsdNextLocalNodeMessage (CsdSchedulerState_t *s)
int	CsdScheduler (int maxmsgs)
void	machine_OffloadAPIProgress (void)
void	CsdScheduleForever (void)
	The main scheduler loop that repeatedly executes messages from a queue, forever.
int	CsdScheduleCount (int maxmsgs)
void	CsdSchedulePoll (void)
void	CsdScheduleNodePoll (void)
void	CmiDeliverSpecificMsg (int handler)
	CpvStaticDeclare (CthThread, CthMainThread)
	CpvStaticDeclare (CthThread, CthSchedulingThread)
	CpvStaticDeclare (CthThread, CthSleepingStandins)
	CpvDeclare (int, CthResumeNormalThreadIdx)
	CpvStaticDeclare (int, CthResumeSchedulingThreadIdx)
	CpvDeclare (int, CthResumeStealableThreadIdx)
	CpvDeclare (int, CthResumeSuspendedStealableThreadIdx)
void	CthStandinCode (void *arg)
static CthThread	CthSuspendNormalThread (void)
void	CthEnqueueSchedulingThread (CthThreadToken *token, int, int, unsigned int *)
CthThread	CthSuspendSchedulingThread (void)
void	CthResumeNormalThread (CthThreadToken *token)
void	CthResumeSchedulingThread (CthThreadToken *token)
void	CthEnqueueNormalThread (CthThreadToken *token, int s, int pb, unsigned int *prio)
void	CthSetStrategyDefault (CthThread t)
void	CthResumeStealableThread (CthThreadToken *token)
void	CthEnqueueStealableThread (CthThreadToken *token, int s, int pb, unsigned int *prio)
void	CthEnqueueSuspendedStealableThread (CthThreadToken *token, int s, int pb, unsigned int *prio)
void	CthResumeSuspendedStealableThread (CthThreadToken *token)
CthThread	CthSuspendStealableThread ()
void	CthSetStrategyWorkStealing (CthThread t)
void	CthSetStrategySuspendedWorkStealing (CthThread t)
void	CthSchedInit ()
void	CsdInit (char **argv)
CdsFifo	CdsFifo_Create (void)
CdsFifo	CdsFifo_Create_len (int len)
void	CdsFifo_Enqueue (CdsFifo q, void *elt)
void *	CdsFifo_Dequeue (CdsFifo q)
void	CdsFifo_Push (CdsFifo q, void *elt)
void *	CdsFifo_Pop (CdsFifo q)
void	CdsFifo_Destroy (CdsFifo q)
void **	CdsFifo_Enumerate (CdsFifo q)
int	CdsFifo_Empty (CdsFifo q)
void *	CdsFifo_Peek (CdsFifo q)
int	CdsFifo_Length (CdsFifo q)
Variables
int	_exitHandlerIdx

---

Detailed Description

The portion of Converse responsible for scheduling the execution of incoming messages.

Converse provides a scheduler for message delivery: methods can be registered to the scheduler, and then messages allocated through CmiAlloc can be sent to the correspondent method in a remote processor. This is done through the converse header (which has few common fields, but is architecture dependent).

In converse the CsdScheduleForever() routine will run an infinite while loop that looks for available messages to process from the unprocessed message queues. The details of the queues and the order in which they are emptied is hidden behind CsdNextMessage(), which is used to dequeue the next message for processing by the converse scheduler. When a message is taken from the queue it is then passed into CmiHandleMessage() which calls the handler associated with the message.

Incoming messages that are destined for Charm++ will be passed to the charm scheduling routines.

---

Typedef Documentation

typedef CkQ<void*> _Fifo

Definition at line 13 of file conv-lists.C.

typedef void* CdsFifo

Definition at line 16 of file conv-lists.h.

---

Function Documentation

void CsdBeginIdle

(

void

)

Definition at line 1584 of file convcore.C.

References CcdCallBacks(), CcdRaiseCondition(), and memory_order_relaxed.

Here is the call graph for this function:

void CsdStillIdle

(

void

)

Definition at line 1601 of file convcore.C.

References CcdRaiseCondition().

Here is the call graph for this function:

void CsdEndIdle

(

void

)

Definition at line 1606 of file convcore.C.

References CcdRaiseCondition(), and memory_order_relaxed.

Referenced by CsdScheduleCount(), and CsdScheduleForever().

Here is the call graph for this function:

Here is the caller graph for this function:

void CmiHandleMessage

(

void *

msg

)

Takes a message and calls its corresponding handler.

Definition at line 1625 of file convcore.C.

References CmiAbort(), h, CmiHandlerInfo::hdlr, and CmiHandlerInfo::userPtr.

Referenced by CldHopHandler(), CldReadytoExec(), CldTokenHandler(), CmiDeliverSpecificMsg(), CmiImmediateMsgHandler(), CpdDeliverMessageInt(), CpdDeliverSingleMessage(), enqueueNcpyMessage(), handleArrayMsgOnChildPostCompletionForRecvBcast(), handleGroupMsgOnChildPostCompletionForRecvBcast(), handleNGMsgOnChildPostCompletionForRecvBcast(), ntohl(), FuncCkLoop::parallelizeFuncHybrid(), persistentDecompressHandler(), and persistentNoDecompressHandler().

Here is the call graph for this function:

Here is the caller graph for this function:

void CmiDeliversInit

(

void

)

Definition at line 1665 of file convcore.C.

Referenced by ConverseCommonInit().

Here is the caller graph for this function:

int CmiDeliverMsgs

(

int

maxmsgs

)

Definition at line 1669 of file convcore.C.

References CsdScheduler().

Here is the call graph for this function:

CpvDeclare	(	void *	,
		CsdObjQueue
	)

void CsdSchedulerState_new

(

CsdSchedulerState_t *

s

)

Definition at line 1678 of file convcore.C.

References CsdSchedulerState_t::gridQ, CsdSchedulerState_t::localCounter, CsdSchedulerState_t::localQ, CsdSchedulerState_t::nodeLock, CsdSchedulerState_t::nodeQ, CsdSchedulerState_t::objQ, CsdSchedulerState_t::schedQ, CsdSchedulerState_t::suspendedTaskQ, and CsdSchedulerState_t::taskQ.

Referenced by ntohl().

Here is the caller graph for this function:

void* CsdNextMessage

(

CsdSchedulerState_t *

s

)

Dequeue and return the next message from the unprocessed message queues.

This function encapsulates the multiple queues that exist for holding unprocessed messages and the rules for the order in which to check them. There are five (5) different Qs that converse uses to store and retrieve unprocessed messages. These are: Q Purpose Type internal DeQ logic -----------------------------------------------------------

• PE offnode pcQ FIFO
• PE onnode CkQ FIFO
• Node offnode pcQ FIFO
• Node onnode prioQ prio-based
• Scheduler prioQ prio-based

The PE queues hold messages that are destined for a specific PE. There is one such queue for every PE within a charm node. The node queues hold messages that are destined to that node. There is only one of each node queue within a charm node. Finally there is also a charm++ message queue for each PE.

The offnode queues are meant for holding messages that arrive from outside the node. The onnode queues hold messages that are generated within the same charm node.

The PE and node level offnode queues are accessed via functions CmiGetNonLocal() and CmiGetNonLocalNodeQ(). These are implemented separately by each machine layer and hide the implementation specifics for each layer.

The PE onnode queue is implemented as a FIFO CkQ and is initialized via a call to CdsFifo_Create(). The node local queue and the scheduler queue are both priority queues. They are initialized via calls to CqsCreate() which gives each of them three separate internal queues for different priority ranges (-ve, 0 and +ve). Access to these queues is via pointers stored in the struct CsdSchedulerState that is passed into this function.

The order in which these queues are checked is described below. The function proceeds to the next queue in the list only if it does not find any messages in the current queue. The first message that is found is returned, terminating the call. (1) offnode queue for this PE (2) onnode queue for this PE (3) offnode queue for this node (4) highest priority msg from onnode queue or scheduler queue

Note:

: Across most (all?) machine layers, the two GetNonLocal functions simply access (after observing adequate locking rigor) structs representing the scheduler state, to dequeue from the queues stored within them. The structs (CmiStateStruct and CmiNodeStateStruct) implement these queues as pc (producer-consumer) queues". The functions also perform other necessary actions like PumpMsgs() etc.

Definition at line 1750 of file convcore.C.

References CdsFifo_Dequeue(), CmiGetNonLocal(), CmiGetNonLocalNodeQ(), CmiSuspendedTaskPop(), CmiTryLock(), CmiUnlock(), CqsDequeue(), CqsEmpty(), CqsGetPriority(), CqsPrioGT(), CsdLocalMax, CsdSchedulerState_t::gridQ, CsdSchedulerState_t::localCounter, CsdSchedulerState_t::localQ, msg, CsdSchedulerState_t::nodeLock, CsdSchedulerState_t::nodeQ, CsdSchedulerState_t::objQ, CsdSchedulerState_t::schedQ, CsdSchedulerState_t::taskQ, and TaskQueuePop().

Referenced by CpdCharmInit(), CsdScheduleCount(), CsdScheduleForever(), and CsdSchedulePoll().

Here is the call graph for this function:

Here is the caller graph for this function:

void* CsdNextLocalNodeMessage

(

CsdSchedulerState_t *

s

)

Definition at line 1823 of file convcore.C.

References CmiLock(), CmiUnlock(), CqsDequeue(), CqsEmpty(), msg, CsdSchedulerState_t::nodeLock, and CsdSchedulerState_t::nodeQ.

Referenced by CsdScheduleNodePoll().

Here is the call graph for this function:

Here is the caller graph for this function:

int CsdScheduler

(

int

maxmsgs

)

Definition at line 1840 of file convcore.C.

References CsdScheduleCount(), CsdScheduleForever(), and CsdSchedulePoll().

Referenced by call_startfn(), CharmLibExit(), CkExit(), CkLocalNodeBranch(), CmiDeliverMsgs(), ConverseRunPE(), CthStandinCode(), and StartInteropScheduler().

Here is the call graph for this function:

Here is the caller graph for this function:

void machine_OffloadAPIProgress

(

void

)

Referenced by CsdScheduleForever().

Here is the caller graph for this function:

void CsdScheduleForever

(

void

)

The main scheduler loop that repeatedly executes messages from a queue, forever.

Definition at line 1884 of file convcore.C.

References CharmLibInterOperate, CsdEndIdle(), CsdNextMessage(), hapiPollEvents(), machine_OffloadAPIProgress(), and msg.

Referenced by CsdScheduler().

Here is the call graph for this function:

Here is the caller graph for this function:

int CsdScheduleCount

(

int

maxmsgs

)

Definition at line 1938 of file convcore.C.

References CsdEndIdle(), CsdNextMessage(), and msg.

Referenced by CsdScheduler().

Here is the call graph for this function:

Here is the caller graph for this function:

void CsdSchedulePoll

(

void

)

Definition at line 1961 of file convcore.C.

References CsdNextMessage(), and msg.

Referenced by CsdScheduler(), and LrtsInitCpuTopo().

Here is the call graph for this function:

Here is the caller graph for this function:

void CsdScheduleNodePoll

(

void

)

Definition at line 1978 of file convcore.C.

References CsdNextLocalNodeMessage(), and msg.

Here is the call graph for this function:

void CmiDeliverSpecificMsg

(

int

handler

)

Definition at line 1993 of file convcore.C.

References CdsFifo_Dequeue(), CdsFifo_Enqueue(), CmiGetNonLocal(), CmiHandleMessage(), and msg.

Referenced by _propMapInit(), CmiCheckAffinity(), and CmiInitCPUAffinity().

Here is the call graph for this function:

Here is the caller graph for this function:

CpvStaticDeclare	(	CthThread	,
		CthMainThread
	)

CpvStaticDeclare	(	CthThread	,
		CthSchedulingThread
	)

CpvStaticDeclare	(	CthThread	,
		CthSleepingStandins
	)

CpvDeclare	(	int	,
		CthResumeNormalThreadIdx
	)

CpvStaticDeclare	(	int	,
		CthResumeSchedulingThreadIdx
	)

CpvDeclare	(	int	,
		CthResumeStealableThreadIdx
	)

CpvDeclare	(	int	,
		CthResumeSuspendedStealableThreadIdx
	)

void CthStandinCode

(

void *

arg

)

Definition at line 2056 of file convcore.C.

References CsdScheduler().

Referenced by CthSuspendSchedulingThread().

Here is the call graph for this function:

Here is the caller graph for this function:

static CthThread CthSuspendNormalThread

(

void

)

[static]

Definition at line 2062 of file convcore.C.

Referenced by CthSetStrategyDefault().

Here is the caller graph for this function:

void CthEnqueueSchedulingThread	(	CthThreadToken *	token,
		int	s,
		int	pb,
		unsigned int *	prio
	)

Definition at line 2159 of file convcore.C.

Referenced by CthResumeSchedulingThread(), CthSchedInit(), and CthSuspendSchedulingThread().

Here is the caller graph for this function:

CthThread CthSuspendSchedulingThread

(

void

)

Definition at line 2070 of file convcore.C.

References CthCreate(), CthEnqueueSchedulingThread(), CthGetNext(), CthSetStrategy(), and CthStandinCode().

Referenced by CthSchedInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthResumeNormalThread

(

CthThreadToken *

token

)

Definition at line 2088 of file convcore.C.

References CthResume(), CthScheduledDecrement(), CthSelf(), CthSetPrev(), CthTraceResume(), free(), PUP::t, and CthThreadToken::thread.

Referenced by CthSchedInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthResumeSchedulingThread

(

CthThreadToken *

token

)

Definition at line 2123 of file convcore.C.

References CthEnqueueSchedulingThread(), CthGetToken(), CthResume(), CthScheduledDecrement(), CthSelf(), CthSetNext(), CthSetPrev(), CthTraceResume(), PUP::t, and CthThreadToken::thread.

Referenced by CthSchedInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthEnqueueNormalThread	(	CthThreadToken *	token,
		int	s,
		int	pb,
		unsigned int *	prio
	)

Definition at line 2152 of file convcore.C.

Referenced by CthSetStrategyDefault().

Here is the caller graph for this function:

void CthSetStrategyDefault

(

CthThread

t

)

Definition at line 2166 of file convcore.C.

References CthEnqueueNormalThread(), CthSetStrategy(), and CthSuspendNormalThread().

Referenced by CpmThread2(), CpmThreadSize2(), Cpthread_create(), and CthThreadBaseInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthResumeStealableThread

(

CthThreadToken *

token

)

Definition at line 2176 of file convcore.C.

References CthResume(), CthScheduledDecrement(), CthSelf(), CthSetNext(), CthSetPrev(), CthTraceResume(), PUP::t, and CthThreadToken::thread.

Referenced by CthSchedInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthEnqueueStealableThread	(	CthThreadToken *	token,
		int	s,
		int	pb,
		unsigned int *	prio
	)

Definition at line 2192 of file convcore.C.

Referenced by CthEnqueueSuspendedStealableThread(), CthResumeSuspendedStealableThread(), and CthSetStrategyWorkStealing().

Here is the caller graph for this function:

void CthEnqueueSuspendedStealableThread	(	CthThreadToken *	token,
		int	s,
		int	pb,
		unsigned int *	prio
	)

Definition at line 2198 of file convcore.C.

References CmiMyRank(), CmiSuspendedTaskEnqueue(), CthEnqueueStealableThread(), CthGetThreadID(), CthSetStrategyWorkStealing(), _CmiObjId::id, PUP::t, and CthThreadToken::thread.

Referenced by CthSetStrategySuspendedWorkStealing().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthResumeSuspendedStealableThread

(

CthThreadToken *

token

)

Definition at line 2212 of file convcore.C.

References CthEnqueueStealableThread().

Referenced by CthSchedInit().

Here is the call graph for this function:

Here is the caller graph for this function:

CthThread CthSuspendStealableThread

(

)

Definition at line 2216 of file convcore.C.

References CmiAbort(), CthGetNext(), and CthSelf().

Referenced by CthSetStrategySuspendedWorkStealing(), and CthSetStrategyWorkStealing().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthSetStrategyWorkStealing

(

CthThread

t

)

Definition at line 2226 of file convcore.C.

References CthEnqueueStealableThread(), CthSetStrategy(), and CthSuspendStealableThread().

Referenced by CthEnqueueSuspendedStealableThread().

Here is the call graph for this function:

Here is the caller graph for this function:

void CthSetStrategySuspendedWorkStealing

(

CthThread

t

)

Definition at line 2231 of file convcore.C.

References CthEnqueueSuspendedStealableThread(), CthSetStrategy(), and CthSuspendStealableThread().

Here is the call graph for this function:

void CthSchedInit

(

)

Definition at line 2237 of file convcore.C.

References CmiRegisterHandler(), CthEnqueueSchedulingThread(), CthResumeNormalThread(), CthResumeSchedulingThread(), CthResumeStealableThread(), CthResumeSuspendedStealableThread(), CthSelf(), CthSetStrategy(), and CthSuspendSchedulingThread().

Referenced by ConverseCommonInit().

Here is the call graph for this function:

Here is the caller graph for this function:

void CsdInit

(

char **

argv

)

Definition at line 2265 of file convcore.C.

References CdsFifo_Create(), CmiCreateLock(), CmiGetArgIntDesc(), CmiLock(), CmiMyPe(), CmiMyRank(), CmiNodeAllBarrier(), CmiPrintf(), CmiTaskQueueInit(), CqsCreate(), CsdLocalMax, and TaskQueueCreate().

Referenced by ConverseCommonInit().

Here is the call graph for this function:

Here is the caller graph for this function:

CdsFifo CdsFifo_Create

(

void

)

Definition at line 15 of file conv-lists.C.

Referenced by CmiStateInit(), CpdBgInit(), Cpthread_cond_init(), Cpthread_mutex_init(), CkObjectMsgQ::create(), CsdInit(), ntohl(), and waitqd_QDChare::waitQD().

Here is the caller graph for this function:

CdsFifo CdsFifo_Create_len

(

int

len

)

Definition at line 16 of file conv-lists.C.

void CdsFifo_Enqueue	(	CdsFifo	q,
		void *	elt
	)

Definition at line 17 of file conv-lists.C.

Referenced by _enqObjQueue(), BgProcessMessageFreezeMode(), CmiDeliverSpecificMsg(), CmiSendSelf(), CpdDeliverMessageInt(), CpdDeliverSingleMessage(), Cpthread_cond_wait(), Cpthread_mutex_lock(), Cpthread_mutex_trylock(), ntohl(), CkObjectMsgQ::process(), and waitqd_QDChare::waitQD().

Here is the caller graph for this function:

void * CdsFifo_Dequeue

(

CdsFifo

q

)

Definition at line 18 of file conv-lists.C.

Referenced by BgProcessMessageFreezeMode(), CmiDeliverSpecificMsg(), CpdDeliverMessageInt(), CpdDeliverSingleMessage(), CpdGetNextMessageConditional(), Cpthread_cond_broadcast(), Cpthread_cond_signal(), CsdNextMessage(), ntohl(), waitqd_QDChare::onQD(), and CkObjectMsgQ::process().

Here is the caller graph for this function:

void CdsFifo_Push	(	CdsFifo	q,
		void *	elt
	)

Definition at line 19 of file conv-lists.C.

void * CdsFifo_Pop

(

CdsFifo

q

)

Definition at line 20 of file conv-lists.C.

Referenced by Cpthread_cond_wait(), and Cpthread_mutex_unlock().

Here is the caller graph for this function:

void CdsFifo_Destroy

(

CdsFifo

q

)

Definition at line 21 of file conv-lists.C.

Referenced by Cpthread_cond_destroy(), Cpthread_mutex_destroy(), waitqd_QDChare::onQD(), and CkObjectMsgQ::~CkObjectMsgQ().

Here is the caller graph for this function:

void ** CdsFifo_Enumerate

(

CdsFifo

q

)

Definition at line 22 of file conv-lists.C.

Referenced by CpdList_localQ::pup().

Here is the caller graph for this function:

int CdsFifo_Empty

(

CdsFifo

q

)

Definition at line 23 of file conv-lists.C.

Referenced by BgProcessMessageFreezeMode(), CpdDeliverSingleMessage(), Cpthread_mutex_destroy(), Cpthread_mutex_trylock(), ntohl(), waitqd_QDChare::onQD(), and PumpMsgsBlocking().

Here is the caller graph for this function:

void * CdsFifo_Peek

(

CdsFifo

q

)

Definition at line 24 of file conv-lists.C.

Referenced by Cpthread_cond_wait(), Cpthread_mutex_lock(), and Cpthread_mutex_unlock().

Here is the caller graph for this function:

int CdsFifo_Length

(

CdsFifo

q

)

Definition at line 27 of file conv-lists.C.

Referenced by CpdList_localQ::getLength(), CkObjectMsgQ::length(), and CpdList_localQ::pup().

Here is the caller graph for this function:

---

Variable Documentation

int _exitHandlerIdx

Definition at line 192 of file init.C.

Referenced by _exitHandler(), _initCharm(), _sendStats(), _sendWarnings(), CkCleanup(), CkContinueExit(), and CkExit().