Manages communication with a single paircalc array. More...

#include <pcCommManager.h>

Public Member Functions
	PCCommManager (const CkIndex2D gspaceIdx, const pc::pcConfig &_cfg, const pc::InstanceIDs _pcHandle)
	Constructor.

	PCCommManager ()

void	sendLeftData (int numPoints, complex *ptr, bool psiV)
	Starts the forward path work (Psi, Lambda and PsiV cases) by multicasting an entry method call to the appropriate PC chare array section.

void	sendRightData (int numPoints, complex *ptr, bool psiV)
	Starts the forward path work (along with startPairCalcLeft()) in the asymmetric (Lambda) case.


CProxySection_PairCalculator	makeOneResultSection_asym (int chunk)
	Initialize an array section that is used to reduce the results from the PCs back to the GSP chares. More...

CProxySection_PairCalculator	makeOneResultSection_asym_column (int chunk)
	initialize plane and column wise section reduction for lambda->gspace More...

CProxySection_PairCalculator	makeOneResultSection_sym1 (int chunk)
	initialize plane and row wise section reduction for psi->gspace

CProxySection_PairCalculator	makeOneResultSection_sym2 (int chunk)
	initialize plane and column wise section reduction for psi->gspace More...

Private Member Functions
void	makeLeftTree ()
	Creates multicast trees to the appropriate PC chare array sections used in the symmetric / asymmetric loops. More...

void	makeRightTree ()
	Creates a multicast tree that includes the PC chare arrays used in the asymmetric loop. More...

void	sendLeftDataMcast (int numPoints, complex *ptr, bool psiV)
	Multicasts the left matrix data to the PC section. More...

void	sendRightDataMcast (int numPoints, complex *ptr, bool psiV)
	Multicasts the right matrix data to the PC section. More...

void	sendLeftDataRDMA (int numPoints, complex *ptr, bool psiV)
	Sends left matrix data via RDMA.

void	sendRightDataRDMA (int numPoints, complex *ptr, bool psiV)
	Sends right matrix data via RDMA.

void	sendLeftRDMARequest (RDMApair_GSP_PC idTkn, int totalsize, CkCallback cb)
	Send RDMA setup requests to all the destination PC chares that will be getting left data.

void	sendRightRDMARequest (RDMApair_GSP_PC idTkn, int totalsize, CkCallback cb)
	Send RDMA setup requests to all the destination PC chares that will be getting right data.

void	setResultProxy (CProxySection_PairCalculator *sectProxy, bool lbsync, CkCallback synccb)
	Send out a dummy mcast to prod CkMulticast into setting up the result reduction trees etc. More...

Private Attributes
CkIndex2D	gspaceIndex
	The array index of the owner GSpace chare.

cp::paircalc::pcConfig	pcCfg
	Input configurations for the paircalcs.

cp::paircalc::InstanceIDs	pcHandle
	Handles to the paircalc array and related entities that I will be managing comm with.

CProxySection_InputDataHandler < CollatorType, CollatorType > *	sectionGettingLeft
	Array section which receives left matrix block data. More...

CProxySection_InputDataHandler < CollatorType, CollatorType > *	sectionGettingRight
	Array section which receives right matrix block data. More...

CkVec< CkArrayIndex4D >	listGettingLeft
	A list of PC array elements which expect left matrix data from owning GSpace chare.

CkVec< CkArrayIndex4D >	listGettingRight
	A list of PC array elements which expect right matrix data from owning GSpace chare.

CkVec< rdmaHandleType >	leftDestinationHandles
	RDMA handles for each PC chare's input data handler that will receive data from the owner of this object (a GSpace[s,p] chare)

CkVec< rdmaHandleType >	rightDestinationHandles

bool	existsLproxy
	True if a proxy for the destination PC array section including a (portion of a) row exists.

bool	existsRproxy
	True if a proxy for the destination PC array section including a (portion of a) column exists.

Friends
class	::CP_State_GSpacePlane

Detailed Description

Manages communication with a single paircalc array.

It needs a handle to an already created paircalc instance and the config settings for that instance Given this info, and the index of its owner (GSpace) chare, it creates appropriate mcast/redn sections and manages these communications. GSpace simply delegates the data sends to the appropriate comm manager instances in every iteration.

Todo:: : This class also manages the RDMA setup requests, although it could also assume the responsibility for processing the ack msg that arrives in GSpace to complete an RDMA handshake.

Definition at line 31 of file pcCommManager.h.

Constructor & Destructor Documentation

cp::gspace::PCCommManager::PCCommManager ( )

inline

Warning: : Just to appease charm migration constructors. pffouggh...

Definition at line 38 of file pcCommManager.h.

Member Function Documentation

void cp::gspace::PCCommManager::makeLeftTree ( )

private

Creates multicast trees to the appropriate PC chare array sections used in the symmetric / asymmetric loops.

Compute the max index along the state dimensions of the PC array

Find the row index of the PC chare that handles this state

If the PC is a symmetric instance, then include only the post-diagonal chares on the row s1, else, include all the PC chares on row s1

If GSpace to PC comm is point to point direct msging

simply create a list of PC chare array indices, with chunk=0 (as the comm list is the same for all chunks)

else, if communication is through section multicasts

Allocate one section proxy for each chunk

Build an array section for each chunk

Delegate the multicast work to an appropriate library

PC chares receiving data have been identified and memorized (either as an array section or a vector of IDs)

Definition at line 30 of file pcCommManager.C.

References existsLproxy, cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::pcConfig::isInputMulticast, cp::paircalc::pcConfig::isSymmetric, listGettingLeft, cp::paircalc::InstanceIDs::mCastMgrGID, cp::paircalc::pcConfig::numChunks, cp::paircalc::pcConfig::numStates, pcCfg, pcHandle, and sectionGettingLeft.

Referenced by sendLeftDataMcast(), and sendLeftRDMARequest().

CProxySection_PairCalculator cp::gspace::PCCommManager::makeOneResultSection_asym ( int chunk )

Initialize an array section that is used to reduce the results from the PCs back to the GSP chares.

initialize plane and row wise section reduction for lambda->gspace

The makeOneResultSection functions all have the same mission. Make one section at a time using only the relevant processors instead of making them all at once. We have each gspaceplane chare initialize its own section. Each section will have S/grainsize members. Such that PC(w,*,y,*) contribute to GSP(y,w).

Symmetric and asymm dynamics case will additionally have PC(w,x,y!=x,*) contributing to GSP(x,w) to fill out the total S/grainsize contributions in each section.

Then return the section proxy.

Definition at line 514 of file pcCommManager.C.

References cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::InstanceIDs::mCastMgrGID, cp::paircalc::pcConfig::numStates, cp::paircalc::InstanceIDs::pcAID, pcCfg, pcHandle, and setResultProxy().

CProxySection_PairCalculator cp::gspace::PCCommManager::makeOneResultSection_asym_column ( int chunk )

initialize plane and column wise section reduction for lambda->gspace

olumn

Definition at line 537 of file pcCommManager.C.

References cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::InstanceIDs::mCastMgrGID, cp::paircalc::pcConfig::numStates, cp::paircalc::InstanceIDs::pcAID, pcCfg, pcHandle, and setResultProxy().

CProxySection_PairCalculator cp::gspace::PCCommManager::makeOneResultSection_sym2 ( int chunk )

initialize plane and column wise section reduction for psi->gspace

olumn

Definition at line 594 of file pcCommManager.C.

References cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::InstanceIDs::mCastMgrGID, cp::paircalc::pcConfig::numStates, cp::paircalc::InstanceIDs::pcAID, pcCfg, pcHandle, and setResultProxy().

void cp::gspace::PCCommManager::makeRightTree ( )

private

Creates a multicast tree that includes the PC chare arrays used in the asymmetric loop.

Compute the max index along the state dimensions of the PC array

If the PC is a symmetric instance, then include only the pre-diagonal chares on the column s2 else, include all the PC chares on column s2

If GSpace to PC comm is point to point direct msging

simply create a list of PC chare array indices, with chunk=0 (as the comm list is the same for all chunks)

else, if communication is through section multicasts

Allocate one section proxy for each chunk

Build an array section for each chunk

Delegate the multicast work to an appropriate library

PC chares receiving data have been identified and memorized (either as an array section or a vector of IDs)

Definition at line 90 of file pcCommManager.C.

References existsRproxy, cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::pcConfig::isInputMulticast, cp::paircalc::pcConfig::isSymmetric, listGettingRight, cp::paircalc::InstanceIDs::mCastMgrGID, cp::paircalc::pcConfig::numChunks, cp::paircalc::pcConfig::numStates, pcCfg, pcHandle, and sectionGettingRight.

Referenced by sendRightDataMcast(), and sendRightRDMARequest().

void cp::gspace::PCCommManager::sendLeftDataMcast	(	int	numPoints,
		complex *	ptr,
		bool	psiV
	)

private

Multicasts the left matrix data to the PC section.

Deposits data as left matrix block with InputHandler chare array For symmetric instances, sends to the post-diagonal row of PCs that correspond to state gspaceIndex.x (including the chare on the chare array diagonal) For asymmetric instances, sends to the whole row of PCs that correspond to state gspaceIndex.x.

If a destination array section doesnt exist, build one

If a left matrix destination section exists, send the data as the left matrix block

else, if the destination section doesnt exist even after attempting to create one

Definition at line 156 of file pcCommManager.C.

References existsLproxy, gspaceIndex, cp::paircalc::pcConfig::inputMsgPriority, cp::paircalc::pcConfig::isDoublePackOn, cp::paircalc::pcConfig::isInputMulticast, cp::paircalc::pcConfig::isSymmetric, listGettingLeft, makeLeftTree(), cp::paircalc::pcConfig::numChunks, pcCfg, pcHandle, paircalcInputMsg::points, and sectionGettingLeft.

Referenced by sendLeftData(), and sendLeftDataRDMA().

void cp::gspace::PCCommManager::sendRightDataMcast	(	int	numPoints,
		complex *	ptr,
		bool	psiV
	)

private

Multicasts the right matrix data to the PC section.

Deposits data as right matrix block with InputHandler chare array For symmetric instances, sends to the strictly pre-diagonal column of PCs that correspond to state gspaceIndex.x For asymmetric instances, sends to the whole row of PCs that correspond to state gspaceIndex.x.

If a destination array section doesnt exist, build one

If a right matrix destination section exists, send the data as the left matrix block

last chunk gets remainder

else, if the destination section doesnt exist even after attempting to create one

Definition at line 243 of file pcCommManager.C.

References existsRproxy, gspaceIndex, cp::paircalc::pcConfig::inputMsgPriority, cp::paircalc::pcConfig::isDoublePackOn, cp::paircalc::pcConfig::isInputMulticast, cp::paircalc::pcConfig::isSymmetric, listGettingRight, makeRightTree(), cp::paircalc::pcConfig::numChunks, pcCfg, pcHandle, paircalcInputMsg::points, and sectionGettingRight.

Referenced by sendRightData(), and sendRightDataRDMA().

void cp::gspace::PCCommManager::setResultProxy	(	CProxySection_PairCalculator *	sectProxy,
		bool	lbsync,
		CkCallback	synccb
	)

private

Send out a dummy mcast to prod CkMulticast into setting up the result reduction trees etc.

send the multcast message to initialize the section tree and set the cookie

Definition at line 484 of file pcCommManager.C.

References cp::paircalc::pcConfig::grainSize, gspaceIndex, cp::paircalc::InstanceIDs::mCastMgrGID, pcCfg, and pcHandle.

Referenced by makeOneResultSection_asym(), makeOneResultSection_asym_column(), makeOneResultSection_sym1(), and makeOneResultSection_sym2().

Friends And Related Function Documentation

friend class ::CP_State_GSpacePlane

friend

Note: : Temporary until paircalc startup moves completely to GSpace

Definition at line 33 of file pcCommManager.h.

Member Data Documentation

CProxySection_InputDataHandler<CollatorType,CollatorType>* cp::gspace::PCCommManager::sectionGettingLeft

private

Array section which receives left matrix block data.

symm instance: post-diagonal chares on row 's' that get data from this GSpace[s,p] chare asymm instance: all chares on row 's' that get data from this GSpace[s,p] chare

Definition at line 86 of file pcCommManager.h.

Referenced by makeLeftTree(), sendLeftDataMcast(), and sendLeftRDMARequest().

CProxySection_InputDataHandler<CollatorType,CollatorType>* cp::gspace::PCCommManager::sectionGettingRight

private

Array section which receives right matrix block data.

symm instance: pre-diagonal chares on column 's' that get data from this GSpace[s,p] chare asymm instance: all chares on column 's' that get data from this GSpace[s,p] chare

Definition at line 93 of file pcCommManager.h.

Referenced by makeRightTree(), sendRightDataMcast(), and sendRightRDMARequest().

The documentation for this class was generated from the following files:

Public Member Functions

Private Member Functions

Private Attributes

Friends

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation