Class that buffers incoming data (via messages/RDMA) till it counts a pre-specified number of arrivals, and then spits the collated data out (packaged in the same incoming message type) via a CkCallback. More...

#include <MessageDataCollator.h>

Public Member Functions
	MessageDataCollator ()

	MessageDataCollator (std::string debugID, CkCallback callbk, const unsigned int nExpected, bool manageBuffer=false, const unsigned int offsetStart=0)
	Collators should be constructed thus. More...

	~MessageDataCollator ()
	Destructor.

void	operator() (msgType *msg)
	PUP method. More...

template<class tokenType >
void	setupRDMA (RDMASetupRequestMsg< tokenType > msg, tokenType replyToken=0)
	Called by a data sender to setup an RDMA link with me (data receiver)

void	operator() (void)
	RDMA equivalent of msg handling. When data arrives via RDMA, this is called for handling it. Gets called from the CmiDirect code via a callback.

void	expectNext ()
	Ask me to notify converse that we're ready for the next batch of messages.

unsigned int	getNumExpected () const
	Get the number of messages this collator has been configured to buffer.

unsigned int	getNumReceived () const
	Check the number of messages that have arrived.

unsigned int	getNumBatchesDone () const
	Get the number of batches of incoming messages already processed.

msgType *	getSampleMsg ()
	Get a copy of a sample message from the last completed batch of messages. Copy managed by the user. More...

Static Public Member Functions
static void	collatorCallback (void *thisObj)
	Fed to CmiDirect as a callback to be triggered when some data is deposited by RDMA.

Private Attributes
std::string	myName
	A string that encodes some kind of ID supplied by the owner of an instance.

CkCallback	uponCompletion
	Callback that is passed in to trigger the post-collation work.

bool	isBufferMine
	If buffer is mine, I manage it, viz. reuse the same buffer for every batch of msgs where possible. If not, I allocate a fresh buffer and let the client manage it.

msgType *	outputMsg
	The message (which has the data buffer) in which the incoming data is collated.

dataType *	dataBuffer
	A pointer to the memory region (contained in outputMsg) where incoming data is collated.

msgType *	sampleMsg
	Handle to a sample message copied from every batch of incoming messages.

unsigned int	numExpected
	Counters to keep track of the messages.

unsigned int	numReceived

unsigned int	numBatchesProcessed

unsigned int	numRows
	Variables related to the message/buffer sizes.

unsigned int	numCols

unsigned int	numUnitsInMsg

unsigned int	bufferSizeInUnits

unsigned int	minSender
	Senders should have an ID accessed by msgType::sender(). More...

CkVec< rdmaHandleType >	RDMAhandles
	A vector of RDMA handles, one for each sender.

bool	isNextBatchViaRDMA
	Booleans indicating if the next/current batch of messages are via RDMA.

bool	isThisBatchViaRDMA

Detailed Description

template<class msgType, typename dataType>
class cp::paircalc::MessageDataCollator< msgType, dataType >

Class that buffers incoming data (via messages/RDMA) till it counts a pre-specified number of arrivals, and then spits the collated data out (packaged in the same incoming message type) via a CkCallback.

Assumptions:

All messages carry the same amount of data
Each batch of data arrives completely only via RDMA or msgs (no interleaving of different methods within a single batch of messages)
Incoming messages are not deleted after processing. This class is not a chare and will let the user code deal with such formalities as needed
If this class manages the data buffer, then the post-collation callback goes to a chare on the same PE so that the output msg is not deleted/moved/resized in any way
The first batch of messages immediately after an RDMA setup (which will involve a bunch of setupRDMA() calls) WILL be via RDMA (and not messages)
expectNext() will be called before every new batch of RDMA arrivals (and only after the current batch is done)
Every time the amount of data in an imminent batch will be more than in any previous batch, the buffer is reallocated. A fresh RDMA setup is required for subsequent RDMA arrivals
Incoming data is treated as 2d (nRows x nCols). This is not binding. Higher / Lower dimensional data can be sent in as a contiguous array as long as msgType has the same API
Some effort has been made to tolerate different data sizes across different batches, but no testing / guarantees as yet

Note

: Class msgType should provide the following methods:

int numRows() const : returns the number of rows in the dataType array in the message
int numCols() const : returns the number of columns in the dataType array in the message
int sender() const : returns the sender ID or a representation of the senderID as an int
dataType* data() : returns a pointer to the data held by the message. Does not strictly have to be of dataType as it is only used by CmiMemcpy.
msgType(const int senderID, const int nCols, const int nRows) : Used to create the output msg that holds the collated data

Definition at line 53 of file MessageDataCollator.h.

Constructor & Destructor Documentation

template<class msgType, typename dataType>

cp::paircalc::MessageDataCollator< msgType, dataType >::MessageDataCollator ( )

inline

Warning: : Do not use this. This exists only to appease charm migration code which screams for default constructors.

Definition at line 58 of file MessageDataCollator.h.

template<class msgType, typename dataType>

cp::paircalc::MessageDataCollator< msgType, dataType >::MessageDataCollator	(	std::string	debugID,
		CkCallback	callbk,
		const unsigned int	nExpected,
		bool	manageBuffer = `false`,
		const unsigned int	offsetStart = `0`
	)

inline

Collators should be constructed thus.

Copying has not been restricted so that collator classes can be passed in as arguments. Hence, be careful when copying collators to make sure multiple collators dont use the same memory region as a data buffer and kill you.

Definition at line 63 of file MessageDataCollator.h.

References cp::paircalc::MessageDataCollator< msgType, dataType >::isBufferMine, cp::paircalc::MessageDataCollator< msgType, dataType >::myName, cp::paircalc::MessageDataCollator< msgType, dataType >::numExpected, and cp::paircalc::MessageDataCollator< msgType, dataType >::RDMAhandles.

Member Function Documentation

template<class msgType , typename dataType >

msgType * MessageDataCollator< msgType, dataType >::getSampleMsg ( )

inline

Get a copy of a sample message from the last completed batch of messages. Copy managed by the user.

Useful in case the msg has info other than the data() field.

These can then be extracted in a case-specific manner, by the client code directly.

Note: : This class makes no guarantees on which message will be held as a sample msg.

Definition at line 267 of file MessageDataCollator.h.

Referenced by PairCalculator::acceptLeftData(), and PairCalculator::acceptRightData().

template<class msgType , typename dataType >

void MessageDataCollator< msgType, dataType >::operator() ( msgType * msg )

PUP method.

Todo:: : Define a PUP method

Makes this a functor so that it can be fed messages by any entity that just knows the MessageHandler API.

Todo:: : Debate if some kind of check is needed to ensure that the message received is the correct iteration etc. For e.g, if the message defined a uniqID() method that would return a unique tag indicating the group the message belonged to, then all messages that belonged to this group could be buffered together. uniqID() could return iteration number if messages from a single iteration are to be grouped, or the sender ID if messages from a single sender are to be grouped together. Other possible scenarios where it is possible to receive messages from separate collation groups might occur.

If we're in the middle of a batch of messages

else, if this is a fresh batch of messages

First, ensure that I was not expecting the next batch of data via RDMA

Since, this is the first message in a new batch, flag this whole batch as NOT via RDMA

Get the message size

Get the required buffer size

If buffer exists AND I manage it AND its not big enough for this batch of messages, delete it

If the buffer is unallocated ...

Get some memory to hold the message data

Zero out the memory region

endif

Ensure that the output message and its data buffer exist

Ensure that this message has the same size as the others in this batch

Compute the offset for this message. This will depend on the sender index and thisIndex.

Copy the data from the message into the contiguous data buffer

Increment the tally of received messages

If the expected number have arrived ...

Increment the number of message batches that have been processed

Create a copy of this message for a sample from this batch

Call the trigger functor

Reset the number received so that we can start collating all over again

If you are not managing the buffer, then you need to forget about it

endif

Do NOT delete the msg; MessageDataCollator is not a chare and doesnt worry about the message formalities. Let the user code handle this

Definition at line 163 of file MessageDataCollator.h.

Member Data Documentation

template<class msgType, typename dataType>

unsigned int cp::paircalc::MessageDataCollator< msgType, dataType >::minSender

private

Senders should have an ID accessed by msgType::sender().

This stores the minimum value of sender(), so that offsets into the memory buffer can be computed

Todo:: : A better way to accomplish that leaves the collator class less rigid might be better

Definition at line 153 of file MessageDataCollator.h.

The documentation for this class was generated from the following file:

MessageDataCollator.h

Public Member Functions

Static Public Member Functions

Private Attributes

Detailed Description

template<class msgType, typename dataType> class cp::paircalc::MessageDataCollator< msgType, dataType >

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation

template<class msgType, typename dataType>
class cp::paircalc::MessageDataCollator< msgType, dataType >