Custom Plug-in Framework API

Custom Plug-in Framework (CPF) interacts with the simulators through the custom plug-in framework API. The API provides pre-defined functions that the simulators have to implement. 

Framework Variables

Variable

Explanation

U4 

unsigned 4-byte integer

U8 

unsigned 8-byte integer

I4 

signed 4-byte integer

I8

signed 8-byte integer

F4

4-byte floating point

F8

8-byte floating point

Data Structures

FuncEventData

The FuncEventData structure represents any function events that occur in a trace on a specific thread and do not span multiple threads. CPF passes a FuncEventData structure to a simulator whenever it encounters any application event, MPI single rank event, or a non-tracing event (such as idle time on a process).  The FuncEventData structure also represents a non-blocking sending because a non-blocking sending occurs only on a specific thread.

The only member of FuncEventData that you can modify using a simulator is the end time of the function event.  The other parameters are all constant because modifying any of these values would greatly change the fundamental parts of the trace.  The constant parameters can be the thread id, function type, size of message (if it is a non-blocking sending), or start time.

Note

The end time of this event should never be less than the start time. CPF checks this rule when returned from the simulator.

Variable

Explanation 

_threadID

The thread id where this function event occurred. 

_func

The function id of this event. You can look up the id using mapping provided.

See Private Functions for details. 

_sizeSent

The size of the message sent by the thread.

This variable is available when the function event is used as part of the collective operation. 

_sizeRecv

The size of the message recieved by the thread.

This variable is available when the function event is used as part of the collective operation. 

_startTime

The start time of this function event. 

_endTime

The end time of this function event.

You can modify this variable, but make sure that it is not less then _startTime

P2PInfo

The P2PInfo data structure represents a point-to-point (P2P) message that occurs in a trace. It is also passed to non-blocking API function to give the simulator access to different aspects of the point-to-point message. The values are slightly different depending on the type of a point-to-point message that this data structure represents. See Private Functions for details.

The following table lists the P2Pinfo variables. All variables, except the end times of the send and receive messages, are constant because modifying any of these values would drastically change the fundamental parts of the trace.

Variable

Explanation 

_sendStartTime

This variable contains the send start time or the time when the message is sent. 

_sendEndTime

This variable is only valid on  sendrecv non-blocking sends (only in processnonblockingsend()) or on sync sends.

This variable can be modified, but should never be less then _sendStartTime.

_recvStartTime

This variable is only valid on sendrecv, sync sends, waitall/waitsomes. 

_recvEndTime

This variable is valid on all cases, can be modified, but should never be less then _recvStartTime

_iRecvStartTime

This variable is only valid when P2P message involves a waitall/waitsome.

This variable represents the start time of i-receive that occurs before the waitall/waitsome for this thread. 

_backwards

This variable indicates that the message send start time > receive end time. 

_sender

The thread id of the sending thread.

_receiver

The thread id of the receiving thread.

_communicator

The MPI communicator.

The P2P message communication is over; there is a communicator to text mapping provided for the simulator in the API.

See Private Functions for details. 

_tag

The MPI tag associated with the message.

_messageSize

The size of the message sent.

_sendFID

The identifier of the sending function.

You can get more information on the function using the mapping provided.

See Private Functions for details.

_recvFID

The identifier of the receiving function.

You can get more information on the function using mapping provided.

See Private Functions for details

CollOpInfo

The CollOpInfo data structure represents a collective operation that occurs in the trace. In a trace, there is a collective operation that connects the same event on each thread in the trace to one another. This CollOpInfo data structure provides this grouping to the simulator. 

All members of the CollOpInfo data structure are constant and cannot be modified except for the FuncEventData events that represent each thread that is involved in the collective operation. To modify end times of the collective operation on each thread, modify the FuncEventData for each thread in the vector in this collective operation. The same rules apply CollOpInfo for FuncEventData

Variable

Explanation 

_firstTime

The earliest time that a thread enters a collective operation. 

_communicator

Communicator on which collective operation is executed. 

_collOpId

The function id of the collective operation. You can look up what it is using mapping provided.

See Private Functions for details. 

_root

The thread id of the root, valid only if the collective operation is a One-To-All or All-To-One. 

_rootIndex

The root's index into the thread vector. 

_threads

The thread vector that represents the individual function events per thread that make up this collective operation.

Public Functions

The following topics list the public API functions and general public functions that CPF calls. When CPF is interacting with the API, it first creates a data structure of type described in Data Structures; then passes a pointer to the simulator's API function.  Any modification to this data structure in the API function automatically reflects changes in CPF.  These functions do not return any values.

void setup()

void setup (const Vector< U4 >, const Vector< U4 >, const Vector< U4 >, const HashMap< U4, const char* >, const char*, const U8, const U8, const U4)

Use this function to pass important information/characteristics about the trace from CPF to the simulator. This function takes care of initialization of the private member variables in Private Members. Do not modify this function.

Function

Description

virtual void Initialize ( ) = 0;

Your simulator needs to implement this API function.

The function is called before the start of processing a trace file, not to be used as a constructor.

This function enables simulator extensions to pre-trace file operations before the executable starts processing the trace file.

virtual void Finalize ( ) = 0;

Your simulator needs to implement this API function.

The function is called at the very end of processing a trace file, not to be used as a destructor. 

This function allows simulator extensions to have a way of doing post-trace file operations before the executable exits.

virtual void ProcessFunctionEvent ( FuncEventInfo * _event_ ) = 0;

Your simulator needs to implement this API function.

This function processes application events that occur in the trace file.

The event object is a regular application event in the trace that is processed from the queues.

virtual void ProcessNonTracingEvent ( FuncEventInfo * _event_ ) = 0;

Your simulator needs to implement this API function.

This function processes non-tracing events that occur in the trace file. This event object is a non tracing event in the trace that is to be processed from the queues.  A non-tracing event is an empty white space at beginning of trace before each thread, or parts of a trace file which have tracing turned off.

virtual void ProcessMPIFunctionEvent ( FuncEventInfo * _event_ ) = 0;

Your simulator needs to implement this API function.

This function processes a single rank MPI event in trace file. This event object is a single rank MPI event in the trace that is to be processed from the queues.  A single rank MPI event is an MPI event that only involves one thread.

virtual void ProcessCollOp ( CollOpInfo * _collop_ ) = 0;

Your simulator needs to implement this API function.

This function processes a collective operation MPI event in trace file. This event object is a collective operation MPI event in the trace that is ready to be processed from the queues.  This one object represents all threads involved in collective operation.  The simulator does not get separate events for each thread involved in the collective operation.

virtual void ProcessP2P ( P2PInfo * _message_ ) = 0;

Your simulator needs to implement this API function.

This function processes a point-to-point MPI event in trace file. This event object is a P2P MPI event in the trace that is to be processed from the queues. This function is called for sendrecv, synchronous, or normal send to receive MPI operations.

virtual void ProcessNonBlockingSend ( FuncEventInfo * _event_, P2PInfo * _message_ ) =0;

Your simulator needs to implement this API function.

This function processes a non-blocking send MPI event in a trace file.

This function gets an event object that represents the send operation and the corresponding P2P message that goes with that send. 

This function is only called for non-synchronous sends, or sends that do not rely on the receive thread.  The start time of the receive thread may not be known.

virtual void ProcessP2PWaitAllSome ( Vector<P2PInfo*, ObjectPointerAllocator<P2PInfo*> > * _messages_ ) = 0;

Your simulator needs to implement this API function. This function processes a waitall and waitsome MPI event in trace file. This function gets a vector of all P2P messages involved with the waitall/waitsome.  There is no individual function events corresponding to the send or receive thread involved in this P2P operation.

Private Members

Member

Explanation

Vector< char* > _commandLineParams;

This vector represents the command-line parameters passed to the simulator by CPF.  It is a vector of char pointers of parameters that do not match any keywords for CPF. These parameters may contain invalid values, and need to be checked by simulator.  

char * _stfPath;

Pathway to the directory of the original STF file loaded by CPF at runtime. 

U8 _stfStart;

Start time of original STF file. 

U8 _stfEnd;

End time of original STF file 

U4 _numThreads;

Number of threads in trace 

F8 _clockResolution;

Resolution length of clock tick in seconds. 

Vector< U4 > _singleRankFuncs;

Vector of single rank function id's. This vector is initialized in an init() call with the id values of every rank function in the trace file. Ranks that are not in the trace file are initialized with the value -1. 

Vector< U4 > _collOpFuncs;

Vector of collective operation id's. This vector is initialized in an init() call with the id values of every collective function in the trace file. Given a function id number, the corresponding collective operation id is returned. 

Vector< U4 > _P2PFuncs;

Vector of P2P identifiers. This vector is initialized in an init() call with the id values of every P2P function in the trace file. This vector returns the related the corresponding collective operation id for every function id number. 

HashMap< U4, const char* > _communicatorsMapped;

HashMap of all communicator names. Lookup is by id of communicator. Data value is strings. 

Private Functions

Function

Explanation 

bool isSingleRankFunc( U4 _id_ )

Checks to see whether the function id is a single rank ID. Returns TRUE if _id_ is a single rank function; FALSE otherwise.  

bool isCollectiveOperation( U4 _id_ )

Checks to see whether the function id matches a collective operation ID. Returns TRUE if _id_ is a collective operation; FALSE otherwise. 

bool isP2P( U4 _id_ )

This function checks to see whether the function id is a P2P ID. Returns TRUE if _id_ is a P2P function; FALSE otherwise. 

bool isAllToAll( U4 _id_ )

Returns TRUE if it is an ALL-TO-ALL collective operation id number; FALSE otherwise. 

bool isOneToAll( U4 _id_ )

Returns TRUE if it is a ONE-TO-ALL collective operation id number; FALSE otherwise. 

bool isAllToOne( U4 _id_ )

Returns TRUE if it is an ALL-TO-ONE collective operation id number; FALSE otherwise. 

bool isAll( U4 _id_ )

Returns TRUE if it is an ALL collective operation id number; FALSE otherwise.

bool isSendP2P( U4 _id_ )

Returns TRUE when the FuncEventData is a P2P send operation; FALSE otherwise.

bool isSyncSendP2P( U4 _id_ )

Returns TRUE when the FuncEventData is a P2P sync send operation; FALSE otherwise.

bool isSendRecv( U4 _id_ )

Returns TRUE when the FuncEventData is a P2P send- receive operation; FALSE otherwise.

bool isRecv( U4 _id_ )

Returns TRUE when the FuncEventData is a P2P receive operation; FALSE otherwise.

Typedef Functions

These functions are essential to providing the plug-in module for CPF. They are already included in DevSim and if you modified DevSim, then you should have properly modified these functions to handle your simulator. The creat_t function creates an instance of your simulator class and passes a reference back to CPF. When CPF has completed processing a trace, it destroys the instance of the simulator through the destroy_t function.

typedef CustomPluginFramework* create_t();
typedef void destroy_t(CustomPluginFramework*);