Msgcom

Overview

Msgcom is a library that was created with the intent of simplifying message passing between processes by utilizing the Multicore Navigator to transport the message.
These processes can reside on:

• The same core
  
• Different cores (ARM or DSP)
  
• Different devices entirely

From a high level perspective, message passing is accomplished through the use of a “reader” core and a “writer” core.  The reader core creates a msgcom channel which the writer core writes a message to.  This message gets placed in a hardware queue associated with the reader.  The reader core will either poll this queue through a msgcom API or be alerted via interrupt when a message has been placed.  One channel provides one-way communication and thus, if bi-directional communication is required, two channels must be created. 

Regarding channels, there are several different properties that define a channel.

Channel Properties

Channel Type

• Simple Queue Channel:  Messages are placed directly into a destination hardware queue that is associated with a reader.
  
• Virtual Channel:  Multiple virtual channels are associated with the same hardware queue.
  
• Queue DMA Channel:  Messages are copied using infrastructure PKTDMA between the writer and the reader.
  
• Proxy Queue Channels:  Indirect channels work over BSD sockets; Enable communications between writer and reader that are not connected to the same Navigator (aka different devices)

Interrupt Type

• No Interrupt:  Reader polls until a message arrives.
  
• Direct Interrupt:  Low-delay system; special queues must be used.
  
• Accumulated Interrupts:  Special queues are used; Reader receives an interrupt when the number of messages crosses a defined threshold.

Blocking and Non-Blocking

• Blocking:  The reader is blocked until a message is available by a software semaphore.
  
• Non-blocking:  The reader polls for a message.  If there is no message, it continues execution. 

MSGCOM Cases

Generic Channel Communication (DSP-DSP)

a. Reader creates a channel ahead of time with a given name (e.g. MyCh1).
b. When the Writer has information to write, it looks for the channel (find).
c. Writer asks for a buffer and writes the message into the buffer.
d. Writer does a “put” to the buffer.  The Multicore Navigator ensures it goes to the proper destination queue.
e. When the Reader calls “get,” it receives the message.
f. The Reader must “free” the message after it is done reading.

Low-Latency Channel Communication Single and Virtual Channel (DSP-DSP)

a. Reader creates a channel based on a pending queue.  The channel is created ahead of time with a given name (e.g. MyCh2).
b. Reader waits for the message by pending on a software semaphore.
c. When Writer has information to write, it looks for the channel (find).
d. Writer asks for buffer and writes the message into the buffer.
e. Writer does a “put” to the buffer.  The Navigator ensures it reaches the appropriate destination queue and generates an interrupt.  The ISR posts the semaphore to the correct channel.
f. The Reader starts processing the message.
g. Virtual channel structure enables usage of a single interrupt to post semaphore to one of many channels.

Reduce Context Switching (DSP-DSP)

a. Reader creates a channel based on an accumulator queue.  The channel is created ahead of time with a given name (e.g. MyCh4).
b. When Writer has information to write, it looks for the channel (find).
c. Writer asks for buffer and writes the message into the buffer.
d. The Writer does a “put” to the buffer.  The Navigator adds the message to an accumulator queue.
e. When the number of messages reaches a water mark, or after a pre-defined time out, the accumulator sends an interrupt to the core.
f. Reader starts processing the message and makes it “free” after it is done.

Generic Channel Communication (ARM-DSP)

a. Reader creates a channel ahead of time with a given name (e.g. MyCh5).
b. When the Writer has information to write, it looks for the channel (find).  The linux kernel is aware of the user space handle.
c. Writer asks for a buffer.  The kernel dedicates a descriptor to the channel and provides the Writer with a pointer to a buffer that is associated with the descriptor.  The Writer writes the message into the buffer.
d. Writer does a “put” to the buffer.  The kernel pushes the descriptor into the right queue.  The Navigator copies the descriptor data and frees the kernel queue.  The Navigator then loads the data into another descriptor and sends it to the appropriate core.
e. When the Reader calls “get,” it receives the message.
f. The Reader must “free” the message after it is done reading.

Low-Latency Channel Communication (ARM-DSP)

a. Reader creates a channel based on a pending queue.  The channel is created ahead of time with a given name (e.g. MyCh6).
b. Reader waits for the message by pending on a software semaphore.
c. When Writer has information to write, it looks for the channel (find).  The kernel space is aware of the handle.
d. Writer asks for a buffer.  The kernel dedicates a descriptor to the channel and provides the Writer with a pointer to a buffer that is associated with the descriptor.  The Writer writes the messages into the buffer.
e. Writer does a “put” to the buffer.  The kernel pushes the descriptor into the right queue.  The Navigator copies the descriptor data and frees the kernel queue.  The Navigator then loads the data into another descriptor, moves it to the right queue, and generates an interrupt.  The ISR posts the semaphore to the correct channel.
f. Reader starts processing the message.
g. Virtual channel structure enables usage of a single interrupt to post semaphore to one of many channels.

Reduce Context Switching (ARM-DSP)

a. Reader creates a channel based on one of the accumulator queues.  The channel is created ahead of time with a given name (e.g. MyCh7).
b. When Writer has information to write, it looks for the channel (find).  The kernel space is aware of the handle.
c. The Writer asks for a buffer.  The kernel dedicates a descriptor to the channel and gives the Writer a pointer to a buffer that is associated with the descriptor.  The Writer writes the message into the buffer.
d. The Writer puts the buffer.  The Kernel pushes the descriptor into the right queue.  The Navigator copies the descriptor data and frees the Kernel queue.  Then the Navigator loads the data into another descriptor.  Next the Navigator adds the message to an accumulator queue.
e. When the number of messages reaches a watermark, or after a pre-defined time out, the accumulator sends an interrupt to the core.
f. Reader starts processing the message and frees it after it is complete.

Generic Channel Communication (ARM-DSP) Code Example

DSP Reader Code

<syntaxhighlight lang='c'> char* channelName; Msgcom_ChannelCfg chConfig; MsgCom_ChHandle chHandle; Ti_Pkt* ptrMessage; uint8_t* ptrDataBuffer;

/* Initialize the channel configuration. */ memset ((void *)&chConfig, 0, sizeof(Msgcom_ChannelCfg));

/* Populate the channel configuration. */ chConfig.mode = Msgcom_ChannelMode_NON_BLOCKING; chConfig.appCallBack = NULL; chConfig.u.queueDMACfg.interruptMode = Msgcom_QueueInterruptMode_NO_INTERRUPT; chConfig.u.queueDMACfg.rxFreeQueueNum = (Qmss_getQIDFromHandle(Pktlib_getInternalHeapQueue(mySharedHeapHandle)));

// Reader channel created by DSP0 to be written to by ARM channelName = "dspReaderChannel";

/* Create the Message communicator channel. */ chHandle = Msgcom_create (channelName, Msgcom_ChannelType_QUEUE_RING, (Msgcom_ChCfg*)&chConfig, &errorCode); if (chHandle == 0) { System_printf ("Error: Unable to open channel Error : %d\n", errorCode); } else { System_printf ("Successfully created channel %s on DSP\n",channelName); }

//Get the message if (Msgcom_getMessage (chHandle, (MsgCom_Buffer**)&ptrMessage) < 0) { System_printf ("Error: Unable to get a message\n"); } if (ptrMessage == NULL) { Task_sleep(2); continue; }

System_printf("Success: Message received on DSP successfully\n");

/* Since this is a QUEUE channel and the descriptors and buffers are located in

• shared memory. It is the responsibility of the application to invalidate the
• packet. */

Pktlib_invalidatePkt(ptrMessage);

/* Message Received: Get the data buffer from the received message. */ Pktlib_getDataBuffer(ptrMessage, (uint8_t**)&ptrDataBuffer, &dataLen);

/* Invalidate the contents of the data buffer. */ appInvalidateBuffer(ptrDataBuffer, dataLen);

/* Cleanup the received message */ Pktlib_freePacket(ptrMessage);

</syntaxhighlight>

ARM Writer Code

<syntaxhighlight lang='c'> Msgcom_ChannelCfg chConfig; MsgCom_ChHandle chHandle; uint32_t i, dataLen = 32; uint8_t* ptrDataBuffer; char* channelName; Msgcom_QRingMsgBuffer qRingPktBuffer;

//Reader channel created by DSP for ARM to find channelName = "dspReaderChannel";

// Wait for message communicator channel to be created on DSP Core 0 while (1) { // Check if communicator channel has been created chHandle = Msgcom_find (channelName, (Msgcom_ChCfg*)&chConfig); if (chHandle != NULL) { printf ("Success: Channel %s found on ARM\n",channelName); break; } sleep (1); }

if ((ptrDataBuffer = udma_mem_block_alloc (memHandle)) == NULL) { printf("Error: udma_mem_block_alloc failed\n"); }

// Populate the data buffer for (i = 0; i < dataLen; i++) ptrDataBuffer [i] = 0xAA; memset ((void *)&qRingPktBuffer, 0, sizeof (Msgcom_QRingMsgBuffer)); qRingPktBuffer.ptrBuffer = (uint8_t *)ptrDataBuffer; qRingPktBuffer.bufferLen = dataLen; qRingPktBuffer.memHandle = memHandle; qRingPktBuffer.metaData = 0;

// Send the message! if (Msgcom_putMessage (chHandle, (MsgCom_Buffer *)&qRingPktBuffer) < 0) { printf ("Error: Unable to send the message over the polling queue ring channel\n"); } else { printf("Success: ARM core successfully put message\n"); } </syntaxhighlight>

Subsystems

Overview

There are several subsystems which are a necessary in order to leverage Msgcom.  The diagram below presents many subsystems, however, this document will focus on those necessary solely for msgcom and not networking.  These include: Agent, JOSH, Message Router, udmalib, PktLib, and Resource Manager (not shown).

Resource Manager

The Resource Manager ensures that system resources can be requested and granted access without conflict.  It throws an error during system initialization if requested resources are greater than system limitations.
Example system resources include:

• General purpose queues
  
• Accumulator channels
  
• Hardware semaphores
  
• Direct interrupt queues
  
• CPINTC interrupts
  
• Memory region requests
  

The Resource Manager’s primary responsibility when utilizing msgcom is to sync resources that are to be viewed on both ARM and DSP, such as msgcom channels. 
ARM and DSP will each maintain their own database of named resources.  Thus, when DSP (or ARM), creates a channel – it updates its own database.  Then, through the help of Agent and JOSH, this named resource is pushed to the ARM (or DSP) and gets updated in its database. 

• Note: More information about Resource Manager can be found within syslib_x_xx_xx_xx/docs/Syslib_User_Guide.pdf.
  

Message Router

Message Router is a daemon that creates special msgcom control channels, also known as “control path.”  The Agent module utilizes these control channels to send system messages, such as creation and deletion of resources, between the ARM and DSP.
Message Router must be uploaded to the ARM and run before the user application as follows:

./msgrouter.out – n 4 –d 10 & 

• Note: More information about using Message Router can be found within syslib_x_xx_xx_xx/docs/Small_Cell_System_Library_Unit_Test.pdf and more high level details found within syslib_x_xx_xx_xx/docs/Syslib_User_Guide.pdf.
  

JOSH (Job Scheduler)

The Job Scheduler is a mechanism which allows a function call made on one processing element to be executed on another processing element.  However, since the format of these function calls changes between ARM and DSP, JOSH is utilized as a common language and prototype for a function call such that both ARM and DSP can understand.

• Note: More information about JOSH can be found within syslib_x_xx_xx_xx/docs/Syslib_User_Guide.pdf.
  
• Note: User application DOES NOT directly exercise any of the JOSH APIs.

Agent

Agent is a critical component used for syncing resources such as msgcom channels.  Agent utilizes msgcom control channels created by Message Router to sync updates about resources – creation, deletion, and modification.  For instance, when DSP creates a channel it performs the following steps:

• DSP will update its Resource Manager Database
  
• Agent creates a JOSH packet saying that this is a new resource with name and corresponding data
  
• This JOSH packet is pushed to ARM by the control channels created by Message Router
  
• ARM then updates its Resource Manager Database with this information
  

In order to use Agent with a msgcom application, one must must create a task (or thread in ARM) that polls these msgcom control channels waiting for a message to arrive.  See below for Agent initialization and polling structure.

• Note: More information about Resource Manager can be found within syslib_x_xx_xx_xx/docs/Syslib_User_Guide.pdf

DSP Agent Initialization and Polling Code

<syntaxhighlight lang='c'> void AgentRxTask(UArg arg0, UArg arg1) {

   int32_t             retVal;
   int32_t             errCode;
   Msgcom_ControlCfg   agentChannelCfg;

   /* NOTE: Do Msgcom_init before the Agent_init. */
   if (Agent_init ((void *) 0xA0000000) < 0)
   {
       printf ("Error: Agent init failed.\n");
       return;
   }
   System_printf ("Debug: Agent was initialized successfully\n");

   /* Initialize the MSGCOM Control Channel Info. */
   memset ((void *)&agentChannelCfg, 0, sizeof(Msgcom_ControlCfg));

   /* Setup a shared heap for the MsgCom library to create the Queue Ring
    * control channel */
   agentChannelCfg.rpcId                                 = DNUM;
   agentChannelCfg.chConfig.mode                         = Msgcom_ChannelMode_NON_BLOCKING;
   agentChannelCfg.chConfig.appCallBack                  = NULL;
   agentChannelCfg.chConfig.u.queueDMACfg.interruptMode  = Msgcom_QueueInterruptMode_NO_INTERRUPT;
   agentChannelCfg.chConfig.u.queueDMACfg.rxFreeQueueNum = Pktlib_getInternalHeapQueue(mySharedHeapHandle);

   /* Initialize the Agent */
   agentHandle = Agent_create (&agentChannelCfg, &errCode);
   if (agentHandle == NULL)
   {
       printf ("Error: Agent creation failed. Error %d\n", errCode);
       return;
   }
   System_printf ("Debug: Agent%d was created successfully\n", DNUM);

   /* Check if the ARM control channels are up. */
   while (1)
   {
       if (Agent_isControlChannelUp (DNUM) < 0)
           continue;
       else
       {
           printf ("Debug: ARM control channels for Agent0 have been created.\n");
           break;
       }
   }

   /* This is a slave loop waiting for messages from the master to arrive */
   while (1)
   {
   	/* Check for any received messages */
   	retVal = Agent_receive (agentHandle, &errCode);
       if (retVal < 0)
       {
           /* Error: Unable to handle it. Check the error code? */
           if (errCode == AGENT_ENOMSG)
           {
               /* No MSG was received; we will retry after some time. */
               Task_sleep(2);
               continue;
           }
           System_printf ("Error: Agent Receive Failed with Error %d\n", errCode);
           return;
       }
   }

} </syntaxhighlight>

ARM Agent Initialization

<syntaxhighlight lang='c'> void agentInitTask(void *arg) {

   Agent_Cfg       agentCfg;
   pthread_t       agentThread;
   int32_t         retVal;

   /* Initialize the agent configuration. */
   memset ((void *)&agentCfg, 0, sizeof(Agent_Cfg));

   /* Populate the Agent Configuration
    *  - The test application is the default process for all SUBMIT Requests
    *  - We are communicating with DSP Core0 */
   agentCfg.localId          = getpid();
   agentCfg.isDefaultProcess = 1;
   agentCfg.remoteId         = 0;

   /* Initialize the Agent */
   agentHandle = Agent_init (&agentCfg);
   if (agentHandle == NULL)
       return;

   /* Debug Message: */
   printf ("Debug: Agent was successfully instantiated (Handle 0x%p)\n", agentHandle);

/* Create the Agent Thread. */ retVal = pthread_create (&agentThread, NULL, (PTHREAD_FN)AgentRxTask, agentHandle); if (retVal < 0) {

   	printf ("Error: Agent Thread failed to start error code %d \n", retVal);
       return;

} } </syntaxhighlight>

ARM Agent Polling

<syntaxhighlight lang='c'> void AgentRxTask(void *arg) {

   Agent_Handle        agentHandle;
   int32_t             errCode;

   /* Get the Agent Handle from the thread. */
   agentHandle = (Agent_Handle)arg;

   /* Execute the thread */
   while (1)
   {
       /* Execute the Agent Receive. */
       if (Agent_receive(agentHandle, &errCode) < 0)
       {
           /* Failure Detected: Use the Error Code to gather more information. */
           if (errCode == AGENT_ENOMSG)
           {
               /* This is an expected error message and can be ignored */
               continue;
           }
           else
           {
               /* All other errors are failures; notify the world & abort the thread. */
               printf ("Error: Agent Receive Failed with Error Code %d\n", errCode);
               break;
           }
       }
   }
   return;

} </syntaxhighlight>

Pktlib & udmalib

The Pktlib provides the packet infrastructure for the DSP side of the application. It is a high level library that allows allocation and manipulation of packets as well as heaps. Udma is essentially the equivalent library for this functionality on the ARM side.

• Note: More information about Pktlib can be found within syslib_x_xx_xx_xx/docs/Syslib_User_Guide.pdf.

Notes on Subsystems

It is not practical to show code snippets exclusive to each subsystem because of the substantial interaction between them.  The best way to gain an understanding of how to use the subsystems in a msgcom application is to look at the demo projects below.

Useful Resources

Examples

• Example msgcom project which tests all channel types can be found in syslib_x_xx_xx_xx/packages/ti/runtime/msgcom/test
  
  • Instructions to build and run project found in syslib_x_xx_xx_xx/docs/Small_Cell_System_Library_Unit_Test.pdf
    
• ARM-DSP bi-directional communication using Generic Channel Communication found with Train-the-Trainer materials 
  

Documentation

• Syslib_User_Guide contains information about msgcom, Resource Manager, Pktlib, Agent, and JOSH.