System Analyzer Tutorial 5B

Tutorial 5B:
Using LoggerStreamer and the PDK Trace Framework.

Background info:

• SYS/BIOS and Linux Multicore Software Development Kits (MCSDK) for C66x, C647x, C645x Processors
• TMS320TCI6614 Literature
• Enabling small cells with KeyStone software

Prerequisites

• The latest SC-MCSDK target content package
• A board with a C66XX device (e.g. TMDXEVM6670L, TMDXEVM6678L or TMDXEVM6614LE)
• CCSv5.2 or later
• System Analyzer 1.1 or later
• Target content package uia_1_01_01_14 or later

Tutorial 5A showed how to create a project that uses LoggerStreamer to store events in a circular array of UIA Packet buffers in memory, and how to use a DSS script to write the UIA Packet buffers into a binary file that can be read and displayed by System analyzer. In this tutorial, we will extend this concept to run on a multiple cores, with the UIA Packets that are produced by LoggerStreamer loggers on each of the cores being consumed by a task running on CPU 0 that copies the UIA packets into DDR3. A Debug Server Scripting (DSS) script is provided which uploads the events from DDR3 and saves them to a binary file that can be analyzed off-line by System Analyzer.

As well as providing a lightweight 'flight-recorder' style of logging events, this project demonstrates the use of inter-processor communication interfaces and the TraceFramework 'producer / consumer' model. (The TraceFramework is provided in the pdk_tci6614 package that is installed as part of the SC-MCSDK.) In the next tutorial, we'll extend this project by streaming the UIAPackets by UDP over bare-metal ethernet to the host PC running System Analyzer.

Building the Tutorial 5B Project

• unzip the Tutorial 5B project (File:Tutorial5B.zip) into a temporary folder
• In CCS, select File / Import... / Code Composer Studio - Existing CCS Eclipse Projects

• set the "Select search-directory" field to the temporary folder that you unzipped the project into
• make sure that the "Copy projects into workspace" checkbox is checked.

• In the C/C++ Projects view, right-click on the project and select Show Build Settings...

• In the left pane, select General, and in the right pane, click on the RTSC tab.
• Verify that you have the correct package versions installed. The project was built and tested with the following:

• XDCtools version: 3.23.3.53
• TCI6614 PDK 1.0.1.14
• UIA 1.1.1.14
• SYS/BIOS 6.33.5.46
• Target: ti.targets.elf.C66
• Platform: ti.platforms.evm6614

• If you are using an evm6670 or evm6678, change the Platform field to match your EVM.

• The build Configuration for Tutorial 5B is set to "Release" (for best performance).

• To change this to Debug mode, right click on the project, select Build Configurations -> Set Active -> Debug.

• To build the project, right click on the project and select Build Project.

A brief overview of the TraceFramework

The tutorial was built with the TraceFramework that ships in pdk_6614_1_00_01_14/packages/ti/instrumentation/traceframework folder. The TraceFramework User Guide (traceframework_ug.pdf) is provided in the docs subfolder.

• The TraceFramework uses the term "producer" to describe software entities that generate buffers of data and pass the data to the TraceFramework via a bufferExchange function. In this tutorial, the application code that logs events can be thought of as the producer, since the events that are logged are written to a buffer by LoggerStreamer, which calls the TraceFramework's bufferExchange function when the packet is full. The following line of .cfg script configures LoggerStreamer to call the TraceFramework's callback function:

• LoggerStreamer.exchangeFxn = '&tf_uiaProducerBufExchange';

• The TraceFramework uses the term "consumer" to describe software entities that wait for buffers of data to be generated and then do something useful with them. In Tutorial 5B, a task running on CPU 0 is configured as the consumer for the producer on each core.
• The application is responsible for configuring an array of buffers for use by the TraceFramework. The TraceFramework maintains a ring of these buffers.

• The tf_uiaProducerBufExchange callback function is configured to raise an interrupt when it is called, notifying any consumers that have registered to receive data from that producer. It then returns the next free buffer in the ring back to LoggerStreamer.
• If the producer generates a new buffer before the consumer can consume the previous buffer, it will result in one less buffer in the ring of buffers being free. In this way, the ring of buffers acts as a sort of giant jitter buffer, soaking up timing variations between the rate of production and the rate of consumption of the buffers.
• If there are no free buffers available when LoggerStreamer calls the buffer exchange function, a buffer is dropped. Thus, for applications where there is a high degree of variability in the timing of the consumer task and the number of events being logged, a larger number of buffers needs to be allocated by the application for use by the trace framework.

• This is one of the key 'tweaking' factors you will likely need to tune in order to optimize the trade off between memory consumption for the buffers vs. not dropping events.

A brief overview of the Tutorial Project

In this tutorial:

• *Log_app.h* : provides important #defines, including N_BUFS - the number of buffers provided for use by the traceFramework
• *application_bufs.c* : defines the buffers used by the traceFramework producers
• *captured_bufs.c*: defines the 'captureBuf' buffers in DDR3 that the consumer on CPU0 copies the buffers that are produced by all cores into. These are the buffers that are uploaded by the DSS script.
• *application_ipc.c*: contains the code used to configure and raise the interrupt that is used to signal to the consumer on CPU0 that the traceFramework has a buffer of data from a producer available
• *uiaosal_sem.c* : contains the code used to configure and raise the semaphore that is used by the consumer task on CPU 0
• *main.c* : the core of the application, including the consumer task and a 'producer' task that logs events.

• Note that events are also logged by the Sys/BIOS Task module and the UIA LogSync module (via the LoggerStreamer logger), so the name of the 'producer' task is a bit misleading. It named this way to keep it more in line with the example application that ships with the traceFramework.

Some requirements on the buffers used to log events:

• buffer length must be a multiple of 4
• buffer start address must be word-aligned
• buffer start address should be cache aligned

• Since the buffers are being logged by different cores, the buffers in the tutorial have been aligned with lines of memory to improve DDR access speeds.
• The cache has been turned off for the region of memory that the buffers are located in to avoid cache coherency problems.

The tutorial logs events using the "LogUC" (Log unconditional) APIs Log_infoUC3 and Log_writeUC0 in order to reduce the CPU overhead. For more information on these APIs, please see Tutorial 1E: LogUC.h - reducing the number of cycles required to log an event.

Running the Tutorial Project

• Launch a debug session for your target device:

• group the first 4 C66XX CPUs on the device by multi-selecting them in the Debug View, right clicking and selecting Group Cores from the context menu
• select the group node that the cores are grouped under, and connect to the group
• select the first core in the group and

• do a system reset (Run / Reset / System Reset)
• run the Global Default Initialization GEL script for the EVM

• For the C6614: Scripts / EVMTCI6614LXE MENU / Global_Default_Setup
• For the C6670: Scripts / LC_EVM_C6670_Functions / Global_Default_Setup

• select the group node and load the project .out file. The cores should all run and halt at main.
• run all the cores for 10 seconds or so, and then halt the cores.
• at this point, the capture buffers in DDR should contain packets from all of the cores.

A Debug Server Script to upload captured events

A Debug Server Scripting (DSS) script named Tutorial5B_readLogBuffers.js should have been installed with your project in the root folder of the project. This script can be run from the scripting console to upload the captured event buffers via JTAG and store them to a binary file.

If you are not using the 6614 EVM with the mezzanine board, you will likely need to make some modifications to this script in order to use it in your setup.

• The following line in the Tutorial5B_readLogBuffers.js script is used to specify which CPU the script will interact with:

• The string passed to debugServer.openSession must match exactly the string displayed in the Debug View for the CPU that you wish the script to interact with:

• To change the script to work with a different CPU or with a different emulator, open the .js file in a text editor (e.g. Notepad++) and change the string to match the text shown in the Debug View for the CPU you wish to use. The string is case sensitive.
• Note: To edit the file in CCS, right-click on the file in The CCS Project Explorer and either

• select Open With / Javascript Editor (CCSv5.3 and later) or, if this reports errors,
• select Open With / Other... / C/C++ Editor. This editor provides syntax coloring and does not try to use Microsoft JScript to parse it.

• The script is tailored to work with the tutorial example, but can be easily modified to work with other applications.

• The script references the "gCaptureBuf" and "gCaptureBufIndex" symbols defined in the tutorial's capture_bufs.c file. If you are using different names for these items in your application's .c code, change the symbol names used in the script to match.
• The following variables in the script must reflect values in your application:
  
  

• The script saves the packets to the folder specified by the strOutputFolderName variable ("c:/ti").

• If you wish to change the directory that the file is written to, edit the string assigned to the strOutputFolderName variable.
• Note that the name of the file MUST be systemAnalyzerData.bin for System Analyzer to be able to recognize it.

An important difference in the script used for Tutorial 5B vs. the one used for the previous tutorial is that it works with buffers from multiple cores that have been copied from the original buffers that the events were logged into. With this arrangement, it is not possible to upload the latest events that have been logged into the buffer that LoggerStreamer is currently writing to, since this buffer has not yet been captured by the 'consumer' task on CPU 0. Also, there may be buffers that have been filled but have not yet been copied by the consumer task. As such, this tutorial does not provide the same level of 'flight recorder' visibility into the events leading up to the halt as is provided by Tutorial 5A. The primary purpose of this tutorial is to show how to interface with the PDK traceframework and to serve as an example for how buffers from multiple cores can be collected and passed to a transport for upload to System Analyer. In this tutorial, the 'transport' is simply some capture buffers in DDR. In the next tutorial, we will send the buffers to System Analyzer as UDP packets over bare-metal Ethernet.

Uploading the captured events

• Open the scripting console (View / Scripting Console)
• run the script by typing loadJSFile("c:/<your project root folder path>/Tutorial5B_readLogBuffers.js")

• The script outputs status messages to the console to provide feedback on what it is doing

This will save the uploaded buffers (each containing a UIA event packet) to c:\ti\systemAnalyzerData.bin To view the events:

• Tools / System Analyzer / Open Binary File
• At the top of the dialog, specify the Folder Name that the binary file is located in (c:/ti)
• Click on the Create UIA Config File button to create a custom configuration for System Analyzer, consisting of an Event Transport set to Type = None and a single endpoint, with the .out file set to the location of the .out file for this tutorial, as shown below:
  

• Save this to the root folder of the tutorial project in the CCS workspace for later use.

• Click the Start button to process the bin file

This will display the events that were captured in a view named "Binary File - : Logs".

To update the System Analyzer Binary File - :Logs view to show the newly uploaded events:

• click the red Stop button in the view toolbar

• 
• the icon will change to a green triangle
• 

• click the button again to upload the events from the same file as you used previously.

Configuring SysBios to use LoggerStreamer to upload SysBios events

The following lines in the project's .cfg file, configure the SysBios Task module to log task-switching events and the Hwi module to log hw interrupts using LoggerStreamer, while disabling the logging of events by the SysBios Load module:

• var LoggingSetup = xdc.useModule('ti.uia.sysbios.LoggingSetup');
• LoggingSetup.eventUploadMode = LoggingSetup.UploadMode_STREAMER;
• LoggingSetup.sysbiosTaskLogging = true;
• LoggingSetup.sysbiosLoadLogging = false;
• LoggingSetup.sysbiosHwiLogging = true;

See Tutorial 3 for more information on how to configure and view SysBios events.

How things work

A good way to understand how the tutorial works is to look at System Analyzer's execution graph. Click on the small black triangle to the right of the "Analyze" label in the Logs view toolbar and select Execution Graph.

Bug Workaround
Prior to CCSv5.4, a bug in the way System Analyzer handles sync point events that are logged by LoggerStreamer can cause all of the events that occur prior to the first recorded sync point event to be timestamped with the local CPU timestamp, and all events after the sync point to be timestamped with the common global timestamp. This causes difficulties when using the Execution graph, as events occurring prior to the sync point event will typically be at the extreme left of the graph's time axis, and events after the sync point event will be clustered at the extreme right of the graph's time axis.

To get around this:

• scroll the view horizontally to the right as much as possible.
• position the mouse just below the time axis and left-click and drag to select a region to zoom into.
• you can use the mouse scroll wheel to zoom in or zoom out from that region, and the scroll bar to reposition the events that are viewed.

Managing screen real-estate
With smaller computer monitors, you may want to reduce the number of rows of events that are displayed so that you can see more of the events of interest. To do this, right click on the view and select Display properties. Uncheck all of the rows that have Hwi in their name execpt for C66xx_0.#Hwi, as shown below:

Using the Execution Graph to help understand what is happening in Tutorial 5B
The screenshot below shows some of the events that were captured prior to the cores halting:

• The (BIOS Scheduler) row activity (shown by the blue line segments in the above screenshot) corresponds to the Task_yield() statement in the producer_app()
• The green [ symbols in the C66xx_0.#Hwi row show when the BIOS Hwi module started handling an interrupt, and the red ] symbols show when the module stopped handling the interrupt. This is also indicated by the line segments in the #Hwi.ti_sysbios_family_c64p_EventCombiner row (shown here in yellow).
• The line segments in the Task.producer_app() row (shown here in green) indicate when the task running the producer_app() function was active
• The line segment in the Task.consumer_app() row for CPU 0 (shown here in red) shows when the task running the consumer_app() function was active.

Here's what happened to cause the events:

Configuration and Start up

• Configuration and setup is handled by code in main() in main.c that, for Core 0 only, does the following:

• creates a task to run the consumer_app function
• calls the CreateConsumer function to create one consumer contract for each C66X core in the device

• The CreateConsumer function in main.c configures each contract to have a "notify_method" of NOTIFY_VIA_REGMASK and sets the "notify_reg_addr" field to &ipcRegs->IPCGR[0].
• Writing 1 into this ipcRegs->IPCGR[0] register generates an interrupt pulse to core 0 (for the C6614, this is interrupt 9). For more information on the use of interrupts for inter-processor communication on C66XX devices, please see section 2.4 of the [Keystone Architecture Chip Interrupt Controller (CIC) User Guide]

• The consumer 'contract' has a "notify_method" of NOTIFY_VIA_REGMASK". This causes the tf_prodBufExchange function to call tf_prodNotifyConsumer (defined in <pdk_tci6614 install folder>/packages/ti/instrumentation/traceframework/producer.h), which writes into the specified register.

• The uiaInitIpc function in application_ipc.c configures the EventCombiner to handle the IPC event, and maps the IPC event to hardware interrupt 9.

The main() function then executes code that is run on all cores to create a 'producer' by calling the CreateUIAProducer function in main.c. It then creates a task to run the producer_app() function, and assigns the contract for the core 0 consumer that is responsible for handling buffers that are generated.

Logging events and generating UIAPacket buffers

• The producer_app() and bios Task and Hwi modules log events to LoggerStreamer. LoggerStreamer stores these events in a buffer passed to it by the trace framework.
• When LoggerStreamer determines that the buffer is full or LoggerStreamer_flush() is called, the module calls the buffer exchange callback function that was configured for it in the logger.cfg file (tf_prodBufExchange)

• The source code for the buffer exchange function (tf_prodBufExchange) is in the producer.c file that is located in <pdk_tci6614 install folder>/packages/ti/instrumentation/traceframework/src

IPC

• The tf_prodBufExchange function scans through the 'contracts' that have been registered by consumers that want to be notified when a buffer exchange happens. The contract that it finds instructs it to write into the register that raises an IPC event on core 0, which has been mapped to HWI interrupt 9.
• The interrupt that results is handled on core 0 by the uiaInterruptHandler function in application_ipc.c, which posts a semaphore.

Copying log buffers into the gCaptureBufRing array in DDR

• The consumer_app() function is pending on this semaphore, and when the semaphore is posted it scans through the array of consumer contracts and calls tf_consProcess (<pdk_tci6614 install folder>/packages/ti/instrumentation/traceframework/src/consumer.c), which checks to see if the ring index of the currently used buffer that the producer maintains has changed or not. If it has changed, it calls the function pointed to by the sendFxn field in the contract, which the CreateConsumer function configured as "notifyData".
• The notifyData function is passed the next full log buffer to process as well as the handle of the contract in the 'param' argument. It determines which core the buffer is associated with and then copies the buffer of event data that it was passed into the appropriate gCaptureBufRing array, updating the index into the array for next time.
• After all of the consumer contracts have been checked, the consumer_app() function waits for the next semaphore to be raised.

This process repeats continuously while the cores are running.

Next:Tutorial 5C: Sending UIA Packets via UDP over bare metal Ethernet using Navigator and the PDK Trace Framework.

Links

• Back up to the System Analyzer Tutorial 5 page
• Back up to the System Analyzer Tutorials page
• Back up to the System Analyzer Home