Compare Loop Performance Across C6000 Compilers

Problem Statement

Your system is working with an older version of the C6000 compiler, and now you have built it with a newer version of the C6000 compiler. You want to know if any of the loops run slower or faster, and if so, which ones.

Introduction

This page introduces a tool for comparing performance of loops between two different versions of the C6000 compiler. The tool is named loopdiff. Loopdiff is a static analysis tool. Running the code is not required. The comparison is done by processing the assembly source output of the compiler.

This page walks through three different methods of using the compiler. One is based on performing the build from the command line with a makefile. The other two are based on performing the build from within Code Composer Studio versions 4 (CCSv4) or 5.1 (CCSv5.1). For each method, this page shows how to employ loopdiff to discover the loops which run slower or faster.

Download and Install

Release v0.20.00

• Release Details (.txt)
• Windows Install Package
• Linux x86 Install Package

Install Directions

Download the package for your system, then execute it.

The Linux package only works when executed from the current directory. Suppose you download it to /some/local/dir. You cannot be in another directory and invoke it like this ...

% /some/local/dir/loopdiff-v0_20_00-Linux-x86-Install

You have to ...

% cd /some/local/dir
% ./loopdiff-v0_20_00-Linux-x86-Install

Makefile Method

Introduction

The modifications are made to this simple makefile.

#-----------------------------------------------------------------------------
# Name the object files
#-----------------------------------------------------------------------------
OBJS := icdct.obj isbt.obj iup.obj iwinm.obj mhead.obj towave.obj wavep.obj

#-----------------------------------------------------------------------------
# Develop C_OPTS: The compiler build options
#-----------------------------------------------------------------------------
# Optimization level 2
C_OPTS := $(C_OPTS) -o=2
# Build for C6400+ processors
C_OPTS := $(C_OPTS) -mv6400+

#-----------------------------------------------------------------------------
# Link build rule
#-----------------------------------------------------------------------------
mp3.out : $(OBJS)
	cl6x -z $(OBJS) -o=$@ lnk.cmd

#-----------------------------------------------------------------------------
# Compile build rule
#-----------------------------------------------------------------------------
%.obj : %.c
	cl6x $(C_OPTS) $<

This section presumes you are comfortable with modifying makefiles, performing builds, and generally working from the command line. This makefile and the methods described for working with it are admittedly simple. Such methods probably do not scale to very large builds. That is not the point here. The point here is to convey the conceptual changes to your makefile based build, and not the actual details of how your particular makefile must change.

Keep the Compiler Generated Assembly

Add these lines to the C_OPTS section of the makefile ...

# Extra SW pipeline info
C_OPTS := $(C_OPTS) -mw

By default, the compiler issues a block comment before every software pipelined loop which details properties of that loop. This option causes the compiler to issue a more verbose form of that comment block. It also causes the compiler to keep the assembly output file. As far as loopdiff is concerned, the important part here is that the assembly file is not deleted. But if you are interested enough in software pipelined loops to read this far, you probably appreciate hearing about -mw.

Build with Both Compiler Versions

Create a directory named "old_ver". Then add these lines to the C_OPTS section of the makefile ...

# Put .asm files in this directory
C_OPTS := $(C_OPTS) -fs=old_ver

The option -fs tells the compiler to place the generated assembly files in the given directory.

Insure the older version of the compiler is in the system path. Then build.

Next, create a directory named "new_ver". Change the just added lines to the C_OPTS section to this ...

# Put .asm files in this directory
C_OPTS := $(C_OPTS) -fs=new_ver

Change the system path to refer to the newer version of the compiler. Then build.

At this point, the directory "old_ver" contains compiler generated assembly from the older version of the compiler, and "new_ver" contains compiler generated assembly from the newer version of the compiler.

Compare Loops

From the directory which contains the sub-directories "old_ver" and "new_ver", issue a command similar to:

% perl /path/to/loopdiff.pl old_ver new_ver > results.txt

Or, if you do not have Perl installed and you are on a Windows PC system, you can use the standalone executable version:

% /path/to/loopdiff.exe old_ver new_ver > results.txt

Loopdiff looks through both directories for assembly files and processes them. Loops are automatically found and compared. A report is printed to the standard output stream. It is captured by this command in the file results.txt. A description of the results is found below.

CCSv4 Method

Presumptions

This section presumes you already have a project built and running under CCSv4. While you may have more than one build configuration for this project, this section presumes the current active configuration is the one used for the final production build.

There are a few different ways to access the interface for changing project build options. This section presumes you select the name of the project, right-click, and choose Build Properties. Other methods of accessing it may look a bit different than the screen shots shown here, but they will be similar.

This section also presumes you have installed the two compiler versions of interest, and configured CCS to use them. If not, the article Compiler Releases discusses how to obtain new compilers, and the article Compiler Installation and Selection discusses how to configure CCS to use them.

Keep the Compiler Generated Assembly

The loopdiff utility processes the compiler generated assembly files. If your project build settings already keep the compiler generated assembly files, then you can skip this step. However, default build configurations do not keep these files. Here is one way to change that.

Bring up the Build Properties. Make sure the Tool Settings tab is selected. Highlight Runtime Model Options. Check the box for Generate verbose software pipelining information (--debug_software_pipeline).

Select --debug_software_pipeline

By default, the compiler issues a block comment before every software pipelined loop which details properties of that loop. This option causes the compiler to issue a more verbose form of that comment block. It also causes the compiler to keep the assembly output file. As far as loopdiff is concerned, the important part here is that the assembly file is not deleted. But if you are interested enough in software pipelined loops to read this far, you probably appreciate hearing about --debug_software_pipeline.

Rebuild the project. Now your project build directory contains assembly source files built with the current version of the compiler.

Rename the Build Configuration

This step is not strictly necessary. But things are a bit easier to manage if the compiler version number is part of the build configuration name.

Bring up the Build Properties. Make sure the Tool Settings tab is selected. Highlight Tool-chain Settings. Note the Code Generation tools version shown. It may be grayed out. At this point, you don't need to change it. Just write down that version number.

Next, notice that on the right end of the Configuration drop-down box is another box labeled Manage.... Click it.

Manage Build Configuration

Highlight the name of the build configuration name to change. Then click Rename. Add the compiler version number to the end of the configuration name, similar to Release_v6_0_23. Click OK until all the dialog boxes are closed.

Create New Build Configuration Which Uses New Compiler

This step is a bit unusual. All at one time, you create a new build configuration which is the same as the current one, except it uses the new version of the compiler.

Bring up the Build Properties. In the leftmost part of the Window, highlight CCS Build. Insure the General tab is selected. In the drop-down box labeled Code Generation tools, choose the new version of the compiler. Click OK.

Change Compiler Version

Next, a dialog titled Save Build Configuration Settings comes up. Select the radio button labeled Create a new build configuration initialized with build options from the existing configuration. Change the New build configuration name so that it includes the version number of the compiler just selected, similar to Release_v7_2_0. Also check the box labeled Set the new build configuration as the active for the project. Click OK.

Create New Configuration

Build the project. A new build directory is created, and like the other build configuration, it contains the compiler generated assembly source files.

CCSv4 Compare Loops

At this point, you have built the project with both configurations, i.e. with both compilers. Next, the loops are compared with the loopdiff utility. At this time the loopdiff utility can only be run from the command line. If it proves to be popular, we may devise a way to invoke it from within CCSv4.

First, determine the directory where the project is based. Do not bring up the build properties by right clicking the project name. Instead, choose the menu item Project -> Properties. In the window on the left, highlight Info. Note the directory name shown by the label Location. Let's call this the project directory. Click OK.

At this point, the difference between methods for CCSv4 and CCSv5.1 is very slight. So, jump down below to the CCSv5.1 discussion of how to compare loops.

CCSv5.1 Method

Much of this section is cut-n-paste from the CCSv4 section. But there are some substantial differences. The main difference is in how the new configuration is constructed. The screen shots are different as well.

Presumptions

This section presumes you already have a project built and running under CCSv5.1. While you may have more than one build configuration for this project, this section presumes the current active configuration is the one used for the final production build.

There are a few different ways to access the interface for changing project build options. This section presumes you select the name of the project, right-click, and choose Build Options.... Other methods of accessing it may look a bit different than the screen shots shown here, but they will be similar.

This section also presumes you have installed the two compiler versions of interest, and configured CCS to use them. If not, the article Compiler Releases discusses how to obtain new compilers, and the article Compiler Installation and Selection discusses how to configure CCS to use them.

Keep the Compiler Generated Assembly

The loopdiff utility processes the compiler generated assembly files. If your project build settings already keep the compiler generated assembly files, then you can skip this step. However, default build configurations do not keep these files. Here is one way to change that.

Bring up the Build Options.... Expand the directory tree like structure on the left to Build/C6000 Compiler. Highlight Runtime Model Options. Check the box for Generate verbose software pipelining information (--debug_software_pipeline, -mw).

Select --debug_software_pipeline

By default, the compiler issues a block comment before every software pipelined loop which details properties of that loop. This option causes the compiler to issue a more verbose form of that comment block. It also causes the compiler to keep the assembly output file. As far as loopdiff is concerned, the important part here is that the assembly file is not deleted. But if you are interested enough in software pipelined loops to read this far, you probably appreciate hearing about --debug_software_pipeline.

Rebuild the project. Now your project build directory contains assembly source files built with the current version of the compiler.

Rename the Build Configuration

This step is not strictly necessary. But things are a bit easier to manage if the compiler version number is part of the build configuration name.

First, record which compiler is currently being used. Bring up the Build Options.... In the lower left corner, click on Show advanced settings. In the left panel, highlight CCS General. Under Advanced settings, note the current compiler version number. The compiler version is 6.1.12 in this example. Click OK.

Current Compiler Version Number

Second, rename the current build configuration. Right click on the project name and choose Build Configurations -> Manage.... Make sure the current build configuration is highlighted. Click Rename.... Type something like Release_v6_1_12 in the Name box. Click OK until all dialog boxes are closed.

As of this writing, CCSv5.1 has a bug related to renaming a build configuration. The steps in this paragraph are a workaround for this bug. The project view for this project includes a folder with the same name as the configuration. When the configuration is renamed, the name of this folder should automatically change to match. But it doesn't. So, rename this folder manually. Right click the folder name (not the project name) and choose Rename.... In the dialog box change the name to match the configuration. Click OK.

Create New Build Configuration Which Uses New Compiler

First, create the new build configuration. Right click on the project name and choose Build Configurations -> Manage.... Click New.... In the Name box enter something which includes the new compiler version, like Release_v7_3_1. Under Copy settings from be sure the Existing configuration radio button is selected, and the corresponding drop down box contains the name of the configuration for the other compiler version. Click OK once. Next, make the new configuration active. Highlight the name of the new configuration and click Set Active. Click OK.

Creating New Compiler Configuration

Second, change the compiler used by the new configuration. Bring up the Build Options.... In the lower left corner, click on Show advanced settings. In the left panel, highlight CCS General. Under Advanced settings, change the drop-down box for Compiler version to use the new compiler. Click OK.

Finally, the build the project. A new build directory is created, and like the other build configuration, it contains the compiler generated assembly source files.

CCSv5.1 Compare Loops

At this point, you have built the project with both configurations, i.e. with both compilers. Next, the loops are compared with the loopdiff utility. At this time the loopdiff utility can only be run from the command line. If it proves to be popular, we may devise a way to invoke it from within CCS.

First, determine the directory where the project is based. Bring up the Build Options.... In the lower left corner, click on Show advanced settings. On the upper left, highlight Resource. A directory path is shown next to Location:. Let's call this the project directory. Click OK.

Determine Project Directory

Now, bring up your favorite command shell. If you don't have one, ask a local expert. On most Windows machines, the menu selection Start -> All Programs -> Accessories -> Command Prompt often works.

Go to the project directory as determined above. For example:

C:\>cd "\Documents and Settings\Your Name Here\workspace_v5_1\mp3"

Examine the directory contents to see the names of the two build configurations created earlier. For example:

C:\Documents and Settings\Your Name Here\workspace_v5_1\mp3>dir
 Volume in drive C has no label.
 Volume Serial Number is 10F5-1C87

 Directory of C:\Documents and Settings\Your Name Here\workspace_v5_1\mp3

01/20/2012  11:29 AM    <DIR>          .
01/20/2012  11:29 AM    <DIR>          ..
01/20/2012  10:03 AM               479 .ccsproject
01/20/2012  11:28 AM            16,885 .cproject
01/20/2012  11:28 AM             2,586 .project
01/20/2012  10:08 AM    <DIR>          .settings

...

01/20/2012  11:11 AM    <DIR>          Release_v6_1_12
01/20/2012  11:30 AM    <DIR>          Release_v7_3_1
09/09/2003  10:42 AM             5,043 tableawd.h
05/20/2011  08:52 AM            12,824 towave.c
08/17/2007  11:54 AM             2,632 wavep.c
09/09/2003  10:42 AM             2,647 wcvt.ch
              21 File(s)        173,524 bytes
               5 Dir(s)  11,244,933,120 bytes free

The configurations in this example are named Release_v6_1_12 and Release_v7_3_1. This is the point where it is useful to have the compiler version numbers embedded in the configuration names. To run the standalone executable version of loopdiff issue a command similar to:

C:\Documents and Settings\Your Name Here\workspace_v5_1\mp3>\path\to\loopdiff.exe Release_v6_1_12 Release_v7_3_1 > results.txt
Done!

Change \path\to to the directory where you have loopdiff installed. Loopdiff looks through both directories for assembly files and processes them. Loops are automatically found and compared. A report is printed to the standard output stream. It is captured by this command in the file results.txt. A description of such results is found below.

If you have Perl installed, you can use it to invoke the Perl script directly. For example:

C:\Documents and Settings\Your Name Here\workspace_v5_1\mp3>perl \path\to\loopdiff.pl Release_v6_1_12 Release_v7_3_1 > results.txt
Done!

Loopdiff Results Explained

This section describes the default output. You can also get the output in a raw CSV form.

Typical results look like this ...

Found 136 loops.
48, or 35.3%, have the same performance.


Loops Which Degraded

6.0.23       7.3.0        Delta  Line  Impl  File
--------------------------------------------------
19           21           -2     311   0     iwinQ.ch
19           21           -2     386   0     iwinQ.ch
...
18.50        19           -0.50  50    0     iwinQ.ch
19.50        20           -0.50  50    0     iwinbQ.ch


Loops Which Improved

6.0.23       7.3.0        Delta  Line  Impl  File
--------------------------------------------------
21           19           2      82    0     icdct.c
5            3            2      186   0     icdct.c
...
0.50         0.25         0.25   324   0     iupL1.ch
0.50         0.25         0.25   356   0     iupini.ch


Summary of Software Pipelined Loops Unique to Version 6.0.23

EII    Line  Impl  File
------------------------
DQ     166   1     isbt.c
DQ     208   0     iupini.ch
...

Summary of Software Pipelined Loops Unique to Version 7.3.0

EII    Line  Impl  File
------------------------
5      205   0     iupini.ch
5      215   0     iupini.ch
...

The first part is a overall summary of how many loops were found, and what percentage of those loops have the same performance. The next part is a table of data about loops generated by both compilers, and the loop performance degraded; that is, the new compiler generated a slower loop. The next part is a similar table of data, but for loops which improved. This is followed by a table of data about loops generated only by the older compiler version. Last is that same table of data, but about loops generated only by the new compiler version.

Table Column Headings

In all tables, each row represents one loop. A description of each column follows:

• 6.0.23 - The version number of the older compiler. The data is the effective initiation interval (EII) for that loop under that version. More background on EII is given below. This column heading will vary with the actual version number of the compiler seen.
• 7.3.0 - The version number of the new compiler. The data is the EII for that loop under that version.
• Delta - The difference between the EII values. Negative numbers indicate a loss in performance. Conversely, positive numbers indicate a gain in performance.
• Line - The source line number where the loop begins
• Impl - The same source line loop can, in some cases, be implemented multiple times in assembly. This number distinguishes among such implementations. More details below.
• File - The source file which contains the loop
• EII - The Effective Initiation Interval of the loop. See below for more detail.

How the Tables are Sorted

The degraded loops table and the improved loops table are both sorted by the Delta column. The degraded loops table starts with the loops with the biggest slow down. Conversely, the improved loops table starts with the loops with the biggest speed up. Ties are broken by sorting on file name, line number, then implementation number.

The unique loops tables are sorted by file name, line number, then implementation number.

Effective Initiation Interval (EII) Explained

The EII summarizes, in one number, the performance of a loop. Smaller numbers are better.

A description of an initiation interval (II) is given here. The effective II (EII) is the II divided by the unroll factor. The unroll factor is the number of copies of the loop in the assembly after unrolling. For a loop that is not unrolled at all, the unroll factor is 1. Thus, for a loop that is not unrolled, EII equals II. In the comment block the compiler issues before every loop, the unroll factor is titled the Loop Unroll Multiple.

Even though two loops may have different II, due to unrolling, the actual performance may be the same. For example, suppose the older compiler pipelines a loop with II=2. Suppose the new compiler pipelines the same loop with II=4, but it unrolls it once (thus divide by 2). The net performance of these loops is the same, which is reflected in that EII=2 for both loops. Note an EII can be less than 1.

In some unusual cases, the EII is not a number, but one of the following abbreviations:

• DQ - Disqualified. The loop was disqualfied from software pipelining for some reason. The most common reason is the loop contains a function call. Inspect the comment block in the compiler generated assembly for the exact reason for this specific loop.
• FAIL - The compiler attempted to software pipeline the loop, but failed for some reason. For example, no register allocation could be found. Again, inspect the comment block in the compiler generated assembly for the exact reason for this specific loop.
• NA - Not applicable. May only appear in the Delta column. This shows when a loop pipelines normally under one compiler version, but is DQ or FAIL under the other compiler version.

Multiple Loop Implementations (Impl) Explained

In some cases, the compiler generates multiple implementations for the same source loop. There are two, rather different, ways this can happen. Despite the difference in how this occurs, it appears similar in the assembly source, and so the handling in loopdiff is the same.

One way is when the compiler automatically chooses to emit multiple implementations of the loop for performance reasons. In such cases, there is a bit of code before the loops which, at runtime, chooses between them. The criteria for the choice may be the number of loop iterations, whether certain pointers do not alias, etc.

The other way is when the source loop appears in an inline function in a header file, and that function is called from multiple places.

No matter how it occurs, the multiple implementations are assigned a number, starting with 0, in the order of which they appear in the assembly. There is no guarantee that, for two different versions of the compiler, the same loop will result in the same number of multiple iterations, or, even if the same number of loops is generated, those loops will appear in the same order. Still, when this occurs, the loops with the same implementation number are compared. At times, this will result in an invalid comparison. Given the information presently available in the assembly files, there is nothing more that can be done.

Limitations, Caveats, Issues

Sometimes the Wrong Loops Are Compared

As already discussed above, there are cases where one loop in the C source results in multiple implementations of that loop in the assembly. Loopdiff assigns each of those assembly loops an implementation number, shown as Impl in the output. Loops with the same Impl are compared. Sometimes, this comparison is invalid.

Sometimes the Wrong File Name is Used

The name of the source file which contains the loop is part of how loops are matched up for comparison. A detailed description of how the file name is determined is beyond the scope of this document. The way it is usually done, with current versions of the compiler (as of this writing), is by using the file name seen in the most recent .dwpsn directive. These Dwarf position directives are issued so CCS can determine the C/C++ source code location for one or more assembly instructions. However, there is a bug where, in some limited cases, the file name in the .dwpsn directive is wrong. If the source for the loop is in an include file, the file name in the Dwarf position directive may be incorrect. The details are in the ClearQuest entry SDSCM00042053, which can be looked up here. This bug is fixed as of these compiler versions:

• 7.3.3
• 7.2.8
• 7.0.6
• 6.1.21
• 6.0.30

As of this writing, only 7.3.3 and 7.2.8 are available. Not all of these versions are planned for a release. Some of them, particularly the older ones, may never be released.

Loopdiff is able to transparently handle either current compiler output, or the slight differences in compiler output due to this bug fix.

Runtime Behavior Not Considered

The total impact any one loop has on overall system performance has several factors. Loopdiff only considers one: the number of CPU cycles required to execute the loop one time. Other considerations include:

• How many times the loop iterates
• How many times the function containing the loop is called
• Cache effects
• Memory system effects

Thus, the loop which loopdiff indicates has degraded the most may not, at runtime, cause that much performance loss. Conversely, a different loop which loopdiff indicates degraded a small amount may, at runtime, cause a great deal of performance loss.

This is a fundamental limitation of any static performance analysis, such as the one performed by loopdiff. All of the factors listed above can only be determined by executing the code and conducting a dynamic performance analysis.

File Names May Not Contain Slashes

A source file name may not contain either a forward slash (/) or a reverse slash (\). The rules on what characters are allowed in file names is difficult to pin down. I think forward slash is never allowed anywhere. And Windows systems do not allow reverse slash. Unix systems do allow a reverse slash in file names. Even so, this is exceedingly rare in practice. So I don't expect this limitation to cause any problems.

Allowing this presumption makes file name parsing much more straightforward.

File Names Should be Unique

As already explained, file names are part of how the loops are paired up for comparison. The file name considered is only the base name of the file, without any directory path information. This does not mean it is impossible to have a file of the same name in two different directories. But it does mean it will be confusing. All you see in loopdiff output is the name of the file. You'll have to work out which directory it came from.

Technical Details

Invoking Loopdiff

The tool is implemented as a Perl script named loopdiff.pl. It is supplied in source form, which can be run the same as any other Perl script. A standalone executable is also supplied, for use on Windows PC systems which do not have Perl installed.

Loopdiff Command Line

If you have Perl installed, you can always see the latest details on the command line with this command:

% perldoc loopdiff.pl

Loopdiff operates upon directories, and not file names. List out the directories which contain the compiler generated assembly files. Those directories, and any sub-directories they contain, are searched for assembly source files, in a manner similar to the Unix find command. If no directories are given, the search begins from the current directory.

As of this writing, these are the options:

• --help : Output usage and quit
• --asm_extension=s : Extension for assembly files
• -ea=s : Extension for assembly files
• --verbose : Print names of files as they are processed
• --csv_dump : Dump out the raw loop data in CSV format

Single letter abbreviations also work, e.g. -h for help, -v for verbose.

Dump Raw Loop Information

Use the option --csv_dump to get the loop performance data in comma separated values (CSV) format. The data is not processed or analyzed in any way. CSV data can be trivially loaded into Excel and other programs for custom analysis. Here is an example

File Name,Line Number,Impl Number,Version,II,Unroll
"icdct.c",67,0,6.0.23,19,1
"icdct.c",67,0,7.3.0,19,1
"icdct.c",82,0,6.0.23,21,1
"icdct.c",82,0,7.3.0,19,1
...

The first row gives the data column headings. Each subsequent row is data for one loop. The column headings are:

• File Name - The name of the source file
• Line Number - The line number where the loop starts
• Impl Number - The loop implementation number, described above
• Version - The version of the compiler which produced this loop
• II - The initiation interval
• Unroll - How many times the loop is unrolled

Recognizing Assembly Source Files

Assembly source files are identified by the file extension. By default, if the file extension is .asm, or begins with .s (e.g. .s62, .s55, .etc), then it is presumed to be an assembly source file. You can override the default with the options --asm_extension=s or -ea=s. Note these same options, with similar usage, are available with the C6000 compiler.

One difference is the meaning of something like -ea=s*. It is common to think that means s followed by anything. For Perl regular expressions, that is not what it means. It means zero or more s characters, and thus will match any file extension, even none at all. To have that effect, you have to write -ea=s.* instead. Note the addition of the '.' dot. The script warns you when your -ea choice would match any file extension.

BSD License

The loopdiff tool is released under the BSD license. Details are in the license wording at the end of the Perl source code.

How to Obtain Support

Post your questions and suspected bugs to the compiler forum with the tag loopdiff. Please use the tag. Use of the tag triggers an e-mail to the current support team. Without the tag, the issue may go unnoticed for some time.

Compiler Option Details

Compiler options can be expressed in two ways. All options have a long form that starts with two dashes. Many options, though not all, also have an equivalent shorter form that begins with one dash. The makefile portion of this article uses short form options, and the CCS portions use long form options. Here is a table which shows both forms of the options used in this article.

Long Form	Short Form
--silicon_version=id	-mv=id
--opt_level=n	-o=n
--debug_software_pipeline	-mw
--asm_directory=dir	-fs=dir